QEMU VM crashes when enabling KVM

QEMU VM crashes when enabling KVM

[Wayne Li]

[-- Attachment #1: Type: text/plain, Size: 1824 bytes --]

Dear QEMU list members,

So we developed a virtual machine that that runs on QEMU.  The virtual
machine emulates a certain piece of equipment running on a PowerPC 7457
processor.  And we are running the virtual machine on a T4240-RDB which as
a PowerPC e6500 processor.  I can't give any more details than that as this
project has military applications.

The problem I'm having right now is when I use the -enable-kvm option when
starting the virtual machine, the virtual machine crashes like so:

kvm error: missing PVR setting capability
kvm_init_vcpu failed: Function not implemented
common_qemu_exit() has been called

Now I am fairly sure KVM is actually enabled on the system.  Finding that
out was another story that spanned a couple of months.  But long story
short, lsmod doesn't show that the KVM kernel module is running.  But
that's because KVM is built-in and it can't actually be built as a loadable
kernel module in this particular system.

So I'm not really sure what could be the problem.  Though I was thinking if
I understood the error better that might help?  Following the code I see
that the "Missing PVR setting capability." is called when a variable called
"cap_segstate" is 0:

if (!cap_segstate) {
            fprintf(stderr, "kvm error: missing PVR setting capability\n");
            return -ENOSYS;
}

And the cap_segstate variable is set by the following function:

cap_segstate = kvm_check_extension(s, KVM_CAP_PPC_SEGSTATE);

From there I'm pretty lost; I simply don't understand what any of these
terms mean.  Do any of you guys know what PVR setting capability,
cap_segstate, or KVM_CAP_PPC_SEGSTATE mean?  Do I have to set up any kind
of communication between my virtual machine and KVM?  Any of you guys know
what's going on here?  I'm just pretty lost right now haha.

-Thanks, Wayne Li

[-- Attachment #2: Type: text/html, Size: 2235 bytes --]

Re: QEMU VM crashes when enabling KVM

[Paolo Bonzini]

On 11/12/19 22:23, Wayne Li wrote:
> 
> Now I am fairly sure KVM is actually enabled on the system.  Finding
> that out was another story that spanned a couple of months.  But long
> story short, lsmod doesn't show that the KVM kernel module is running. 
> But that's because KVM is built-in and it can't actually be built as a
> loadable kernel module in this particular system.
> 
> So I'm not really sure what could be the problem.  Though I was thinking
> if I understood the error better that might help?  Following the code I
> see that the "Missing PVR setting capability." is called when a variable
> called "cap_segstate" is 0:
> 
> if (!cap_segstate) {
>             fprintf(stderr, "kvm error: missing PVR setting capability\n");
>             return -ENOSYS;
> }
> 
> And the cap_segstate variable is set by the following function:
> 
> cap_segstate = kvm_check_extension(s, KVM_CAP_PPC_SEGSTATE);

You are not saying how you are running QEMU.  I think you are using a
CPU model that requires a Book3S KVM.

Paolo

Re: QEMU VM crashes when enabling KVM

[Wayne Li]

[-- Attachment #1: Type: text/plain, Size: 1728 bytes --]

We wrote a project that is created on top of the QEMU source code; it calls
functions from the QEMU code.  I run the executable created by compiling
that project/QEMU code.  Anyway, looking at the following documentation:

https://www.kernel.org/doc/Documentation/powerpc/cpu_families.txt

It looks like the PowerPC 7457 is Book3S and the PowerPC e6500 is BookE.
Is that why you think I require a Book3S KVM?  Exactly why do you feel this
way?  Also would that mean my team would need to go and buy a board with a
Book3S processor?

-Thanks!, Wayne Li

From my understanding

On Wed, Dec 11, 2019 at 7:16 PM Paolo Bonzini <pbonzini@redhat.com> wrote:

> On 11/12/19 22:23, Wayne Li wrote:
> >
> > Now I am fairly sure KVM is actually enabled on the system.  Finding
> > that out was another story that spanned a couple of months.  But long
> > story short, lsmod doesn't show that the KVM kernel module is running.
> > But that's because KVM is built-in and it can't actually be built as a
> > loadable kernel module in this particular system.
> >
> > So I'm not really sure what could be the problem.  Though I was thinking
> > if I understood the error better that might help?  Following the code I
> > see that the "Missing PVR setting capability." is called when a variable
> > called "cap_segstate" is 0:
> >
> > if (!cap_segstate) {
> >             fprintf(stderr, "kvm error: missing PVR setting
> capability\n");
> >             return -ENOSYS;
> > }
> >
> > And the cap_segstate variable is set by the following function:
> >
> > cap_segstate = kvm_check_extension(s, KVM_CAP_PPC_SEGSTATE);
>
> You are not saying how you are running QEMU.  I think you are using a
> CPU model that requires a Book3S KVM.
>
> Paolo
>
>

[-- Attachment #2: Type: text/html, Size: 2414 bytes --]

Re: QEMU VM crashes when enabling KVM

[Paolo Bonzini]

On 12/12/19 02:59, Wayne Li wrote:
> We wrote a project that is created on top of the QEMU source code; it
> calls functions from the QEMU code.  I run the executable created by
> compiling that project/QEMU code.  Anyway, looking at the following
> documentation:
> 
> https://www.kernel.org/doc/Documentation/powerpc/cpu_families.txt
> 
> It looks like the PowerPC 7457 is Book3S and the PowerPC e6500 is
> BookE.  Is that why you think I require a Book3S KVM?  Exactly why do
> you feel this way?  Also would that mean my team would need to go and
> buy a board with a Book3S processor?

CCing the PPC maintainer.  There are aspects of BookE and Book3S that
are different and not really interchangeable in the privileged interface.

Paolo

> -Thanks!, Wayne Li
> 
> From my understanding
> 
> On Wed, Dec 11, 2019 at 7:16 PM Paolo Bonzini <pbonzini@redhat.com
> <mailto:pbonzini@redhat.com>> wrote:
> 
>     On 11/12/19 22:23, Wayne Li wrote:
>     >
>     > Now I am fairly sure KVM is actually enabled on the system.  Finding
>     > that out was another story that spanned a couple of months.  But long
>     > story short, lsmod doesn't show that the KVM kernel module is
>     running. 
>     > But that's because KVM is built-in and it can't actually be built as a
>     > loadable kernel module in this particular system.
>     >
>     > So I'm not really sure what could be the problem.  Though I was
>     thinking
>     > if I understood the error better that might help?  Following the
>     code I
>     > see that the "Missing PVR setting capability." is called when a
>     variable
>     > called "cap_segstate" is 0:
>     >
>     > if (!cap_segstate) {
>     >             fprintf(stderr, "kvm error: missing PVR setting
>     capability\n");
>     >             return -ENOSYS;
>     > }
>     >
>     > And the cap_segstate variable is set by the following function:
>     >
>     > cap_segstate = kvm_check_extension(s, KVM_CAP_PPC_SEGSTATE);
> 
>     You are not saying how you are running QEMU.  I think you are using a
>     CPU model that requires a Book3S KVM.
> 
>     Paolo
> 

Re: QEMU VM crashes when enabling KVM

[Wayne Li]

[-- Attachment #1: Type: text/plain, Size: 2662 bytes --]

Dear David Gibson,

I know you are under no obligation to respond, but if it's possible for you
to find the time to respond to my question, I would be extremely grateful.
My team at Boeing has been stuck trying to get KVM working for our project
for the last few months.  A good explanation of why this isn't possible
would be absolutely critical.

-Thanks, Wayne Li

On Thu, Dec 12, 2019 at 1:17 AM Paolo Bonzini <pbonzini@redhat.com> wrote:

> On 12/12/19 02:59, Wayne Li wrote:
> > We wrote a project that is created on top of the QEMU source code; it
> > calls functions from the QEMU code.  I run the executable created by
> > compiling that project/QEMU code.  Anyway, looking at the following
> > documentation:
> >
> > https://www.kernel.org/doc/Documentation/powerpc/cpu_families.txt
> >
> > It looks like the PowerPC 7457 is Book3S and the PowerPC e6500 is
> > BookE.  Is that why you think I require a Book3S KVM?  Exactly why do
> > you feel this way?  Also would that mean my team would need to go and
> > buy a board with a Book3S processor?
>
> CCing the PPC maintainer.  There are aspects of BookE and Book3S that
> are different and not really interchangeable in the privileged interface.
>
> Paolo
>
> > -Thanks!, Wayne Li
> >
> > From my understanding
> >
> > On Wed, Dec 11, 2019 at 7:16 PM Paolo Bonzini <pbonzini@redhat.com
> > <mailto:pbonzini@redhat.com>> wrote:
> >
> >     On 11/12/19 22:23, Wayne Li wrote:
> >     >
> >     > Now I am fairly sure KVM is actually enabled on the system.
> Finding
> >     > that out was another story that spanned a couple of months.  But
> long
> >     > story short, lsmod doesn't show that the KVM kernel module is
> >     running.
> >     > But that's because KVM is built-in and it can't actually be built
> as a
> >     > loadable kernel module in this particular system.
> >     >
> >     > So I'm not really sure what could be the problem.  Though I was
> >     thinking
> >     > if I understood the error better that might help?  Following the
> >     code I
> >     > see that the "Missing PVR setting capability." is called when a
> >     variable
> >     > called "cap_segstate" is 0:
> >     >
> >     > if (!cap_segstate) {
> >     >             fprintf(stderr, "kvm error: missing PVR setting
> >     capability\n");
> >     >             return -ENOSYS;
> >     > }
> >     >
> >     > And the cap_segstate variable is set by the following function:
> >     >
> >     > cap_segstate = kvm_check_extension(s, KVM_CAP_PPC_SEGSTATE);
> >
> >     You are not saying how you are running QEMU.  I think you are using a
> >     CPU model that requires a Book3S KVM.
> >
> >     Paolo
> >
>
>

[-- Attachment #2: Type: text/html, Size: 3801 bytes --]

Re: QEMU VM crashes when enabling KVM

[BALATON Zoltan]

Hello,

I'm no expert on PPC KVM and don't have much experience with it myself so 
what I say may or may not be correct, it's just my understanding. 
Nevertheless I share it in the hope that it may help and maybe those who 
know will correct me.

> On Thu, Dec 12, 2019 at 1:17 AM Paolo Bonzini <pbonzini@redhat.com> wrote:
>> On 12/12/19 02:59, Wayne Li wrote:
>>> We wrote a project that is created on top of the QEMU source code; it
>>> calls functions from the QEMU code.  I run the executable created by
>>> compiling that project/QEMU code.  Anyway, looking at the following
>>> documentation:
>>>
>>> https://www.kernel.org/doc/Documentation/powerpc/cpu_families.txt
>>>
>>> It looks like the PowerPC 7457 is Book3S and the PowerPC e6500 is
>>> BookE.  Is that why you think I require a Book3S KVM?  Exactly why do
>>> you feel this way?  Also would that mean my team would need to go and
>>> buy a board with a Book3S processor?
>>
>> CCing the PPC maintainer.  There are aspects of BookE and Book3S that
>> are different and not really interchangeable in the privileged interface.

It's best to look at the kvm sources in Linux under linux/arch/powerpc/kvm 
or also ask on the appropriate Linux list for KVM specific questions. 
AFAIK KVM between BookE and BookS CPUs doesn't work, at least running 
BookE guest on a BookS host doesn't work which is what we've tried. You 
seem to want the opposite direction: running BookS code on BookE host but 
I think PPC KVM works best on BookS host running BookS code and apart from 
that there's some support for running BookE code on BookE host for e500 
but that's not much supported any more and that's it, other combinations 
are likely untested and don't work. So it's best to have host CPU match 
target (running e500 code on e6500 host or run 7457 code on same or newer 
host CPU, but the more different the host from the guest the most problems 
are to be expected). There are also HV and PR KVM where HV needs new 
enough CPU that supports it (e6500 probably does) but guest CPU cannot be 
different than host with HV KVM. With PR KVM there's some more freedom 
about host and guest as only user space runs on host CPU with privileged 
instructions are software emulated but mixing BookE and BookS is not 
supported even with PR KVM if I'm not mistaken. So you may have better 
luck with some BookS host but I can't tell for sure.

Regrads,
BALATON Zoltan

Re: QEMU VM crashes when enabling KVM

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 4919 bytes --]

On Thu, Dec 12, 2019 at 10:40:44AM -0600, Wayne Li wrote:
> Dear David Gibson,
> 
> I know you are under no obligation to respond, but if it's possible for you
> to find the time to respond to my question, I would be extremely grateful.
> My team at Boeing has been stuck trying to get KVM working for our project
> for the last few months.  A good explanation of why this isn't possible
> would be absolutely critical.

As you can see from that diagram, the history ppc CPUs is quite a bit
more diverse than x86.  Although they're all very similar from the
point of view of userspace code, they're quite different for
privileged kernel code: they have different MMUs, different privileged
registers amongst other things.

Because of this there are several different KVM implementations.

1) KVM HV

This one uses the virtualization facilities of BookS CPUs (present
since POWER4 / 970, but only well supported from POWER7 onwards).
Those don't allow much to virtualize the guest cpu model, so it
assumes the guest cpu is the same as the host.

So, both your guest and host CPUs rule this one out.

2) Book3E KVM

Uses the virtualization features of recent enough Freescale Book E
CPUs.  I don't know a lot about this or its limitations.  The e6500
might well have these features, but I'm pretty sure it can only
emulate BookE cpus for the guest.

So, your guest rules this one out.

3) KVM PR

This one operates by running the entire guest in user mode, and
emulating all privileged instructions.  It's slow (relative to
hardware assisted KVM models), but it's flexible.

In theory, this one can do what you want, but there are a bunch of
caveats:

  * Emulating all the privileged instructions for a whole bunch of cpu
  variants is a huge task, and KVM PR is now barely maintained.  There
  are lots of gaps in coverage.

  * I'm not sure if it was ever really implemented for BookE hosts.

  * Although there aren't many, there are a few differences between
  userland instructions between cpu variants, mostly because of new
  additions.  I think 7457 is an old enough design that this probably
  won't cause you troube, but I'm not certain.



> 
> -Thanks, Wayne Li
> 
> On Thu, Dec 12, 2019 at 1:17 AM Paolo Bonzini <pbonzini@redhat.com> wrote:
> 
> > On 12/12/19 02:59, Wayne Li wrote:
> > > We wrote a project that is created on top of the QEMU source code; it
> > > calls functions from the QEMU code.  I run the executable created by
> > > compiling that project/QEMU code.  Anyway, looking at the following
0> > > documentation:
> > >
> > > https://www.kernel.org/doc/Documentation/powerpc/cpu_families.txt
> > >
> > > It looks like the PowerPC 7457 is Book3S and the PowerPC e6500 is
> > > BookE.  Is that why you think I require a Book3S KVM?  Exactly why do
> > > you feel this way?  Also would that mean my team would need to go and
> > > buy a board with a Book3S processor?
> >
> > CCing the PPC maintainer.  There are aspects of BookE and Book3S that
> > are different and not really interchangeable in the privileged interface.
> >
> > Paolo
> >
> > > -Thanks!, Wayne Li
> > >
> > > From my understanding
> > >
> > > On Wed, Dec 11, 2019 at 7:16 PM Paolo Bonzini <pbonzini@redhat.com
> > > <mailto:pbonzini@redhat.com>> wrote:
> > >
> > >     On 11/12/19 22:23, Wayne Li wrote:
> > >     >
> > >     > Now I am fairly sure KVM is actually enabled on the system.
> > Finding
> > >     > that out was another story that spanned a couple of months.  But
> > long
> > >     > story short, lsmod doesn't show that the KVM kernel module is
> > >     running.
> > >     > But that's because KVM is built-in and it can't actually be built
> > as a
> > >     > loadable kernel module in this particular system.
> > >     >
> > >     > So I'm not really sure what could be the problem.  Though I was
> > >     thinking
> > >     > if I understood the error better that might help?  Following the
> > >     code I
> > >     > see that the "Missing PVR setting capability." is called when a
> > >     variable
> > >     > called "cap_segstate" is 0:
> > >     >
> > >     > if (!cap_segstate) {
> > >     >             fprintf(stderr, "kvm error: missing PVR setting
> > >     capability\n");
> > >     >             return -ENOSYS;
> > >     > }
> > >     >
> > >     > And the cap_segstate variable is set by the following function:
> > >     >
> > >     > cap_segstate = kvm_check_extension(s, KVM_CAP_PPC_SEGSTATE);
> > >
> > >     You are not saying how you are running QEMU.  I think you are using a
> > >     CPU model that requires a Book3S KVM.
> > >
> > >     Paolo
> > >
> >
> >

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: QEMU VM crashes when enabling KVM

[Wayne Li]

[-- Attachment #1: Type: text/plain, Size: 5279 bytes --]

Thanks David and Zalton for the awesome explanations.  They're very helpful
to us!

-Thanks, Wayne Li

On Thu, Dec 12, 2019 at 9:49 PM David Gibson <david@gibson.dropbear.id.au>
wrote:

> On Thu, Dec 12, 2019 at 10:40:44AM -0600, Wayne Li wrote:
> > Dear David Gibson,
> >
> > I know you are under no obligation to respond, but if it's possible for
> you
> > to find the time to respond to my question, I would be extremely
> grateful.
> > My team at Boeing has been stuck trying to get KVM working for our
> project
> > for the last few months.  A good explanation of why this isn't possible
> > would be absolutely critical.
>
> As you can see from that diagram, the history ppc CPUs is quite a bit
> more diverse than x86.  Although they're all very similar from the
> point of view of userspace code, they're quite different for
> privileged kernel code: they have different MMUs, different privileged
> registers amongst other things.
>
> Because of this there are several different KVM implementations.
>
> 1) KVM HV
>
> This one uses the virtualization facilities of BookS CPUs (present
> since POWER4 / 970, but only well supported from POWER7 onwards).
> Those don't allow much to virtualize the guest cpu model, so it
> assumes the guest cpu is the same as the host.
>
> So, both your guest and host CPUs rule this one out.
>
> 2) Book3E KVM
>
> Uses the virtualization features of recent enough Freescale Book E
> CPUs.  I don't know a lot about this or its limitations.  The e6500
> might well have these features, but I'm pretty sure it can only
> emulate BookE cpus for the guest.
>
> So, your guest rules this one out.
>
> 3) KVM PR
>
> This one operates by running the entire guest in user mode, and
> emulating all privileged instructions.  It's slow (relative to
> hardware assisted KVM models), but it's flexible.
>
> In theory, this one can do what you want, but there are a bunch of
> caveats:
>
>   * Emulating all the privileged instructions for a whole bunch of cpu
>   variants is a huge task, and KVM PR is now barely maintained.  There
>   are lots of gaps in coverage.
>
>   * I'm not sure if it was ever really implemented for BookE hosts.
>
>   * Although there aren't many, there are a few differences between
>   userland instructions between cpu variants, mostly because of new
>   additions.  I think 7457 is an old enough design that this probably
>   won't cause you troube, but I'm not certain.
>
>
>
> >
> > -Thanks, Wayne Li
> >
> > On Thu, Dec 12, 2019 at 1:17 AM Paolo Bonzini <pbonzini@redhat.com>
> wrote:
> >
> > > On 12/12/19 02:59, Wayne Li wrote:
> > > > We wrote a project that is created on top of the QEMU source code; it
> > > > calls functions from the QEMU code.  I run the executable created by
> > > > compiling that project/QEMU code.  Anyway, looking at the following
> 0> > > documentation:
> > > >
> > > > https://www.kernel.org/doc/Documentation/powerpc/cpu_families.txt
> > > >
> > > > It looks like the PowerPC 7457 is Book3S and the PowerPC e6500 is
> > > > BookE.  Is that why you think I require a Book3S KVM?  Exactly why do
> > > > you feel this way?  Also would that mean my team would need to go and
> > > > buy a board with a Book3S processor?
> > >
> > > CCing the PPC maintainer.  There are aspects of BookE and Book3S that
> > > are different and not really interchangeable in the privileged
> interface.
> > >
> > > Paolo
> > >
> > > > -Thanks!, Wayne Li
> > > >
> > > > From my understanding
> > > >
> > > > On Wed, Dec 11, 2019 at 7:16 PM Paolo Bonzini <pbonzini@redhat.com
> > > > <mailto:pbonzini@redhat.com>> wrote:
> > > >
> > > >     On 11/12/19 22:23, Wayne Li wrote:
> > > >     >
> > > >     > Now I am fairly sure KVM is actually enabled on the system.
> > > Finding
> > > >     > that out was another story that spanned a couple of months.
> But
> > > long
> > > >     > story short, lsmod doesn't show that the KVM kernel module is
> > > >     running.
> > > >     > But that's because KVM is built-in and it can't actually be
> built
> > > as a
> > > >     > loadable kernel module in this particular system.
> > > >     >
> > > >     > So I'm not really sure what could be the problem.  Though I was
> > > >     thinking
> > > >     > if I understood the error better that might help?  Following
> the
> > > >     code I
> > > >     > see that the "Missing PVR setting capability." is called when a
> > > >     variable
> > > >     > called "cap_segstate" is 0:
> > > >     >
> > > >     > if (!cap_segstate) {
> > > >     >             fprintf(stderr, "kvm error: missing PVR setting
> > > >     capability\n");
> > > >     >             return -ENOSYS;
> > > >     > }
> > > >     >
> > > >     > And the cap_segstate variable is set by the following function:
> > > >     >
> > > >     > cap_segstate = kvm_check_extension(s, KVM_CAP_PPC_SEGSTATE);
> > > >
> > > >     You are not saying how you are running QEMU.  I think you are
> using a
> > > >     CPU model that requires a Book3S KVM.
> > > >
> > > >     Paolo
> > > >
> > >
> > >
>
> --
> David Gibson                    | I'll have my music baroque, and my code
> david AT gibson.dropbear.id.au  | minimalist, thank you.  NOT _the_
> _other_
>                                 | _way_ _around_!
> http://www.ozlabs.org/~dgibson
>

[-- Attachment #2: Type: text/html, Size: 7332 bytes --]