(no subject)

[Wayne Li]

[-- Attachment #1: Type: text/plain, Size: 1536 bytes --]

Dear QEMU list members,

This will kind of be a repost but I'd like to post my question again
because I've gained some more knowledge that makes me feel that my question
would be easier to answer.  So we developed a custom-made QEMU VM that
emulates a custom machine that has an e5500 processor.  I'm running this VM
on a T4240-RDB board which has an e6500 processor and I'm trying to get the
VM running with KVM enabled.  The problem I'm having is the program counter
refuses to increment at all.  It just stays at the address 0xFFFFFFFC.  On
a run without KVM enabled, the VM will also start executing at this same
address but the program counter beings to increment immediately.  I know
this is a custom QEMU VM and maybe some of the startup stuff we do could be
causing problems, but what could possibly stop the program counter from
incrementing altogether?

Also, I do have another side question.  When running with KVM enabled, I
see the kernel-level ioctl call KVM_RUN running and then returning over and
over again (by the way before the VM kinda grinds to a halt I only see QEMU
make the KVM_RUN call twice, but the kernel-level ioctl function is being
called over and over again for some reason).  And each time the KVM_RUN
call returns, the return-from-interrupt takes the VM to the address
0xFFFFFFFC.  What is the KVM_RUN ioctl call used for?  Why is it being
called over and over again?  Maybe if I understood this better I'd be able
to figure out what's stopping my program counter from incrementing.

-Thanks, Wayne Li

[-- Attachment #2: Type: text/html, Size: 1669 bytes --]

Re: Program counter stuck at 0xFFFFFFFC when emulating e5500 processor on T4240-RDB board

[Philippe Mathieu-Daudé]

Hello,

On 2/20/20 6:40 AM, Wayne Li wrote:
> Dear QEMU list members,

No subject: it is unlikely your question get noticed...

Also you didn't Cc'ed the people who might help you (doing that for you):

./scripts/get_maintainer.pl -f target/ppc/cpu-models.c
David Gibson <david@gibson.dropbear.id.au> (maintainer:PowerPC TCG CPUs)
qemu-ppc@nongnu.org (open list:PowerPC TCG CPUs)
qemu-devel@nongnu.org (open list:All patches CC here)

> This will kind of be a repost but I'd like to post my question again 
> because I've gained some more knowledge that makes me feel that my 
> question would be easier to answer.  So we developed a custom-made QEMU 
> VM that emulates a custom machine that has an e5500 processor.  I'm 
> running this VM on a T4240-RDB board which has an e6500 processor and 
> I'm trying to get the VM running with KVM enabled.  The problem I'm 
> having is the program counter refuses to increment at all.  It just 
> stays at the address 0xFFFFFFFC.  On a run without KVM enabled, the VM 
> will also start executing at this same address but the program counter 
> beings to increment immediately.  I know this is a custom QEMU VM and 
> maybe some of the startup stuff we do could be causing problems, but 
> what could possibly stop the program counter from incrementing altogether?
> 
> Also, I do have another side question.  When running with KVM enabled, I 
> see the kernel-level ioctl call KVM_RUN running and then returning over 
> and over again (by the way before the VM kinda grinds to a halt I only 
> see QEMU make the KVM_RUN call twice, but the kernel-level ioctl 
> function is being called over and over again for some reason).  And each 
> time the KVM_RUN call returns, the return-from-interrupt takes the VM to 
> the address 0xFFFFFFFC.  What is the KVM_RUN ioctl call used for?  Why 
> is it being called over and over again?  Maybe if I understood this 
> better I'd be able to figure out what's stopping my program counter from 
> incrementing.
> 
> -Thanks, Wayne Li

Re:

[Peter Maydell]

On Thu, 20 Feb 2020 at 05:41, Wayne Li <waynli329@gmail.com> wrote:
> Also, I do have another side question.  When running with KVM enabled, I
>see the kernel-level ioctl call KVM_RUN running and then returning over
>and over again (by the way before the VM kinda grinds to a halt I only see
>QEMU make the KVM_RUN call twice, but the kernel-level ioctl function
>is being called over and over again for some reason).  And each time the
>KVM_RUN call returns, the return-from-interrupt takes the VM to the
>address 0xFFFFFFFC.  What is the KVM_RUN ioctl call used for?  Why
>is it being called over and over again?  Maybe if I understood this better
>I'd be able to figure out what's stopping my program counter from incrementing.

KVM_RUN is the ioctl which QEMU uses to tell the kernel "hey, please
start running the guest". The kernel will not return control to QEMU
(ie the syscall will not return) until something happens that needs
QEMU's intervention, of which the main one is "guest made a device
access that QEMU must handle". (You can see this run loop in
accel/kvm/kvm-all.c:kvm_cpu_exec()). So in normal operation,
QEMU will do a bunch of setup, and then call KVM_RUN, which
will cause the guest to run, probably for quite a long time. The
ioctl will return when the guest does some IO a QEMU-provided
device, and QEMU will then do that device emulation, set up the
kvm_run struct fields to hold the results of the device access,
and call KVM_RUN again. The cycle continues like this until the
VM is shut down. If the guest goes into an infinite loop, you
should expect that KVM_RUN will never return, as there's never
any need for the kernel to ask QEMU to do anything.

Also important to note: before we call KVM_RUN we push
the CPU register state up to the kernel (using various arch-specific
ioctls), and after that the 'live copy' of the data is in the kernel,
and values in the CPU state struct in QEMU are stale. We only
get the up to date data out of the kernel when we need it, by
calling kvm_cpu_synchronize_state(). So if your 0xfffffffc value
comes from the CPU state struct and you're reading it at a
point before the state has been synchronized back from the
kernel it's not going to be the actual PC value.

The KVM kernel API is comparatively well documented at
https://www.kernel.org/doc/html/latest/virt/kvm/api.html
though the section on KVM_RUN is pretty weak.

thanks
-- PMM

Re:

[Wayne Li]

[-- Attachment #1: Type: text/plain, Size: 3111 bytes --]

Thanks for the KVM_RUN explanation, that makes a lot of sense.  Though I'm
pretty sure I'm getting the live program counter value because I use the
"info registers" command in QEMU monitor and that calls
kvm_cpu_synchronize_state().  And also to be more sure, I added a few
assembly instructions into the kernel code that stored a value that
constantly incremented and the program counter into two general purpose
registers.  I was able to see both of these in QEMU monitor so the program
counter must actually be stuck at 0xfffffffc.

On Thu, Feb 20, 2020 at 3:57 AM Peter Maydell <peter.maydell@linaro.org>
wrote:

> On Thu, 20 Feb 2020 at 05:41, Wayne Li <waynli329@gmail.com> wrote:
> > Also, I do have another side question.  When running with KVM enabled, I
> >see the kernel-level ioctl call KVM_RUN running and then returning over
> >and over again (by the way before the VM kinda grinds to a halt I only see
> >QEMU make the KVM_RUN call twice, but the kernel-level ioctl function
> >is being called over and over again for some reason).  And each time the
> >KVM_RUN call returns, the return-from-interrupt takes the VM to the
> >address 0xFFFFFFFC.  What is the KVM_RUN ioctl call used for?  Why
> >is it being called over and over again?  Maybe if I understood this better
> >I'd be able to figure out what's stopping my program counter from
> incrementing.
>
> KVM_RUN is the ioctl which QEMU uses to tell the kernel "hey, please
> start running the guest". The kernel will not return control to QEMU
> (ie the syscall will not return) until something happens that needs
> QEMU's intervention, of which the main one is "guest made a device
> access that QEMU must handle". (You can see this run loop in
> accel/kvm/kvm-all.c:kvm_cpu_exec()). So in normal operation,
> QEMU will do a bunch of setup, and then call KVM_RUN, which
> will cause the guest to run, probably for quite a long time. The
> ioctl will return when the guest does some IO a QEMU-provided
> device, and QEMU will then do that device emulation, set up the
> kvm_run struct fields to hold the results of the device access,
> and call KVM_RUN again. The cycle continues like this until the
> VM is shut down. If the guest goes into an infinite loop, you
> should expect that KVM_RUN will never return, as there's never
> any need for the kernel to ask QEMU to do anything.
>
> Also important to note: before we call KVM_RUN we push
> the CPU register state up to the kernel (using various arch-specific
> ioctls), and after that the 'live copy' of the data is in the kernel,
> and values in the CPU state struct in QEMU are stale. We only
> get the up to date data out of the kernel when we need it, by
> calling kvm_cpu_synchronize_state(). So if your 0xfffffffc value
> comes from the CPU state struct and you're reading it at a
> point before the state has been synchronized back from the
> kernel it's not going to be the actual PC value.
>
> The KVM kernel API is comparatively well documented at
> https://www.kernel.org/doc/html/latest/virt/kvm/api.html
> though the section on KVM_RUN is pretty weak.
>
> thanks
> -- PMM
>

[-- Attachment #2: Type: text/html, Size: 3811 bytes --]

Re: Program counter stuck at 0xFFFFFFFC when emulating e5500 processor on T4240-RDB board

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 3021 bytes --]

On Thu, Feb 20, 2020 at 09:44:13AM +0100, Philippe Mathieu-Daudé wrote:
> Hello,
> 
> On 2/20/20 6:40 AM, Wayne Li wrote:
> > Dear QEMU list members,
> 
> No subject: it is unlikely your question get noticed...
> 
> Also you didn't Cc'ed the people who might help you (doing that for you):
> 
> ./scripts/get_maintainer.pl -f target/ppc/cpu-models.c
> David Gibson <david@gibson.dropbear.id.au> (maintainer:PowerPC TCG CPUs)
> qemu-ppc@nongnu.org (open list:PowerPC TCG CPUs)
> qemu-devel@nongnu.org (open list:All patches CC here)

Right.  Unfortunately the embedded cpu support is barely maintained,
so there's only a little bit of help I can give.

> > This will kind of be a repost but I'd like to post my question again
> > because I've gained some more knowledge that makes me feel that my
> > question would be easier to answer.  So we developed a custom-made QEMU
> > VM that emulates a custom machine that has an e5500 processor.  I'm
> > running this VM on a T4240-RDB board which has an e6500 processor and
> > I'm trying to get the VM running with KVM enabled.  The problem I'm
> > having is the program counter refuses to increment at all.  It just
> > stays at the address 0xFFFFFFFC.  On a run without KVM enabled, the VM
> > will also start executing at this same address but the program counter
> > beings to increment immediately.  I know this is a custom QEMU VM and
> > maybe some of the startup stuff we do could be causing problems, but
> > what could possibly stop the program counter from incrementing
> > altogether?
> > 
> > Also, I do have another side question.  When running with KVM enabled, I
> > see the kernel-level ioctl call KVM_RUN running and then returning over
> > and over again (by the way before the VM kinda grinds to a halt I only
> > see QEMU make the KVM_RUN call twice, but the kernel-level ioctl
> > function is being called over and over again for some reason).  And each
> > time the KVM_RUN call returns, the return-from-interrupt takes the VM to
> > the address 0xFFFFFFFC.  What is the KVM_RUN ioctl call used for?  Why
> > is it being called over and over again?  Maybe if I understood this
> > better I'd be able to figure out what's stopping my program counter from
> > incrementing.

I don't really know if KVM even attempts to support e5500 cpus.  We'd
need to at least know what your host kernel is and what KVM variant it
is using.  What's the instruction at 0xfffffffc which seems not to be
handled here properly?  It sounds like you might be getting some sort
of rolling trap, because KVM isn't handling that instruction correctly.

The KVM_RUN is what causes KVM to actual enter the guest and execute
instructions.  It returns when something triggers a guest exit - a
trap or various other events can do this.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]