Thanks for the KVM_RUN explanation, that makes a lot of sense.  Though I'm
pretty sure I'm getting the live program counter value because I use the
"info registers" command in QEMU monitor and that calls
kvm_cpu_synchronize_state().  And also to be more sure, I added a few
assembly instructions into the kernel code that stored a value that
constantly incremented and the program counter into two general purpose
registers.  I was able to see both of these in QEMU monitor so the program
counter must actually be stuck at 0xfffffffc.

On Thu, Feb 20, 2020 at 3:57 AM Peter Maydell <peter.maydell@linaro.org>
wrote:

> On Thu, 20 Feb 2020 at 05:41, Wayne Li <waynli329@gmail.com> wrote:
> > Also, I do have another side question.  When running with KVM enabled, I
> >see the kernel-level ioctl call KVM_RUN running and then returning over
> >and over again (by the way before the VM kinda grinds to a halt I only see
> >QEMU make the KVM_RUN call twice, but the kernel-level ioctl function
> >is being called over and over again for some reason).  And each time the
> >KVM_RUN call returns, the return-from-interrupt takes the VM to the
> >address 0xFFFFFFFC.  What is the KVM_RUN ioctl call used for?  Why
> >is it being called over and over again?  Maybe if I understood this better
> >I'd be able to figure out what's stopping my program counter from
> incrementing.
>
> KVM_RUN is the ioctl which QEMU uses to tell the kernel "hey, please
> start running the guest". The kernel will not return control to QEMU
> (ie the syscall will not return) until something happens that needs
> QEMU's intervention, of which the main one is "guest made a device
> access that QEMU must handle". (You can see this run loop in
> accel/kvm/kvm-all.c:kvm_cpu_exec()). So in normal operation,
> QEMU will do a bunch of setup, and then call KVM_RUN, which
> will cause the guest to run, probably for quite a long time. The
> ioctl will return when the guest does some IO a QEMU-provided
> device, and QEMU will then do that device emulation, set up the
> kvm_run struct fields to hold the results of the device access,
> and call KVM_RUN again. The cycle continues like this until the
> VM is shut down. If the guest goes into an infinite loop, you
> should expect that KVM_RUN will never return, as there's never
> any need for the kernel to ask QEMU to do anything.
>
> Also important to note: before we call KVM_RUN we push
> the CPU register state up to the kernel (using various arch-specific
> ioctls), and after that the 'live copy' of the data is in the kernel,
> and values in the CPU state struct in QEMU are stale. We only
> get the up to date data out of the kernel when we need it, by
> calling kvm_cpu_synchronize_state(). So if your 0xfffffffc value
> comes from the CPU state struct and you're reading it at a
> point before the state has been synchronized back from the
> kernel it's not going to be the actual PC value.
>
> The KVM kernel API is comparatively well documented at
> https://www.kernel.org/doc/html/latest/virt/kvm/api.html
> though the section on KVM_RUN is pretty weak.
>
> thanks
> -- PMM
>