Extracting PC information from QEMU/KVM during single-step

Extracting PC information from QEMU/KVM during single-step

[Steven Raasch]

[-- Attachment #1: Type: text/plain, Size: 1698 bytes --]

Hi -

I'm trying to create a hack that will allow me to extract an instruction
trace from QEMU/KVM (i386). The KVM part is important (see below).

Background:

   - I have used KVM to create a snapshot of a windows-10 guest running a
   graphics-intensive app. The *original* issue is that the app does not
   execute correctly when re-started from the snapshot using TCG (it doesn't
   crash, but it doesn't run correctly, either). So, using the existing "-d
   in_asm" won't work. It seemed to me that hacking in tracing to KVM should
   be easier than figuring out why the app doesn't work.
   - I've poked around the tracing mechanism in the TCG, and extracted what
   I need to dump instructions and then added that to kvm_cpu_exec().
   - I'm setting DEBUG & single-step modes by calling cpu_single_step()
   from the top of kvm_vcpu_thread_fn().
   - in kvm_cpu_exec() I wait until I get a KVM_EXIT_DEBUG signal before
   logging the instruction.

I have the output of TCG "-d in_asm" from the beginning of the execution,
and I'm comparing the KVM output with that.

What I don't have right is the PC of the instruction that's been executed.
The TCG is clearly sane, but the KVM output is not.

My best thought was to extract the PC from kvm_run (run->debug.arch.pc)
after the KVM_RUN ioctl, but that doesn't match up. I also tried
kvm_vcpu_ioctl() with KVM_GET_REGS, and grabbing the rip from cpu->env.rip.
I didn't expect any of these to be *exactly* right, but I thought they
would lead me to something sane.

Using run->debug.arch.pc gives me the right address for the first
instruction, but nothing makes sense after that.

Can anyone help me get onto the right track?

Thanks!

-Steve

[-- Attachment #2: Type: text/html, Size: 7789 bytes --]

Re: Extracting PC information from QEMU/KVM during single-step

[Alexander Bulekov]

On 210623 1514, Steven Raasch wrote:
> Hi -
> 
> I'm trying to create a hack that will allow me to extract an instruction
> trace from QEMU/KVM (i386). The KVM part is important (see below).
> 
> Background:
> 
>    - I have used KVM to create a snapshot of a windows-10 guest running a
>    graphics-intensive app. The *original* issue is that the app does not
>    execute correctly when re-started from the snapshot using TCG (it doesn't
>    crash, but it doesn't run correctly, either). So, using the existing "-d
>    in_asm" won't work. It seemed to me that hacking in tracing to KVM should
>    be easier than figuring out why the app doesn't work.
>    - I've poked around the tracing mechanism in the TCG, and extracted what
>    I need to dump instructions and then added that to kvm_cpu_exec().
>    - I'm setting DEBUG & single-step modes by calling cpu_single_step()
>    from the top of kvm_vcpu_thread_fn().
>    - in kvm_cpu_exec() I wait until I get a KVM_EXIT_DEBUG signal before
>    logging the instruction.
> 
> I have the output of TCG "-d in_asm" from the beginning of the execution,
> and I'm comparing the KVM output with that.
> 
> What I don't have right is the PC of the instruction that's been executed.
> The TCG is clearly sane, but the KVM output is not.
> 
> My best thought was to extract the PC from kvm_run (run->debug.arch.pc)
> after the KVM_RUN ioctl, but that doesn't match up. I also tried
> kvm_vcpu_ioctl() with KVM_GET_REGS, and grabbing the rip from cpu->env.rip.
> I didn't expect any of these to be *exactly* right, but I thought they
> would lead me to something sane.
> 
> Using run->debug.arch.pc gives me the right address for the first
> instruction, but nothing makes sense after that.
> 
> Can anyone help me get onto the right track?
> 
> Thanks!
> 
> -Steve

Is there some reason you can't do this using qemu's gdbstub and gdb? It
supports single-stepping under KVM. 

-Alex

Re: Extracting PC information from QEMU/KVM during single-step

[Steven Raasch]

[-- Attachment #1: Type: text/plain, Size: 2756 bytes --]

Alex -

Our application is similar to a high-end game in that it adjusts it's
code-path depending on how "fast" it is running. Our previous attempts to
trace it's execution have run into issues due to the overhead of the trace
collection.

My hope had been that by using QEMU, I could control the "hardware" that
the app is running on such that the app doesn't know that it's not running
at full speed. In my mind, this means instrumenting the VM itself.

Using GDB is my second choice, as I'm not sure that I can gather the trace
w/out slowing down the app's execution. I haven't dug into how GDB
interacts with QEMU and GDB-scripting as of yet.

-Steve


On Wed, Jun 23, 2021 at 8:46 PM Alexander Bulekov <alxndr@bu.edu> wrote:

> On 210623 1514, Steven Raasch wrote:
> > Hi -
> >
> > I'm trying to create a hack that will allow me to extract an instruction
> > trace from QEMU/KVM (i386). The KVM part is important (see below).
> >
> > Background:
> >
> >    - I have used KVM to create a snapshot of a windows-10 guest running a
> >    graphics-intensive app. The *original* issue is that the app does not
> >    execute correctly when re-started from the snapshot using TCG (it
> doesn't
> >    crash, but it doesn't run correctly, either). So, using the existing
> "-d
> >    in_asm" won't work. It seemed to me that hacking in tracing to KVM
> should
> >    be easier than figuring out why the app doesn't work.
> >    - I've poked around the tracing mechanism in the TCG, and extracted
> what
> >    I need to dump instructions and then added that to kvm_cpu_exec().
> >    - I'm setting DEBUG & single-step modes by calling cpu_single_step()
> >    from the top of kvm_vcpu_thread_fn().
> >    - in kvm_cpu_exec() I wait until I get a KVM_EXIT_DEBUG signal before
> >    logging the instruction.
> >
> > I have the output of TCG "-d in_asm" from the beginning of the execution,
> > and I'm comparing the KVM output with that.
> >
> > What I don't have right is the PC of the instruction that's been
> executed.
> > The TCG is clearly sane, but the KVM output is not.
> >
> > My best thought was to extract the PC from kvm_run (run->debug.arch.pc)
> > after the KVM_RUN ioctl, but that doesn't match up. I also tried
> > kvm_vcpu_ioctl() with KVM_GET_REGS, and grabbing the rip from
> cpu->env.rip.
> > I didn't expect any of these to be *exactly* right, but I thought they
> > would lead me to something sane.
> >
> > Using run->debug.arch.pc gives me the right address for the first
> > instruction, but nothing makes sense after that.
> >
> > Can anyone help me get onto the right track?
> >
> > Thanks!
> >
> > -Steve
>
> Is there some reason you can't do this using qemu's gdbstub and gdb? It
> supports single-stepping under KVM.
>
> -Alex
>

[-- Attachment #2: Type: text/html, Size: 3573 bytes --]

Re: Extracting PC information from QEMU/KVM during single-step

[Alexander Bulekov]

I see. I haven't encountered the corrupted RIP problem you described -
maybe someone else has seen it.
Even if the overhead of the gdb ipc is prohibitive, looking at gdbstub.c
might be useful for comparing with your tracing code.  Under the hood,
the gdb stub is calling the same cpu_single_step function you mentioned,
and has code for reading the registers.
Good luck!
-Alex

On 210624 0849, Steven Raasch wrote:
> Alex -
> 
> Our application is similar to a high-end game in that it adjusts it's
> code-path depending on how "fast" it is running. Our previous attempts to
> trace it's execution have run into issues due to the overhead of the trace
> collection.
> 
> My hope had been that by using QEMU, I could control the "hardware" that
> the app is running on such that the app doesn't know that it's not running
> at full speed. In my mind, this means instrumenting the VM itself.
> 
> Using GDB is my second choice, as I'm not sure that I can gather the trace
> w/out slowing down the app's execution. I haven't dug into how GDB
> interacts with QEMU and GDB-scripting as of yet.
> 
> -Steve
> 
> 
> On Wed, Jun 23, 2021 at 8:46 PM Alexander Bulekov <alxndr@bu.edu> wrote:
> 
> > On 210623 1514, Steven Raasch wrote:
> > > Hi -
> > >
> > > I'm trying to create a hack that will allow me to extract an instruction
> > > trace from QEMU/KVM (i386). The KVM part is important (see below).
> > >
> > > Background:
> > >
> > >    - I have used KVM to create a snapshot of a windows-10 guest running a
> > >    graphics-intensive app. The *original* issue is that the app does not
> > >    execute correctly when re-started from the snapshot using TCG (it
> > doesn't
> > >    crash, but it doesn't run correctly, either). So, using the existing
> > "-d
> > >    in_asm" won't work. It seemed to me that hacking in tracing to KVM
> > should
> > >    be easier than figuring out why the app doesn't work.
> > >    - I've poked around the tracing mechanism in the TCG, and extracted
> > what
> > >    I need to dump instructions and then added that to kvm_cpu_exec().
> > >    - I'm setting DEBUG & single-step modes by calling cpu_single_step()
> > >    from the top of kvm_vcpu_thread_fn().
> > >    - in kvm_cpu_exec() I wait until I get a KVM_EXIT_DEBUG signal before
> > >    logging the instruction.
> > >
> > > I have the output of TCG "-d in_asm" from the beginning of the execution,
> > > and I'm comparing the KVM output with that.
> > >
> > > What I don't have right is the PC of the instruction that's been
> > executed.
> > > The TCG is clearly sane, but the KVM output is not.
> > >
> > > My best thought was to extract the PC from kvm_run (run->debug.arch.pc)
> > > after the KVM_RUN ioctl, but that doesn't match up. I also tried
> > > kvm_vcpu_ioctl() with KVM_GET_REGS, and grabbing the rip from
> > cpu->env.rip.
> > > I didn't expect any of these to be *exactly* right, but I thought they
> > > would lead me to something sane.
> > >
> > > Using run->debug.arch.pc gives me the right address for the first
> > > instruction, but nothing makes sense after that.
> > >
> > > Can anyone help me get onto the right track?
> > >
> > > Thanks!
> > >
> > > -Steve
> >
> > Is there some reason you can't do this using qemu's gdbstub and gdb? It
> > supports single-stepping under KVM.
> >
> > -Alex
> >

Re: Extracting PC information from QEMU/KVM during single-step

[Peter Maydell]

On Wed, 23 Jun 2021 at 22:10, Steven Raasch <sraasch@gmail.com> wrote:
> I have used KVM to create a snapshot of a windows-10 guest running a graphics-intensive app. The *original* issue is that the app does not execute correctly when re-started from the snapshot using TCG (it doesn't crash, but it doesn't run correctly, either).

I'm not sure that taking a snapshot with KVM and then resuming under TCG
is really tested. So I'm not very surprised that it doesn't work.

> I'm setting DEBUG & single-step modes by calling cpu_single_step() from the top of kvm_vcpu_thread_fn().
> in kvm_cpu_exec() I wait until I get a KVM_EXIT_DEBUG signal before logging the instruction.

If your app can cope with the slowdown involved in taking a VM exit
after every instruction (which will be massive), then it can probably
also handle the extra overhead on top of that of the gdbstub communication
protocol. So it's probably simplest just to connect to QEMU's gdbstub and
do the single-stepping that way.

The other approach to this would be to see if intel's perf monitor
stuff (which I know nothing about) has some kind of execution-trace
capture support and if that works when passing through the PMU to a
KVM guest.

thanks
-- PMM

Re: Extracting PC information from QEMU/KVM during single-step

[Steven Raasch]

[-- Attachment #1: Type: text/plain, Size: 2297 bytes --]

Understood with your KVM/TCG snapshot comment. I thought it was worth a try.

NOTE: I do not yet understand how gdb interacts with the virtual machine. I
have experience with GDB, but only at a linux app-debug level. I don't grok
how gdb on a linux host works with QEMU running a windows guest.
My *assumption* is that the VM continues to run while an app is being
debugged with GDB can be stopped, stepped, etc. If this is the case, I
would expect that the VM's sense of time will continue to move forward
while the app is paused. This would be an issue for my time-sensitive app.

If I slow down the entire QEMU system with my hacks, then my expectation is
that the time for both the VM and the app will slow down similarly (if I
decouple the VM time from real-world time using the -rtc command-line
argument).

So...
   1) Are my assumptions close?
   2) Can someone point me to information on using gdb with QEMU/KVM?

Thanks!
-S




On Thu, Jun 24, 2021 at 11:23 AM Peter Maydell <peter.maydell@linaro.org>
wrote:

> On Wed, 23 Jun 2021 at 22:10, Steven Raasch <sraasch@gmail.com> wrote:
> > I have used KVM to create a snapshot of a windows-10 guest running a
> graphics-intensive app. The *original* issue is that the app does not
> execute correctly when re-started from the snapshot using TCG (it doesn't
> crash, but it doesn't run correctly, either).
>
> I'm not sure that taking a snapshot with KVM and then resuming under TCG
> is really tested. So I'm not very surprised that it doesn't work.
>
> > I'm setting DEBUG & single-step modes by calling cpu_single_step() from
> the top of kvm_vcpu_thread_fn().
> > in kvm_cpu_exec() I wait until I get a KVM_EXIT_DEBUG signal before
> logging the instruction.
>
> If your app can cope with the slowdown involved in taking a VM exit
> after every instruction (which will be massive), then it can probably
> also handle the extra overhead on top of that of the gdbstub communication
> protocol. So it's probably simplest just to connect to QEMU's gdbstub and
> do the single-stepping that way.
>
> The other approach to this would be to see if intel's perf monitor
> stuff (which I know nothing about) has some kind of execution-trace
> capture support and if that works when passing through the PMU to a
> KVM guest.
>
> thanks
> -- PMM
>

[-- Attachment #2: Type: text/html, Size: 2955 bytes --]

Re: Extracting PC information from QEMU/KVM during single-step

[Peter Maydell]

On Thu, 24 Jun 2021 at 18:09, Steven Raasch <sraasch@gmail.com> wrote:
> NOTE: I do not yet understand how gdb interacts with the virtual machine. I have experience with GDB, but only at a linux app-debug level. I don't grok how gdb on a linux host works with QEMU running a windows guest.
> My *assumption* is that the VM continues to run while an app is being debugged with GDB can be stopped, stepped, etc. If this is the case, I would expect that the VM's sense of time will continue to move forward while the app is paused. This would be an issue for my time-sensitive app.

No, that's not how it works. A gdb connected to QEMU's gdbstub is a
bit like a hardware JTAG debugger connected to a real CPU, if that
helps. When gdb gets control the entire VM is stopped; stepping
steps one instruction of whatever the VM is doing. gdb and the
gdbstub have no understanding of individual processes running
inside the guest OS -- single stepping will happily step through
the app, into interrupt handlers, across the OS context switching
and into other processes, etc.

-- PMM

Re: Extracting PC information from QEMU/KVM during single-step

[Steven Raasch]

[-- Attachment #1: Type: text/plain, Size: 1362 bytes --]

Peter -

Thanks for the clarification, that makes perfect sense.

In this case, using GDB is clearly the best way to go about the tracing.

I appreciate your time!

-S

On Thu, Jun 24, 2021 at 1:51 PM Peter Maydell <peter.maydell@linaro.org>
wrote:

> On Thu, 24 Jun 2021 at 18:09, Steven Raasch <sraasch@gmail.com> wrote:
> > NOTE: I do not yet understand how gdb interacts with the virtual
> machine. I have experience with GDB, but only at a linux app-debug level. I
> don't grok how gdb on a linux host works with QEMU running a windows guest.
> > My *assumption* is that the VM continues to run while an app is being
> debugged with GDB can be stopped, stepped, etc. If this is the case, I
> would expect that the VM's sense of time will continue to move forward
> while the app is paused. This would be an issue for my time-sensitive app.
>
> No, that's not how it works. A gdb connected to QEMU's gdbstub is a
> bit like a hardware JTAG debugger connected to a real CPU, if that
> helps. When gdb gets control the entire VM is stopped; stepping
> steps one instruction of whatever the VM is doing. gdb and the
> gdbstub have no understanding of individual processes running
> inside the guest OS -- single stepping will happily step through
> the app, into interrupt handlers, across the OS context switching
> and into other processes, etc.
>
> -- PMM
>

[-- Attachment #2: Type: text/html, Size: 1857 bytes --]