Re: Causing VMEXITs when kprobes are hit in the guest VM

[Arnabjyoti Kalita <akalita@cs.stonybrook.edu>]

Dear Sean and all,

When a VMEXIT happens of type "KVM_EXIT_DEBUG" because a hardware
breakpoint was triggered when an instruction was about to be executed,
does the instruction where the breakpoint was placed actually execute
before the VMEXIT happens?

I am attempting to record the occurrence of the debug exception in
userspace. I do not want to do anything extra with the debug
exception. I have modified the kernel code (handle_exception_nmi) to
do something like this -

case BP_VECTOR:
    /*
     * Update instruction length as we may reinject #BP from
     * user space while in guest debugging mode. Reading it for
     * #DB as well causes no harm, it is not used in that case.
     */
      vmx->vcpu.arch.event_exit_inst_len = vmcs_read32(VM_EXIT_INSTRUCTION_LEN);
      kvm_run->exit_reason = KVM_EXIT_DEBUG;
      ......
      kvm_run->debug.arch.pc = vmcs_readl(GUEST_CS_BASE) + rip;
      kvm_run->debug.arch.exception = ex_no;
      kvm_rip_write(vcpu, rip + vmcs_read32(VM_EXIT_INSTRUCTION_LEN));
   <---Change : update RIP here
      break;

This allows the guest to proceed after the hardware breakpoint
exception was triggered. However, the guest kernel keeps running into
page fault at arbitrary points in time. So, I'm not sure if I need to
handle something else too.

I have modified the userspace code to not trigger any exception, it
just records the occurence of this VMEXIT and lets the guest continue.

Is this the right approach?

Thank you very much.

Best Regards,
Arnabjyoti Kalita


On Fri, May 6, 2022 at 10:44 AM Arnabjyoti Kalita
<akalita@cs.stonybrook.edu> wrote:
>
> Dear Sean,
>
> Thank you very much for your answer.
>
> I managed to set a Hardware Breakpoint using the API you provided, in
> userspace, and managed to get a VMEXIT of type "KVM_EXIT_DEBUG". I
> will not re-inject the #BP since I do not necessarily want the guest
> to do anything with it. All I will do is just record the CPU ID of the
> cpu (in userspace) that caused this breakpoint and let the guest
> continue execution.
>
> Best Regards,
> Arnabjyoti Kalita
>
> On Wed, May 4, 2022 at 2:15 AM Sean Christopherson <seanjc@google.com> wrote:
> >
> > On Sun, May 01, 2022, Arnabjyoti Kalita wrote:
> > > Hello all,
> > >
> > > I intend to run a kernel module inside my guest VM. The kernel module
> > > sets kprobes on a couple of functions in the linux kernel. After
> > > registering the kprobes successfully, I can see that the kprobes are
> > > being hit repeatedly.
> > >
> > > I would like to cause a VMEXIT when these kprobes are hit. I know that
> > > kprobes use a breakpoint instruction (INT 3) to successfully execute
> > > the pre and post handlers. This would mean that the execution of the
> > > instruction INT 3 should technically cause a VMEXIT.
> >
> > No, it should cause #BP.  KVM doesn't intercept #BP by default because there's no
> > reason to do so.
> >
> > > However, I do not get any software exception type VMEXITs when these kprobes
> > > are hit.
> > >
> > > I have used the commands "perf kvm stat record" and "perf kvm stat
> > > report --event=vmexit" to try and observe the VMEXIT reasons and I do
> > > not see any VMEXIT of type "EXCEPTION_NMI" being returned in the
> > > period that the kprobe was being hit.
> > >
> > > My host uses a modified Linux kernel 5.8.0 while my guest runs a 4.4.0
> > > Linux kernel. Both the guest and the host use the x86_64 architecture.
> > > I am using QEMU version 5.0.1. What changes are needed in the Linux
> > > kernel to make sure that I get an exception in the form of a VMEXIT
> > > whenever the kprobes are hit?
> >
> > This can be done entirely from userspace by enabling KVM_GUESTDBG_USE_SW_BP, e.g.
> >
> >         struct kvm_guest_debug debug;
> >
> >         memset(&debug, 0, sizeof(debug));
> >         debug.control = KVM_GUESTDBG_ENABLE | KVM_GUESTDBG_USE_SW_BP;
> >         ioctl(vcpu->fd, KVM_SET_GUEST_DEBUG, &debug);
> >
> > That will intercept #BP and exit to userspace with KVM_EXIT_DEBUG.  Note, it's
> > userspace's responsibility to re-inject the #BP if userspace wants to forward the
> > #BP to the guest.
> >
> > There's a bit more info in Documentation/virt/kvm/api.rst under KVM_SET_GUEST_DEBUG.