Re: Intercepting RDTSC instruction by causing a VMEXIT

[Arnabjyoti Kalita <akalita@cs.stonybrook.edu>]

Hello Sean,

Thank you for all your help. I've now managed to cause a VMEXIT in the
guest. I've also identified which guest registers to access to read
and record the tsc counter value when the vmexit handler is called in
userspace.

Best Regards,
Arnabjyoti Kalita

On Tue, Apr 27, 2021 at 7:49 AM Arnabjyoti Kalita
<akalita@cs.stonybrook.edu> wrote:
>
> Hello Sean,
>
> Thank you very much again. I didn't know about the tracepointing
> methodology. This is very helpful, indeed.
>
> In addition to my requirement of recording tsc values, I am trying to
> track when the VMEXITs happen in userspace(QEMU). This allows me to
> correlate recorded TSC values with the VMEXIT sequence. So, causing a
> VMEXIT in userspace is absolutely essential to me and I would love to
> have some pointers on how to do it.
>
> I run a server within the guest and I'm okay with performance dropping
> down to within a factor of 2 compared to running the server on native
> hardware. I run this experiment for about 20-30 seconds.
>
> Best Regards,
> Arnabjyoti Kalita
>
>
> On Mon, Apr 26, 2021 at 9:35 PM Sean Christopherson <seanjc@google.com> wrote:
> >
> > On Mon, Apr 26, 2021, Arnabjyoti Kalita wrote:
> > > Hello Sean,
> > >
> > > Thank you very much for your answer again. I could actually see that the
> > > RDTSC handler gets called successfully. I added a "printk" statement to the
> > > handler and I can see the messages being printed out to the kernel syslog.
> > >
> > > However, I am not sure if a VMEXIT is happening in userspace. I use QEMU
> > > and I do not see any notification from QEMU that tells me that a VMEXIT
> > > happened due to RDTSC being executed. Is there a mechanism to confirm this?
> >
> > Without further modification, KVM will _not_ exit to userspace in this case.
> >
> > > My actual requirement to record tsc values read out as a result of RDTSC
> > > execution still stands.
> >
> > Your requirement didn't clarify that userspace needed to record the values ;-)
> >
> > Forcing an exit to userspace is very doable, but will tank guest performance and
> > possibly interfere with whatever you're trying to do.  I would try adding a
> > tracepoint and using that to capture the TSC values.  Adding guest RIP, etc...
> > as needed is trivial.
> >
> > diff --git a/arch/x86/kvm/trace.h b/arch/x86/kvm/trace.h
> > index a61c015870e3..e962e813ba04 100644
> > --- a/arch/x86/kvm/trace.h
> > +++ b/arch/x86/kvm/trace.h
> > @@ -821,6 +821,23 @@ TRACE_EVENT(
> >                   __entry->gpa_match ? "GPA" : "GVA")
> >  );
> >
> > +TRACE_EVENT(kvm_emulate_rdtsc,
> > +       TP_PROTO(unsigned int vcpu_id, __u64 tsc),
> > +       TP_ARGS(vcpu_id, tsc),
> > +
> > +       TP_STRUCT__entry(
> > +               __field( unsigned int,  vcpu_id         )
> > +               __field(        __u64,  tsc             )
> > +       ),
> > +
> > +       TP_fast_assign(
> > +               __entry->vcpu_id                = vcpu_id;
> > +               __entry->tsc                    = tsc;
> > +       ),
> > +
> > +       TP_printk("vcpu=%u tsc=%llu", __entry->vcpu_id, __entry->tsc)
> > +);
> > +
> >  TRACE_EVENT(kvm_write_tsc_offset,
> >         TP_PROTO(unsigned int vcpu_id, __u64 previous_tsc_offset,
> >                  __u64 next_tsc_offset),
> > diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
> > index 61bf0b86770b..1fbeef520349 100644
> > --- a/arch/x86/kvm/vmx/vmx.c
> > +++ b/arch/x86/kvm/vmx/vmx.c
> > @@ -5254,6 +5254,8 @@ static int handle_rdtsc(struct kvm_vcpu *vcpu)
> >  {
> >         u64 tsc = kvm_read_l1_tsc(vcpu, rdtsc());
> >
> > +       trace_kvm_emulate_rdtsc(vcpu->vcpu_id, tsc);
> > +
> >         kvm_rax_write(vcpu, tsc & -1u);
> >         kvm_rdx_write(vcpu, (tsc >> 32) & -1u);
> >         return kvm_skip_emulated_instruction(vcpu);