Re: Intercepting RDTSC instruction by causing a VMEXIT

[Arnabjyoti Kalita <akalita@cs.stonybrook.edu>]

Hello Sean,

Thank you very much again. I didn't know about the tracepointing
methodology. This is very helpful, indeed.

In addition to my requirement of recording tsc values, I am trying to
track when the VMEXITs happen in userspace(QEMU). This allows me to
correlate recorded TSC values with the VMEXIT sequence. So, causing a
VMEXIT in userspace is absolutely essential to me and I would love to
have some pointers on how to do it.

I run a server within the guest and I'm okay with performance dropping
down to within a factor of 2 compared to running the server on native
hardware. I run this experiment for about 20-30 seconds.

Best Regards,
Arnabjyoti Kalita


On Mon, Apr 26, 2021 at 9:35 PM Sean Christopherson <seanjc@google.com> wrote:
>
> On Mon, Apr 26, 2021, Arnabjyoti Kalita wrote:
> > Hello Sean,
> >
> > Thank you very much for your answer again. I could actually see that the
> > RDTSC handler gets called successfully. I added a "printk" statement to the
> > handler and I can see the messages being printed out to the kernel syslog.
> >
> > However, I am not sure if a VMEXIT is happening in userspace. I use QEMU
> > and I do not see any notification from QEMU that tells me that a VMEXIT
> > happened due to RDTSC being executed. Is there a mechanism to confirm this?
>
> Without further modification, KVM will _not_ exit to userspace in this case.
>
> > My actual requirement to record tsc values read out as a result of RDTSC
> > execution still stands.
>
> Your requirement didn't clarify that userspace needed to record the values ;-)
>
> Forcing an exit to userspace is very doable, but will tank guest performance and
> possibly interfere with whatever you're trying to do.  I would try adding a
> tracepoint and using that to capture the TSC values.  Adding guest RIP, etc...
> as needed is trivial.
>
> diff --git a/arch/x86/kvm/trace.h b/arch/x86/kvm/trace.h
> index a61c015870e3..e962e813ba04 100644
> --- a/arch/x86/kvm/trace.h
> +++ b/arch/x86/kvm/trace.h
> @@ -821,6 +821,23 @@ TRACE_EVENT(
>                   __entry->gpa_match ? "GPA" : "GVA")
>  );
>
> +TRACE_EVENT(kvm_emulate_rdtsc,
> +       TP_PROTO(unsigned int vcpu_id, __u64 tsc),
> +       TP_ARGS(vcpu_id, tsc),
> +
> +       TP_STRUCT__entry(
> +               __field( unsigned int,  vcpu_id         )
> +               __field(        __u64,  tsc             )
> +       ),
> +
> +       TP_fast_assign(
> +               __entry->vcpu_id                = vcpu_id;
> +               __entry->tsc                    = tsc;
> +       ),
> +
> +       TP_printk("vcpu=%u tsc=%llu", __entry->vcpu_id, __entry->tsc)
> +);
> +
>  TRACE_EVENT(kvm_write_tsc_offset,
>         TP_PROTO(unsigned int vcpu_id, __u64 previous_tsc_offset,
>                  __u64 next_tsc_offset),
> diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
> index 61bf0b86770b..1fbeef520349 100644
> --- a/arch/x86/kvm/vmx/vmx.c
> +++ b/arch/x86/kvm/vmx/vmx.c
> @@ -5254,6 +5254,8 @@ static int handle_rdtsc(struct kvm_vcpu *vcpu)
>  {
>         u64 tsc = kvm_read_l1_tsc(vcpu, rdtsc());
>
> +       trace_kvm_emulate_rdtsc(vcpu->vcpu_id, tsc);
> +
>         kvm_rax_write(vcpu, tsc & -1u);
>         kvm_rdx_write(vcpu, (tsc >> 32) & -1u);
>         return kvm_skip_emulated_instruction(vcpu);