KVM Archive on lore.kernel.org
 help / color / Atom feed
* Intercepting RDTSC instruction by causing a VMEXIT
@ 2021-04-17  4:54 Arnabjyoti Kalita
  2021-04-19 16:46 ` Sean Christopherson
  0 siblings, 1 reply; 7+ messages in thread
From: Arnabjyoti Kalita @ 2021-04-17  4:54 UTC (permalink / raw)
  To: kvm

Hello all,

I'm having a requirement to record values obtained by reading tsc clock.

The command line I use to start QEMU in KVM mode is as below -

sudo ./qemu-system-x86_64 -m 1024 --machine pc-i440fx-2.5 -cpu
qemu64,-vme,-x2apic,-kvmclock,+lahf_lm,+3dnowprefetch,+vmx -enable-kvm
-netdev tap,id=tap1,ifname=tap0,script=no,downscript=no -device
virtio-net-pci,netdev=tap1,mac=00:00:00:00:00:00 -drive
file=~/os_images_for_qemu/ubuntu-16.04.server.qcow2,format=qcow2,if=none,id=img-direct
-device virtio-blk-pci,drive=img-direct

I am using QEMU version 2.11.92 and the guest kernel is a
4.4.0-116-generic. I use the CPU model "qemu64" because I have a
requirement to create a snapshot of this guest and load the snapshot
in TCG mode. The generic CPU model helps, in this regard.

Now when the guest is running, I want to intercept all rdtsc
instructions and record the tsc clock values. I know that for this to
happen, the CPU_BASED_RDTSC_EXITING flag needs to exist for the
particular CPU model.

How do I start adding support for causing VMEXIT upon rdtsc execution?

I see that a fairly recent commit in QEMU helps adding nested VMX
controls to named CPU models, but not "qemu64". Can I extend this
commit to add these controls to "qemu64" as well? Will making this
change immediately add support for intercepting VMEXITS for "qemu64"
CPU?

Thank you very much.

Best Regards,
Arnabjyoti Kalita

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: Intercepting RDTSC instruction by causing a VMEXIT
  2021-04-17  4:54 Intercepting RDTSC instruction by causing a VMEXIT Arnabjyoti Kalita
@ 2021-04-19 16:46 ` Sean Christopherson
  2021-04-20  2:33   ` Arnabjyoti Kalita
  0 siblings, 1 reply; 7+ messages in thread
From: Sean Christopherson @ 2021-04-19 16:46 UTC (permalink / raw)
  To: Arnabjyoti Kalita; +Cc: kvm

On Sat, Apr 17, 2021, Arnabjyoti Kalita wrote:
> Hello all,
> 
> I'm having a requirement to record values obtained by reading tsc clock.
> 
> The command line I use to start QEMU in KVM mode is as below -
> 
> sudo ./qemu-system-x86_64 -m 1024 --machine pc-i440fx-2.5 -cpu
> qemu64,-vme,-x2apic,-kvmclock,+lahf_lm,+3dnowprefetch,+vmx -enable-kvm
> -netdev tap,id=tap1,ifname=tap0,script=no,downscript=no -device
> virtio-net-pci,netdev=tap1,mac=00:00:00:00:00:00 -drive
> file=~/os_images_for_qemu/ubuntu-16.04.server.qcow2,format=qcow2,if=none,id=img-direct
> -device virtio-blk-pci,drive=img-direct
> 
> I am using QEMU version 2.11.92 and the guest kernel is a
> 4.4.0-116-generic. I use the CPU model "qemu64" because I have a
> requirement to create a snapshot of this guest and load the snapshot
> in TCG mode. The generic CPU model helps, in this regard.
> 
> Now when the guest is running, I want to intercept all rdtsc
> instructions and record the tsc clock values. I know that for this to
> happen, the CPU_BASED_RDTSC_EXITING flag needs to exist for the
> particular CPU model.
> 
> How do I start adding support for causing VMEXIT upon rdtsc execution?

This requires a KVM change.  The below should do the trick.

diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index c05e6e2854b5..f000728e4319 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -2453,7 +2453,8 @@ static __init int setup_vmcs_config(struct vmcs_config *vmcs_conf,
              CPU_BASED_MWAIT_EXITING |
              CPU_BASED_MONITOR_EXITING |
              CPU_BASED_INVLPG_EXITING |
-             CPU_BASED_RDPMC_EXITING;
+             CPU_BASED_RDPMC_EXITING |
+             CPU_BASED_RDTSC_EXITING;

        opt = CPU_BASED_TPR_SHADOW |
              CPU_BASED_USE_MSR_BITMAPS |
@@ -5194,6 +5195,15 @@ static int handle_invlpg(struct kvm_vcpu *vcpu)
        return kvm_skip_emulated_instruction(vcpu);
 }

+static int handle_rdtsc(struct kvm_vcpu *vcpu)
+{
+       u64 tsc = kvm_read_l1_tsc(vcpu, rdtsc());
+
+       kvm_rax_write(vcpu, tsc & -1u);
+       kvm_rdx_write(vcpu, (tsc >> 32) & -1u);
+       return kvm_skip_emulated_instruction(vcpu);
+}
+
 static int handle_apic_access(struct kvm_vcpu *vcpu)
 {
        if (likely(fasteoi)) {
@@ -5605,6 +5615,7 @@ static int (*kvm_vmx_exit_handlers[])(struct kvm_vcpu *vcpu) = {
        [EXIT_REASON_INVD]                    = kvm_emulate_invd,
        [EXIT_REASON_INVLPG]                  = handle_invlpg,
        [EXIT_REASON_RDPMC]                   = kvm_emulate_rdpmc,
+       [EXIT_REASON_RDTSC]                   = handle_rdtsc,
        [EXIT_REASON_VMCALL]                  = kvm_emulate_hypercall,
        [EXIT_REASON_VMCLEAR]                 = handle_vmx_instruction,
        [EXIT_REASON_VMLAUNCH]                = handle_vmx_instruction,
 
> I see that a fairly recent commit in QEMU helps adding nested VMX
> controls to named CPU models, but not "qemu64". Can I extend this
> commit to add these controls to "qemu64" as well? Will making this
> change immediately add support for intercepting VMEXITS for "qemu64"
> CPU?

Are you actually running a nested guest?

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: Intercepting RDTSC instruction by causing a VMEXIT
  2021-04-19 16:46 ` Sean Christopherson
@ 2021-04-20  2:33   ` Arnabjyoti Kalita
       [not found]     ` <CAJGDS+GT1mKHz6K=qHQf54S_97ym=nRP12MfO6OSEOpLYGht=A@mail.gmail.com>
  0 siblings, 1 reply; 7+ messages in thread
From: Arnabjyoti Kalita @ 2021-04-20  2:33 UTC (permalink / raw)
  To: Sean Christopherson; +Cc: kvm

Hello Sean,

Thank you very much for your answer. I'm hoping the inlined changes
should be enough to see RDTSC interception.

No, I'm actually not running a nested guest, even though vmx is enabled.

Best Regards,
Arnabjyoti Kalita

On Mon, Apr 19, 2021 at 10:16 PM Sean Christopherson <seanjc@google.com> wrote:
>
> On Sat, Apr 17, 2021, Arnabjyoti Kalita wrote:
> > Hello all,
> >
> > I'm having a requirement to record values obtained by reading tsc clock.
> >
> > The command line I use to start QEMU in KVM mode is as below -
> >
> > sudo ./qemu-system-x86_64 -m 1024 --machine pc-i440fx-2.5 -cpu
> > qemu64,-vme,-x2apic,-kvmclock,+lahf_lm,+3dnowprefetch,+vmx -enable-kvm
> > -netdev tap,id=tap1,ifname=tap0,script=no,downscript=no -device
> > virtio-net-pci,netdev=tap1,mac=00:00:00:00:00:00 -drive
> > file=~/os_images_for_qemu/ubuntu-16.04.server.qcow2,format=qcow2,if=none,id=img-direct
> > -device virtio-blk-pci,drive=img-direct
> >
> > I am using QEMU version 2.11.92 and the guest kernel is a
> > 4.4.0-116-generic. I use the CPU model "qemu64" because I have a
> > requirement to create a snapshot of this guest and load the snapshot
> > in TCG mode. The generic CPU model helps, in this regard.
> >
> > Now when the guest is running, I want to intercept all rdtsc
> > instructions and record the tsc clock values. I know that for this to
> > happen, the CPU_BASED_RDTSC_EXITING flag needs to exist for the
> > particular CPU model.
> >
> > How do I start adding support for causing VMEXIT upon rdtsc execution?
>
> This requires a KVM change.  The below should do the trick.
>
> diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
> index c05e6e2854b5..f000728e4319 100644
> --- a/arch/x86/kvm/vmx/vmx.c
> +++ b/arch/x86/kvm/vmx/vmx.c
> @@ -2453,7 +2453,8 @@ static __init int setup_vmcs_config(struct vmcs_config *vmcs_conf,
>               CPU_BASED_MWAIT_EXITING |
>               CPU_BASED_MONITOR_EXITING |
>               CPU_BASED_INVLPG_EXITING |
> -             CPU_BASED_RDPMC_EXITING;
> +             CPU_BASED_RDPMC_EXITING |
> +             CPU_BASED_RDTSC_EXITING;
>
>         opt = CPU_BASED_TPR_SHADOW |
>               CPU_BASED_USE_MSR_BITMAPS |
> @@ -5194,6 +5195,15 @@ static int handle_invlpg(struct kvm_vcpu *vcpu)
>         return kvm_skip_emulated_instruction(vcpu);
>  }
>
> +static int handle_rdtsc(struct kvm_vcpu *vcpu)
> +{
> +       u64 tsc = kvm_read_l1_tsc(vcpu, rdtsc());
> +
> +       kvm_rax_write(vcpu, tsc & -1u);
> +       kvm_rdx_write(vcpu, (tsc >> 32) & -1u);
> +       return kvm_skip_emulated_instruction(vcpu);
> +}
> +
>  static int handle_apic_access(struct kvm_vcpu *vcpu)
>  {
>         if (likely(fasteoi)) {
> @@ -5605,6 +5615,7 @@ static int (*kvm_vmx_exit_handlers[])(struct kvm_vcpu *vcpu) = {
>         [EXIT_REASON_INVD]                    = kvm_emulate_invd,
>         [EXIT_REASON_INVLPG]                  = handle_invlpg,
>         [EXIT_REASON_RDPMC]                   = kvm_emulate_rdpmc,
> +       [EXIT_REASON_RDTSC]                   = handle_rdtsc,
>         [EXIT_REASON_VMCALL]                  = kvm_emulate_hypercall,
>         [EXIT_REASON_VMCLEAR]                 = handle_vmx_instruction,
>         [EXIT_REASON_VMLAUNCH]                = handle_vmx_instruction,
>
> > I see that a fairly recent commit in QEMU helps adding nested VMX
> > controls to named CPU models, but not "qemu64". Can I extend this
> > commit to add these controls to "qemu64" as well? Will making this
> > change immediately add support for intercepting VMEXITS for "qemu64"
> > CPU?
>
> Are you actually running a nested guest?

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: Intercepting RDTSC instruction by causing a VMEXIT
       [not found]     ` <CAJGDS+GT1mKHz6K=qHQf54S_97ym=nRP12MfO6OSEOpLYGht=A@mail.gmail.com>
@ 2021-04-26  1:51       ` Arnabjyoti Kalita
  2021-04-26 16:05       ` Sean Christopherson
  1 sibling, 0 replies; 7+ messages in thread
From: Arnabjyoti Kalita @ 2021-04-26  1:51 UTC (permalink / raw)
  To: Sean Christopherson; +Cc: kvm

Hello Sean,

Thank you very much for your answer again. I could actually see that
the RDTSC handler gets called successfully. I added a "printk"
statement to the handler and I can see the messages being printed out
to the kernel syslog.

However, I am not sure if a VMEXIT is happening in userspace. I use
QEMU and I do not see any notification from QEMU that tells me that a
VMEXIT happened due to RDTSC being executed. Is there a mechanism to
confirm this?

My actual requirement to record tsc values read out as a result of
RDTSC execution still stands.

Best Regards,
Arnabjyoti Kalita


On Mon, Apr 26, 2021 at 7:19 AM Arnabjyoti Kalita
<akalita@cs.stonybrook.edu> wrote:
>
> Hello Sean,
>
> Thank you very much for your answer again. I could actually see that the RDTSC handler gets called successfully. I added a "printk" statement to the handler and I can see the messages being printed out to the kernel syslog.
>
> However, I am not sure if a VMEXIT is happening in userspace. I use QEMU and I do not see any notification from QEMU that tells me that a VMEXIT happened due to RDTSC being executed. Is there a mechanism to confirm this?
>
> My actual requirement to record tsc values read out as a result of RDTSC execution still stands.
>
> Best Regards,
> Arnabjyoti Kalita
>
>
>
>
> On Tue, 20 Apr 2021, 08:03 Arnabjyoti Kalita, <akalita@cs.stonybrook.edu> wrote:
>>
>> Hello Sean,
>>
>> Thank you very much for your answer. I'm hoping the inlined changes
>> should be enough to see RDTSC interception.
>>
>> No, I'm actually not running a nested guest, even though vmx is enabled.
>>
>> Best Regards,
>> Arnabjyoti Kalita
>>
>> On Mon, Apr 19, 2021 at 10:16 PM Sean Christopherson <seanjc@google.com> wrote:
>> >
>> > On Sat, Apr 17, 2021, Arnabjyoti Kalita wrote:
>> > > Hello all,
>> > >
>> > > I'm having a requirement to record values obtained by reading tsc clock.
>> > >
>> > > The command line I use to start QEMU in KVM mode is as below -
>> > >
>> > > sudo ./qemu-system-x86_64 -m 1024 --machine pc-i440fx-2.5 -cpu
>> > > qemu64,-vme,-x2apic,-kvmclock,+lahf_lm,+3dnowprefetch,+vmx -enable-kvm
>> > > -netdev tap,id=tap1,ifname=tap0,script=no,downscript=no -device
>> > > virtio-net-pci,netdev=tap1,mac=00:00:00:00:00:00 -drive
>> > > file=~/os_images_for_qemu/ubuntu-16.04.server.qcow2,format=qcow2,if=none,id=img-direct
>> > > -device virtio-blk-pci,drive=img-direct
>> > >
>> > > I am using QEMU version 2.11.92 and the guest kernel is a
>> > > 4.4.0-116-generic. I use the CPU model "qemu64" because I have a
>> > > requirement to create a snapshot of this guest and load the snapshot
>> > > in TCG mode. The generic CPU model helps, in this regard.
>> > >
>> > > Now when the guest is running, I want to intercept all rdtsc
>> > > instructions and record the tsc clock values. I know that for this to
>> > > happen, the CPU_BASED_RDTSC_EXITING flag needs to exist for the
>> > > particular CPU model.
>> > >
>> > > How do I start adding support for causing VMEXIT upon rdtsc execution?
>> >
>> > This requires a KVM change.  The below should do the trick.
>> >
>> > diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
>> > index c05e6e2854b5..f000728e4319 100644
>> > --- a/arch/x86/kvm/vmx/vmx.c
>> > +++ b/arch/x86/kvm/vmx/vmx.c
>> > @@ -2453,7 +2453,8 @@ static __init int setup_vmcs_config(struct vmcs_config *vmcs_conf,
>> >               CPU_BASED_MWAIT_EXITING |
>> >               CPU_BASED_MONITOR_EXITING |
>> >               CPU_BASED_INVLPG_EXITING |
>> > -             CPU_BASED_RDPMC_EXITING;
>> > +             CPU_BASED_RDPMC_EXITING |
>> > +             CPU_BASED_RDTSC_EXITING;
>> >
>> >         opt = CPU_BASED_TPR_SHADOW |
>> >               CPU_BASED_USE_MSR_BITMAPS |
>> > @@ -5194,6 +5195,15 @@ static int handle_invlpg(struct kvm_vcpu *vcpu)
>> >         return kvm_skip_emulated_instruction(vcpu);
>> >  }
>> >
>> > +static int handle_rdtsc(struct kvm_vcpu *vcpu)
>> > +{
>> > +       u64 tsc = kvm_read_l1_tsc(vcpu, rdtsc());
>> > +
>> > +       kvm_rax_write(vcpu, tsc & -1u);
>> > +       kvm_rdx_write(vcpu, (tsc >> 32) & -1u);
>> > +       return kvm_skip_emulated_instruction(vcpu);
>> > +}
>> > +
>> >  static int handle_apic_access(struct kvm_vcpu *vcpu)
>> >  {
>> >         if (likely(fasteoi)) {
>> > @@ -5605,6 +5615,7 @@ static int (*kvm_vmx_exit_handlers[])(struct kvm_vcpu *vcpu) = {
>> >         [EXIT_REASON_INVD]                    = kvm_emulate_invd,
>> >         [EXIT_REASON_INVLPG]                  = handle_invlpg,
>> >         [EXIT_REASON_RDPMC]                   = kvm_emulate_rdpmc,
>> > +       [EXIT_REASON_RDTSC]                   = handle_rdtsc,
>> >         [EXIT_REASON_VMCALL]                  = kvm_emulate_hypercall,
>> >         [EXIT_REASON_VMCLEAR]                 = handle_vmx_instruction,
>> >         [EXIT_REASON_VMLAUNCH]                = handle_vmx_instruction,
>> >
>> > > I see that a fairly recent commit in QEMU helps adding nested VMX
>> > > controls to named CPU models, but not "qemu64". Can I extend this
>> > > commit to add these controls to "qemu64" as well? Will making this
>> > > change immediately add support for intercepting VMEXITS for "qemu64"
>> > > CPU?
>> >
>> > Are you actually running a nested guest?

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: Intercepting RDTSC instruction by causing a VMEXIT
       [not found]     ` <CAJGDS+GT1mKHz6K=qHQf54S_97ym=nRP12MfO6OSEOpLYGht=A@mail.gmail.com>
  2021-04-26  1:51       ` Arnabjyoti Kalita
@ 2021-04-26 16:05       ` Sean Christopherson
  2021-04-27  2:19         ` Arnabjyoti Kalita
  1 sibling, 1 reply; 7+ messages in thread
From: Sean Christopherson @ 2021-04-26 16:05 UTC (permalink / raw)
  To: Arnabjyoti Kalita; +Cc: kvm

On Mon, Apr 26, 2021, Arnabjyoti Kalita wrote:
> Hello Sean,
> 
> Thank you very much for your answer again. I could actually see that the
> RDTSC handler gets called successfully. I added a "printk" statement to the
> handler and I can see the messages being printed out to the kernel syslog.
> 
> However, I am not sure if a VMEXIT is happening in userspace. I use QEMU
> and I do not see any notification from QEMU that tells me that a VMEXIT
> happened due to RDTSC being executed. Is there a mechanism to confirm this?

Without further modification, KVM will _not_ exit to userspace in this case.

> My actual requirement to record tsc values read out as a result of RDTSC
> execution still stands.

Your requirement didn't clarify that userspace needed to record the values ;-)

Forcing an exit to userspace is very doable, but will tank guest performance and
possibly interfere with whatever you're trying to do.  I would try adding a
tracepoint and using that to capture the TSC values.  Adding guest RIP, etc...
as needed is trivial.

diff --git a/arch/x86/kvm/trace.h b/arch/x86/kvm/trace.h
index a61c015870e3..e962e813ba04 100644
--- a/arch/x86/kvm/trace.h
+++ b/arch/x86/kvm/trace.h
@@ -821,6 +821,23 @@ TRACE_EVENT(
                  __entry->gpa_match ? "GPA" : "GVA")
 );

+TRACE_EVENT(kvm_emulate_rdtsc,
+       TP_PROTO(unsigned int vcpu_id, __u64 tsc),
+       TP_ARGS(vcpu_id, tsc),
+
+       TP_STRUCT__entry(
+               __field( unsigned int,  vcpu_id         )
+               __field(        __u64,  tsc             )
+       ),
+
+       TP_fast_assign(
+               __entry->vcpu_id                = vcpu_id;
+               __entry->tsc                    = tsc;
+       ),
+
+       TP_printk("vcpu=%u tsc=%llu", __entry->vcpu_id, __entry->tsc)
+);
+
 TRACE_EVENT(kvm_write_tsc_offset,
        TP_PROTO(unsigned int vcpu_id, __u64 previous_tsc_offset,
                 __u64 next_tsc_offset),
diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index 61bf0b86770b..1fbeef520349 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -5254,6 +5254,8 @@ static int handle_rdtsc(struct kvm_vcpu *vcpu)
 {
        u64 tsc = kvm_read_l1_tsc(vcpu, rdtsc());

+       trace_kvm_emulate_rdtsc(vcpu->vcpu_id, tsc);
+
        kvm_rax_write(vcpu, tsc & -1u);
        kvm_rdx_write(vcpu, (tsc >> 32) & -1u);
        return kvm_skip_emulated_instruction(vcpu);

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: Intercepting RDTSC instruction by causing a VMEXIT
  2021-04-26 16:05       ` Sean Christopherson
@ 2021-04-27  2:19         ` Arnabjyoti Kalita
  2021-04-30 17:16           ` Arnabjyoti Kalita
  0 siblings, 1 reply; 7+ messages in thread
From: Arnabjyoti Kalita @ 2021-04-27  2:19 UTC (permalink / raw)
  To: Sean Christopherson; +Cc: kvm

Hello Sean,

Thank you very much again. I didn't know about the tracepointing
methodology. This is very helpful, indeed.

In addition to my requirement of recording tsc values, I am trying to
track when the VMEXITs happen in userspace(QEMU). This allows me to
correlate recorded TSC values with the VMEXIT sequence. So, causing a
VMEXIT in userspace is absolutely essential to me and I would love to
have some pointers on how to do it.

I run a server within the guest and I'm okay with performance dropping
down to within a factor of 2 compared to running the server on native
hardware. I run this experiment for about 20-30 seconds.

Best Regards,
Arnabjyoti Kalita


On Mon, Apr 26, 2021 at 9:35 PM Sean Christopherson <seanjc@google.com> wrote:
>
> On Mon, Apr 26, 2021, Arnabjyoti Kalita wrote:
> > Hello Sean,
> >
> > Thank you very much for your answer again. I could actually see that the
> > RDTSC handler gets called successfully. I added a "printk" statement to the
> > handler and I can see the messages being printed out to the kernel syslog.
> >
> > However, I am not sure if a VMEXIT is happening in userspace. I use QEMU
> > and I do not see any notification from QEMU that tells me that a VMEXIT
> > happened due to RDTSC being executed. Is there a mechanism to confirm this?
>
> Without further modification, KVM will _not_ exit to userspace in this case.
>
> > My actual requirement to record tsc values read out as a result of RDTSC
> > execution still stands.
>
> Your requirement didn't clarify that userspace needed to record the values ;-)
>
> Forcing an exit to userspace is very doable, but will tank guest performance and
> possibly interfere with whatever you're trying to do.  I would try adding a
> tracepoint and using that to capture the TSC values.  Adding guest RIP, etc...
> as needed is trivial.
>
> diff --git a/arch/x86/kvm/trace.h b/arch/x86/kvm/trace.h
> index a61c015870e3..e962e813ba04 100644
> --- a/arch/x86/kvm/trace.h
> +++ b/arch/x86/kvm/trace.h
> @@ -821,6 +821,23 @@ TRACE_EVENT(
>                   __entry->gpa_match ? "GPA" : "GVA")
>  );
>
> +TRACE_EVENT(kvm_emulate_rdtsc,
> +       TP_PROTO(unsigned int vcpu_id, __u64 tsc),
> +       TP_ARGS(vcpu_id, tsc),
> +
> +       TP_STRUCT__entry(
> +               __field( unsigned int,  vcpu_id         )
> +               __field(        __u64,  tsc             )
> +       ),
> +
> +       TP_fast_assign(
> +               __entry->vcpu_id                = vcpu_id;
> +               __entry->tsc                    = tsc;
> +       ),
> +
> +       TP_printk("vcpu=%u tsc=%llu", __entry->vcpu_id, __entry->tsc)
> +);
> +
>  TRACE_EVENT(kvm_write_tsc_offset,
>         TP_PROTO(unsigned int vcpu_id, __u64 previous_tsc_offset,
>                  __u64 next_tsc_offset),
> diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
> index 61bf0b86770b..1fbeef520349 100644
> --- a/arch/x86/kvm/vmx/vmx.c
> +++ b/arch/x86/kvm/vmx/vmx.c
> @@ -5254,6 +5254,8 @@ static int handle_rdtsc(struct kvm_vcpu *vcpu)
>  {
>         u64 tsc = kvm_read_l1_tsc(vcpu, rdtsc());
>
> +       trace_kvm_emulate_rdtsc(vcpu->vcpu_id, tsc);
> +
>         kvm_rax_write(vcpu, tsc & -1u);
>         kvm_rdx_write(vcpu, (tsc >> 32) & -1u);
>         return kvm_skip_emulated_instruction(vcpu);

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: Intercepting RDTSC instruction by causing a VMEXIT
  2021-04-27  2:19         ` Arnabjyoti Kalita
@ 2021-04-30 17:16           ` Arnabjyoti Kalita
  0 siblings, 0 replies; 7+ messages in thread
From: Arnabjyoti Kalita @ 2021-04-30 17:16 UTC (permalink / raw)
  To: Sean Christopherson; +Cc: kvm

Hello Sean,

Thank you for all your help. I've now managed to cause a VMEXIT in the
guest. I've also identified which guest registers to access to read
and record the tsc counter value when the vmexit handler is called in
userspace.

Best Regards,
Arnabjyoti Kalita

On Tue, Apr 27, 2021 at 7:49 AM Arnabjyoti Kalita
<akalita@cs.stonybrook.edu> wrote:
>
> Hello Sean,
>
> Thank you very much again. I didn't know about the tracepointing
> methodology. This is very helpful, indeed.
>
> In addition to my requirement of recording tsc values, I am trying to
> track when the VMEXITs happen in userspace(QEMU). This allows me to
> correlate recorded TSC values with the VMEXIT sequence. So, causing a
> VMEXIT in userspace is absolutely essential to me and I would love to
> have some pointers on how to do it.
>
> I run a server within the guest and I'm okay with performance dropping
> down to within a factor of 2 compared to running the server on native
> hardware. I run this experiment for about 20-30 seconds.
>
> Best Regards,
> Arnabjyoti Kalita
>
>
> On Mon, Apr 26, 2021 at 9:35 PM Sean Christopherson <seanjc@google.com> wrote:
> >
> > On Mon, Apr 26, 2021, Arnabjyoti Kalita wrote:
> > > Hello Sean,
> > >
> > > Thank you very much for your answer again. I could actually see that the
> > > RDTSC handler gets called successfully. I added a "printk" statement to the
> > > handler and I can see the messages being printed out to the kernel syslog.
> > >
> > > However, I am not sure if a VMEXIT is happening in userspace. I use QEMU
> > > and I do not see any notification from QEMU that tells me that a VMEXIT
> > > happened due to RDTSC being executed. Is there a mechanism to confirm this?
> >
> > Without further modification, KVM will _not_ exit to userspace in this case.
> >
> > > My actual requirement to record tsc values read out as a result of RDTSC
> > > execution still stands.
> >
> > Your requirement didn't clarify that userspace needed to record the values ;-)
> >
> > Forcing an exit to userspace is very doable, but will tank guest performance and
> > possibly interfere with whatever you're trying to do.  I would try adding a
> > tracepoint and using that to capture the TSC values.  Adding guest RIP, etc...
> > as needed is trivial.
> >
> > diff --git a/arch/x86/kvm/trace.h b/arch/x86/kvm/trace.h
> > index a61c015870e3..e962e813ba04 100644
> > --- a/arch/x86/kvm/trace.h
> > +++ b/arch/x86/kvm/trace.h
> > @@ -821,6 +821,23 @@ TRACE_EVENT(
> >                   __entry->gpa_match ? "GPA" : "GVA")
> >  );
> >
> > +TRACE_EVENT(kvm_emulate_rdtsc,
> > +       TP_PROTO(unsigned int vcpu_id, __u64 tsc),
> > +       TP_ARGS(vcpu_id, tsc),
> > +
> > +       TP_STRUCT__entry(
> > +               __field( unsigned int,  vcpu_id         )
> > +               __field(        __u64,  tsc             )
> > +       ),
> > +
> > +       TP_fast_assign(
> > +               __entry->vcpu_id                = vcpu_id;
> > +               __entry->tsc                    = tsc;
> > +       ),
> > +
> > +       TP_printk("vcpu=%u tsc=%llu", __entry->vcpu_id, __entry->tsc)
> > +);
> > +
> >  TRACE_EVENT(kvm_write_tsc_offset,
> >         TP_PROTO(unsigned int vcpu_id, __u64 previous_tsc_offset,
> >                  __u64 next_tsc_offset),
> > diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
> > index 61bf0b86770b..1fbeef520349 100644
> > --- a/arch/x86/kvm/vmx/vmx.c
> > +++ b/arch/x86/kvm/vmx/vmx.c
> > @@ -5254,6 +5254,8 @@ static int handle_rdtsc(struct kvm_vcpu *vcpu)
> >  {
> >         u64 tsc = kvm_read_l1_tsc(vcpu, rdtsc());
> >
> > +       trace_kvm_emulate_rdtsc(vcpu->vcpu_id, tsc);
> > +
> >         kvm_rax_write(vcpu, tsc & -1u);
> >         kvm_rdx_write(vcpu, (tsc >> 32) & -1u);
> >         return kvm_skip_emulated_instruction(vcpu);

^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, back to index

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-04-17  4:54 Intercepting RDTSC instruction by causing a VMEXIT Arnabjyoti Kalita
2021-04-19 16:46 ` Sean Christopherson
2021-04-20  2:33   ` Arnabjyoti Kalita
     [not found]     ` <CAJGDS+GT1mKHz6K=qHQf54S_97ym=nRP12MfO6OSEOpLYGht=A@mail.gmail.com>
2021-04-26  1:51       ` Arnabjyoti Kalita
2021-04-26 16:05       ` Sean Christopherson
2021-04-27  2:19         ` Arnabjyoti Kalita
2021-04-30 17:16           ` Arnabjyoti Kalita

KVM Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/kvm/0 kvm/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 kvm kvm/ https://lore.kernel.org/kvm \
		kvm@vger.kernel.org
	public-inbox-index kvm

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.kvm


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git