All of lore.kernel.org
 help / color / mirror / Atom feed
* Controlling the guest TSC on x86
       [not found] <DM4PR12MB52648CB3F874AC3B7C41A19DB1309@DM4PR12MB5264.namprd12.prod.outlook.com>
@ 2021-06-15 12:20 ` Stephan Tobies
  2021-06-15 16:58   ` Jim Mattson
  0 siblings, 1 reply; 4+ messages in thread
From: Stephan Tobies @ 2021-06-15 12:20 UTC (permalink / raw)
  To: kvm

Good afternoon!

We are looking at the use of KVM on x86 to emulate an x86 processor in a Virtual Prototyping/SystemC context. The requirements are such that the guest OS should be ideally run unmodified (i.e., in this case ideally without any drivers that know and exploit the fact that the guest is not running on real HW but as a KVM guest). 

For this, we also would like to control the TSC (as observed by the guest via rdtsc and related instructions) in such a way that time is apparently stopped whenever the guest is not actively executing in KVM_RUN.

I must admit that I am confused by the multitude of mechanism and MSRs that are available in this context. So, how would one best achieve to (approximately) stop the increment of the TSC when the guest is not running. If this is important, we are also not using the in-chip APIC but are using our own SystemC models. Also, are there extra considerations when running multiple virtual processors?

Any pointers would be greatly appreciated!

Thanks and best regards

Stephan Tobies


^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: Controlling the guest TSC on x86
  2021-06-15 12:20 ` Controlling the guest TSC on x86 Stephan Tobies
@ 2021-06-15 16:58   ` Jim Mattson
  2021-06-16 12:24     ` Stephan Tobies
  0 siblings, 1 reply; 4+ messages in thread
From: Jim Mattson @ 2021-06-15 16:58 UTC (permalink / raw)
  To: Stephan Tobies; +Cc: kvm

On Tue, Jun 15, 2021 at 5:20 AM Stephan Tobies
<Stephan.Tobies@synopsys.com> wrote:
>
> Good afternoon!
>
> We are looking at the use of KVM on x86 to emulate an x86 processor in a Virtual Prototyping/SystemC context. The requirements are such that the guest OS should be ideally run unmodified (i.e., in this case ideally without any drivers that know and exploit the fact that the guest is not running on real HW but as a KVM guest).
>
> For this, we also would like to control the TSC (as observed by the guest via rdtsc and related instructions) in such a way that time is apparently stopped whenever the guest is not actively executing in KVM_RUN.
>
> I must admit that I am confused by the multitude of mechanism and MSRs that are available in this context. So, how would one best achieve to (approximately) stop the increment of the TSC when the guest is not running. If this is important, we are also not using the in-chip APIC but are using our own SystemC models. Also, are there extra considerations when running multiple virtual processors?
>
> Any pointers would be greatly appreciated!
>
> Thanks and best regards
>
> Stephan Tobies

You can use the VM-exit MSR-save list to save the value of the TSC on
VM-exit, and then you can adjust the TSC offset field in the VMCS just
before VM-entry to subtract any time the vCPU wasn't in VMX non-root
operation. There will be some slop here, since the guest TSC will run
during VM-entry and part of VM-exit. However, this is just the
beginning of your troubles. It will be impossible to keep the TSCs
synchronized across multiple vCPUs this way. Moreover, the TSC time
domain will get out of sync with other time domains, such as the APIC
time domain and the RTC time domain. Maybe it's enough to report to
the guest that CPUID.80000007H:EDX.INVARIANT_TSC[bit 8] is zero, but I
suspect you are in for a lot of headaches.

^ permalink raw reply	[flat|nested] 4+ messages in thread

* RE: Controlling the guest TSC on x86
  2021-06-15 16:58   ` Jim Mattson
@ 2021-06-16 12:24     ` Stephan Tobies
  2021-06-16 17:46       ` Jim Mattson
  0 siblings, 1 reply; 4+ messages in thread
From: Stephan Tobies @ 2021-06-16 12:24 UTC (permalink / raw)
  To: Jim Mattson; +Cc: kvm

Hi Jim,

Thanks for the feedback. Some of the headache that you mention is hopefully avoided by the fact that both APIC and RTC time domain are also running under the control of the (external) SystemC scheduler, but we will be on the lookup for trouble.

Somehow I still have not been able to get it to work, probably because I have not been able to match your description with my external view of KVM. What is wrong with the following algorithm (assuming that we want to run guest and host at the same speed):

1) on each return from ioctl(..., KVM_RUN, ...) read the host TSC with rdtsc
2) before each call to ioctl(..., KVM_RUN, ...) read the host TSC again and set the MSR_IA32_TSC_ADJUST so the sum of such accumulated deltas

Why would I need to read the guest TSC, as you have mentioned in your answer?

Do I need to do something else? Because to me it looks like just writing the MSR_IA32_TSC_ADJUST does not lead to a recomputation of the guest TSC. (Or is this the answer to my previous question?)

Best regards

Stephan

-----Original Message-----
From: Jim Mattson <jmattson@google.com> 
Sent: Tuesday, June 15, 2021 6:59 PM
To: Stephan Tobies <tobies@synopsys.com>
Cc: kvm@vger.kernel.org
Subject: Re: Controlling the guest TSC on x86

On Tue, Jun 15, 2021 at 5:20 AM Stephan Tobies <Stephan.Tobies@synopsys.com> wrote:
>
> Good afternoon!
>
> We are looking at the use of KVM on x86 to emulate an x86 processor in a Virtual Prototyping/SystemC context. The requirements are such that the guest OS should be ideally run unmodified (i.e., in this case ideally without any drivers that know and exploit the fact that the guest is not running on real HW but as a KVM guest).
>
> For this, we also would like to control the TSC (as observed by the guest via rdtsc and related instructions) in such a way that time is apparently stopped whenever the guest is not actively executing in KVM_RUN.
>
> I must admit that I am confused by the multitude of mechanism and MSRs that are available in this context. So, how would one best achieve to (approximately) stop the increment of the TSC when the guest is not running. If this is important, we are also not using the in-chip APIC but are using our own SystemC models. Also, are there extra considerations when running multiple virtual processors?
>
> Any pointers would be greatly appreciated!
>
> Thanks and best regards
>
> Stephan Tobies

You can use the VM-exit MSR-save list to save the value of the TSC on VM-exit, and then you can adjust the TSC offset field in the VMCS just before VM-entry to subtract any time the vCPU wasn't in VMX non-root operation. There will be some slop here, since the guest TSC will run during VM-entry and part of VM-exit. However, this is just the beginning of your troubles. It will be impossible to keep the TSCs synchronized across multiple vCPUs this way. Moreover, the TSC time domain will get out of sync with other time domains, such as the APIC time domain and the RTC time domain. Maybe it's enough to report to the guest that CPUID.80000007H:EDX.INVARIANT_TSC[bit 8] is zero, but I suspect you are in for a lot of headaches.

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: Controlling the guest TSC on x86
  2021-06-16 12:24     ` Stephan Tobies
@ 2021-06-16 17:46       ` Jim Mattson
  0 siblings, 0 replies; 4+ messages in thread
From: Jim Mattson @ 2021-06-16 17:46 UTC (permalink / raw)
  To: Stephan Tobies; +Cc: kvm

On Wed, Jun 16, 2021 at 5:24 AM Stephan Tobies
<Stephan.Tobies@synopsys.com> wrote:
>
> Hi Jim,
>
> Thanks for the feedback. Some of the headache that you mention is hopefully avoided by the fact that both APIC and RTC time domain are also running under the control of the (external) SystemC scheduler, but we will be on the lookup for trouble.
>
> Somehow I still have not been able to get it to work, probably because I have not been able to match your description with my external view of KVM. What is wrong with the following algorithm (assuming that we want to run guest and host at the same speed):
>
> 1) on each return from ioctl(..., KVM_RUN, ...) read the host TSC with rdtsc
> 2) before each call to ioctl(..., KVM_RUN, ...) read the host TSC again and set the MSR_IA32_TSC_ADJUST so the sum of such accumulated deltas

That should be the inverse of the sum of the accumulated deltas.

> Why would I need to read the guest TSC, as you have mentioned in your answer?

You seem to have a lot more tolerance for slop than I was expecting,
if you are happy to count time spent in the host kernel. I was trying
to get the tightest bounds possible on time spent actually running
guest code.

> Do I need to do something else? Because to me it looks like just writing the MSR_IA32_TSC_ADJUST does not lead to a recomputation of the guest TSC. (Or is this the answer to my previous question?)

Hmmm. Does the guest CPUID information enumerate support for
IA32_TSC_ADJUST? (i.e., is
CPUID.(EAX=07H,ECX=0):EBX.IA32_TSC_ADJUST[bit 1] set in the guest?)

> Best regards
>
> Stephan
>
> -----Original Message-----
> From: Jim Mattson <jmattson@google.com>
> Sent: Tuesday, June 15, 2021 6:59 PM
> To: Stephan Tobies <tobies@synopsys.com>
> Cc: kvm@vger.kernel.org
> Subject: Re: Controlling the guest TSC on x86
>
> On Tue, Jun 15, 2021 at 5:20 AM Stephan Tobies <Stephan.Tobies@synopsys.com> wrote:
> >
> > Good afternoon!
> >
> > We are looking at the use of KVM on x86 to emulate an x86 processor in a Virtual Prototyping/SystemC context. The requirements are such that the guest OS should be ideally run unmodified (i.e., in this case ideally without any drivers that know and exploit the fact that the guest is not running on real HW but as a KVM guest).
> >
> > For this, we also would like to control the TSC (as observed by the guest via rdtsc and related instructions) in such a way that time is apparently stopped whenever the guest is not actively executing in KVM_RUN.
> >
> > I must admit that I am confused by the multitude of mechanism and MSRs that are available in this context. So, how would one best achieve to (approximately) stop the increment of the TSC when the guest is not running. If this is important, we are also not using the in-chip APIC but are using our own SystemC models. Also, are there extra considerations when running multiple virtual processors?
> >
> > Any pointers would be greatly appreciated!
> >
> > Thanks and best regards
> >
> > Stephan Tobies
>
> You can use the VM-exit MSR-save list to save the value of the TSC on VM-exit, and then you can adjust the TSC offset field in the VMCS just before VM-entry to subtract any time the vCPU wasn't in VMX non-root operation. There will be some slop here, since the guest TSC will run during VM-entry and part of VM-exit. However, this is just the beginning of your troubles. It will be impossible to keep the TSCs synchronized across multiple vCPUs this way. Moreover, the TSC time domain will get out of sync with other time domains, such as the APIC time domain and the RTC time domain. Maybe it's enough to report to the guest that CPUID.80000007H:EDX.INVARIANT_TSC[bit 8] is zero, but I suspect you are in for a lot of headaches.

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2021-06-16 17:47 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
     [not found] <DM4PR12MB52648CB3F874AC3B7C41A19DB1309@DM4PR12MB5264.namprd12.prod.outlook.com>
2021-06-15 12:20 ` Controlling the guest TSC on x86 Stephan Tobies
2021-06-15 16:58   ` Jim Mattson
2021-06-16 12:24     ` Stephan Tobies
2021-06-16 17:46       ` Jim Mattson

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.