Controlling the guest TSC on x86

Controlling the guest TSC on x86

[Stephan Tobies]

Good afternoon!

We are looking at the use of KVM on x86 to emulate an x86 processor in a Virtual Prototyping/SystemC context. The requirements are such that the guest OS should be ideally run unmodified (i.e., in this case ideally without any drivers that know and exploit the fact that the guest is not running on real HW but as a KVM guest). 

For this, we also would like to control the TSC (as observed by the guest via rdtsc and related instructions) in such a way that time is apparently stopped whenever the guest is not actively executing in KVM_RUN.

I must admit that I am confused by the multitude of mechanism and MSRs that are available in this context. So, how would one best achieve to (approximately) stop the increment of the TSC when the guest is not running. If this is important, we are also not using the in-chip APIC but are using our own SystemC models. Also, are there extra considerations when running multiple virtual processors?

Any pointers would be greatly appreciated!

Thanks and best regards

Stephan Tobies

Re: Controlling the guest TSC on x86

[Jim Mattson]

On Tue, Jun 15, 2021 at 5:20 AM Stephan Tobies
<Stephan.Tobies@synopsys.com> wrote:
>
> Good afternoon!
>
> We are looking at the use of KVM on x86 to emulate an x86 processor in a Virtual Prototyping/SystemC context. The requirements are such that the guest OS should be ideally run unmodified (i.e., in this case ideally without any drivers that know and exploit the fact that the guest is not running on real HW but as a KVM guest).
>
> For this, we also would like to control the TSC (as observed by the guest via rdtsc and related instructions) in such a way that time is apparently stopped whenever the guest is not actively executing in KVM_RUN.
>
> I must admit that I am confused by the multitude of mechanism and MSRs that are available in this context. So, how would one best achieve to (approximately) stop the increment of the TSC when the guest is not running. If this is important, we are also not using the in-chip APIC but are using our own SystemC models. Also, are there extra considerations when running multiple virtual processors?
>
> Any pointers would be greatly appreciated!
>
> Thanks and best regards
>
> Stephan Tobies

You can use the VM-exit MSR-save list to save the value of the TSC on
VM-exit, and then you can adjust the TSC offset field in the VMCS just
before VM-entry to subtract any time the vCPU wasn't in VMX non-root
operation. There will be some slop here, since the guest TSC will run
during VM-entry and part of VM-exit. However, this is just the
beginning of your troubles. It will be impossible to keep the TSCs
synchronized across multiple vCPUs this way. Moreover, the TSC time
domain will get out of sync with other time domains, such as the APIC
time domain and the RTC time domain. Maybe it's enough to report to
the guest that CPUID.80000007H:EDX.INVARIANT_TSC[bit 8] is zero, but I
suspect you are in for a lot of headaches.

RE: Controlling the guest TSC on x86

[Stephan Tobies]

Hi Jim,

Thanks for the feedback. Some of the headache that you mention is hopefully avoided by the fact that both APIC and RTC time domain are also running under the control of the (external) SystemC scheduler, but we will be on the lookup for trouble.

Somehow I still have not been able to get it to work, probably because I have not been able to match your description with my external view of KVM. What is wrong with the following algorithm (assuming that we want to run guest and host at the same speed):

1) on each return from ioctl(..., KVM_RUN, ...) read the host TSC with rdtsc
2) before each call to ioctl(..., KVM_RUN, ...) read the host TSC again and set the MSR_IA32_TSC_ADJUST so the sum of such accumulated deltas

Why would I need to read the guest TSC, as you have mentioned in your answer?

Do I need to do something else? Because to me it looks like just writing the MSR_IA32_TSC_ADJUST does not lead to a recomputation of the guest TSC. (Or is this the answer to my previous question?)

Best regards

Stephan

-----Original Message-----
From: Jim Mattson <jmattson@google.com> 
Sent: Tuesday, June 15, 2021 6:59 PM
To: Stephan Tobies <tobies@synopsys.com>
Cc: kvm@vger.kernel.org
Subject: Re: Controlling the guest TSC on x86

On Tue, Jun 15, 2021 at 5:20 AM Stephan Tobies <Stephan.Tobies@synopsys.com> wrote:
>
> Good afternoon!
>
> We are looking at the use of KVM on x86 to emulate an x86 processor in a Virtual Prototyping/SystemC context. The requirements are such that the guest OS should be ideally run unmodified (i.e., in this case ideally without any drivers that know and exploit the fact that the guest is not running on real HW but as a KVM guest).
>
> For this, we also would like to control the TSC (as observed by the guest via rdtsc and related instructions) in such a way that time is apparently stopped whenever the guest is not actively executing in KVM_RUN.
>
> I must admit that I am confused by the multitude of mechanism and MSRs that are available in this context. So, how would one best achieve to (approximately) stop the increment of the TSC when the guest is not running. If this is important, we are also not using the in-chip APIC but are using our own SystemC models. Also, are there extra considerations when running multiple virtual processors?
>
> Any pointers would be greatly appreciated!
>
> Thanks and best regards
>
> Stephan Tobies

You can use the VM-exit MSR-save list to save the value of the TSC on VM-exit, and then you can adjust the TSC offset field in the VMCS just before VM-entry to subtract any time the vCPU wasn't in VMX non-root operation. There will be some slop here, since the guest TSC will run during VM-entry and part of VM-exit. However, this is just the beginning of your troubles. It will be impossible to keep the TSCs synchronized across multiple vCPUs this way. Moreover, the TSC time domain will get out of sync with other time domains, such as the APIC time domain and the RTC time domain. Maybe it's enough to report to the guest that CPUID.80000007H:EDX.INVARIANT_TSC[bit 8] is zero, but I suspect you are in for a lot of headaches.

Re: Controlling the guest TSC on x86

[Jim Mattson]

On Wed, Jun 16, 2021 at 5:24 AM Stephan Tobies
<Stephan.Tobies@synopsys.com> wrote:
>
> Hi Jim,
>
> Thanks for the feedback. Some of the headache that you mention is hopefully avoided by the fact that both APIC and RTC time domain are also running under the control of the (external) SystemC scheduler, but we will be on the lookup for trouble.
>
> Somehow I still have not been able to get it to work, probably because I have not been able to match your description with my external view of KVM. What is wrong with the following algorithm (assuming that we want to run guest and host at the same speed):
>
> 1) on each return from ioctl(..., KVM_RUN, ...) read the host TSC with rdtsc
> 2) before each call to ioctl(..., KVM_RUN, ...) read the host TSC again and set the MSR_IA32_TSC_ADJUST so the sum of such accumulated deltas

That should be the inverse of the sum of the accumulated deltas.

> Why would I need to read the guest TSC, as you have mentioned in your answer?

You seem to have a lot more tolerance for slop than I was expecting,
if you are happy to count time spent in the host kernel. I was trying
to get the tightest bounds possible on time spent actually running
guest code.

> Do I need to do something else? Because to me it looks like just writing the MSR_IA32_TSC_ADJUST does not lead to a recomputation of the guest TSC. (Or is this the answer to my previous question?)

Hmmm. Does the guest CPUID information enumerate support for
IA32_TSC_ADJUST? (i.e., is
CPUID.(EAX=07H,ECX=0):EBX.IA32_TSC_ADJUST[bit 1] set in the guest?)

> Best regards
>
> Stephan
>
> -----Original Message-----
> From: Jim Mattson <jmattson@google.com>
> Sent: Tuesday, June 15, 2021 6:59 PM
> To: Stephan Tobies <tobies@synopsys.com>
> Cc: kvm@vger.kernel.org
> Subject: Re: Controlling the guest TSC on x86
>
> On Tue, Jun 15, 2021 at 5:20 AM Stephan Tobies <Stephan.Tobies@synopsys.com> wrote:
> >
> > Good afternoon!
> >
> > We are looking at the use of KVM on x86 to emulate an x86 processor in a Virtual Prototyping/SystemC context. The requirements are such that the guest OS should be ideally run unmodified (i.e., in this case ideally without any drivers that know and exploit the fact that the guest is not running on real HW but as a KVM guest).
> >
> > For this, we also would like to control the TSC (as observed by the guest via rdtsc and related instructions) in such a way that time is apparently stopped whenever the guest is not actively executing in KVM_RUN.
> >
> > I must admit that I am confused by the multitude of mechanism and MSRs that are available in this context. So, how would one best achieve to (approximately) stop the increment of the TSC when the guest is not running. If this is important, we are also not using the in-chip APIC but are using our own SystemC models. Also, are there extra considerations when running multiple virtual processors?
> >
> > Any pointers would be greatly appreciated!
> >
> > Thanks and best regards
> >
> > Stephan Tobies
>
> You can use the VM-exit MSR-save list to save the value of the TSC on VM-exit, and then you can adjust the TSC offset field in the VMCS just before VM-entry to subtract any time the vCPU wasn't in VMX non-root operation. There will be some slop here, since the guest TSC will run during VM-entry and part of VM-exit. However, this is just the beginning of your troubles. It will be impossible to keep the TSCs synchronized across multiple vCPUs this way. Moreover, the TSC time domain will get out of sync with other time domains, such as the APIC time domain and the RTC time domain. Maybe it's enough to report to the guest that CPUID.80000007H:EDX.INVARIANT_TSC[bit 8] is zero, but I suspect you are in for a lot of headaches.