[RFC PATCH 0/6] x86/kvm: Virtual suspend time injection support

[RFC PATCH 0/6] x86/kvm: Virtual suspend time injection support

[Hikaru Nishida]

Hi,

This patch series adds virtual suspend time injection support to KVM.

Before this change, if the host goes into suspended state while the
guest is running, the guest will experience a time jump after the host's
resume. This can confuse some services in the guest since they can't
detect if the system went into suspend or not by comparing
CLOCK_BOOTTIME and CLOCK_MONOTONIC.

To solve this problem, we wanted to add a way to adjust the guest clocks
without actually suspending the guests. However, there was no way to
modify a gap between CLOCK_BOOTTIME and CLOCK_MONOTONIC without actually
suspending the guests. Therefore, this series introduce a new struct
called kvm_host_suspend_time to share the suspend time between host and
guest and a mechanism to inject a suspend time to the guest while
keeping
monotonicity of the clocks.

Could you take a look and let me know how we can improve the patches if
they are doing something wrong?

Thanks,

Hikaru Nishida



Hikaru Nishida (6):
  x86/kvm: Reserve KVM_FEATURE_HOST_SUSPEND_TIME and
    MSR_KVM_HOST_SUSPEND_TIME
  x86/kvm: Add a struct and constants for virtual suspend time injection
  x86/kvm: Add CONFIG_KVM_VIRT_SUSPEND_TIMING
  x86/kvm: Add a host side support for virtual suspend time injection
  x86/kvm: Add CONFIG_KVM_VIRT_SUSPEND_TIMING_GUEST
  x86/kvm: Add a guest side support for virtual suspend time injection

 Documentation/virt/kvm/cpuid.rst     |  3 +
 Documentation/virt/kvm/msr.rst       | 29 +++++++++
 arch/x86/Kconfig                     | 13 ++++
 arch/x86/include/asm/kvm_host.h      |  5 ++
 arch/x86/include/asm/kvm_para.h      |  9 +++
 arch/x86/include/uapi/asm/kvm_para.h |  6 ++
 arch/x86/kernel/kvmclock.c           | 25 ++++++++
 arch/x86/kvm/Kconfig                 | 13 ++++
 arch/x86/kvm/cpuid.c                 |  4 ++
 arch/x86/kvm/x86.c                   | 89 +++++++++++++++++++++++++++-
 include/linux/kvm_host.h             |  7 +++
 include/linux/timekeeper_internal.h  |  4 ++
 kernel/time/timekeeping.c            | 31 ++++++++++
 13 files changed, 237 insertions(+), 1 deletion(-)

-- 
2.31.1.498.g6c1eba8ee3d-goog

[RFC PATCH 1/6] x86/kvm: Reserve KVM_FEATURE_HOST_SUSPEND_TIME and MSR_KVM_HOST_SUSPEND_TIME

[Hikaru Nishida]

No functional change; just add documentation for
KVM_FEATURE_HOST_SUSPEND_TIME and its corresponding
MSR_KVM_HOST_SUSPEND_TIME to support virtual suspend timing injection in
later patches.

Signed-off-by: Hikaru Nishida <hikalium@chromium.org>
---

 Documentation/virt/kvm/cpuid.rst |  3 +++
 Documentation/virt/kvm/msr.rst   | 29 +++++++++++++++++++++++++++++
 2 files changed, 32 insertions(+)

diff --git a/Documentation/virt/kvm/cpuid.rst b/Documentation/virt/kvm/cpuid.rst
index cf62162d4be2..c7cb581b9a9b 100644
--- a/Documentation/virt/kvm/cpuid.rst
+++ b/Documentation/virt/kvm/cpuid.rst
@@ -96,6 +96,9 @@ KVM_FEATURE_MSI_EXT_DEST_ID        15          guest checks this feature bit
                                                before using extended destination
                                                ID bits in MSI address bits 11-5.
 
+KVM_FEATURE_HOST_SUSPEND_TIME      16          host suspend time information
+                                               is available at msr 0x4b564d08.
+
 KVM_FEATURE_CLOCKSOURCE_STABLE_BIT 24          host will warn if no guest-side
                                                per-cpu warps are expected in
                                                kvmclock
diff --git a/Documentation/virt/kvm/msr.rst b/Documentation/virt/kvm/msr.rst
index e37a14c323d2..de96743245c9 100644
--- a/Documentation/virt/kvm/msr.rst
+++ b/Documentation/virt/kvm/msr.rst
@@ -376,3 +376,32 @@ data:
 	write '1' to bit 0 of the MSR, this causes the host to re-scan its queue
 	and check if there are more notifications pending. The MSR is available
 	if KVM_FEATURE_ASYNC_PF_INT is present in CPUID.
+
+MSR_KVM_HOST_SUSPEND_TIME:
+	0x4b564d08
+
+data:
+	8-byte alignment physical address of a memory area which must be
+	in guest RAM, plus an enable bit in bit 0. This memory is expected to
+	hold a copy of the following structure::
+
+	 struct kvm_host_suspend_time {
+		__u64   suspend_time_ns;
+	 };
+
+	whose data will be filled in by the hypervisor.
+	If the guest register this structure through the MSR write, the host
+	will stop all the clocks including TSCs observed by the guest during
+	the host's suspension and report the duration of suspend through this
+	structure. Fields have the following meanings:
+
+	host_suspend_time_ns:
+		Total number of nanoseconds passed during the host's suspend
+		while the VM is running. This value will be increasing
+		monotonically.
+
+	Note that although MSRs are per-CPU entities, the effect of this
+	particular MSR is global.
+
+	Availability of this MSR must be checked via bit 16 in 0x4000001 cpuid
+	leaf prior to usage.
-- 
2.31.1.498.g6c1eba8ee3d-goog

Re: [RFC PATCH 0/6] x86/kvm: Virtual suspend time injection support

[Maxim Levitsky]

On Mon, 2021-04-26 at 18:06 +0900, Hikaru Nishida wrote:
> Hi,
> 
> This patch series adds virtual suspend time injection support to KVM.
> 
> Before this change, if the host goes into suspended state while the
> guest is running, the guest will experience a time jump after the host's
> resume. This can confuse some services in the guest since they can't
> detect if the system went into suspend or not by comparing
> CLOCK_BOOTTIME and CLOCK_MONOTONIC.
> 
> To solve this problem, we wanted to add a way to adjust the guest clocks
> without actually suspending the guests. However, there was no way to
> modify a gap between CLOCK_BOOTTIME and CLOCK_MONOTONIC without actually
> suspending the guests. Therefore, this series introduce a new struct
> called kvm_host_suspend_time to share the suspend time between host and
> guest and a mechanism to inject a suspend time to the guest while
> keeping
> monotonicity of the clocks.
> 
> Could you take a look and let me know how we can improve the patches if
> they are doing something wrong?
> 
> Thanks,
> 
> Hikaru Nishida
> 

I haven't yet looked at that, but in my experience when I suspend the host
with VMs running, after resume all my VMs complain something about TSC watchdog
and stop using it. The TSC is stable/synchornized, but after resume it does
reset to 0 on all CPUs.

I use INVTSC flag for all my VMs.
I haven't investigated this futher yet.

Just my 0.2 cents.

Best regards,
	Maxim Levitsky

> 
> 
> Hikaru Nishida (6):
>   x86/kvm: Reserve KVM_FEATURE_HOST_SUSPEND_TIME and
>     MSR_KVM_HOST_SUSPEND_TIME
>   x86/kvm: Add a struct and constants for virtual suspend time injection
>   x86/kvm: Add CONFIG_KVM_VIRT_SUSPEND_TIMING
>   x86/kvm: Add a host side support for virtual suspend time injection
>   x86/kvm: Add CONFIG_KVM_VIRT_SUSPEND_TIMING_GUEST
>   x86/kvm: Add a guest side support for virtual suspend time injection
> 
>  Documentation/virt/kvm/cpuid.rst     |  3 +
>  Documentation/virt/kvm/msr.rst       | 29 +++++++++
>  arch/x86/Kconfig                     | 13 ++++
>  arch/x86/include/asm/kvm_host.h      |  5 ++
>  arch/x86/include/asm/kvm_para.h      |  9 +++
>  arch/x86/include/uapi/asm/kvm_para.h |  6 ++
>  arch/x86/kernel/kvmclock.c           | 25 ++++++++
>  arch/x86/kvm/Kconfig                 | 13 ++++
>  arch/x86/kvm/cpuid.c                 |  4 ++
>  arch/x86/kvm/x86.c                   | 89 +++++++++++++++++++++++++++-
>  include/linux/kvm_host.h             |  7 +++
>  include/linux/timekeeper_internal.h  |  4 ++
>  kernel/time/timekeeping.c            | 31 ++++++++++
>  13 files changed, 237 insertions(+), 1 deletion(-)
> 

Re: [RFC PATCH 1/6] x86/kvm: Reserve KVM_FEATURE_HOST_SUSPEND_TIME and MSR_KVM_HOST_SUSPEND_TIME

[David Edmondson]

On Monday, 2021-04-26 at 18:06:40 +09, Hikaru Nishida wrote:

> No functional change; just add documentation for
> KVM_FEATURE_HOST_SUSPEND_TIME and its corresponding
> MSR_KVM_HOST_SUSPEND_TIME to support virtual suspend timing injection in
> later patches.
>
> Signed-off-by: Hikaru Nishida <hikalium@chromium.org>
> ---
>
>  Documentation/virt/kvm/cpuid.rst |  3 +++
>  Documentation/virt/kvm/msr.rst   | 29 +++++++++++++++++++++++++++++
>  2 files changed, 32 insertions(+)
>
> diff --git a/Documentation/virt/kvm/cpuid.rst b/Documentation/virt/kvm/cpuid.rst
> index cf62162d4be2..c7cb581b9a9b 100644
> --- a/Documentation/virt/kvm/cpuid.rst
> +++ b/Documentation/virt/kvm/cpuid.rst
> @@ -96,6 +96,9 @@ KVM_FEATURE_MSI_EXT_DEST_ID        15          guest checks this feature bit
>                                                 before using extended destination
>                                                 ID bits in MSI address bits 11-5.
>  
> +KVM_FEATURE_HOST_SUSPEND_TIME      16          host suspend time information
> +                                               is available at msr 0x4b564d08.
> +
>  KVM_FEATURE_CLOCKSOURCE_STABLE_BIT 24          host will warn if no guest-side
>                                                 per-cpu warps are expected in
>                                                 kvmclock
> diff --git a/Documentation/virt/kvm/msr.rst b/Documentation/virt/kvm/msr.rst
> index e37a14c323d2..de96743245c9 100644
> --- a/Documentation/virt/kvm/msr.rst
> +++ b/Documentation/virt/kvm/msr.rst
> @@ -376,3 +376,32 @@ data:
>  	write '1' to bit 0 of the MSR, this causes the host to re-scan its queue
>  	and check if there are more notifications pending. The MSR is available
>  	if KVM_FEATURE_ASYNC_PF_INT is present in CPUID.
> +
> +MSR_KVM_HOST_SUSPEND_TIME:
> +	0x4b564d08
> +
> +data:
> +	8-byte alignment physical address of a memory area which must be
> +	in guest RAM, plus an enable bit in bit 0. This memory is expected to
> +	hold a copy of the following structure::
> +
> +	 struct kvm_host_suspend_time {
> +		__u64   suspend_time_ns;
> +	 };
> +
> +	whose data will be filled in by the hypervisor.
> +	If the guest register this structure through the MSR write, the host
> +	will stop all the clocks including TSCs observed by the guest during
> +	the host's suspension and report the duration of suspend through this
> +	structure. Fields have the following meanings:
> +
> +	host_suspend_time_ns:

s/host_suspend_time_ns/suspend_time_ns/

> +		Total number of nanoseconds passed during the host's suspend
> +		while the VM is running. This value will be increasing
> +		monotonically.
> +
> +	Note that although MSRs are per-CPU entities, the effect of this
> +	particular MSR is global.
> +
> +	Availability of this MSR must be checked via bit 16 in 0x4000001 cpuid
> +	leaf prior to usage.
> -- 
> 2.31.1.498.g6c1eba8ee3d-goog

dme.
-- 
I do believe it's Madame Joy.