[PATCH] xen: preserve native TSC speed during migration between identical hosts

[Olaf Hering <olaf@aepfle.de>]

After migrating a domU to another identical host a performance drop can
be observed. One reason is that before migration TSC was accessed at
native speed, after migration TSC has to be emulated. This happens
because the measured CPU frequency is not accurate, the values differ
even between reboots.

To avoid the emulation a tolerance range can be specified during boot
with "vtsc-tolerance=N".  If the frequency expected by the domU is
within the range, TSC access from the domU will remain native. If the
domU is migrated to another machine type TSC might be emulated.

Signed-off-by: Olaf Hering <olaf@aepfle.de>
---
 docs/man/xen-tscmode.pod.7          | 11 +++++++----
 docs/misc/xen-command-line.markdown | 10 ++++++++++
 xen/arch/x86/time.c                 | 14 +++++++++++++-
 3 files changed, 30 insertions(+), 5 deletions(-)

diff --git a/docs/man/xen-tscmode.pod.7 b/docs/man/xen-tscmode.pod.7
index 3bbc96f201..e04af70855 100644
--- a/docs/man/xen-tscmode.pod.7
+++ b/docs/man/xen-tscmode.pod.7
@@ -206,10 +206,13 @@ TSC-safe, rdtsc will execute at hardware speed; if it is not, rdtsc
 will be emulated.  Once a virtual machine is save/restored or migrated,
 however, there are two possibilities: TSC remains native IF the source
 physical machine and target physical machine have the same TSC frequency
-(or, for HVM/PVH guests, if TSC scaling support is available); else TSC
-is emulated.  Note that, though emulated, the "apparent" TSC frequency
-will be the TSC frequency of the initial physical machine, even after
-migration.
+(or, for HVM/PVH guests, if TSC scaling support is available); else TSC is
+emulated. The Xen cmdline option "vtsc-tolerance" allows host admins to
+specify a tolerance range in case the measured frequency of supposedly
+identical machines differs slightly. If the frequency on the target machine
+is within the range the tsc_mode remains native.
+Note that, though emulated, the "apparent" TSC frequency will be the TSC
+frequency of the initial physical machine, even after migration.
 
 For environments where both TSC-safeness AND highest performance
 even across migration is a requirement, application code can be specially
diff --git a/docs/misc/xen-command-line.markdown b/docs/misc/xen-command-line.markdown
index 44d99852aa..ff92975a15 100644
--- a/docs/misc/xen-command-line.markdown
+++ b/docs/misc/xen-command-line.markdown
@@ -1723,6 +1723,16 @@ Note that if **watchdog** option is also specified vpmu will be turned off.
 As the virtualisation is not 100% safe, don't use the vpmu flag on
 production systems (see http://xenbits.xen.org/xsa/advisory-163.html)!
 
+### vtsc-tolerance
+> `= <integer>`
+
+> Default: `0`
+
+Specify the tolerated difference of pCPUs clock frequency in kHz.
+This option affects only domUs which have tsc\_mode=default enabled.
+If the frequency expected by a domU is within the tolerance range tsc
+will remain native. Otherwise tsc emulation will be used for the domU.
+
 ### vwfi
 > `= trap | native
 
diff --git a/xen/arch/x86/time.c b/xen/arch/x86/time.c
index 655af33cb3..b290e8aca9 100644
--- a/xen/arch/x86/time.c
+++ b/xen/arch/x86/time.c
@@ -41,6 +41,9 @@
 static char __initdata opt_clocksource[10];
 string_param("clocksource", opt_clocksource);
 
+static unsigned int __read_mostly opt_vtsc_tolerance;
+integer_param("vtsc-tolerance", opt_vtsc_tolerance);
+
 unsigned long __read_mostly cpu_khz;  /* CPU clock frequency in kHz. */
 DEFINE_SPINLOCK(rtc_lock);
 unsigned long pit0_ticks;
@@ -2009,6 +2012,8 @@ void tsc_set_info(struct domain *d,
                   uint32_t tsc_mode, uint64_t elapsed_nsec,
                   uint32_t gtsc_khz, uint32_t incarnation)
 {
+    uint32_t khz_diff, tolerated;
+
     if ( is_idle_domain(d) || is_hardware_domain(d) )
     {
         d->arch.vtsc = 0;
@@ -2024,6 +2029,13 @@ void tsc_set_info(struct domain *d,
         d->arch.vtsc_offset = get_s_time() - elapsed_nsec;
         d->arch.tsc_khz = gtsc_khz ?: cpu_khz;
         set_time_scale(&d->arch.vtsc_to_ns, d->arch.tsc_khz * 1000);
+        if (!opt_vtsc_tolerance) {
+            tolerated = d->arch.tsc_khz == cpu_khz;
+        } else {
+            khz_diff = cpu_khz > d->arch.tsc_khz ?
+                       cpu_khz - d->arch.tsc_khz : d->arch.tsc_khz - cpu_khz;
+            tolerated = khz_diff <= opt_vtsc_tolerance;
+        }
 
         /*
          * In default mode use native TSC if the host has safe TSC and
@@ -2033,7 +2045,7 @@ void tsc_set_info(struct domain *d,
          * d->arch.tsc_khz == cpu_khz. Thus no need to check incarnation.
          */
         if ( tsc_mode == TSC_MODE_DEFAULT && host_tsc_is_safe() &&
-             (d->arch.tsc_khz == cpu_khz ||
+             (tolerated ||
               (is_hvm_domain(d) &&
                hvm_get_tsc_scaling_ratio(d->arch.tsc_khz))) )
         {

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel