[PATCH v6 0/5] Add VMX TSC scaling support

[PATCH v6 0/5] Add VMX TSC scaling support

[Haozhong Zhang]

This patchset adds support for VMX TSC scaling feature which is
available on Intel Skylake Server CPU. The specification of VMX TSC
scaling can be found at
http://www.intel.com/content/www/us/en/processors/timestamp-counter-scaling-virtualization-white-paper.html

VMX TSC scaling allows guest TSC which is read by guest rdtsc(p)
instructions increases in a rate that is customized by the hypervisor
and can be different than the host TSC frequency. Basically, VMX TSC
scaling adds a 64-bit field called TSC multiplier in VMCS so that, if
VMX TSC scaling is enabled, TSC read by guest rdtsc(p) instructions
will be calculated by the following formula:

  guest EDX:EAX = (Host TSC * TSC multiplier) >> 48 + VMX TSC Offset

where, Host TSC = Host MSR_IA32_TSC + Host MSR_IA32_TSC_ADJUST.

If the destination host supports VMX TSC scaling, this patchset allows
guest programs in a HVM container in the default TSC mode or PVRDTSCP
(native_paravirt) TSC mode to observe the same TSC frequency across
the migration.

Changes in v6:
 * v5 patch 1 has been committed so it's not included in v6.
 * v5 patch 1 - 5 correspond to v4 patch 2 - 6.
 * Adjust inline assembly in patch 1 and 2 per Jan's comments.
 * Name and type adjustments in patch 1 and 4.
 * Patch 3 and 5 are not changed.

Changes in v5:
 * v4 patch 1 - 3 have been committed so they are not included in v5.
 * v4 patch 5 "x86: Add functions for 64-bit integer arithmetic" is
   removed in v5. All math64 functions are inlined in v5.
 * v5 patch 1 corresponds to v4 patch 4.
   v5 patch 2 - 6 correspond to v4 patch 6 - 10.
 * Other changes are logged in each patch respectively.
 * Previous R-b and A-b (from Boris Ostrovsky, Jan Beulich and Kevin
   Tian) for all patches except patch 4 are removed because of above
   changes.

Changes in v4:
 * v3 patch 1&2 have been committed so they are not included in v4.
 * v3 patch 11 "x86/hvm: Detect TSC scaling through hvm_funcs" is merged
   early into v4 patch 4 "x86/hvm: Collect information of TSC scaling ratio".
 * v4 patch 1 - 8 correspond to v3 patch 3 - 10.
   v4 patch 9 - 10 correspond to v3 patch 12 - 13.
 * Other changes are logged in each patch respectively.

Changes in v3:
 * v2 patch 1&2 have been merged so they do not appear in v3.
 * Patch 1 - 6 correspond to v2 patch 3 - 8. Patch 7 is new.
   Patch 8 - 13 correspond to v2 patch 9 - 14.
 * Other changes are logged in each patch respectively.

Changes in v2:
 * Remove unnecessary v1 patch 1&13.
 * Add and move all bug-fix patches to the beginning of this series.
   (Patch 1 - 6)
 * Update changes in tsc_set_info() and tsc_get_info() to make both
   functions consistent with each other. (Patch 2 - 4)
 * Move a part of scaling logic out of [vmx|svm]_set_tsc_offset().
   (Patch 7)
 * Remove redundant hvm_funcs.tsc_scaling_ratio_rsvd. (Patch 8)
 * Reimplement functions that calculate TSC ratio and scale TSC.
   (Patch 9&10)
 * Merge setting VMX TSC multiplier into patch 13.
 * Move initialing tsc_scaling_ratio in VMX ahead to
   vmx_vcpu_initialise() so as to make construct_vmcs() naturally
   use this field instead of a constant. (Patch 13)
 * Update documents related to tsc_mode.
 * Other code cleanup and style fixes.

Haozhong Zhang (5):
  x86/hvm: Setup TSC scaling ratio
  x86/hvm: Replace architecture TSC scaling by a common function
  x86/hvm: Move saving/loading vcpu's TSC to common code
  vmx: Add VMX RDTSC(P) scaling support
  docs: Add descriptions of TSC scaling in xl.cfg and tscmode.txt

 docs/man/xl.cfg.pod.5              | 14 +++++++-
 docs/misc/tscmode.txt              | 21 ++++++++++++
 xen/arch/x86/hvm/hvm.c             | 69 ++++++++++++++++++++++++++++++++++++--
 xen/arch/x86/hvm/svm/svm.c         | 15 +--------
 xen/arch/x86/hvm/vmx/vmcs.c        | 12 +++++--
 xen/arch/x86/hvm/vmx/vmx.c         | 22 +++++++++---
 xen/arch/x86/time.c                | 13 ++++---
 xen/include/asm-x86/hvm/domain.h   |  2 ++
 xen/include/asm-x86/hvm/hvm.h      | 12 ++++++-
 xen/include/asm-x86/hvm/svm/svm.h  |  3 --
 xen/include/asm-x86/hvm/vmx/vmcs.h |  7 ++++
 11 files changed, 157 insertions(+), 33 deletions(-)

-- 
2.7.2


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[PATCH v6 1/5] x86/hvm: Setup TSC scaling ratio

[Haozhong Zhang]

This patch adds a field tsc_scaling_ratio in struct hvm_domain to record
the per-domain TSC scaling ratio, and sets it in tsc_set_info().

Before setting the per-domain TSC scaling ratio, we check its validity
in tsc_set_info(). If an invalid ratio is given, we will leave the
default value in tsc_scaling_ratio (i.e. ratio = 1) and setup guest TSC
as if no TSC scaling is used:
* For TSC_MODE_FAULT,
  - if a user-specified TSC frequency is given, we will set the guest
    TSC frequency to it; otherwise, we set it to the host TSC frequency.
  - if guest TSC frequency does not equal to host TSC frequency, we will
    emulate guest TSC (i.e. d->arch.vtsc is set to 1). In both cases,
    guest TSC runs in the guest TSC frequency.
* For TSC_MODE_PVRDTSCP,
  - we set the guest TSC frequency to the host TSC frequency.
  - guest rdtsc is executed natively in the host TSC frequency as
    before.
  - if rdtscp is not available to guest, it will be emulated; otherwise,
    it will be executed natively. In both cases, guest rdtscp gets TSC
    in the host TSC frequency as before.

Signed-off-by: Haozhong Zhang <haozhong.zhang@intel.com>
---
CC: Keir Fraser <keir@xen.org>
CC: Jan Beulich <jbeulich@suse.com>
CC: Andrew Cooper <andrew.cooper3@citrix.com>
CC: Boris Ostrovsky <boris.ostrovsky@oracle.com>
CC: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
CC: Aravind Gopalakrishnan <Aravind.Gopalakrishnan@amd.com>
CC: Kevin Tian <kevin.tian@intel.com>
---
Changes in v6:
 For inline assembly in hvm_get_tsc_scaling_ratio():
 * Add a check before inline assembly to avoid #DE from divq.
 * Replace salq by shlq.
 * Split inputs and outputs for dummy and ratio.
 * Turn to named arguments.
 * Put assembly and 'asm (' in the same line.
 Misc:
 * Rename macro hvm_vcpu_tsc_scaling_ratio() into hvm_tsc_scaling_ratio()
   and change its argument type to struct domain *.
---
 xen/arch/x86/hvm/hvm.c            | 38 ++++++++++++++++++++++++++++++++++++++
 xen/arch/x86/hvm/svm/svm.c        |  4 ++--
 xen/arch/x86/time.c               | 10 ++++++++--
 xen/include/asm-x86/hvm/domain.h  |  2 ++
 xen/include/asm-x86/hvm/hvm.h     |  8 ++++++++
 xen/include/asm-x86/hvm/svm/svm.h |  3 ---
 6 files changed, 58 insertions(+), 7 deletions(-)

diff --git a/xen/arch/x86/hvm/hvm.c b/xen/arch/x86/hvm/hvm.c
index f46d53c..6c32e99 100644
--- a/xen/arch/x86/hvm/hvm.c
+++ b/xen/arch/x86/hvm/hvm.c
@@ -298,6 +298,41 @@ int hvm_set_guest_pat(struct vcpu *v, u64 guest_pat)
     return 1;
 }
 
+/*
+ * Get the ratio to scale host TSC frequency to gtsc_khz. zero will be
+ * returned if TSC scaling is unavailable or ratio cannot be handled
+ * by host CPU. Otherwise, a non-zero ratio will be returned.
+ */
+u64 hvm_get_tsc_scaling_ratio(u32 gtsc_khz)
+{
+    u8 ratio_frac_bits = hvm_funcs.tsc_scaling.ratio_frac_bits;
+    u64 max_ratio = hvm_funcs.tsc_scaling.max_ratio;
+    u64 ratio, dummy;
+
+    if ( !hvm_tsc_scaling_supported )
+        return 0;
+
+    /*
+     * Return early if the quotient is too large to fit in the integral
+     * part of TSC scaling ratio. This also avoids #DE from the following
+     * divq when the quotient can not fit in a 64-bit integer.
+     */
+    if ( gtsc_khz / cpu_khz > (max_ratio >> ratio_frac_bits) )
+        return 0;
+
+    /* ratio = (gtsc_khz << hvm_funcs.tsc_scaling.ratio_frac_bits) / cpu_khz */
+    asm ( "shldq %[frac],%[gkhz],%[zero] ; "
+          "shlq  %[frac],%[gkhz]         ; "
+          "divq  %[hkhz]                   "
+          : "=d" (dummy), "=a" (ratio)
+          : [frac] "c" (ratio_frac_bits),
+            [gkhz] "a" ((u64) gtsc_khz),
+            [zero] "d" (0ULL),
+            [hkhz] "rm" ((u64) cpu_khz) );
+
+    return ratio > max_ratio ? 0 : ratio;
+}
+
 void hvm_set_guest_tsc_fixed(struct vcpu *v, u64 guest_tsc, u64 at_tsc)
 {
     uint64_t tsc;
@@ -1641,6 +1676,9 @@ int hvm_domain_initialise(struct domain *d)
     register_portio_handler(d, 0xe9, 1, hvm_print_line);
     register_portio_handler(d, 0xcf8, 4, hvm_access_cf8);
 
+    if ( hvm_tsc_scaling_supported )
+        d->arch.hvm_domain.tsc_scaling_ratio = hvm_default_tsc_scaling_ratio;
+
     rc = hvm_funcs.domain_initialise(d);
     if ( rc != 0 )
         goto fail2;
diff --git a/xen/arch/x86/hvm/svm/svm.c b/xen/arch/x86/hvm/svm/svm.c
index b22d4a1..7172f25 100644
--- a/xen/arch/x86/hvm/svm/svm.c
+++ b/xen/arch/x86/hvm/svm/svm.c
@@ -823,7 +823,7 @@ static uint64_t svm_scale_tsc(const struct vcpu *v, uint64_t tsc)
 {
     ASSERT(cpu_has_tsc_ratio && !v->domain->arch.vtsc);
 
-    return scale_tsc(tsc, vcpu_tsc_ratio(v));
+    return scale_tsc(tsc, hvm_tsc_scaling_ratio(v->domain));
 }
 
 static uint64_t svm_get_tsc_offset(uint64_t host_tsc, uint64_t guest_tsc,
@@ -1000,7 +1000,7 @@ static inline void svm_tsc_ratio_save(struct vcpu *v)
 static inline void svm_tsc_ratio_load(struct vcpu *v)
 {
     if ( cpu_has_tsc_ratio && !v->domain->arch.vtsc ) 
-        wrmsrl(MSR_AMD64_TSC_RATIO, vcpu_tsc_ratio(v));
+        wrmsrl(MSR_AMD64_TSC_RATIO, hvm_tsc_scaling_ratio(v->domain));
 }
 
 static void svm_ctxt_switch_from(struct vcpu *v)
diff --git a/xen/arch/x86/time.c b/xen/arch/x86/time.c
index 2248dfa..fda9692 100644
--- a/xen/arch/x86/time.c
+++ b/xen/arch/x86/time.c
@@ -1865,7 +1865,8 @@ void tsc_set_info(struct domain *d,
          */
         if ( tsc_mode == TSC_MODE_DEFAULT && host_tsc_is_safe() &&
              (has_hvm_container_domain(d) ?
-              d->arch.tsc_khz == cpu_khz || hvm_tsc_scaling_supported :
+              (d->arch.tsc_khz == cpu_khz ||
+               hvm_get_tsc_scaling_ratio(d->arch.tsc_khz)) :
               incarnation == 0) )
         {
     case TSC_MODE_NEVER_EMULATE:
@@ -1879,7 +1880,8 @@ void tsc_set_info(struct domain *d,
         d->arch.vtsc = !boot_cpu_has(X86_FEATURE_RDTSCP) ||
                        !host_tsc_is_safe();
         enable_tsc_scaling = has_hvm_container_domain(d) &&
-                             hvm_tsc_scaling_supported && !d->arch.vtsc;
+                             !d->arch.vtsc &&
+                             hvm_get_tsc_scaling_ratio(gtsc_khz ?: cpu_khz);
         d->arch.tsc_khz = (enable_tsc_scaling && gtsc_khz) ? gtsc_khz : cpu_khz;
         set_time_scale(&d->arch.vtsc_to_ns, d->arch.tsc_khz * 1000 );
         d->arch.ns_to_vtsc = scale_reciprocal(d->arch.vtsc_to_ns);
@@ -1897,6 +1899,10 @@ void tsc_set_info(struct domain *d,
     d->arch.incarnation = incarnation + 1;
     if ( has_hvm_container_domain(d) )
     {
+        if ( hvm_tsc_scaling_supported && !d->arch.vtsc )
+            d->arch.hvm_domain.tsc_scaling_ratio =
+                hvm_get_tsc_scaling_ratio(d->arch.tsc_khz);
+
         hvm_set_rdtsc_exiting(d, d->arch.vtsc);
         if ( d->vcpu && d->vcpu[0] && incarnation == 0 )
         {
diff --git a/xen/include/asm-x86/hvm/domain.h b/xen/include/asm-x86/hvm/domain.h
index 2446586..4406be9 100644
--- a/xen/include/asm-x86/hvm/domain.h
+++ b/xen/include/asm-x86/hvm/domain.h
@@ -143,6 +143,8 @@ struct hvm_domain {
      */
     uint64_t sync_tsc;
 
+    uint64_t tsc_scaling_ratio;
+
     unsigned long *io_bitmap;
 
     /* List of permanently write-mapped pages. */
diff --git a/xen/include/asm-x86/hvm/hvm.h b/xen/include/asm-x86/hvm/hvm.h
index 82d0f9d..ddb1e33 100644
--- a/xen/include/asm-x86/hvm/hvm.h
+++ b/xen/include/asm-x86/hvm/hvm.h
@@ -272,6 +272,14 @@ u64 hvm_get_guest_tsc_fixed(struct vcpu *v, u64 at_tsc);
 #define hvm_tsc_scaling_supported \
     (!!hvm_funcs.tsc_scaling.ratio_frac_bits)
 
+#define hvm_default_tsc_scaling_ratio \
+    (1ULL << hvm_funcs.tsc_scaling.ratio_frac_bits)
+
+#define hvm_tsc_scaling_ratio(d) \
+    ((d)->arch.hvm_domain.tsc_scaling_ratio)
+
+u64 hvm_get_tsc_scaling_ratio(u32 gtsc_khz);
+
 int hvm_set_mode(struct vcpu *v, int mode);
 void hvm_init_guest_time(struct domain *d);
 void hvm_set_guest_time(struct vcpu *v, u64 guest_time);
diff --git a/xen/include/asm-x86/hvm/svm/svm.h b/xen/include/asm-x86/hvm/svm/svm.h
index d60ec23..c954b7e 100644
--- a/xen/include/asm-x86/hvm/svm/svm.h
+++ b/xen/include/asm-x86/hvm/svm/svm.h
@@ -97,9 +97,6 @@ extern u32 svm_feature_flags;
 /* TSC rate */
 #define DEFAULT_TSC_RATIO       0x0000000100000000ULL
 #define TSC_RATIO_RSVD_BITS     0xffffff0000000000ULL
-#define TSC_RATIO(g_khz, h_khz) ( (((u64)(g_khz)<<32)/(u64)(h_khz)) & \
-                                  ~TSC_RATIO_RSVD_BITS )
-#define vcpu_tsc_ratio(v)       TSC_RATIO((v)->domain->arch.tsc_khz, cpu_khz)
 
 extern void svm_host_osvw_reset(void);
 extern void svm_host_osvw_init(void);
-- 
2.7.2


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[PATCH v6 2/5] x86/hvm: Replace architecture TSC scaling by a common function

[Haozhong Zhang]

This patch implements a common function hvm_scale_tsc() to scale TSC by
using TSC scaling information collected by architecture code.

Signed-off-by: Haozhong Zhang <haozhong.zhang@intel.com>
---
CC: Keir Fraser <keir@xen.org>
CC: Jan Beulich <jbeulich@suse.com>
CC: Andrew Cooper <andrew.cooper3@citrix.com>
CC: Boris Ostrovsky <boris.ostrovsky@oracle.com>
CC: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
CC: Aravind Gopalakrishnan <Aravind.Gopalakrishnan@amd.com>
CC: Kevin Tian <kevin.tian@intel.com>
---
Changes in v6:
 * Turn to named arguments for inline assembly in hvm_scale_tsc().
 * I don't take R-b from Jan Beulich and Kevin Tian because of above changes.
---
 xen/arch/x86/hvm/hvm.c        | 21 +++++++++++++++++++--
 xen/arch/x86/hvm/svm/svm.c    |  8 --------
 xen/arch/x86/time.c           |  3 +--
 xen/include/asm-x86/hvm/hvm.h |  3 +--
 4 files changed, 21 insertions(+), 14 deletions(-)

diff --git a/xen/arch/x86/hvm/hvm.c b/xen/arch/x86/hvm/hvm.c
index 6c32e99..25be45c 100644
--- a/xen/arch/x86/hvm/hvm.c
+++ b/xen/arch/x86/hvm/hvm.c
@@ -333,6 +333,23 @@ u64 hvm_get_tsc_scaling_ratio(u32 gtsc_khz)
     return ratio > max_ratio ? 0 : ratio;
 }
 
+u64 hvm_scale_tsc(const struct domain *d, u64 tsc)
+{
+    u64 ratio = d->arch.hvm_domain.tsc_scaling_ratio;
+    u64 dummy;
+
+    if ( ratio == hvm_default_tsc_scaling_ratio )
+        return tsc;
+
+    /* tsc = (tsc * ratio) >> hvm_funcs.tsc_scaling.ratio_frac_bits */
+    asm ( "mulq %[ratio]; shrdq %[frac],%%rdx,%[tsc]"
+          : [tsc] "+a" (tsc), "=d" (dummy)
+          : [frac] "c" (hvm_funcs.tsc_scaling.ratio_frac_bits),
+            [ratio] "rm" (ratio) );
+
+    return tsc;
+}
+
 void hvm_set_guest_tsc_fixed(struct vcpu *v, u64 guest_tsc, u64 at_tsc)
 {
     uint64_t tsc;
@@ -347,7 +364,7 @@ void hvm_set_guest_tsc_fixed(struct vcpu *v, u64 guest_tsc, u64 at_tsc)
     {
         tsc = at_tsc ?: rdtsc();
         if ( hvm_tsc_scaling_supported )
-            tsc = hvm_funcs.tsc_scaling.scale_tsc(v, tsc);
+            tsc = hvm_scale_tsc(v->domain, tsc);
     }
 
     delta_tsc = guest_tsc - tsc;
@@ -379,7 +396,7 @@ u64 hvm_get_guest_tsc_fixed(struct vcpu *v, uint64_t at_tsc)
     {
         tsc = at_tsc ?: rdtsc();
         if ( hvm_tsc_scaling_supported )
-            tsc = hvm_funcs.tsc_scaling.scale_tsc(v, tsc);
+            tsc = hvm_scale_tsc(v->domain, tsc);
     }
 
     return tsc + v->arch.hvm_vcpu.cache_tsc_offset;
diff --git a/xen/arch/x86/hvm/svm/svm.c b/xen/arch/x86/hvm/svm/svm.c
index 7172f25..979d226 100644
--- a/xen/arch/x86/hvm/svm/svm.c
+++ b/xen/arch/x86/hvm/svm/svm.c
@@ -819,13 +819,6 @@ static uint64_t scale_tsc(uint64_t host_tsc, uint64_t ratio)
     return scaled_host_tsc;
 }
 
-static uint64_t svm_scale_tsc(const struct vcpu *v, uint64_t tsc)
-{
-    ASSERT(cpu_has_tsc_ratio && !v->domain->arch.vtsc);
-
-    return scale_tsc(tsc, hvm_tsc_scaling_ratio(v->domain));
-}
-
 static uint64_t svm_get_tsc_offset(uint64_t host_tsc, uint64_t guest_tsc,
     uint64_t ratio)
 {
@@ -2291,7 +2284,6 @@ static struct hvm_function_table __initdata svm_function_table = {
 
     .tsc_scaling = {
         .max_ratio = ~TSC_RATIO_RSVD_BITS,
-        .scale_tsc = svm_scale_tsc,
     },
 };
 
diff --git a/xen/arch/x86/time.c b/xen/arch/x86/time.c
index fda9692..687e39b 100644
--- a/xen/arch/x86/time.c
+++ b/xen/arch/x86/time.c
@@ -816,8 +816,7 @@ static void __update_vcpu_system_time(struct vcpu *v, int force)
     {
         if ( has_hvm_container_domain(d) && hvm_tsc_scaling_supported )
         {
-            tsc_stamp            =
-                hvm_funcs.tsc_scaling.scale_tsc(v, t->local_tsc_stamp);
+            tsc_stamp            = hvm_scale_tsc(d, t->local_tsc_stamp);
             _u.tsc_to_system_mul = d->arch.vtsc_to_ns.mul_frac;
             _u.tsc_shift         = d->arch.vtsc_to_ns.shift;
         }
diff --git a/xen/include/asm-x86/hvm/hvm.h b/xen/include/asm-x86/hvm/hvm.h
index ddb1e33..c5c9328 100644
--- a/xen/include/asm-x86/hvm/hvm.h
+++ b/xen/include/asm-x86/hvm/hvm.h
@@ -231,8 +231,6 @@ struct hvm_function_table {
         uint8_t  ratio_frac_bits;
         /* maximum-allowed TSC scaling ratio */
         uint64_t max_ratio;
-
-        uint64_t (*scale_tsc)(const struct vcpu *v, uint64_t tsc);
     } tsc_scaling;
 };
 
@@ -278,6 +276,7 @@ u64 hvm_get_guest_tsc_fixed(struct vcpu *v, u64 at_tsc);
 #define hvm_tsc_scaling_ratio(d) \
     ((d)->arch.hvm_domain.tsc_scaling_ratio)
 
+u64 hvm_scale_tsc(const struct domain *d, u64 tsc);
 u64 hvm_get_tsc_scaling_ratio(u32 gtsc_khz);
 
 int hvm_set_mode(struct vcpu *v, int mode);
-- 
2.7.2


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[PATCH v6 3/5] x86/hvm: Move saving/loading vcpu's TSC to common code

[Haozhong Zhang]

Both VMX and SVM save/load vcpu's TSC when saving/loading vcpu's
context, so this patch moves saving/loading vcpu's TSC to the common
functions hvm_[save|load]_cpu_ctxt().

Signed-off-by: Haozhong Zhang <haozhong.zhang@intel.com>
Acked-by: Jan Beulich <jbeulich@suse.com>
Acked-by: Kevin Tian <kevin.tian@intel.com>
Reviewed-by: Boris Ostrovsky <boris.ostrovsky@oracle.com>
---
CC: Keir Fraser <keir@xen.org>
CC: Jan Beulich <jbeulich@suse.com>
CC: Andrew Cooper <andrew.cooper3@citrix.com>
CC: Boris Ostrovsky <boris.ostrovsky@oracle.com>
CC: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
CC: Aravind Gopalakrishnan <Aravind.Gopalakrishnan@amd.com>
CC: Jun Nakajima <jun.nakajima@intel.com>
CC: Kevin Tian <kevin.tian@intel.com>
---
 xen/arch/x86/hvm/hvm.c     | 4 ++++
 xen/arch/x86/hvm/svm/svm.c | 5 -----
 xen/arch/x86/hvm/vmx/vmx.c | 5 -----
 3 files changed, 4 insertions(+), 10 deletions(-)

diff --git a/xen/arch/x86/hvm/hvm.c b/xen/arch/x86/hvm/hvm.c
index 25be45c..50e5a5c 100644
--- a/xen/arch/x86/hvm/hvm.c
+++ b/xen/arch/x86/hvm/hvm.c
@@ -1818,6 +1818,8 @@ static int hvm_save_cpu_ctxt(struct domain *d, hvm_domain_context_t *h)
         /* Architecture-specific vmcs/vmcb bits */
         hvm_funcs.save_cpu_ctxt(v, &ctxt);
 
+        ctxt.tsc = hvm_get_guest_tsc_fixed(v, d->arch.hvm_domain.sync_tsc);
+
         ctxt.msr_tsc_aux = hvm_msr_tsc_aux(v);
 
         hvm_get_segment_register(v, x86_seg_idtr, &seg);
@@ -2123,6 +2125,8 @@ static int hvm_load_cpu_ctxt(struct domain *d, hvm_domain_context_t *h)
 
     v->arch.hvm_vcpu.msr_tsc_aux = ctxt.msr_tsc_aux;
 
+    hvm_set_guest_tsc_fixed(v, ctxt.tsc, d->arch.hvm_domain.sync_tsc);
+
     seg.limit = ctxt.idtr_limit;
     seg.base = ctxt.idtr_base;
     hvm_set_segment_register(v, x86_seg_idtr, &seg);
diff --git a/xen/arch/x86/hvm/svm/svm.c b/xen/arch/x86/hvm/svm/svm.c
index 979d226..7634c3f 100644
--- a/xen/arch/x86/hvm/svm/svm.c
+++ b/xen/arch/x86/hvm/svm/svm.c
@@ -357,9 +357,6 @@ static void svm_save_cpu_state(struct vcpu *v, struct hvm_hw_cpu *data)
     data->msr_syscall_mask = vmcb->sfmask;
     data->msr_efer         = v->arch.hvm_vcpu.guest_efer;
     data->msr_flags        = -1ULL;
-
-    data->tsc = hvm_get_guest_tsc_fixed(v,
-                                        v->domain->arch.hvm_domain.sync_tsc);
 }
 
 
@@ -374,8 +371,6 @@ static void svm_load_cpu_state(struct vcpu *v, struct hvm_hw_cpu *data)
     vmcb->sfmask     = data->msr_syscall_mask;
     v->arch.hvm_vcpu.guest_efer = data->msr_efer;
     svm_update_guest_efer(v);
-
-    hvm_set_guest_tsc_fixed(v, data->tsc, v->domain->arch.hvm_domain.sync_tsc);
 }
 
 static void svm_save_vmcb_ctxt(struct vcpu *v, struct hvm_hw_cpu *ctxt)
diff --git a/xen/arch/x86/hvm/vmx/vmx.c b/xen/arch/x86/hvm/vmx/vmx.c
index 2b266e7..b949d9c 100644
--- a/xen/arch/x86/hvm/vmx/vmx.c
+++ b/xen/arch/x86/hvm/vmx/vmx.c
@@ -604,9 +604,6 @@ static void vmx_save_cpu_state(struct vcpu *v, struct hvm_hw_cpu *data)
     data->msr_lstar        = guest_state->msrs[VMX_INDEX_MSR_LSTAR];
     data->msr_star         = guest_state->msrs[VMX_INDEX_MSR_STAR];
     data->msr_syscall_mask = guest_state->msrs[VMX_INDEX_MSR_SYSCALL_MASK];
-
-    data->tsc = hvm_get_guest_tsc_fixed(v,
-                                        v->domain->arch.hvm_domain.sync_tsc);
 }
 
 static void vmx_load_cpu_state(struct vcpu *v, struct hvm_hw_cpu *data)
@@ -621,8 +618,6 @@ static void vmx_load_cpu_state(struct vcpu *v, struct hvm_hw_cpu *data)
 
     v->arch.hvm_vmx.cstar     = data->msr_cstar;
     v->arch.hvm_vmx.shadow_gs = data->shadow_gs;
-
-    hvm_set_guest_tsc_fixed(v, data->tsc, v->domain->arch.hvm_domain.sync_tsc);
 }
 
 
-- 
2.7.2


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[PATCH v6 4/5] vmx: Add VMX RDTSC(P) scaling support

[Haozhong Zhang]

This patch adds the initialization and setup code for VMX TSC scaling.

Signed-off-by: Haozhong Zhang <haozhong.zhang@intel.com>
Acked-by: Jan Beulich <jbeulich@suse.com>
Acked-by: Kevin Tian <kevin.tian@intel.com>
---
CC: Keir Fraser <keir@xen.org>
CC: Jan Beulich <jbeulich@suse.com>
CC: Andrew Cooper <andrew.cooper3@citrix.com>
CC: Jun Nakajima <jun.nakajima@intel.com>
CC: Kevin Tian <kevin.tian@intel.com>
---
Changes in v6:
 * Adapt to the macro renaming in patch 1.
 * No functionality changes.
---
 xen/arch/x86/hvm/hvm.c             |  6 ++++++
 xen/arch/x86/hvm/vmx/vmcs.c        | 12 +++++++++---
 xen/arch/x86/hvm/vmx/vmx.c         | 17 +++++++++++++++++
 xen/include/asm-x86/hvm/hvm.h      |  3 +++
 xen/include/asm-x86/hvm/vmx/vmcs.h |  7 +++++++
 5 files changed, 42 insertions(+), 3 deletions(-)

diff --git a/xen/arch/x86/hvm/hvm.c b/xen/arch/x86/hvm/hvm.c
index 50e5a5c..9059bbc 100644
--- a/xen/arch/x86/hvm/hvm.c
+++ b/xen/arch/x86/hvm/hvm.c
@@ -2123,6 +2123,9 @@ static int hvm_load_cpu_ctxt(struct domain *d, hvm_domain_context_t *h)
     if ( hvm_funcs.load_cpu_ctxt(v, &ctxt) < 0 )
         return -EINVAL;
 
+    if ( hvm_funcs.tsc_scaling.setup )
+        hvm_funcs.tsc_scaling.setup(v);
+
     v->arch.hvm_vcpu.msr_tsc_aux = ctxt.msr_tsc_aux;
 
     hvm_set_guest_tsc_fixed(v, ctxt.tsc, d->arch.hvm_domain.sync_tsc);
@@ -5638,6 +5641,9 @@ void hvm_vcpu_reset_state(struct vcpu *v, uint16_t cs, uint16_t ip)
     hvm_set_segment_register(v, x86_seg_gdtr, &reg);
     hvm_set_segment_register(v, x86_seg_idtr, &reg);
 
+    if ( hvm_funcs.tsc_scaling.setup )
+        hvm_funcs.tsc_scaling.setup(v);
+
     /* Sync AP's TSC with BSP's. */
     v->arch.hvm_vcpu.cache_tsc_offset =
         v->domain->vcpu[0]->arch.hvm_vcpu.cache_tsc_offset;
diff --git a/xen/arch/x86/hvm/vmx/vmcs.c b/xen/arch/x86/hvm/vmx/vmcs.c
index 5bc3c74..d506f80 100644
--- a/xen/arch/x86/hvm/vmx/vmcs.c
+++ b/xen/arch/x86/hvm/vmx/vmcs.c
@@ -149,6 +149,7 @@ static void __init vmx_display_features(void)
     P(cpu_has_vmx_vmfunc, "VM Functions");
     P(cpu_has_vmx_virt_exceptions, "Virtualisation Exceptions");
     P(cpu_has_vmx_pml, "Page Modification Logging");
+    P(cpu_has_vmx_tsc_scaling, "TSC Scaling");
 #undef P
 
     if ( !printed )
@@ -243,7 +244,8 @@ static int vmx_init_vmcs_config(void)
                SECONDARY_EXEC_ENABLE_VM_FUNCTIONS |
                SECONDARY_EXEC_ENABLE_VIRT_EXCEPTIONS |
                SECONDARY_EXEC_XSAVES |
-               SECONDARY_EXEC_PCOMMIT);
+               SECONDARY_EXEC_PCOMMIT |
+               SECONDARY_EXEC_TSC_SCALING);
         rdmsrl(MSR_IA32_VMX_MISC, _vmx_misc_cap);
         if ( _vmx_misc_cap & VMX_MISC_VMWRITE_ALL )
             opt |= SECONDARY_EXEC_ENABLE_VMCS_SHADOWING;
@@ -1000,7 +1002,7 @@ static int construct_vmcs(struct vcpu *v)
     __vmwrite(PIN_BASED_VM_EXEC_CONTROL, vmx_pin_based_exec_control);
 
     v->arch.hvm_vmx.exec_control = vmx_cpu_based_exec_control;
-    if ( d->arch.vtsc )
+    if ( d->arch.vtsc && !cpu_has_vmx_tsc_scaling )
         v->arch.hvm_vmx.exec_control |= CPU_BASED_RDTSC_EXITING;
 
     v->arch.hvm_vmx.secondary_exec_control = vmx_secondary_exec_control;
@@ -1288,6 +1290,9 @@ static int construct_vmcs(struct vcpu *v)
     if ( cpu_has_vmx_xsaves )
         __vmwrite(XSS_EXIT_BITMAP, 0);
 
+    if ( cpu_has_vmx_tsc_scaling )
+        __vmwrite(TSC_MULTIPLIER, d->arch.hvm_domain.tsc_scaling_ratio);
+
     vmx_vmcs_exit(v);
 
     /* PVH: paging mode is updated by arch_set_info_guest(). */
@@ -1870,7 +1875,8 @@ void vmcs_dump_vcpu(struct vcpu *v)
            vmr32(VM_EXIT_REASON), vmr(EXIT_QUALIFICATION));
     printk("IDTVectoring: info=%08x errcode=%08x\n",
            vmr32(IDT_VECTORING_INFO), vmr32(IDT_VECTORING_ERROR_CODE));
-    printk("TSC Offset = 0x%016lx\n", vmr(TSC_OFFSET));
+    printk("TSC Offset = 0x%016lx  TSC Multiplier = 0x%016lx\n",
+           vmr(TSC_OFFSET), vmr(TSC_MULTIPLIER));
     if ( (v->arch.hvm_vmx.exec_control & CPU_BASED_TPR_SHADOW) ||
          (vmx_pin_based_exec_control & PIN_BASED_POSTED_INTERRUPT) )
         printk("TPR Threshold = 0x%02x  PostedIntrVec = 0x%02x\n",
diff --git a/xen/arch/x86/hvm/vmx/vmx.c b/xen/arch/x86/hvm/vmx/vmx.c
index b949d9c..a201b74 100644
--- a/xen/arch/x86/hvm/vmx/vmx.c
+++ b/xen/arch/x86/hvm/vmx/vmx.c
@@ -1122,6 +1122,16 @@ static void vmx_handle_cd(struct vcpu *v, unsigned long value)
     }
 }
 
+static void vmx_setup_tsc_scaling(struct vcpu *v)
+{
+    if ( !hvm_tsc_scaling_supported || v->domain->arch.vtsc )
+        return;
+
+    vmx_vmcs_enter(v);
+    __vmwrite(TSC_MULTIPLIER, hvm_tsc_scaling_ratio(v->domain));
+    vmx_vmcs_exit(v);
+}
+
 static void vmx_set_tsc_offset(struct vcpu *v, u64 offset, u64 at_tsc)
 {
     vmx_vmcs_enter(v);
@@ -2016,6 +2026,10 @@ static struct hvm_function_table __initdata vmx_function_table = {
     .altp2m_vcpu_update_vmfunc_ve = vmx_vcpu_update_vmfunc_ve,
     .altp2m_vcpu_emulate_ve = vmx_vcpu_emulate_ve,
     .altp2m_vcpu_emulate_vmfunc = vmx_vcpu_emulate_vmfunc,
+    .tsc_scaling = {
+        .max_ratio = VMX_TSC_MULTIPLIER_MAX,
+        .setup     = vmx_setup_tsc_scaling,
+    },
 };
 
 /* Handle VT-d posted-interrupt when VCPU is running. */
@@ -2120,6 +2134,9 @@ const struct hvm_function_table * __init start_vmx(void)
          && cpu_has_vmx_secondary_exec_control )
         vmx_function_table.pvh_supported = 1;
 
+    if ( cpu_has_vmx_tsc_scaling )
+        vmx_function_table.tsc_scaling.ratio_frac_bits = 48;
+
     setup_vmcs_dump();
 
     return &vmx_function_table;
diff --git a/xen/include/asm-x86/hvm/hvm.h b/xen/include/asm-x86/hvm/hvm.h
index c5c9328..12209d5 100644
--- a/xen/include/asm-x86/hvm/hvm.h
+++ b/xen/include/asm-x86/hvm/hvm.h
@@ -231,6 +231,9 @@ struct hvm_function_table {
         uint8_t  ratio_frac_bits;
         /* maximum-allowed TSC scaling ratio */
         uint64_t max_ratio;
+
+        /* Architecture function to setup TSC scaling ratio */
+        void (*setup)(struct vcpu *v);
     } tsc_scaling;
 };
 
diff --git a/xen/include/asm-x86/hvm/vmx/vmcs.h b/xen/include/asm-x86/hvm/vmx/vmcs.h
index a5e7aee..0d7c6d2 100644
--- a/xen/include/asm-x86/hvm/vmx/vmcs.h
+++ b/xen/include/asm-x86/hvm/vmx/vmcs.h
@@ -237,6 +237,7 @@ extern u32 vmx_vmentry_control;
 #define SECONDARY_EXEC_ENABLE_VIRT_EXCEPTIONS   0x00040000
 #define SECONDARY_EXEC_XSAVES                   0x00100000
 #define SECONDARY_EXEC_PCOMMIT                  0x00200000
+#define SECONDARY_EXEC_TSC_SCALING              0x02000000
 extern u32 vmx_secondary_exec_control;
 
 #define VMX_EPT_EXEC_ONLY_SUPPORTED                         0x00000001
@@ -259,6 +260,8 @@ extern u64 vmx_ept_vpid_cap;
 #define VMX_MISC_CR3_TARGET                     0x01ff0000
 #define VMX_MISC_VMWRITE_ALL                    0x20000000
 
+#define VMX_TSC_MULTIPLIER_MAX                  0xffffffffffffffffULL
+
 #define cpu_has_wbinvd_exiting \
     (vmx_secondary_exec_control & SECONDARY_EXEC_WBINVD_EXITING)
 #define cpu_has_vmx_virtualize_apic_accesses \
@@ -306,6 +309,9 @@ extern u64 vmx_ept_vpid_cap;
     (vmx_secondary_exec_control & SECONDARY_EXEC_XSAVES)
 #define cpu_has_vmx_pcommit \
     (vmx_secondary_exec_control & SECONDARY_EXEC_PCOMMIT)
+#define cpu_has_vmx_tsc_scaling \
+    (vmx_secondary_exec_control & SECONDARY_EXEC_TSC_SCALING)
+
 #define VMCS_RID_TYPE_MASK              0x80000000
 
 /* GUEST_INTERRUPTIBILITY_INFO flags. */
@@ -380,6 +386,7 @@ enum vmcs_field {
     VMWRITE_BITMAP                  = 0x00002028,
     VIRT_EXCEPTION_INFO             = 0x0000202a,
     XSS_EXIT_BITMAP                 = 0x0000202c,
+    TSC_MULTIPLIER                  = 0x00002032,
     GUEST_PHYSICAL_ADDRESS          = 0x00002400,
     VMCS_LINK_POINTER               = 0x00002800,
     GUEST_IA32_DEBUGCTL             = 0x00002802,
-- 
2.7.2


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[PATCH v6 5/5] docs: Add descriptions of TSC scaling in xl.cfg and tscmode.txt

[Haozhong Zhang]

Signed-off-by: Haozhong Zhang <haozhong.zhang@intel.com>
Reviewed-by: Kevin Tian <kevin.tian@intel.com>
---
CC: Ian Campbell <ian.campbell@citrix.com> 
CC: Ian Jackson <ian.jackson@eu.citrix.com>
CC: Jan Beulich <jbeulich@suse.com> 
CC: Keir Fraser <keir@xen.org> 
CC: Tim Deegan <tim@xen.org>
CC: Kevin Tian <kevin.tian@intel.com>
---
 docs/man/xl.cfg.pod.5 | 14 +++++++++++++-
 docs/misc/tscmode.txt | 21 +++++++++++++++++++++
 2 files changed, 34 insertions(+), 1 deletion(-)

diff --git a/docs/man/xl.cfg.pod.5 b/docs/man/xl.cfg.pod.5
index 40690bd..56b1117 100644
--- a/docs/man/xl.cfg.pod.5
+++ b/docs/man/xl.cfg.pod.5
@@ -1313,9 +1313,17 @@ deprecated. Options are:
 
 =item B<"default">
 
-Guest rdtsc/p executed natively when monotonicity can be guaranteed
+Guest rdtsc/p is executed natively when monotonicity can be guaranteed
 and emulated otherwise (with frequency scaled if necessary).
 
+If a HVM container in B<default> TSC mode is created on a host that
+provides constant host TSC, its guest TSC frequency will be the same
+as the host. If it is later migrated to another host that provide
+constant host TSC and supports Intel VMX TSC scaling/AMD SVM TSC
+ratio, its guest TSC frequency will be the same before and after
+migration, and guest rdtsc/p will be executed natively as well after
+migration.
+
 =item B<"always_emulate">
 
 Guest rdtsc/p always emulated at 1GHz (kernel and user). Guest rdtsc/p
@@ -1337,6 +1345,10 @@ determine when a restore/migration has occurred and assumes guest
 obtains/uses pvclock-like mechanism to adjust for monotonicity and
 frequency changes.
 
+If a HVM container in B<native_paravirt> TSC mode can execute both guest
+rdtsc and guest rdtscp natively, then the guest TSC frequency will be
+determined in the similar way to that of B<default> TSC mode.
+
 =back
 
 Please see F<docs/misc/tscmode.txt> for more information on this option.
diff --git a/docs/misc/tscmode.txt b/docs/misc/tscmode.txt
index e8c84e8..01ee060 100644
--- a/docs/misc/tscmode.txt
+++ b/docs/misc/tscmode.txt
@@ -297,3 +297,24 @@ and also much faster than nearly all OS-provided time mechanisms.
 While pvrtscp is too complex for most apps, certain enterprise
 TSC-sensitive high-TSC-frequency apps may find it useful to
 obtain a significant performance gain.
+
+Hardware TSC Scaling
+
+Intel VMX TSC scaling and AMD SVM TSC ratio allow the guest TSC read
+by guest rdtsc/p increasing in a different frequency than the host
+TSC frequency.
+
+If a HVM container in default TSC mode (tsc_mode=0) or PVRDTSCP mode
+(tsc_mode=3) is created on a host that provides constant TSC, its
+guest TSC frequency will be the same as the host. If it is later
+migrated to another host that provides constant TSC and supports Intel
+VMX TSC scaling/AMD SVM TSC ratio, its guest TSC frequency will be the
+same before and after migration.
+
+For above HVM container in default TSC mode (tsc_mode=0), if above
+hosts support rdtscp, both guest rdtsc and rdtscp instructions will be
+executed natively before and after migration.
+
+For above HVM container in PVRDTSCP mode (tsc_mode=3), if the
+destination host does not support rdtscp, the guest rdtscp instruction
+will be emulated with the guest TSC frequency.
-- 
2.7.2


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [PATCH v6 1/5] x86/hvm: Setup TSC scaling ratio

[Jan Beulich]

>>> On 28.02.16 at 13:54, <haozhong.zhang@intel.com> wrote:
> This patch adds a field tsc_scaling_ratio in struct hvm_domain to record
> the per-domain TSC scaling ratio, and sets it in tsc_set_info().
> 
> Before setting the per-domain TSC scaling ratio, we check its validity
> in tsc_set_info(). If an invalid ratio is given, we will leave the
> default value in tsc_scaling_ratio (i.e. ratio = 1) and setup guest TSC
> as if no TSC scaling is used:
> * For TSC_MODE_FAULT,
>   - if a user-specified TSC frequency is given, we will set the guest
>     TSC frequency to it; otherwise, we set it to the host TSC frequency.
>   - if guest TSC frequency does not equal to host TSC frequency, we will
>     emulate guest TSC (i.e. d->arch.vtsc is set to 1). In both cases,
>     guest TSC runs in the guest TSC frequency.
> * For TSC_MODE_PVRDTSCP,
>   - we set the guest TSC frequency to the host TSC frequency.
>   - guest rdtsc is executed natively in the host TSC frequency as
>     before.
>   - if rdtscp is not available to guest, it will be emulated; otherwise,
>     it will be executed natively. In both cases, guest rdtscp gets TSC
>     in the host TSC frequency as before.
> 
> Signed-off-by: Haozhong Zhang <haozhong.zhang@intel.com>

Reviewed-by: Jan Beulich <jbeulich@suse.com>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [PATCH v6 2/5] x86/hvm: Replace architecture TSC scaling by a common function

[Jan Beulich]

>>> On 28.02.16 at 13:54, <haozhong.zhang@intel.com> wrote:
> --- a/xen/arch/x86/hvm/hvm.c
> +++ b/xen/arch/x86/hvm/hvm.c
> @@ -333,6 +333,23 @@ u64 hvm_get_tsc_scaling_ratio(u32 gtsc_khz)
>      return ratio > max_ratio ? 0 : ratio;
>  }
>  
> +u64 hvm_scale_tsc(const struct domain *d, u64 tsc)
> +{
> +    u64 ratio = d->arch.hvm_domain.tsc_scaling_ratio;
> +    u64 dummy;
> +
> +    if ( ratio == hvm_default_tsc_scaling_ratio )
> +        return tsc;
> +
> +    /* tsc = (tsc * ratio) >> hvm_funcs.tsc_scaling.ratio_frac_bits */
> +    asm ( "mulq %[ratio]; shrdq %[frac],%%rdx,%[tsc]"
> +          : [tsc] "+a" (tsc), "=d" (dummy)

Is mixing named and positional asm() operands supported by all
gcc versions we care about? Also strictly speaking "=d" needs
to be switched to "=&d".

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [PATCH v6 1/5] x86/hvm: Setup TSC scaling ratio

[Boris Ostrovsky]

On 02/29/2016 08:41 AM, Jan Beulich wrote:
>>>> On 28.02.16 at 13:54, <haozhong.zhang@intel.com> wrote:
>> This patch adds a field tsc_scaling_ratio in struct hvm_domain to record
>> the per-domain TSC scaling ratio, and sets it in tsc_set_info().
>>
>> Before setting the per-domain TSC scaling ratio, we check its validity
>> in tsc_set_info(). If an invalid ratio is given, we will leave the
>> default value in tsc_scaling_ratio (i.e. ratio = 1) and setup guest TSC
>> as if no TSC scaling is used:
>> * For TSC_MODE_FAULT,

s/TSC_MODE_FAULT/TSC_MODE_DEFAULT/

-boris

>>    - if a user-specified TSC frequency is given, we will set the guest
>>      TSC frequency to it; otherwise, we set it to the host TSC frequency.
>>    - if guest TSC frequency does not equal to host TSC frequency, we will
>>      emulate guest TSC (i.e. d->arch.vtsc is set to 1). In both cases,
>>      guest TSC runs in the guest TSC frequency.
>> * For TSC_MODE_PVRDTSCP,
>>    - we set the guest TSC frequency to the host TSC frequency.
>>    - guest rdtsc is executed natively in the host TSC frequency as
>>      before.
>>    - if rdtscp is not available to guest, it will be emulated; otherwise,
>>      it will be executed natively. In both cases, guest rdtscp gets TSC
>>      in the host TSC frequency as before.
>>
>> Signed-off-by: Haozhong Zhang <haozhong.zhang@intel.com>
> Reviewed-by: Jan Beulich <jbeulich@suse.com>
>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [PATCH v6 1/5] x86/hvm: Setup TSC scaling ratio

[Haozhong Zhang]

On 02/29/16 08:49, Boris Ostrovsky wrote:
> On 02/29/2016 08:41 AM, Jan Beulich wrote:
> >>>>On 28.02.16 at 13:54, <haozhong.zhang@intel.com> wrote:
> >>This patch adds a field tsc_scaling_ratio in struct hvm_domain to record
> >>the per-domain TSC scaling ratio, and sets it in tsc_set_info().
> >>
> >>Before setting the per-domain TSC scaling ratio, we check its validity
> >>in tsc_set_info(). If an invalid ratio is given, we will leave the
> >>default value in tsc_scaling_ratio (i.e. ratio = 1) and setup guest TSC
> >>as if no TSC scaling is used:
> >>* For TSC_MODE_FAULT,
> 
> s/TSC_MODE_FAULT/TSC_MODE_DEFAULT/
>

will fix in the next version.

Thanks,
Haozhong

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [PATCH v6 1/5] x86/hvm: Setup TSC scaling ratio

[Jan Beulich]

>>> On 29.02.16 at 14:49, <boris.ostrovsky@oracle.com> wrote:
> On 02/29/2016 08:41 AM, Jan Beulich wrote:
>>>>> On 28.02.16 at 13:54, <haozhong.zhang@intel.com> wrote:
>>> This patch adds a field tsc_scaling_ratio in struct hvm_domain to record
>>> the per-domain TSC scaling ratio, and sets it in tsc_set_info().
>>>
>>> Before setting the per-domain TSC scaling ratio, we check its validity
>>> in tsc_set_info(). If an invalid ratio is given, we will leave the
>>> default value in tsc_scaling_ratio (i.e. ratio = 1) and setup guest TSC
>>> as if no TSC scaling is used:
>>> * For TSC_MODE_FAULT,
> 
> s/TSC_MODE_FAULT/TSC_MODE_DEFAULT/

If that's the only issue, I would offer to fix it up while committing.
Committing, though, requires the ack on the SVM changes to be
given (again).

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [PATCH v6 2/5] x86/hvm: Replace architecture TSC scaling by a common function

[Haozhong Zhang]

On 02/29/16 06:44, Jan Beulich wrote:
> >>> On 28.02.16 at 13:54, <haozhong.zhang@intel.com> wrote:
> > --- a/xen/arch/x86/hvm/hvm.c
> > +++ b/xen/arch/x86/hvm/hvm.c
> > @@ -333,6 +333,23 @@ u64 hvm_get_tsc_scaling_ratio(u32 gtsc_khz)
> >      return ratio > max_ratio ? 0 : ratio;
> >  }
> >  
> > +u64 hvm_scale_tsc(const struct domain *d, u64 tsc)
> > +{
> > +    u64 ratio = d->arch.hvm_domain.tsc_scaling_ratio;
> > +    u64 dummy;
> > +
> > +    if ( ratio == hvm_default_tsc_scaling_ratio )
> > +        return tsc;
> > +
> > +    /* tsc = (tsc * ratio) >> hvm_funcs.tsc_scaling.ratio_frac_bits */
> > +    asm ( "mulq %[ratio]; shrdq %[frac],%%rdx,%[tsc]"
> > +          : [tsc] "+a" (tsc), "=d" (dummy)
> 
> Is mixing named and positional asm() operands supported by all
> gcc versions we care about?

I see in Config.mk the oldest version is gcc 4.1. I'll test on that
version.

> Also strictly speaking "=d" needs to be switched to "=&d".
>

I'll fix in the next version.

Thanks,
Haozhong

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [PATCH v6 1/5] x86/hvm: Setup TSC scaling ratio

[Boris Ostrovsky]

On 02/29/2016 08:55 AM, Jan Beulich wrote:
>>>> On 29.02.16 at 14:49, <boris.ostrovsky@oracle.com> wrote:
>> On 02/29/2016 08:41 AM, Jan Beulich wrote:
>>>>>> On 28.02.16 at 13:54, <haozhong.zhang@intel.com> wrote:
>>>> This patch adds a field tsc_scaling_ratio in struct hvm_domain to record
>>>> the per-domain TSC scaling ratio, and sets it in tsc_set_info().
>>>>
>>>> Before setting the per-domain TSC scaling ratio, we check its validity
>>>> in tsc_set_info(). If an invalid ratio is given, we will leave the
>>>> default value in tsc_scaling_ratio (i.e. ratio = 1) and setup guest TSC
>>>> as if no TSC scaling is used:
>>>> * For TSC_MODE_FAULT,
>> s/TSC_MODE_FAULT/TSC_MODE_DEFAULT/
> If that's the only issue, I would offer to fix it up while committing.
> Committing, though, requires the ack on the SVM changes to be
> given (again).
>

For SVM bits:

Acked-by: Boris Ostrovsky <boris.ostrovsky@oracle.com>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [PATCH v6 2/5] x86/hvm: Replace architecture TSC scaling by a common function

[Boris Ostrovsky]

On 02/28/2016 07:54 AM, Haozhong Zhang wrote:
> This patch implements a common function hvm_scale_tsc() to scale TSC by
> using TSC scaling information collected by architecture code.
>
> Signed-off-by: Haozhong Zhang <haozhong.zhang@intel.com>
> ---
> CC: Keir Fraser <keir@xen.org>
> CC: Jan Beulich <jbeulich@suse.com>
> CC: Andrew Cooper <andrew.cooper3@citrix.com>
> CC: Boris Ostrovsky <boris.ostrovsky@oracle.com>
> CC: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
> CC: Aravind Gopalakrishnan <Aravind.Gopalakrishnan@amd.com>
> CC: Kevin Tian <kevin.tian@intel.com>

SVM bits:

Acked-by: Boris Ostrovsky <boris.ostrovsky@oracle.com>



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[PATCH v7 2/5] x86/hvm: Replace architecture TSC scaling by a common function

[Haozhong Zhang]

This patch implements a common function hvm_scale_tsc() to scale TSC by
using TSC scaling information collected by architecture code.

Signed-off-by: Haozhong Zhang <haozhong.zhang@intel.com>
Acked-by: Boris Ostrovsky <boris.ostrovsky@oracle.com> for SVM bits
---
CC: Keir Fraser <keir@xen.org>
CC: Jan Beulich <jbeulich@suse.com>
CC: Andrew Cooper <andrew.cooper3@citrix.com>
CC: Boris Ostrovsky <boris.ostrovsky@oracle.com>
CC: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
CC: Aravind Gopalakrishnan <Aravind.Gopalakrishnan@amd.com>
CC: Kevin Tian <kevin.tian@intel.com>
---
Mixing named and positional operands is tested with gcc 4.1.2. It can 
be compiled without any errors/warnings and provides the same result at 
runtime as that compiled by newer version of gcc (v4.8.4).

Changes in v7:
 For inline assembly in hvm_scale_tsc():
 * Add '&' to the restriction of output operand 'dummy'.
---
 xen/arch/x86/hvm/hvm.c        | 21 +++++++++++++++++++--
 xen/arch/x86/hvm/svm/svm.c    |  8 --------
 xen/arch/x86/time.c           |  3 +--
 xen/include/asm-x86/hvm/hvm.h |  3 +--
 4 files changed, 21 insertions(+), 14 deletions(-)

diff --git a/xen/arch/x86/hvm/hvm.c b/xen/arch/x86/hvm/hvm.c
index 6c32e99..2d87de4 100644
--- a/xen/arch/x86/hvm/hvm.c
+++ b/xen/arch/x86/hvm/hvm.c
@@ -333,6 +333,23 @@ u64 hvm_get_tsc_scaling_ratio(u32 gtsc_khz)
     return ratio > max_ratio ? 0 : ratio;
 }
 
+u64 hvm_scale_tsc(const struct domain *d, u64 tsc)
+{
+    u64 ratio = d->arch.hvm_domain.tsc_scaling_ratio;
+    u64 dummy;
+
+    if ( ratio == hvm_default_tsc_scaling_ratio )
+        return tsc;
+
+    /* tsc = (tsc * ratio) >> hvm_funcs.tsc_scaling.ratio_frac_bits */
+    asm ( "mulq %[ratio]; shrdq %[frac],%%rdx,%[tsc]"
+          : [tsc] "+a" (tsc), "=&d" (dummy)
+          : [frac] "c" (hvm_funcs.tsc_scaling.ratio_frac_bits),
+            [ratio] "rm" (ratio) );
+
+    return tsc;
+}
+
 void hvm_set_guest_tsc_fixed(struct vcpu *v, u64 guest_tsc, u64 at_tsc)
 {
     uint64_t tsc;
@@ -347,7 +364,7 @@ void hvm_set_guest_tsc_fixed(struct vcpu *v, u64 guest_tsc, u64 at_tsc)
     {
         tsc = at_tsc ?: rdtsc();
         if ( hvm_tsc_scaling_supported )
-            tsc = hvm_funcs.tsc_scaling.scale_tsc(v, tsc);
+            tsc = hvm_scale_tsc(v->domain, tsc);
     }
 
     delta_tsc = guest_tsc - tsc;
@@ -379,7 +396,7 @@ u64 hvm_get_guest_tsc_fixed(struct vcpu *v, uint64_t at_tsc)
     {
         tsc = at_tsc ?: rdtsc();
         if ( hvm_tsc_scaling_supported )
-            tsc = hvm_funcs.tsc_scaling.scale_tsc(v, tsc);
+            tsc = hvm_scale_tsc(v->domain, tsc);
     }
 
     return tsc + v->arch.hvm_vcpu.cache_tsc_offset;
diff --git a/xen/arch/x86/hvm/svm/svm.c b/xen/arch/x86/hvm/svm/svm.c
index 7172f25..979d226 100644
--- a/xen/arch/x86/hvm/svm/svm.c
+++ b/xen/arch/x86/hvm/svm/svm.c
@@ -819,13 +819,6 @@ static uint64_t scale_tsc(uint64_t host_tsc, uint64_t ratio)
     return scaled_host_tsc;
 }
 
-static uint64_t svm_scale_tsc(const struct vcpu *v, uint64_t tsc)
-{
-    ASSERT(cpu_has_tsc_ratio && !v->domain->arch.vtsc);
-
-    return scale_tsc(tsc, hvm_tsc_scaling_ratio(v->domain));
-}
-
 static uint64_t svm_get_tsc_offset(uint64_t host_tsc, uint64_t guest_tsc,
     uint64_t ratio)
 {
@@ -2291,7 +2284,6 @@ static struct hvm_function_table __initdata svm_function_table = {
 
     .tsc_scaling = {
         .max_ratio = ~TSC_RATIO_RSVD_BITS,
-        .scale_tsc = svm_scale_tsc,
     },
 };
 
diff --git a/xen/arch/x86/time.c b/xen/arch/x86/time.c
index fda9692..687e39b 100644
--- a/xen/arch/x86/time.c
+++ b/xen/arch/x86/time.c
@@ -816,8 +816,7 @@ static void __update_vcpu_system_time(struct vcpu *v, int force)
     {
         if ( has_hvm_container_domain(d) && hvm_tsc_scaling_supported )
         {
-            tsc_stamp            =
-                hvm_funcs.tsc_scaling.scale_tsc(v, t->local_tsc_stamp);
+            tsc_stamp            = hvm_scale_tsc(d, t->local_tsc_stamp);
             _u.tsc_to_system_mul = d->arch.vtsc_to_ns.mul_frac;
             _u.tsc_shift         = d->arch.vtsc_to_ns.shift;
         }
diff --git a/xen/include/asm-x86/hvm/hvm.h b/xen/include/asm-x86/hvm/hvm.h
index ddb1e33..c5c9328 100644
--- a/xen/include/asm-x86/hvm/hvm.h
+++ b/xen/include/asm-x86/hvm/hvm.h
@@ -231,8 +231,6 @@ struct hvm_function_table {
         uint8_t  ratio_frac_bits;
         /* maximum-allowed TSC scaling ratio */
         uint64_t max_ratio;
-
-        uint64_t (*scale_tsc)(const struct vcpu *v, uint64_t tsc);
     } tsc_scaling;
 };
 
@@ -278,6 +276,7 @@ u64 hvm_get_guest_tsc_fixed(struct vcpu *v, u64 at_tsc);
 #define hvm_tsc_scaling_ratio(d) \
     ((d)->arch.hvm_domain.tsc_scaling_ratio)
 
+u64 hvm_scale_tsc(const struct domain *d, u64 tsc);
 u64 hvm_get_tsc_scaling_ratio(u32 gtsc_khz);
 
 int hvm_set_mode(struct vcpu *v, int mode);
-- 
2.4.8


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [PATCH v7 2/5] x86/hvm: Replace architecture TSC scaling by a common function

[Jan Beulich]

>>> On 01.03.16 at 02:39, <haozhong.zhang@intel.com> wrote:
> This patch implements a common function hvm_scale_tsc() to scale TSC by
> using TSC scaling information collected by architecture code.
> 
> Signed-off-by: Haozhong Zhang <haozhong.zhang@intel.com>
> Acked-by: Boris Ostrovsky <boris.ostrovsky@oracle.com> for SVM bits

Reviewed-by: Jan Beulich <jbeulich@suse.com>

> ---
> Mixing named and positional operands is tested with gcc 4.1.2. It can 
> be compiled without any errors/warnings and provides the same result at 
> runtime as that compiled by newer version of gcc (v4.8.4).

Thanks for making sure of this.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [PATCH v6 5/5] docs: Add descriptions of TSC scaling in xl.cfg and tscmode.txt

[Ian Jackson]

Haozhong Zhang writes ("[PATCH v6 5/5] docs: Add descriptions of TSC scaling in xl.cfg and tscmode.txt"):
> Signed-off-by: Haozhong Zhang <haozhong.zhang@intel.com>
> Reviewed-by: Kevin Tian <kevin.tian@intel.com>

This seems relatively clear.  I'm happy to take your word for it that
this is the behaviour.

If anyone feels that the docs in this patch are wrong, they can send a
followup.  So:

Committed-by: Ian Jackson <ian.jackson@eu.citrix.com>

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [PATCH v6 5/5] docs: Add descriptions of TSC scaling in xl.cfg and tscmode.txt

[Jan Beulich]

>>> On 01.03.16 at 15:09, <Ian.Jackson@eu.citrix.com> wrote:
> Haozhong Zhang writes ("[PATCH v6 5/5] docs: Add descriptions of TSC scaling 
> in xl.cfg and tscmode.txt"):
>> Signed-off-by: Haozhong Zhang <haozhong.zhang@intel.com>
>> Reviewed-by: Kevin Tian <kevin.tian@intel.com>
> 
> This seems relatively clear.  I'm happy to take your word for it that
> this is the behaviour.
> 
> If anyone feels that the docs in this patch are wrong, they can send a
> followup.  So:
> 
> Committed-by: Ian Jackson <ian.jackson@eu.citrix.com>

Oh, thanks. I was about to write a reply asking for a tools side
ack on the xl doc change.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel