[PATCH 0/6] Intel Processor Trace virtulization enabling

[PATCH 0/6] Intel Processor Trace virtulization enabling

[Luwei Kang]

Hi All,

Here is a patch-series which adding Processor Trace enabling in XEN guest. You can get It's software developer manuals from:
https://software.intel.com/sites/default/files/managed/c5/15/architecture-instruction-set-extensions-programming-reference.pdf
In Chapter 5 INTEL PROCESSOR TRACE: VMX IMPROVEMENTS.

Introduction:
Intel Processor Trace (Intel PT) is an extension of Intel Architecture that captures information about software execution using dedicated hardware facilities that cause only minimal performance perturbation to the software being traced. Details on the Intel PT infrastructure and trace capabilities can be found in the Intel 64 and IA-32 Architectures Software Developer’s Manual, Volume 3C.

The suite of architecture changes serve to simplify the process of virtualizing Intel PT for use by a guest software. There are two primary elements to this new architecture support for VMX support improvements made for Intel PT.
1. Addition of a new guest IA32_RTIT_CTL value field to the VMCS.
  — This serves to speed and simplify the process of disabling trace on VM exit, and restoring it on VM entry.
2. Enabling use of EPT to redirect PT output.
  — This enables the VMM to elect to virtualize the PT output buffer using EPT. In this mode, the CPU will treat PT output addresses as Guest Physical Addresses (GPAs) and translate them using EPT. This means that Intel PT output reads (of the ToPA table) and writes (of trace output) can cause EPT violations, and other output events.


Luwei Kang (6):
  x86: add a flag to enable Intel processor trace
  x86: configure vmcs for Intel processor trace virtualization
  x86: add intel proecessor trace support for cpuid
  x86: add intel processor trace context
  x86: implement intel processor trace context switch
  x86: Pass through intel processor trace MSRs

 docs/misc/xen-command-line.markdown         |   7 ++
 tools/libxc/xc_cpuid_x86.c                  |  12 ++-
 xen/arch/x86/cpu/Makefile                   |   1 +
 xen/arch/x86/cpu/intel_pt.c                 | 114 ++++++++++++++++++++++++++++
 xen/arch/x86/cpuid.c                        |  22 ++++++
 xen/arch/x86/domctl.c                       |   4 +
 xen/arch/x86/hvm/vmx/vmcs.c                 |  36 +++++++--
 xen/arch/x86/hvm/vmx/vmx.c                  |   4 +
 xen/include/asm-x86/cpufeature.h            |   1 +
 xen/include/asm-x86/cpuid.h                 |  12 ++-
 xen/include/asm-x86/hvm/vmx/vmcs.h          |  12 +++
 xen/include/asm-x86/intel_pt.h              |  47 ++++++++++++
 xen/include/asm-x86/msr-index.h             |  16 ++++
 xen/include/public/arch-x86/cpufeatureset.h |   1 +
 14 files changed, 281 insertions(+), 8 deletions(-)
 create mode 100644 xen/arch/x86/cpu/intel_pt.c
 create mode 100644 xen/include/asm-x86/intel_pt.h

-- 
1.8.3.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH 1/6] x86: add a flag to enable Intel processor trace

[Luwei Kang]

This patch add a flag to enable Intel PT (Intel processor trace).
Default value is 1 (enabled).

Signed-off-by: Luwei Kang <luwei.kang@intel.com>
---
 docs/misc/xen-command-line.markdown |  7 +++++++
 xen/arch/x86/cpu/Makefile           |  1 +
 xen/arch/x86/cpu/intel_pt.c         | 27 +++++++++++++++++++++++++++
 xen/include/asm-x86/intel_pt.h      | 26 ++++++++++++++++++++++++++
 4 files changed, 61 insertions(+)
 create mode 100644 xen/arch/x86/cpu/intel_pt.c
 create mode 100644 xen/include/asm-x86/intel_pt.h

diff --git a/docs/misc/xen-command-line.markdown b/docs/misc/xen-command-line.markdown
index eb4995e..a376932 100644
--- a/docs/misc/xen-command-line.markdown
+++ b/docs/misc/xen-command-line.markdown
@@ -1009,6 +1009,13 @@ debug hypervisor only).
 ### idle\_latency\_factor
 > `= <integer>`
 
+### intel\_pt
+> `= <boolean>`
+
+> Default: `true`
+
+Flag to enable Intel Processor Trace.
+
 ### ioapic\_ack
 > `= old | new`
 
diff --git a/xen/arch/x86/cpu/Makefile b/xen/arch/x86/cpu/Makefile
index 74f23ae..33d7a74 100644
--- a/xen/arch/x86/cpu/Makefile
+++ b/xen/arch/x86/cpu/Makefile
@@ -8,3 +8,4 @@ obj-y += intel.o
 obj-y += intel_cacheinfo.o
 obj-y += mwait-idle.o
 obj-y += vpmu.o vpmu_amd.o vpmu_intel.o
+obj-y += intel_pt.o
diff --git a/xen/arch/x86/cpu/intel_pt.c b/xen/arch/x86/cpu/intel_pt.c
new file mode 100644
index 0000000..be06d55
--- /dev/null
+++ b/xen/arch/x86/cpu/intel_pt.c
@@ -0,0 +1,27 @@
+/*
+ * intel_pt.c: Support Intel Processor Trace Virtualization.
+ *
+ * Copyright (c) 2017, Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License along with
+ * this program; If not, see <http://www.gnu.org/licenses/>.
+ *
+ * Author: Luwei Kang <luwei.kang@intel.com>
+ */
+
+#include <xen/types.h>
+#include <xen/cache.h>
+#include <xen/init.h>
+
+/* intel_pt: Flag to enable Intel Processor Trace (default on). */
+bool_t __read_mostly opt_intel_pt = 1;
+boolean_param("intel_pt", opt_intel_pt);
diff --git a/xen/include/asm-x86/intel_pt.h b/xen/include/asm-x86/intel_pt.h
new file mode 100644
index 0000000..3b3a047
--- /dev/null
+++ b/xen/include/asm-x86/intel_pt.h
@@ -0,0 +1,26 @@
+/*
+ * intel_pt.h: Intel Processor Trace virtualization for HVM domain.
+ *
+ * Copyright (c) 2017, Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License along with
+ * this program; If not, see <http://www.gnu.org/licenses/>.
+ *
+ * Author: Luwei Kang <luwei.kang@intel.com>
+ */
+
+#ifndef __ASM_X86_HVM_INTEL_PT_H_
+#define __ASM_X86_HVM_INTEL_PT_H_
+
+extern bool_t opt_intel_pt;
+
+#endif /* __ASM_X86_HVM_INTEL_PT_H_ */
-- 
1.8.3.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH 2/6] x86: configure vmcs for Intel processor trace virtualization

[Luwei Kang]

This patch configure VMCS to make Intel PT output address can be
treat as guest physical address and translated by EPT when
intel_pt option is true.
There have some constraint condition on VMCS configuration,
otherwise will cause VM entry failed.

1. If the “Guest PT uses Guest Physical Addresses” execution
   control is 1, the “Clear IA32_RTIT_CTL on exit” exit
   control and the “Load IA32_RTIT_CTL on entry” entry
   control must also be 1.

2. If the “Guest PT uses Guest Physical Addresses” execution
   control is 1, the "enable EPT" execution control must
   also be 1.

Signed-off-by: Luwei Kang <luwei.kang@intel.com>
---
 xen/arch/x86/hvm/vmx/vmcs.c        | 36 +++++++++++++++++++++++++++++++-----
 xen/include/asm-x86/hvm/vmx/vmcs.h |  7 +++++++
 2 files changed, 38 insertions(+), 5 deletions(-)

diff --git a/xen/arch/x86/hvm/vmx/vmcs.c b/xen/arch/x86/hvm/vmx/vmcs.c
index f62fe7e..8cd57b5 100644
--- a/xen/arch/x86/hvm/vmx/vmcs.c
+++ b/xen/arch/x86/hvm/vmx/vmcs.c
@@ -40,6 +40,7 @@
 #include <asm/shadow.h>
 #include <asm/tboot.h>
 #include <asm/apic.h>
+#include <asm/intel_pt.h>
 
 static bool_t __read_mostly opt_vpid_enabled = 1;
 boolean_param("vpid", opt_vpid_enabled);
@@ -242,6 +243,9 @@ static int vmx_init_vmcs_config(void)
         rdmsrl(MSR_IA32_VMX_MISC, _vmx_misc_cap);
         if ( _vmx_misc_cap & VMX_MISC_VMWRITE_ALL )
             opt |= SECONDARY_EXEC_ENABLE_VMCS_SHADOWING;
+        if ( _vmx_misc_cap & VMX_MISC_PT_ENABLE )
+            opt |= SECONDARY_EXEC_PT_USE_GPA |
+                   SECONDARY_EXEC_CONCEAL_PT_PIP;
         if ( opt_vpid_enabled )
             opt |= SECONDARY_EXEC_ENABLE_VPID;
         if ( opt_unrestricted_guest_enabled )
@@ -343,7 +347,8 @@ static int vmx_init_vmcs_config(void)
 
     min = VM_EXIT_ACK_INTR_ON_EXIT;
     opt = VM_EXIT_SAVE_GUEST_PAT | VM_EXIT_LOAD_HOST_PAT |
-          VM_EXIT_CLEAR_BNDCFGS;
+          VM_EXIT_CLEAR_BNDCFGS | VM_EXIT_CONCEAL_PT_PIP |
+          VM_EXIT_CLEAR_IA32_RTIT_CTL;
     min |= VM_EXIT_IA32E_MODE;
     _vmx_vmexit_control = adjust_vmx_controls(
         "VMExit Control", min, opt, MSR_IA32_VMX_EXIT_CTLS, &mismatch);
@@ -383,13 +388,28 @@ static int vmx_init_vmcs_config(void)
         _vmx_secondary_exec_control &= ~SECONDARY_EXEC_ENABLE_VIRT_EXCEPTIONS;
 
     min = 0;
-    opt = VM_ENTRY_LOAD_GUEST_PAT | VM_ENTRY_LOAD_BNDCFGS;
+    opt = VM_ENTRY_LOAD_GUEST_PAT | VM_ENTRY_LOAD_BNDCFGS |
+          VM_ENTRY_CONCEAL_PT_PIP | VM_ENTRY_LOAD_IA32_RTIT_CTL;
     _vmx_vmentry_control = adjust_vmx_controls(
         "VMEntry Control", min, opt, MSR_IA32_VMX_ENTRY_CTLS, &mismatch);
 
     if ( mismatch )
         return -EINVAL;
 
+    if ( !(_vmx_secondary_exec_control & SECONDARY_EXEC_ENABLE_EPT) ||
+         !(_vmx_secondary_exec_control & SECONDARY_EXEC_PT_USE_GPA) ||
+         !(_vmx_vmexit_control & VM_EXIT_CLEAR_IA32_RTIT_CTL) ||
+         !(_vmx_vmentry_control & VM_ENTRY_LOAD_IA32_RTIT_CTL) )
+    {
+        _vmx_secondary_exec_control &= ~(SECONDARY_EXEC_PT_USE_GPA |
+                                         SECONDARY_EXEC_CONCEAL_PT_PIP);
+        _vmx_vmexit_control &= ~(VM_EXIT_CONCEAL_PT_PIP |
+                                 VM_EXIT_CLEAR_IA32_RTIT_CTL);
+        _vmx_vmentry_control &= ~(VM_ENTRY_CONCEAL_PT_PIP |
+                                  VM_ENTRY_LOAD_IA32_RTIT_CTL);
+        opt_intel_pt = 0;
+    }
+
     if ( !vmx_pin_based_exec_control )
     {
         /* First time through. */
@@ -1032,10 +1052,16 @@ static int construct_vmcs(struct vcpu *v)
         v->arch.hvm_vmx.secondary_exec_control &= 
             ~(SECONDARY_EXEC_ENABLE_EPT | 
               SECONDARY_EXEC_UNRESTRICTED_GUEST |
-              SECONDARY_EXEC_ENABLE_INVPCID);
+              SECONDARY_EXEC_ENABLE_INVPCID |
+              SECONDARY_EXEC_PT_USE_GPA |
+              SECONDARY_EXEC_CONCEAL_PT_PIP);
         vmexit_ctl &= ~(VM_EXIT_SAVE_GUEST_PAT |
-                        VM_EXIT_LOAD_HOST_PAT);
-        vmentry_ctl &= ~VM_ENTRY_LOAD_GUEST_PAT;
+                        VM_EXIT_LOAD_HOST_PAT |
+                        VM_EXIT_CONCEAL_PT_PIP |
+                        VM_EXIT_CLEAR_IA32_RTIT_CTL);
+        vmentry_ctl &= ~(VM_ENTRY_LOAD_GUEST_PAT |
+                         VM_ENTRY_CONCEAL_PT_PIP |
+                         VM_ENTRY_LOAD_IA32_RTIT_CTL);
     }
 
     /* Disable Virtualize x2APIC mode by default. */
diff --git a/xen/include/asm-x86/hvm/vmx/vmcs.h b/xen/include/asm-x86/hvm/vmx/vmcs.h
index 8fb9e3c..bd8a128 100644
--- a/xen/include/asm-x86/hvm/vmx/vmcs.h
+++ b/xen/include/asm-x86/hvm/vmx/vmcs.h
@@ -220,6 +220,8 @@ extern u32 vmx_pin_based_exec_control;
 #define VM_EXIT_LOAD_HOST_EFER          0x00200000
 #define VM_EXIT_SAVE_PREEMPT_TIMER      0x00400000
 #define VM_EXIT_CLEAR_BNDCFGS           0x00800000
+#define VM_EXIT_CONCEAL_PT_PIP          0x01000000
+#define VM_EXIT_CLEAR_IA32_RTIT_CTL     0x02000000
 extern u32 vmx_vmexit_control;
 
 #define VM_ENTRY_IA32E_MODE             0x00000200
@@ -229,6 +231,8 @@ extern u32 vmx_vmexit_control;
 #define VM_ENTRY_LOAD_GUEST_PAT         0x00004000
 #define VM_ENTRY_LOAD_GUEST_EFER        0x00008000
 #define VM_ENTRY_LOAD_BNDCFGS           0x00010000
+#define VM_ENTRY_CONCEAL_PT_PIP         0x00020000
+#define VM_ENTRY_LOAD_IA32_RTIT_CTL     0x00040000
 extern u32 vmx_vmentry_control;
 
 #define SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES 0x00000001
@@ -247,7 +251,9 @@ extern u32 vmx_vmentry_control;
 #define SECONDARY_EXEC_ENABLE_VMCS_SHADOWING    0x00004000
 #define SECONDARY_EXEC_ENABLE_PML               0x00020000
 #define SECONDARY_EXEC_ENABLE_VIRT_EXCEPTIONS   0x00040000
+#define SECONDARY_EXEC_CONCEAL_PT_PIP           0x00080000
 #define SECONDARY_EXEC_XSAVES                   0x00100000
+#define SECONDARY_EXEC_PT_USE_GPA               0x01000000
 #define SECONDARY_EXEC_TSC_SCALING              0x02000000
 extern u32 vmx_secondary_exec_control;
 
@@ -268,6 +274,7 @@ extern u32 vmx_secondary_exec_control;
 #define VMX_VPID_INVVPID_SINGLE_CONTEXT_RETAINING_GLOBAL 0x80000000000ULL
 extern u64 vmx_ept_vpid_cap;
 
+#define VMX_MISC_PT_ENABLE                      0x00004000
 #define VMX_MISC_CR3_TARGET                     0x01ff0000
 #define VMX_MISC_VMWRITE_ALL                    0x20000000
 
-- 
1.8.3.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH 3/6] x86: add intel proecessor trace support for cpuid

[Luwei Kang]

This patch add Intel processor trace support
for cpuid handling.

Signed-off-by: Luwei Kang <luwei.kang@intel.com>
---
 tools/libxc/xc_cpuid_x86.c                  | 12 ++++++++++--
 xen/arch/x86/cpuid.c                        | 22 ++++++++++++++++++++++
 xen/arch/x86/domctl.c                       |  4 ++++
 xen/include/asm-x86/cpufeature.h            |  1 +
 xen/include/asm-x86/cpuid.h                 | 12 +++++++++++-
 xen/include/public/arch-x86/cpufeatureset.h |  1 +
 6 files changed, 49 insertions(+), 3 deletions(-)

diff --git a/tools/libxc/xc_cpuid_x86.c b/tools/libxc/xc_cpuid_x86.c
index d890935..0e0575c 100644
--- a/tools/libxc/xc_cpuid_x86.c
+++ b/tools/libxc/xc_cpuid_x86.c
@@ -38,7 +38,7 @@ enum {
 #define clear_feature(idx, dst) ((dst) &= ~bitmaskof(idx))
 #define set_feature(idx, dst)   ((dst) |=  bitmaskof(idx))
 
-#define DEF_MAX_BASE 0x0000000du
+#define DEF_MAX_BASE 0x00000014u
 #define DEF_MAX_INTELEXT  0x80000008u
 #define DEF_MAX_AMDEXT    0x8000001cu
 
@@ -471,6 +471,7 @@ static void xc_cpuid_hvm_policy(xc_interface *xch,
     case 0x00000002: /* Intel cache info (dumped by AMD policy) */
     case 0x00000004: /* Intel cache info (dumped by AMD policy) */
     case 0x0000000a: /* Architectural Performance Monitor Features */
+    case 0x00000014: /* Intel Processor Trace Features */
     case 0x80000002: /* Processor name string */
     case 0x80000003: /* ... continued         */
     case 0x80000004: /* ... continued         */
@@ -757,12 +758,19 @@ int xc_cpuid_apply_policy(xc_interface *xch, domid_t domid,
                 continue;
         }
 
+        if ( input[0] == 0x14 )
+        {
+            input[1]++;
+            if ( input[1] == 1 )
+                continue;
+        }
+
         input[0]++;
         if ( !(input[0] & 0x80000000u) && (input[0] > base_max ) )
             input[0] = 0x80000000u;
 
         input[1] = XEN_CPUID_INPUT_UNUSED;
-        if ( (input[0] == 4) || (input[0] == 7) )
+        if ( (input[0] == 4) || (input[0] == 7) || (input[0] == 0x14) )
             input[1] = 0;
         else if ( input[0] == 0xd )
             input[1] = 1; /* Xen automatically calculates almost everything. */
diff --git a/xen/arch/x86/cpuid.c b/xen/arch/x86/cpuid.c
index 5ee82d3..c3d56fd 100644
--- a/xen/arch/x86/cpuid.c
+++ b/xen/arch/x86/cpuid.c
@@ -9,6 +9,7 @@
 #include <asm/paging.h>
 #include <asm/processor.h>
 #include <asm/xstate.h>
+#include <asm/intel_pt.h>
 
 const uint32_t known_features[] = INIT_KNOWN_FEATURES;
 const uint32_t special_features[] = INIT_SPECIAL_FEATURES;
@@ -487,7 +488,19 @@ void recalculate_cpuid_policy(struct domain *d)
             __clear_bit(X86_FEATURE_VMX, max_fs);
             __clear_bit(X86_FEATURE_SVM, max_fs);
         }
+
+        /*
+         * Hide Intel Processor trace feature when hardware not support
+         * PT-VMX or intel_pt option is disabled.
+         */
+        if ( !opt_intel_pt )
+        {
+            __clear_bit(X86_FEATURE_INTEL_PT, max_fs);
+            zero_leaves(p->intel_pt.raw, 0, ARRAY_SIZE(p->intel_pt.raw) - 1);
+        }
     }
+    else
+        zero_leaves(p->intel_pt.raw, 0, ARRAY_SIZE(p->intel_pt.raw) - 1);
 
     /*
      * Allow the toolstack to set HTT, X2APIC and CMP_LEGACY.  These bits
@@ -634,6 +647,15 @@ void guest_cpuid(const struct vcpu *v, uint32_t leaf,
             *res = p->feat.raw[subleaf];
             break;
 
+        case 0x14:
+            ASSERT(p->intel_pt.max_subleaf < ARRAY_SIZE(p->intel_pt.raw));
+            if ( subleaf > min_t(uint32_t, p->intel_pt.max_subleaf,
+                                 ARRAY_SIZE(p->intel_pt.raw) - 1) )
+                return;
+
+            *res = p->intel_pt.raw[subleaf];
+            break;
+
         case XSTATE_CPUID:
             if ( !p->basic.xsave || subleaf >= ARRAY_SIZE(p->xstate.raw) )
                 return;
diff --git a/xen/arch/x86/domctl.c b/xen/arch/x86/domctl.c
index 1b208f9..405b31e 100644
--- a/xen/arch/x86/domctl.c
+++ b/xen/arch/x86/domctl.c
@@ -100,6 +100,10 @@ static int update_domain_cpuid_info(struct domain *d,
             p->feat.raw[ctl->input[1]] = leaf;
             break;
 
+        case 0x14:
+            p->intel_pt.raw[ctl->input[1]] = leaf;
+            break;
+
         case XSTATE_CPUID:
             p->xstate.raw[ctl->input[1]] = leaf;
             break;
diff --git a/xen/include/asm-x86/cpufeature.h b/xen/include/asm-x86/cpufeature.h
index 84cc51d..8956667 100644
--- a/xen/include/asm-x86/cpufeature.h
+++ b/xen/include/asm-x86/cpufeature.h
@@ -95,6 +95,7 @@
 #define cpu_has_mpx             boot_cpu_has(X86_FEATURE_MPX)
 #define cpu_has_rdseed          boot_cpu_has(X86_FEATURE_RDSEED)
 #define cpu_has_smap            boot_cpu_has(X86_FEATURE_SMAP)
+#define cpu_has_intel_pt        boot_cpu_has(X86_FEATURE_INTEL_PT)
 #define cpu_has_sha             boot_cpu_has(X86_FEATURE_SHA)
 
 /* CPUID level 0x80000007.edx */
diff --git a/xen/include/asm-x86/cpuid.h b/xen/include/asm-x86/cpuid.h
index d2dd841..39f06aa 100644
--- a/xen/include/asm-x86/cpuid.h
+++ b/xen/include/asm-x86/cpuid.h
@@ -61,10 +61,11 @@ extern struct cpuidmasks cpuidmask_defaults;
 /* Whether or not cpuid faulting is available for the current domain. */
 DECLARE_PER_CPU(bool, cpuid_faulting_enabled);
 
-#define CPUID_GUEST_NR_BASIC      (0xdu + 1)
+#define CPUID_GUEST_NR_BASIC      (0x14u + 1)
 #define CPUID_GUEST_NR_FEAT       (0u + 1)
 #define CPUID_GUEST_NR_CACHE      (5u + 1)
 #define CPUID_GUEST_NR_XSTATE     (62u + 1)
+#define CPUID_GUEST_NR_INTEL_PT   (1u + 1)
 #define CPUID_GUEST_NR_EXTD_INTEL (0x8u + 1)
 #define CPUID_GUEST_NR_EXTD_AMD   (0x1cu + 1)
 #define CPUID_GUEST_NR_EXTD       MAX(CPUID_GUEST_NR_EXTD_INTEL, \
@@ -169,6 +170,15 @@ struct cpuid_policy
         } comp[CPUID_GUEST_NR_XSTATE];
     } xstate;
 
+    /* Structured feature leaf: 0x00000014[xx] */
+    union {
+        struct cpuid_leaf raw[CPUID_GUEST_NR_INTEL_PT];
+        struct {
+            /* Subleaf 0. */
+            uint32_t max_subleaf;
+        };
+    } intel_pt;
+
     /* Extended leaves: 0x800000xx */
     union {
         struct cpuid_leaf raw[CPUID_GUEST_NR_EXTD];
diff --git a/xen/include/public/arch-x86/cpufeatureset.h b/xen/include/public/arch-x86/cpufeatureset.h
index 0ee3ea3..b5648f7 100644
--- a/xen/include/public/arch-x86/cpufeatureset.h
+++ b/xen/include/public/arch-x86/cpufeatureset.h
@@ -215,6 +215,7 @@ XEN_CPUFEATURE(SMAP,          5*32+20) /*S  Supervisor Mode Access Prevention */
 XEN_CPUFEATURE(AVX512IFMA,    5*32+21) /*A  AVX-512 Integer Fused Multiply Add */
 XEN_CPUFEATURE(CLFLUSHOPT,    5*32+23) /*A  CLFLUSHOPT instruction */
 XEN_CPUFEATURE(CLWB,          5*32+24) /*A  CLWB instruction */
+XEN_CPUFEATURE(INTEL_PT,      5*32+25) /*H  Intel Processor Trace */
 XEN_CPUFEATURE(AVX512PF,      5*32+26) /*A  AVX-512 Prefetch Instructions */
 XEN_CPUFEATURE(AVX512ER,      5*32+27) /*A  AVX-512 Exponent & Reciprocal Instrs */
 XEN_CPUFEATURE(AVX512CD,      5*32+28) /*A  AVX-512 Conflict Detection Instrs */
-- 
1.8.3.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH 4/6] x86: add intel processor trace context

[Luwei Kang]

This patch add Intel processor trace context
date structure for guest.

Signed-off-by: Luwei Kang <luwei.kang@intel.com>
---
 xen/include/asm-x86/hvm/vmx/vmcs.h |  3 +++
 xen/include/asm-x86/intel_pt.h     | 17 +++++++++++++++++
 xen/include/asm-x86/msr-index.h    | 16 ++++++++++++++++
 3 files changed, 36 insertions(+)

diff --git a/xen/include/asm-x86/hvm/vmx/vmcs.h b/xen/include/asm-x86/hvm/vmx/vmcs.h
index bd8a128..33ec3e6 100644
--- a/xen/include/asm-x86/hvm/vmx/vmcs.h
+++ b/xen/include/asm-x86/hvm/vmx/vmcs.h
@@ -20,6 +20,7 @@
 
 #include <asm/hvm/io.h>
 #include <irq_vectors.h>
+#include <asm/intel_pt.h>
 
 extern void vmcs_dump_vcpu(struct vcpu *v);
 extern void setup_vmcs_dump(void);
@@ -171,6 +172,8 @@ struct arch_vmx_struct {
      * pCPU and wakeup the related vCPU.
      */
     struct pi_blocking_vcpu pi_blocking;
+
+    struct pt_desc       pt_desc;
 };
 
 int vmx_create_vmcs(struct vcpu *v);
diff --git a/xen/include/asm-x86/intel_pt.h b/xen/include/asm-x86/intel_pt.h
index 3b3a047..78e3a37 100644
--- a/xen/include/asm-x86/intel_pt.h
+++ b/xen/include/asm-x86/intel_pt.h
@@ -21,6 +21,23 @@
 #ifndef __ASM_X86_HVM_INTEL_PT_H_
 #define __ASM_X86_HVM_INTEL_PT_H_
 
+#include <asm/msr-index.h>
+
+struct pt_ctx {
+    u64 ctl;
+    u64 status;
+    u64 output_base;
+    u64 output_mask_ptrs;
+    u64 cr3_match;
+    u64 addr[NUM_MSR_IA32_RTIT_ADDR];
+};
+
+struct pt_desc {
+    bool intel_pt_enabled;
+    unsigned int addr_num;
+    struct pt_ctx guest_pt_ctx;
+};
+
 extern bool_t opt_intel_pt;
 
 #endif /* __ASM_X86_HVM_INTEL_PT_H_ */
diff --git a/xen/include/asm-x86/msr-index.h b/xen/include/asm-x86/msr-index.h
index b99c623..d160d44 100644
--- a/xen/include/asm-x86/msr-index.h
+++ b/xen/include/asm-x86/msr-index.h
@@ -528,4 +528,20 @@
 #define MSR_PKGC9_IRTL			0x00000634
 #define MSR_PKGC10_IRTL			0x00000635
 
+/* Intel PT MSRs */
+#define MSR_IA32_RTIT_CTL              0x00000570
+#define MSR_IA32_RTIT_STATUS           0x00000571
+#define MSR_IA32_RTIT_CR3_MATCH        0x00000572
+#define MSR_IA32_RTIT_OUTPUT_BASE      0x00000560
+#define MSR_IA32_RTIT_OUTPUT_MASK_PTRS 0x00000561
+#define MSR_IA32_RTIT_ADDR0_A          0x00000580
+#define MSR_IA32_RTIT_ADDR0_B          0x00000581
+#define MSR_IA32_RTIT_ADDR1_A          0x00000582
+#define MSR_IA32_RTIT_ADDR1_B          0x00000583
+#define MSR_IA32_RTIT_ADDR2_A          0x00000584
+#define MSR_IA32_RTIT_ADDR2_B          0x00000585
+#define MSR_IA32_RTIT_ADDR3_A          0x00000586
+#define MSR_IA32_RTIT_ADDR3_B          0x00000587
+#define NUM_MSR_IA32_RTIT_ADDR         8
+
 #endif /* __ASM_MSR_INDEX_H */
-- 
1.8.3.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH 5/6] x86: implement intel processor trace context switch

[Luwei Kang]

This patch implement Intel proecessor trace context switch.

Signed-off-by: Luwei Kang <luwei.kang@intel.com>
---
 xen/arch/x86/cpu/intel_pt.c        | 79 ++++++++++++++++++++++++++++++++++++++
 xen/arch/x86/hvm/vmx/vmx.c         |  4 ++
 xen/include/asm-x86/hvm/vmx/vmcs.h |  2 +
 xen/include/asm-x86/intel_pt.h     |  4 ++
 4 files changed, 89 insertions(+)

diff --git a/xen/arch/x86/cpu/intel_pt.c b/xen/arch/x86/cpu/intel_pt.c
index be06d55..411b922 100644
--- a/xen/arch/x86/cpu/intel_pt.c
+++ b/xen/arch/x86/cpu/intel_pt.c
@@ -21,7 +21,86 @@
 #include <xen/types.h>
 #include <xen/cache.h>
 #include <xen/init.h>
+#include <asm/hvm/vmx/vmx.h>
+#include <asm/intel_pt.h>
 
 /* intel_pt: Flag to enable Intel Processor Trace (default on). */
 bool_t __read_mostly opt_intel_pt = 1;
 boolean_param("intel_pt", opt_intel_pt);
+
+static inline void pt_load_msr(struct pt_ctx *ctx, u32 addr_num)
+{
+    u32 i;
+    wrmsrl(MSR_IA32_RTIT_STATUS, ctx->status);
+    wrmsrl(MSR_IA32_RTIT_OUTPUT_BASE, ctx->output_base);
+    wrmsrl(MSR_IA32_RTIT_OUTPUT_MASK_PTRS, ctx->output_mask_ptrs);
+    wrmsrl(MSR_IA32_RTIT_CR3_MATCH, ctx->cr3_match);
+    for ( i = 0; i < addr_num; i++ )
+        wrmsrl(MSR_IA32_RTIT_ADDR0_A + i, ctx->addr[i]);
+}
+
+static inline void pt_save_msr(struct pt_ctx *ctx, u32 addr_num)
+{
+    u32 i;
+    rdmsrl(MSR_IA32_RTIT_STATUS, ctx->status);
+    rdmsrl(MSR_IA32_RTIT_OUTPUT_BASE, ctx->output_base);
+    rdmsrl(MSR_IA32_RTIT_OUTPUT_MASK_PTRS, ctx->output_mask_ptrs);
+    rdmsrl(MSR_IA32_RTIT_CR3_MATCH, ctx->cr3_match);
+    for ( i = 0; i < addr_num; i++ )
+        rdmsrl(MSR_IA32_RTIT_ADDR0_A + i, ctx->addr[i]);
+}
+
+void pt_guest_enter(struct vcpu *v)
+{
+    struct pt_desc *pt = &v->arch.hvm_vmx.pt_desc;
+
+    if ( pt->intel_pt_enabled )
+    {
+        vmx_vmcs_enter(v);
+        __vmwrite(GUEST_IA32_RTIT_CTL, pt->guest_pt_ctx.ctl);
+        vmx_vmcs_exit(v);
+
+        pt_load_msr(&pt->guest_pt_ctx, pt->addr_num);
+    }
+}
+
+void pt_guest_exit(struct vcpu *v)
+{
+    struct pt_desc *pt = &v->arch.hvm_vmx.pt_desc;
+
+    if ( pt->intel_pt_enabled )
+    {
+        vmx_vmcs_enter(v);
+        __vmread(GUEST_IA32_RTIT_CTL, &pt->guest_pt_ctx.ctl);
+        vmx_vmcs_exit(v);
+
+        pt_save_msr(&pt->guest_pt_ctx, pt->addr_num);
+    }
+}
+
+void pt_vcpu_init(struct vcpu *v)
+{
+    struct pt_desc *pt = &v->arch.hvm_vmx.pt_desc;
+    unsigned int eax, ebx, ecx, edx;
+
+    memset(pt, 0, sizeof(struct pt_desc));
+    pt->intel_pt_enabled = false;
+
+    if ( !cpu_has_intel_pt || !opt_intel_pt ||
+         !(v->arch.hvm_vmx.secondary_exec_control & SECONDARY_EXEC_PT_USE_GPA) )
+        return;
+
+    /* get the number of address ranges */
+    if ( cpuid_eax(0x14) == 1 )
+        cpuid_count(0x14, 1, &eax, &ebx, &ecx, &edx);
+    else
+        return;
+
+    pt->addr_num = eax & 0x7;
+    pt->guest_pt_ctx.output_mask_ptrs = 0x7F;
+    pt->intel_pt_enabled = true;
+
+    vmx_vmcs_enter(v);
+    __vmwrite(GUEST_IA32_RTIT_CTL, 0);
+    vmx_vmcs_exit(v);
+}
diff --git a/xen/arch/x86/hvm/vmx/vmx.c b/xen/arch/x86/hvm/vmx/vmx.c
index a5c2bd7..ea23dbd 100644
--- a/xen/arch/x86/hvm/vmx/vmx.c
+++ b/xen/arch/x86/hvm/vmx/vmx.c
@@ -468,6 +468,8 @@ static int vmx_vcpu_initialise(struct vcpu *v)
     if ( v->vcpu_id == 0 )
         v->arch.user_regs.rax = 1;
 
+    pt_vcpu_init(v);
+
     return 0;
 }
 
@@ -3492,6 +3494,7 @@ void vmx_vmexit_handler(struct cpu_user_regs *regs)
     __vmread(GUEST_RSP,    &regs->rsp);
     __vmread(GUEST_RFLAGS, &regs->rflags);
 
+    pt_guest_exit(v);
     hvm_invalidate_regs_fields(regs);
 
     if ( paging_mode_hap(v->domain) )
@@ -4260,6 +4263,7 @@ bool vmx_vmenter_helper(const struct cpu_user_regs *regs)
         }
     }
 
+    pt_guest_enter(curr);
  out:
     if ( unlikely(curr->arch.hvm_vmx.lbr_fixup_enabled) )
         lbr_fixup();
diff --git a/xen/include/asm-x86/hvm/vmx/vmcs.h b/xen/include/asm-x86/hvm/vmx/vmcs.h
index 33ec3e6..46c386f 100644
--- a/xen/include/asm-x86/hvm/vmx/vmcs.h
+++ b/xen/include/asm-x86/hvm/vmx/vmcs.h
@@ -421,6 +421,8 @@ enum vmcs_field {
     GUEST_PDPTE0                    = 0x0000280a,
 #define GUEST_PDPTE(n) (GUEST_PDPTE0 + (n) * 2) /* n = 0...3 */
     GUEST_BNDCFGS                   = 0x00002812,
+    GUEST_IA32_RTIT_CTL             = 0x00002814,
+    GUEST_IA32_RTIT_CTL_HIGH        = 0x00002815,
     HOST_PAT                        = 0x00002c00,
     HOST_EFER                       = 0x00002c02,
     HOST_PERF_GLOBAL_CTRL           = 0x00002c04,
diff --git a/xen/include/asm-x86/intel_pt.h b/xen/include/asm-x86/intel_pt.h
index 78e3a37..29ea466 100644
--- a/xen/include/asm-x86/intel_pt.h
+++ b/xen/include/asm-x86/intel_pt.h
@@ -40,4 +40,8 @@ struct pt_desc {
 
 extern bool_t opt_intel_pt;
 
+void pt_vcpu_init(struct vcpu *v);
+void pt_guest_enter(struct vcpu *v);
+void pt_guest_exit(struct vcpu *v);
+
 #endif /* __ASM_X86_HVM_INTEL_PT_H_ */
-- 
1.8.3.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH 6/6] x86: Pass through intel processor trace MSRs

[Luwei Kang]

This patch pass through Intel processor trace MSRs
to guest.

Signed-off-by: Luwei Kang <luwei.kang@intel.com>
---
 xen/arch/x86/cpu/intel_pt.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/xen/arch/x86/cpu/intel_pt.c b/xen/arch/x86/cpu/intel_pt.c
index 411b922..8e83b87 100644
--- a/xen/arch/x86/cpu/intel_pt.c
+++ b/xen/arch/x86/cpu/intel_pt.c
@@ -82,6 +82,7 @@ void pt_vcpu_init(struct vcpu *v)
 {
     struct pt_desc *pt = &v->arch.hvm_vmx.pt_desc;
     unsigned int eax, ebx, ecx, edx;
+    int i;
 
     memset(pt, 0, sizeof(struct pt_desc));
     pt->intel_pt_enabled = false;
@@ -102,5 +103,12 @@ void pt_vcpu_init(struct vcpu *v)
 
     vmx_vmcs_enter(v);
     __vmwrite(GUEST_IA32_RTIT_CTL, 0);
+    vmx_clear_msr_intercept(v, MSR_IA32_RTIT_CTL, VMX_MSR_RW);
+    vmx_clear_msr_intercept(v, MSR_IA32_RTIT_STATUS, VMX_MSR_RW);
+    vmx_clear_msr_intercept(v, MSR_IA32_RTIT_OUTPUT_BASE, VMX_MSR_RW);
+    vmx_clear_msr_intercept(v, MSR_IA32_RTIT_OUTPUT_MASK_PTRS, VMX_MSR_RW);
+    vmx_clear_msr_intercept(v, MSR_IA32_RTIT_CR3_MATCH, VMX_MSR_RW);
+    for ( i = 0; i < pt->addr_num; i++ )
+        vmx_clear_msr_intercept(v, MSR_IA32_RTIT_ADDR0_A + i, VMX_MSR_RW);
     vmx_vmcs_exit(v);
 }
-- 
1.8.3.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 0/6] Intel Processor Trace virtulization enabling

[Andrew Cooper]

On 21/10/17 21:02, Luwei Kang wrote:
> Hi All,
>
> Here is a patch-series which adding Processor Trace enabling in XEN guest. You can get It's software developer manuals from:
> https://software.intel.com/sites/default/files/managed/c5/15/architecture-instruction-set-extensions-programming-reference.pdf
> In Chapter 5 INTEL PROCESSOR TRACE: VMX IMPROVEMENTS.
>
> Introduction:
> Intel Processor Trace (Intel PT) is an extension of Intel Architecture that captures information about software execution using dedicated hardware facilities that cause only minimal performance perturbation to the software being traced. Details on the Intel PT infrastructure and trace capabilities can be found in the Intel 64 and IA-32 Architectures Software Developer’s Manual, Volume 3C.
>
> The suite of architecture changes serve to simplify the process of virtualizing Intel PT for use by a guest software. There are two primary elements to this new architecture support for VMX support improvements made for Intel PT.
> 1. Addition of a new guest IA32_RTIT_CTL value field to the VMCS.
>   — This serves to speed and simplify the process of disabling trace on VM exit, and restoring it on VM entry.
> 2. Enabling use of EPT to redirect PT output.
>   — This enables the VMM to elect to virtualize the PT output buffer using EPT. In this mode, the CPU will treat PT output addresses as Guest Physical Addresses (GPAs) and translate them using EPT. This means that Intel PT output reads (of the ToPA table) and writes (of trace output) can cause EPT violations, and other output events.

Hello,

Having read the new proposed extensions, I've got some architecture
questions before diving into the patches themselves.

First of all, is this technology expected to end up in Icelake, or
something later?

In Vol 3, the existing VMX support describes a number of scenarios
(system wide, VMM-only, VM-only, guest aware), which require the use of
MSR load lists to atomically alter the IA32_RTIT_* msrs.

Obviously, system wide mode is incompatible with also allowing the guest
to use PT itself, but what about Xen wanting to use PT for itself, and
for the guest to use PT as well?

Previously, this appears to be possible using the MSR load lists (albeit
with Xen needing to shadow the ToPA records to cause the packet stream
to end up in the right place).

However, the new VM consistency checks require that using EPT
redirection requires clear/load CTL on exit/entry be set, and having
load on entry set requires the host TraceEn to be clear.

Therefore, as far as I can see, allowing a guest to use PT via EPT now
prohibits Xen also using PT for its own purposes outside of non-root mode.

Is this intentional and/or expected, or have I misunderstood something
in the manuals?

Thanks,

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 0/6] Intel Processor Trace virtulization enabling

[Kang, Luwei]

> > Hi All,
> >
> > Here is a patch-series which adding Processor Trace enabling in XEN guest. You can get It's software developer manuals from:
> > https://software.intel.com/sites/default/files/managed/c5/15/architect
> > ure-instruction-set-extensions-programming-reference.pdf
> > In Chapter 5 INTEL PROCESSOR TRACE: VMX IMPROVEMENTS.
> >
> > Introduction:
> > Intel Processor Trace (Intel PT) is an extension of Intel Architecture that captures information about software execution using
> dedicated hardware facilities that cause only minimal performance perturbation to the software being traced. Details on the Intel PT
> infrastructure and trace capabilities can be found in the Intel 64 and IA-32 Architectures Software Developer’s Manual, Volume 3C.
> >
> > The suite of architecture changes serve to simplify the process of virtualizing Intel PT for use by a guest software. There are two
> primary elements to this new architecture support for VMX support improvements made for Intel PT.
> > 1. Addition of a new guest IA32_RTIT_CTL value field to the VMCS.
> >   — This serves to speed and simplify the process of disabling trace on VM exit, and restoring it on VM entry.
> > 2. Enabling use of EPT to redirect PT output.
> >   — This enables the VMM to elect to virtualize the PT output buffer using EPT. In this mode, the CPU will treat PT output
> addresses as Guest Physical Addresses (GPAs) and translate them using EPT. This means that Intel PT output reads (of the ToPA
> table) and writes (of trace output) can cause EPT violations, and other output events.
> 
> Hello,
> 
> Having read the new proposed extensions, I've got some architecture questions before diving into the patches themselves.
> 
> First of all, is this technology expected to end up in Icelake, or something later?

Yes, this would be enabled on Icelake.

> 
> In Vol 3, the existing VMX support describes a number of scenarios (system wide, VMM-only, VM-only, guest aware), which require
> the use of MSR load lists to atomically alter the IA32_RTIT_* msrs.

Currently, I just enabling the guest only mode(VM-only) in my patches.

> 
> Obviously, system wide mode is incompatible with also allowing the guest to use PT itself, 

Yes, system mode(collect trace packets of the entire system) can't work with guest only mode at the same time.

> but what about Xen wanting to use PT for itself, and for the guest to use PT as well?

I think this can be named by host-guest mode. This may need add new command or interface to enable PT in Xen hypervisor. Trace vmm-only and guest simultaneous and output to their respective buffer.

> 
> Previously, this appears to be possible using the MSR load lists (albeit with Xen needing to shadow the ToPA records to cause the
> packet stream to end up in the right place).

Yes.

> 
> However, the new VM consistency checks require that using EPT redirection requires clear/load CTL on exit/entry be set, and having
> load on entry set requires the host TraceEn to be clear.

Yes. New bits added in VMCS can make sure PT be disabled before VM exit and IA32_RTIT_CTL MSR will be written with the value of the associated Guest State field of the VMCS on VM entry. 

Thanks for your response.

Luwei Kang

> 
> Therefore, as far as I can see, allowing a guest to use PT via EPT now prohibits Xen also using PT for its own purposes outside of non-
> root mode.
> 
> Is this intentional and/or expected, or have I misunderstood something in the manuals?
> 
> Thanks,
> 
> ~Andrew
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 0/6] Intel Processor Trace virtulization enabling

[Andrew Cooper]

On 26/10/17 05:13, Kang, Luwei wrote:
>>> Hi All,
>>>
>>> Here is a patch-series which adding Processor Trace enabling in XEN guest. You can get It's software developer manuals from:
>>> https://software.intel.com/sites/default/files/managed/c5/15/architect
>>> ure-instruction-set-extensions-programming-reference.pdf
>>> In Chapter 5 INTEL PROCESSOR TRACE: VMX IMPROVEMENTS.
>>>
>>> Introduction:
>>> Intel Processor Trace (Intel PT) is an extension of Intel Architecture that captures information about software execution using
>> dedicated hardware facilities that cause only minimal performance perturbation to the software being traced. Details on the Intel PT
>> infrastructure and trace capabilities can be found in the Intel 64 and IA-32 Architectures Software Developer’s Manual, Volume 3C.
>>> The suite of architecture changes serve to simplify the process of virtualizing Intel PT for use by a guest software. There are two
>> primary elements to this new architecture support for VMX support improvements made for Intel PT.
>>> 1. Addition of a new guest IA32_RTIT_CTL value field to the VMCS.
>>>   — This serves to speed and simplify the process of disabling trace on VM exit, and restoring it on VM entry.
>>> 2. Enabling use of EPT to redirect PT output.
>>>   — This enables the VMM to elect to virtualize the PT output buffer using EPT. In this mode, the CPU will treat PT output
>> addresses as Guest Physical Addresses (GPAs) and translate them using EPT. This means that Intel PT output reads (of the ToPA
>> table) and writes (of trace output) can cause EPT violations, and other output events.
>>
>> Hello,
>>
>> Having read the new proposed extensions, I've got some architecture questions before diving into the patches themselves.
>>
>> First of all, is this technology expected to end up in Icelake, or something later?
> Yes, this would be enabled on Icelake.

Thanks.

>
>> In Vol 3, the existing VMX support describes a number of scenarios (system wide, VMM-only, VM-only, guest aware), which require
>> the use of MSR load lists to atomically alter the IA32_RTIT_* msrs.
> Currently, I just enabling the guest only mode(VM-only) in my patches.

That is fine.  I'm not asking you to implement these modes; I am just
trying to work out how they would interact.

>
>> Obviously, system wide mode is incompatible with also allowing the guest to use PT itself, 
> Yes, system mode(collect trace packets of the entire system) can't work with guest only mode at the same time.
>
>> but what about Xen wanting to use PT for itself, and for the guest to use PT as well?
> I think this can be named by host-guest mode. This may need add new command or interface to enable PT in Xen hypervisor. Trace vmm-only and guest simultaneous and output to their respective buffer.
>
>> Previously, this appears to be possible using the MSR load lists (albeit with Xen needing to shadow the ToPA records to cause the
>> packet stream to end up in the right place).
> Yes.
>
>> However, the new VM consistency checks require that using EPT redirection requires clear/load CTL on exit/entry be set, and having
>> load on entry set requires the host TraceEn to be clear.
> Yes. New bits added in VMCS can make sure PT be disabled before VM exit and IA32_RTIT_CTL MSR will be written with the value of the associated Guest State field of the VMCS on VM entry.

I am not sure I explained myself clearly.

It appears that it is possible to implement host-guest mode using MSR
load lists, because all the host configuration gets replaced by guest
configuration on vmentry, and host configuration gets restored at vmexit.

However with these PT extension, a new restriction is that a vmentry
failure will occur if we try to load the guest RTIT_CTL value while the
current RTIT_CTL.TraceEn is set.

As far as I can tell, this prohibits host-guest mode from working,
unless we tolerate having Xen clear RTIT_CTL before restoring guest GPR
state.

Is this correct, or have I missed something?

Thanks,

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 0/6] Intel Processor Trace virtulization enabling

[Kang, Luwei]

> >>> Hi All,
> >>>
> >>> Here is a patch-series which adding Processor Trace enabling in XEN guest. You can get It's software developer manuals from:
> >>> https://software.intel.com/sites/default/files/managed/c5/15/archite
> >>> ct ure-instruction-set-extensions-programming-reference.pdf
> >>> In Chapter 5 INTEL PROCESSOR TRACE: VMX IMPROVEMENTS.
> >>>
> >>> Introduction:
> >>> Intel Processor Trace (Intel PT) is an extension of Intel
> >>> Architecture that captures information about software execution
> >>> using
> >> dedicated hardware facilities that cause only minimal performance
> >> perturbation to the software being traced. Details on the Intel PT infrastructure and trace capabilities can be found in the Intel 64
> and IA-32 Architectures Software Developer’s Manual, Volume 3C.
> >>> The suite of architecture changes serve to simplify the process of
> >>> virtualizing Intel PT for use by a guest software. There are two
> >> primary elements to this new architecture support for VMX support improvements made for Intel PT.
> >>> 1. Addition of a new guest IA32_RTIT_CTL value field to the VMCS.
> >>>   — This serves to speed and simplify the process of disabling trace on VM exit, and restoring it on VM entry.
> >>> 2. Enabling use of EPT to redirect PT output.
> >>>   — This enables the VMM to elect to virtualize the PT output buffer
> >>> using EPT. In this mode, the CPU will treat PT output
> >> addresses as Guest Physical Addresses (GPAs) and translate them using
> >> EPT. This means that Intel PT output reads (of the ToPA
> >> table) and writes (of trace output) can cause EPT violations, and other output events.
> >>
> >> Hello,
> >>
> >> Having read the new proposed extensions, I've got some architecture questions before diving into the patches themselves.
> >>
> >> First of all, is this technology expected to end up in Icelake, or something later?
> > Yes, this would be enabled on Icelake.
> 
> Thanks.
> 
> >
> >> In Vol 3, the existing VMX support describes a number of scenarios
> >> (system wide, VMM-only, VM-only, guest aware), which require the use of MSR load lists to atomically alter the IA32_RTIT_*
> msrs.
> > Currently, I just enabling the guest only mode(VM-only) in my patches.
> 
> That is fine.  I'm not asking you to implement these modes; I am just trying to work out how they would interact.

System-Wide: trace Xen hypervisor and guest and output to host buffer; 
VMM-only: trace Xen hypervisor only and output to host buffer.;
VM-only: trace guest only and output to guest buffer;

> 
> >
> >> Obviously, system wide mode is incompatible with also allowing the
> >> guest to use PT itself,
> > Yes, system mode(collect trace packets of the entire system) can't work with guest only mode at the same time.
> >
> >> but what about Xen wanting to use PT for itself, and for the guest to use PT as well?
> > I think this can be named by host-guest mode. This may need add new command or interface to enable PT in Xen hypervisor.
> Trace vmm-only and guest simultaneous and output to their respective buffer.
> >
> >> Previously, this appears to be possible using the MSR load lists
> >> (albeit with Xen needing to shadow the ToPA records to cause the packet stream to end up in the right place).
> > Yes.
> >
> >> However, the new VM consistency checks require that using EPT
> >> redirection requires clear/load CTL on exit/entry be set, and having load on entry set requires the host TraceEn to be clear.
> > Yes. New bits added in VMCS can make sure PT be disabled before VM exit and IA32_RTIT_CTL MSR will be written with the value
> of the associated Guest State field of the VMCS on VM entry.
> 
> I am not sure I explained myself clearly.
> 
> It appears that it is possible to implement host-guest mode using MSR load lists, because all the host configuration gets replaced by
> guest configuration on vmentry, and host configuration gets restored at vmexit.

Yes, correct.

> 
> However with these PT extension, a new restriction is that a vmentry failure will occur if we try to load the guest RTIT_CTL value
> while the current RTIT_CTL.TraceEn is set.

Yes, as description in "Intel® architecture instruction set extensions programming reference" 5.2.3. 
If the “Load IA32_RTIT_CTL on entry” is 1, IA32_RTIT_CTL.TraceEn must be zero. Otherwise will VM entry fails by loading processor state from the guest-state area of the VMCS.

> 
> As far as I can tell, this prohibits host-guest mode from working, unless we tolerate having Xen clear RTIT_CTL before restoring guest
> GPR state.

Yes. If implement host-guest mode we should clear RTIT_CTL first and restore the guest state before VM-entry .
1. Clear RTIT_CTL before configuration MSRs. A WRMSR to any of these configuration MSRs that begins and ends with IA32_RTIT_CTL.TraceEn set will #GP fault. Packet generation must be disabled before the configuration MSRs can be changed. (SDM 36.2.7.1)
2. Restore other Intel PT MSRs.
3. Restore guest RTIT_CTL state to VMCS guest IA32_RTIT_CTL.

Thanks,
Luwei Kang
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel