[PATCH v1 0/3] Remove the limitation of Intel PT CPUID info

[PATCH v1 0/3] Remove the limitation of Intel PT CPUID info

[Luwei Kang]

The Intel PT feature includes some sub-features(CPUID.(EAX=14H,ECX=0H)) and
these sub-features are different on different HW platforms. To make the live
migration safety(get the same CPUID info with same cpu model on different HW
platform), the current Intel PT CPUID information is set to a constant
value(from ICELAKE Server).

It will block the new feature in the later HW platform. what's more, the
support of "IP payloads" will disable the Intel PT in KVM guest(patch 1) but
it will come soon.

This patchset remove this limitation and expose all the capabilities to
KVM guest. As it will break the live migration safe, Intel PT will be
masked as unmigratable.

Luwei Kang (3):
  i386: Remove the limitation of IP payloads for Intel PT
  i386: Remove the CPUID limitation of Intel PT
  i386: Mark the 'INTEL_PT' CPUID bit as unmigratable

 target/i386/cpu.c | 69 ++++---------------------------------------------------
 1 file changed, 5 insertions(+), 64 deletions(-)

-- 
1.8.3.1

[PATCH v1 1/3] i386: Remove the limitation of IP payloads for Intel PT

[Luwei Kang]

The Intel PT packets which contain IP payloads will have LIP values, and it
will include the CS base component if the CPUID.(EAX=14H,ECX=0H).ECX.[bit31]
is set. But it will disabled the Intel PT in kvm guest because of the need
of live migration safe(c078ca9 i386: Disable Intel PT if packets IP payloads
have LIP values).

This patch will revert the previous limitation because the Intel new hardware
will set this bit and LIP == RIP for most/all real code.

Signed-off-by: Luwei Kang <luwei.kang@intel.com>
---
 target/i386/cpu.c | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index 69f518a..8c0d1e4 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -688,8 +688,6 @@ static CPUCacheInfo legacy_l3_cache = {
  * bit[02]: Support Single-Range Output scheme;
  */
 #define INTEL_PT_MINIMAL_ECX     0x7
-/* generated packets which contain IP payloads have LIP values */
-#define INTEL_PT_IP_LIP          (1 << 31)
 #define INTEL_PT_ADDR_RANGES_NUM 0x2 /* Number of configurable address ranges */
 #define INTEL_PT_ADDR_RANGES_NUM_MASK 0x3
 #define INTEL_PT_MTC_BITMAP      (0x0249 << 16) /* Support ART(0,3,6,9) */
@@ -6281,8 +6279,7 @@ static void x86_cpu_filter_features(X86CPU *cpu, bool verbose)
            ((eax_1 & INTEL_PT_ADDR_RANGES_NUM_MASK) <
                                            INTEL_PT_ADDR_RANGES_NUM) ||
            ((ebx_1 & (INTEL_PT_PSB_BITMAP | INTEL_PT_CYCLE_BITMAP)) !=
-                (INTEL_PT_PSB_BITMAP | INTEL_PT_CYCLE_BITMAP)) ||
-           (ecx_0 & INTEL_PT_IP_LIP)) {
+                (INTEL_PT_PSB_BITMAP | INTEL_PT_CYCLE_BITMAP))) {
             /*
              * Processor Trace capabilities aren't configurable, so if the
              * host can't emulate the capabilities we report on
-- 
1.8.3.1

[PATCH v1 2/3] i386: Remove the CPUID limitation of Intel PT

[Luwei Kang]

To make Intel PT live migration safe and get same CPUID information
with same CPU model on diffrent host. CPUID[14] is set to constant
value in "e37a5c7 i386: Add Intel Processor Trace feature support".
But it will block the new features of Intel PT. This patch will
remove this limitation and expose all the capabilities to guest.

Signed-off-by: Luwei Kang <luwei.kang@intel.com>
---
 target/i386/cpu.c | 65 ++++---------------------------------------------------
 1 file changed, 4 insertions(+), 61 deletions(-)

diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index 8c0d1e4..4d9e203 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -667,33 +667,6 @@ static CPUCacheInfo legacy_l3_cache = {
 #define L2_ITLB_4K_ASSOC       4
 #define L2_ITLB_4K_ENTRIES   512
 
-/* CPUID Leaf 0x14 constants: */
-#define INTEL_PT_MAX_SUBLEAF     0x1
-/*
- * bit[00]: IA32_RTIT_CTL.CR3 filter can be set to 1 and IA32_RTIT_CR3_MATCH
- *          MSR can be accessed;
- * bit[01]: Support Configurable PSB and Cycle-Accurate Mode;
- * bit[02]: Support IP Filtering, TraceStop filtering, and preservation
- *          of Intel PT MSRs across warm reset;
- * bit[03]: Support MTC timing packet and suppression of COFI-based packets;
- */
-#define INTEL_PT_MINIMAL_EBX     0xf
-/*
- * bit[00]: Tracing can be enabled with IA32_RTIT_CTL.ToPA = 1 and
- *          IA32_RTIT_OUTPUT_BASE and IA32_RTIT_OUTPUT_MASK_PTRS MSRs can be
- *          accessed;
- * bit[01]: ToPA tables can hold any number of output entries, up to the
- *          maximum allowed by the MaskOrTableOffset field of
- *          IA32_RTIT_OUTPUT_MASK_PTRS;
- * bit[02]: Support Single-Range Output scheme;
- */
-#define INTEL_PT_MINIMAL_ECX     0x7
-#define INTEL_PT_ADDR_RANGES_NUM 0x2 /* Number of configurable address ranges */
-#define INTEL_PT_ADDR_RANGES_NUM_MASK 0x3
-#define INTEL_PT_MTC_BITMAP      (0x0249 << 16) /* Support ART(0,3,6,9) */
-#define INTEL_PT_CYCLE_BITMAP    0x1fff         /* Support 0,2^(0~11) */
-#define INTEL_PT_PSB_BITMAP      (0x003f << 16) /* Support 2K,4K,8K,16K,32K,64K */
-
 static void x86_cpu_vendor_words2str(char *dst, uint32_t vendor1,
                                      uint32_t vendor2, uint32_t vendor3)
 {
@@ -5538,14 +5511,10 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count,
             break;
         }
 
-        if (count == 0) {
-            *eax = INTEL_PT_MAX_SUBLEAF;
-            *ebx = INTEL_PT_MINIMAL_EBX;
-            *ecx = INTEL_PT_MINIMAL_ECX;
-        } else if (count == 1) {
-            *eax = INTEL_PT_MTC_BITMAP | INTEL_PT_ADDR_RANGES_NUM;
-            *ebx = INTEL_PT_PSB_BITMAP | INTEL_PT_CYCLE_BITMAP;
-        }
+        *eax = kvm_arch_get_supported_cpuid(cs->kvm_state, 0x14, count, R_EAX);
+        *ebx = kvm_arch_get_supported_cpuid(cs->kvm_state, 0x14, count, R_EBX);
+        *ecx = kvm_arch_get_supported_cpuid(cs->kvm_state, 0x14, count, R_ECX);
+        *edx = kvm_arch_get_supported_cpuid(cs->kvm_state, 0x14, count, R_EDX);
         break;
     }
     case 0x40000000:
@@ -6262,32 +6231,6 @@ static void x86_cpu_filter_features(X86CPU *cpu, bool verbose)
         uint64_t unavailable_features = requested_features & ~host_feat;
         mark_unavailable_features(cpu, w, unavailable_features, prefix);
     }
-
-    if ((env->features[FEAT_7_0_EBX] & CPUID_7_0_EBX_INTEL_PT) &&
-        kvm_enabled()) {
-        KVMState *s = CPU(cpu)->kvm_state;
-        uint32_t eax_0 = kvm_arch_get_supported_cpuid(s, 0x14, 0, R_EAX);
-        uint32_t ebx_0 = kvm_arch_get_supported_cpuid(s, 0x14, 0, R_EBX);
-        uint32_t ecx_0 = kvm_arch_get_supported_cpuid(s, 0x14, 0, R_ECX);
-        uint32_t eax_1 = kvm_arch_get_supported_cpuid(s, 0x14, 1, R_EAX);
-        uint32_t ebx_1 = kvm_arch_get_supported_cpuid(s, 0x14, 1, R_EBX);
-
-        if (!eax_0 ||
-           ((ebx_0 & INTEL_PT_MINIMAL_EBX) != INTEL_PT_MINIMAL_EBX) ||
-           ((ecx_0 & INTEL_PT_MINIMAL_ECX) != INTEL_PT_MINIMAL_ECX) ||
-           ((eax_1 & INTEL_PT_MTC_BITMAP) != INTEL_PT_MTC_BITMAP) ||
-           ((eax_1 & INTEL_PT_ADDR_RANGES_NUM_MASK) <
-                                           INTEL_PT_ADDR_RANGES_NUM) ||
-           ((ebx_1 & (INTEL_PT_PSB_BITMAP | INTEL_PT_CYCLE_BITMAP)) !=
-                (INTEL_PT_PSB_BITMAP | INTEL_PT_CYCLE_BITMAP))) {
-            /*
-             * Processor Trace capabilities aren't configurable, so if the
-             * host can't emulate the capabilities we report on
-             * cpu_x86_cpuid(), intel-pt can't be enabled on the current host.
-             */
-            mark_unavailable_features(cpu, FEAT_7_0_EBX, CPUID_7_0_EBX_INTEL_PT, prefix);
-        }
-    }
 }
 
 static void x86_cpu_realizefn(DeviceState *dev, Error **errp)
-- 
1.8.3.1

[PATCH v1 3/3] i386: Mark the 'INTEL_PT' CPUID bit as unmigratable

[Luwei Kang]

After expose all the capabilities of Intel PT to KVM guest, the guest Intel
PT CPUID information may difference with same guest cpu model on differnt
hardware. It will block the live migration. This patch will mark the Intel
PT feature as unmigratable.

Signed-off-by: Luwei Kang <luwei.kang@intel.com>
---
 target/i386/cpu.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index 4d9e203..caee8b1 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -1024,6 +1024,7 @@ static FeatureWordInfo feature_word_info[FEATURE_WORDS] = {
             .reg = R_EBX,
         },
         .tcg_features = TCG_7_0_EBX_FEATURES,
+        .unmigratable_flags = CPUID_7_0_EBX_INTEL_PT,
     },
     [FEAT_7_0_ECX] = {
         .type = CPUID_FEATURE_WORD,
-- 
1.8.3.1

RE: [PATCH v1 0/3] Remove the limitation of Intel PT CPUID info

[Kang, Luwei]

> -----Original Message-----
> From: Kang, Luwei <luwei.kang@intel.com>
> Sent: Tuesday, February 25, 2020 5:38 AM
> To: pbonzini@redhat.com; rth@twiddle.net; ehabkost@redhat.com
> Cc: qemu-devel@nongnu.org; Strong, Beeman <beeman.strong@intel.com>;
> Kang, Luwei <luwei.kang@intel.com>
> Subject: [PATCH v1 0/3] Remove the limitation of Intel PT CPUID info
> 
> The Intel PT feature includes some sub-features(CPUID.(EAX=14H,ECX=0H))
> and these sub-features are different on different HW platforms. To make the
> live migration safety(get the same CPUID info with same cpu model on
> different HW platform), the current Intel PT CPUID information is set to a
> constant value(from ICELAKE Server).
> 
> It will block the new feature in the later HW platform. what's more, the support
> of "IP payloads" will disable the Intel PT in KVM guest(patch 1) but it will come
> soon.
> 
> This patchset remove this limitation and expose all the capabilities to KVM
> guest. As it will break the live migration safe, Intel PT will be masked as
> unmigratable.

Ping.

Thanks,
Luwei Kang

> 
> Luwei Kang (3):
>   i386: Remove the limitation of IP payloads for Intel PT
>   i386: Remove the CPUID limitation of Intel PT
>   i386: Mark the 'INTEL_PT' CPUID bit as unmigratable
> 
>  target/i386/cpu.c | 69 ++++---------------------------------------------------
>  1 file changed, 5 insertions(+), 64 deletions(-)
> 
> --
> 1.8.3.1

Re: [PATCH v1 0/3] Remove the limitation of Intel PT CPUID info

[Eduardo Habkost]

Hi Luwei Kang,

I was looking for info on intel-pt and just saw this series, and
it was never reviewed or merged (sorry for missing it!).  Is this
still the approach we want to follow for intel-pt?

I'm CCing Jiri Denemark because this might be relevant for a
libvirt issue related to intel-pt we were investigating[1].

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1853972


On Mon, Mar 30, 2020 at 09:56:09AM +0000, Kang, Luwei wrote:
> > -----Original Message-----
> > From: Kang, Luwei <luwei.kang@intel.com>
> > Sent: Tuesday, February 25, 2020 5:38 AM
> > To: pbonzini@redhat.com; rth@twiddle.net; ehabkost@redhat.com
> > Cc: qemu-devel@nongnu.org; Strong, Beeman <beeman.strong@intel.com>;
> > Kang, Luwei <luwei.kang@intel.com>
> > Subject: [PATCH v1 0/3] Remove the limitation of Intel PT CPUID info
> > 
> > The Intel PT feature includes some sub-features(CPUID.(EAX=14H,ECX=0H))
> > and these sub-features are different on different HW platforms. To make the
> > live migration safety(get the same CPUID info with same cpu model on
> > different HW platform), the current Intel PT CPUID information is set to a
> > constant value(from ICELAKE Server).
> > 
> > It will block the new feature in the later HW platform. what's more, the support
> > of "IP payloads" will disable the Intel PT in KVM guest(patch 1) but it will come
> > soon.
> > 
> > This patchset remove this limitation and expose all the capabilities to KVM
> > guest. As it will break the live migration safe, Intel PT will be masked as
> > unmigratable.
> 
> Ping.
> 
> Thanks,
> Luwei Kang
> 
> > 
> > Luwei Kang (3):
> >   i386: Remove the limitation of IP payloads for Intel PT
> >   i386: Remove the CPUID limitation of Intel PT
> >   i386: Mark the 'INTEL_PT' CPUID bit as unmigratable
> > 
> >  target/i386/cpu.c | 69 ++++---------------------------------------------------
> >  1 file changed, 5 insertions(+), 64 deletions(-)
> > 
> > --
> > 1.8.3.1
> 

-- 
Eduardo

RE: [PATCH v1 0/3] Remove the limitation of Intel PT CPUID info

[Kang, Luwei]

Hi Eduardo,
    This patch set will remove some limitations of Intel PT CPUID information.
    1. The "IP payloads" feature will disable the Intel PT in guests and it will be coming soon.
    2. To make the live migration safe, we set the Intel PT CPUID as a constant value(Icelake server CPUID). It will mask off the new feature of Intel PT.

    About this issue https://bugzilla.redhat.com/show_bug.cgi?id=1853972, Intel PT is disabled in the guest by default, we should use "-cpu Icelake-Server,+intel-pt" to enable the Intel PT.

Thanks,
Luwei Kang

> -----Original Message-----
> From: Eduardo Habkost <ehabkost@redhat.com>
> Sent: Saturday, September 19, 2020 6:03 AM
> To: Kang, Luwei <luwei.kang@intel.com>
> Cc: pbonzini@redhat.com; rth@twiddle.net; qemu-devel@nongnu.org; Strong,
> Beeman <beeman.strong@intel.com>; Jiri Denemark
> <jdenemar@redhat.com>; Robert Hoo <robert.hu@linux.intel.com>
> Subject: Re: [PATCH v1 0/3] Remove the limitation of Intel PT CPUID info
> 
> Hi Luwei Kang,
> 
> I was looking for info on intel-pt and just saw this series, and it was never
> reviewed or merged (sorry for missing it!).  Is this still the approach we want to
> follow for intel-pt?
> 
> I'm CCing Jiri Denemark because this might be relevant for a libvirt issue related
> to intel-pt we were investigating[1].
> 
> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1853972
> 
> 
> On Mon, Mar 30, 2020 at 09:56:09AM +0000, Kang, Luwei wrote:
> > > -----Original Message-----
> > > From: Kang, Luwei <luwei.kang@intel.com>
> > > Sent: Tuesday, February 25, 2020 5:38 AM
> > > To: pbonzini@redhat.com; rth@twiddle.net; ehabkost@redhat.com
> > > Cc: qemu-devel@nongnu.org; Strong, Beeman
> <beeman.strong@intel.com>;
> > > Kang, Luwei <luwei.kang@intel.com>
> > > Subject: [PATCH v1 0/3] Remove the limitation of Intel PT CPUID info
> > >
> > > The Intel PT feature includes some
> > > sub-features(CPUID.(EAX=14H,ECX=0H))
> > > and these sub-features are different on different HW platforms. To
> > > make the live migration safety(get the same CPUID info with same cpu
> > > model on different HW platform), the current Intel PT CPUID
> > > information is set to a constant value(from ICELAKE Server).
> > >
> > > It will block the new feature in the later HW platform. what's more,
> > > the support of "IP payloads" will disable the Intel PT in KVM
> > > guest(patch 1) but it will come soon.
> > >
> > > This patchset remove this limitation and expose all the capabilities
> > > to KVM guest. As it will break the live migration safe, Intel PT
> > > will be masked as unmigratable.
> >
> > Ping.
> >
> > Thanks,
> > Luwei Kang
> >
> > >
> > > Luwei Kang (3):
> > >   i386: Remove the limitation of IP payloads for Intel PT
> > >   i386: Remove the CPUID limitation of Intel PT
> > >   i386: Mark the 'INTEL_PT' CPUID bit as unmigratable
> > >
> > >  target/i386/cpu.c | 69
> > > ++++---------------------------------------------------
> > >  1 file changed, 5 insertions(+), 64 deletions(-)
> > >
> > > --
> > > 1.8.3.1
> >
> 
> --
> Eduardo

Re: [PATCH v1 0/3] Remove the limitation of Intel PT CPUID info

[Eduardo Habkost]

On Mon, Sep 21, 2020 at 07:49:22AM +0000, Kang, Luwei wrote:
> Hi Eduardo,
>     This patch set will remove some limitations of Intel PT CPUID information.
>     1. The "IP payloads" feature will disable the Intel PT in guests and it will be coming soon.
>     2. To make the live migration safe, we set the Intel PT CPUID as a constant value(Icelake server CPUID). It will mask off the new feature of Intel PT.

Isn't this series doing the opposite of 2?  It replaces all
constant CPUID values with kvm_arch_get_supported_cpuid(), making
the feature unavailable in migration-safe mode.

Does it mean the plan is to drop intel-pt migration support
entirely?

> 
>     About this issue https://bugzilla.redhat.com/show_bug.cgi?id=1853972, Intel PT is disabled in the guest by default, we should use "-cpu Icelake-Server,+intel-pt" to enable the Intel PT.

That's correct.  The point of the BZ is that libvirt
mode=host-model was expected to include intel-pt automatically
when available.  With this series, the request in the BZ stops
making sense (because intel-pt won't be migration-safe anymore),
but I'm not sure yet that's really the plan.


> 
> Thanks,
> Luwei Kang
> 
> > -----Original Message-----
> > From: Eduardo Habkost <ehabkost@redhat.com>
> > Sent: Saturday, September 19, 2020 6:03 AM
> > To: Kang, Luwei <luwei.kang@intel.com>
> > Cc: pbonzini@redhat.com; rth@twiddle.net; qemu-devel@nongnu.org; Strong,
> > Beeman <beeman.strong@intel.com>; Jiri Denemark
> > <jdenemar@redhat.com>; Robert Hoo <robert.hu@linux.intel.com>
> > Subject: Re: [PATCH v1 0/3] Remove the limitation of Intel PT CPUID info
> > 
> > Hi Luwei Kang,
> > 
> > I was looking for info on intel-pt and just saw this series, and it was never
> > reviewed or merged (sorry for missing it!).  Is this still the approach we want to
> > follow for intel-pt?
> > 
> > I'm CCing Jiri Denemark because this might be relevant for a libvirt issue related
> > to intel-pt we were investigating[1].
> > 
> > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1853972
> > 
> > 
> > On Mon, Mar 30, 2020 at 09:56:09AM +0000, Kang, Luwei wrote:
> > > > -----Original Message-----
> > > > From: Kang, Luwei <luwei.kang@intel.com>
> > > > Sent: Tuesday, February 25, 2020 5:38 AM
> > > > To: pbonzini@redhat.com; rth@twiddle.net; ehabkost@redhat.com
> > > > Cc: qemu-devel@nongnu.org; Strong, Beeman
> > <beeman.strong@intel.com>;
> > > > Kang, Luwei <luwei.kang@intel.com>
> > > > Subject: [PATCH v1 0/3] Remove the limitation of Intel PT CPUID info
> > > >
> > > > The Intel PT feature includes some
> > > > sub-features(CPUID.(EAX=14H,ECX=0H))
> > > > and these sub-features are different on different HW platforms. To
> > > > make the live migration safety(get the same CPUID info with same cpu
> > > > model on different HW platform), the current Intel PT CPUID
> > > > information is set to a constant value(from ICELAKE Server).
> > > >
> > > > It will block the new feature in the later HW platform. what's more,
> > > > the support of "IP payloads" will disable the Intel PT in KVM
> > > > guest(patch 1) but it will come soon.
> > > >
> > > > This patchset remove this limitation and expose all the capabilities
> > > > to KVM guest. As it will break the live migration safe, Intel PT
> > > > will be masked as unmigratable.
> > >
> > > Ping.
> > >
> > > Thanks,
> > > Luwei Kang
> > >
> > > >
> > > > Luwei Kang (3):
> > > >   i386: Remove the limitation of IP payloads for Intel PT
> > > >   i386: Remove the CPUID limitation of Intel PT
> > > >   i386: Mark the 'INTEL_PT' CPUID bit as unmigratable
> > > >
> > > >  target/i386/cpu.c | 69
> > > > ++++---------------------------------------------------
> > > >  1 file changed, 5 insertions(+), 64 deletions(-)
> > > >
> > > > --
> > > > 1.8.3.1
> > >
> > 
> > --
> > Eduardo
> 

-- 
Eduardo

RE: [PATCH v1 0/3] Remove the limitation of Intel PT CPUID info

[Kang, Luwei]

> > Hi Eduardo,
> >     This patch set will remove some limitations of Intel PT CPUID information.
> >     1. The "IP payloads" feature will disable the Intel PT in guests and it will be
> coming soon.
> >     2. To make the live migration safe, we set the Intel PT CPUID as a constant
> value(Icelake server CPUID). It will mask off the new feature of Intel PT.
> 
> Isn't this series doing the opposite of 2?  It replaces all constant CPUID values
> with kvm_arch_get_supported_cpuid(), making the feature unavailable in
> migration-safe mode.

Yes, This series will expose all the HW capabilities to KVM guest if the Intel PT is supported in the guest.

> 
> Does it mean the plan is to drop intel-pt migration support entirely?

I don't want to drop intel-pt live migration feature. As discussed with you before, the Intel PT feature includes some sub-features and may be different on each HW platform. Expose all the capabilities to the guest can't make live migration safe. Do you have any new proposals?

Thanks,
Luwei Kang

> 
> >
> >     About this issue https://bugzilla.redhat.com/show_bug.cgi?id=1853972,
> Intel PT is disabled in the guest by default, we should use "-cpu Icelake-
> Server,+intel-pt" to enable the Intel PT.
> 
> That's correct.  The point of the BZ is that libvirt mode=host-model was
> expected to include intel-pt automatically when available.  With this series, the
> request in the BZ stops making sense (because intel-pt won't be migration-safe
> anymore), but I'm not sure yet that's really the plan.
> 
> 
> >
> > Thanks,
> > Luwei Kang
> >
> > > -----Original Message-----
> > > From: Eduardo Habkost <ehabkost@redhat.com>
> > > Sent: Saturday, September 19, 2020 6:03 AM
> > > To: Kang, Luwei <luwei.kang@intel.com>
> > > Cc: pbonzini@redhat.com; rth@twiddle.net; qemu-devel@nongnu.org;
> > > Strong, Beeman <beeman.strong@intel.com>; Jiri Denemark
> > > <jdenemar@redhat.com>; Robert Hoo <robert.hu@linux.intel.com>
> > > Subject: Re: [PATCH v1 0/3] Remove the limitation of Intel PT CPUID
> > > info
> > >
> > > Hi Luwei Kang,
> > >
> > > I was looking for info on intel-pt and just saw this series, and it
> > > was never reviewed or merged (sorry for missing it!).  Is this still
> > > the approach we want to follow for intel-pt?
> > >
> > > I'm CCing Jiri Denemark because this might be relevant for a libvirt
> > > issue related to intel-pt we were investigating[1].
> > >
> > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1853972
> > >
> > >
> > > On Mon, Mar 30, 2020 at 09:56:09AM +0000, Kang, Luwei wrote:
> > > > > -----Original Message-----
> > > > > From: Kang, Luwei <luwei.kang@intel.com>
> > > > > Sent: Tuesday, February 25, 2020 5:38 AM
> > > > > To: pbonzini@redhat.com; rth@twiddle.net; ehabkost@redhat.com
> > > > > Cc: qemu-devel@nongnu.org; Strong, Beeman
> > > <beeman.strong@intel.com>;
> > > > > Kang, Luwei <luwei.kang@intel.com>
> > > > > Subject: [PATCH v1 0/3] Remove the limitation of Intel PT CPUID
> > > > > info
> > > > >
> > > > > The Intel PT feature includes some
> > > > > sub-features(CPUID.(EAX=14H,ECX=0H))
> > > > > and these sub-features are different on different HW platforms.
> > > > > To make the live migration safety(get the same CPUID info with
> > > > > same cpu model on different HW platform), the current Intel PT
> > > > > CPUID information is set to a constant value(from ICELAKE Server).
> > > > >
> > > > > It will block the new feature in the later HW platform. what's
> > > > > more, the support of "IP payloads" will disable the Intel PT in
> > > > > KVM guest(patch 1) but it will come soon.
> > > > >
> > > > > This patchset remove this limitation and expose all the
> > > > > capabilities to KVM guest. As it will break the live migration
> > > > > safe, Intel PT will be masked as unmigratable.
> > > >
> > > > Ping.
> > > >
> > > > Thanks,
> > > > Luwei Kang
> > > >
> > > > >
> > > > > Luwei Kang (3):
> > > > >   i386: Remove the limitation of IP payloads for Intel PT
> > > > >   i386: Remove the CPUID limitation of Intel PT
> > > > >   i386: Mark the 'INTEL_PT' CPUID bit as unmigratable
> > > > >
> > > > >  target/i386/cpu.c | 69
> > > > > ++++---------------------------------------------------
> > > > >  1 file changed, 5 insertions(+), 64 deletions(-)
> > > > >
> > > > > --
> > > > > 1.8.3.1
> > > >
> > >
> > > --
> > > Eduardo
> >
> 
> --
> Eduardo

Re: [PATCH v1 0/3] Remove the limitation of Intel PT CPUID info

[Eduardo Habkost]

On Wed, Sep 23, 2020 at 02:52:50AM +0000, Kang, Luwei wrote:
> > > Hi Eduardo,
> > >     This patch set will remove some limitations of Intel PT CPUID information.
> > >     1. The "IP payloads" feature will disable the Intel PT in guests and it will be
> > coming soon.
> > >     2. To make the live migration safe, we set the Intel PT CPUID as a constant
> > value(Icelake server CPUID). It will mask off the new feature of Intel PT.
> > 
> > Isn't this series doing the opposite of 2?  It replaces all constant CPUID values
> > with kvm_arch_get_supported_cpuid(), making the feature unavailable in
> > migration-safe mode.
> 
> Yes, This series will expose all the HW capabilities to KVM
> guest if the Intel PT is supported in the guest.
> 
> > 
> > Does it mean the plan is to drop intel-pt migration support entirely?
> 
> I don't want to drop intel-pt live migration feature. As
> discussed with you before, the Intel PT feature includes some
> sub-features and may be different on each HW platform. Expose
> all the capabilities to the guest can't make live migration
> safe. Do you have any new proposals?

To support live migration, we need the set of features seen by
the guest be determined only by the input given to QEMU, not host
capabilities.  It can be via:
(1) explicit "-cpu ...,+feat,feat=..." flags;
(2) through data in the CPU model table; or
(3) by hardcoding the same value for all configurations.

The current solution is (3).  (2) is probably the best solution,
with the assumption that the host can always emulate features
from an older CPU in a newer CPU.  If there are features that
can't be emulated if migrating to a newer CPU, a more explicit
configuration mechanism (1) might be better, because not being
able to migrate a VM to newer hardware is inconvenient.

None of those approaches prevent us from implementing passthrough
mode for "-cpu host".  Wouldn't that be preferred instead of
removing support for live migration?

> 
> Thanks,
> Luwei Kang
> 
> > 
> > >
> > >     About this issue https://bugzilla.redhat.com/show_bug.cgi?id=1853972,
> > Intel PT is disabled in the guest by default, we should use "-cpu Icelake-
> > Server,+intel-pt" to enable the Intel PT.
> > 
> > That's correct.  The point of the BZ is that libvirt mode=host-model was
> > expected to include intel-pt automatically when available.  With this series, the
> > request in the BZ stops making sense (because intel-pt won't be migration-safe
> > anymore), but I'm not sure yet that's really the plan.
> > 
> > 
> > >
> > > Thanks,
> > > Luwei Kang
> > >
> > > > -----Original Message-----
> > > > From: Eduardo Habkost <ehabkost@redhat.com>
> > > > Sent: Saturday, September 19, 2020 6:03 AM
> > > > To: Kang, Luwei <luwei.kang@intel.com>
> > > > Cc: pbonzini@redhat.com; rth@twiddle.net; qemu-devel@nongnu.org;
> > > > Strong, Beeman <beeman.strong@intel.com>; Jiri Denemark
> > > > <jdenemar@redhat.com>; Robert Hoo <robert.hu@linux.intel.com>
> > > > Subject: Re: [PATCH v1 0/3] Remove the limitation of Intel PT CPUID
> > > > info
> > > >
> > > > Hi Luwei Kang,
> > > >
> > > > I was looking for info on intel-pt and just saw this series, and it
> > > > was never reviewed or merged (sorry for missing it!).  Is this still
> > > > the approach we want to follow for intel-pt?
> > > >
> > > > I'm CCing Jiri Denemark because this might be relevant for a libvirt
> > > > issue related to intel-pt we were investigating[1].
> > > >
> > > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1853972
> > > >
> > > >
> > > > On Mon, Mar 30, 2020 at 09:56:09AM +0000, Kang, Luwei wrote:
> > > > > > -----Original Message-----
> > > > > > From: Kang, Luwei <luwei.kang@intel.com>
> > > > > > Sent: Tuesday, February 25, 2020 5:38 AM
> > > > > > To: pbonzini@redhat.com; rth@twiddle.net; ehabkost@redhat.com
> > > > > > Cc: qemu-devel@nongnu.org; Strong, Beeman
> > > > <beeman.strong@intel.com>;
> > > > > > Kang, Luwei <luwei.kang@intel.com>
> > > > > > Subject: [PATCH v1 0/3] Remove the limitation of Intel PT CPUID
> > > > > > info
> > > > > >
> > > > > > The Intel PT feature includes some
> > > > > > sub-features(CPUID.(EAX=14H,ECX=0H))
> > > > > > and these sub-features are different on different HW platforms.
> > > > > > To make the live migration safety(get the same CPUID info with
> > > > > > same cpu model on different HW platform), the current Intel PT
> > > > > > CPUID information is set to a constant value(from ICELAKE Server).
> > > > > >
> > > > > > It will block the new feature in the later HW platform. what's
> > > > > > more, the support of "IP payloads" will disable the Intel PT in
> > > > > > KVM guest(patch 1) but it will come soon.
> > > > > >
> > > > > > This patchset remove this limitation and expose all the
> > > > > > capabilities to KVM guest. As it will break the live migration
> > > > > > safe, Intel PT will be masked as unmigratable.
> > > > >
> > > > > Ping.
> > > > >
> > > > > Thanks,
> > > > > Luwei Kang
> > > > >
> > > > > >
> > > > > > Luwei Kang (3):
> > > > > >   i386: Remove the limitation of IP payloads for Intel PT
> > > > > >   i386: Remove the CPUID limitation of Intel PT
> > > > > >   i386: Mark the 'INTEL_PT' CPUID bit as unmigratable
> > > > > >
> > > > > >  target/i386/cpu.c | 69
> > > > > > ++++---------------------------------------------------
> > > > > >  1 file changed, 5 insertions(+), 64 deletions(-)
> > > > > >
> > > > > > --
> > > > > > 1.8.3.1
> > > > >
> > > >
> > > > --
> > > > Eduardo
> > >
> > 
> > --
> > Eduardo
> 

-- 
Eduardo

RE: [PATCH v1 0/3] Remove the limitation of Intel PT CPUID info

[Kang, Luwei]

> > > > Hi Eduardo,
> > > >     This patch set will remove some limitations of Intel PT CPUID
> information.
> > > >     1. The "IP payloads" feature will disable the Intel PT in
> > > > guests and it will be
> > > coming soon.
> > > >     2. To make the live migration safe, we set the Intel PT CPUID
> > > > as a constant
> > > value(Icelake server CPUID). It will mask off the new feature of Intel PT.
> > >
> > > Isn't this series doing the opposite of 2?  It replaces all constant
> > > CPUID values with kvm_arch_get_supported_cpuid(), making the feature
> > > unavailable in migration-safe mode.
> >
> > Yes, This series will expose all the HW capabilities to KVM guest if
> > the Intel PT is supported in the guest.
> >
> > >
> > > Does it mean the plan is to drop intel-pt migration support entirely?
> >
> > I don't want to drop intel-pt live migration feature. As discussed
> > with you before, the Intel PT feature includes some sub-features and
> > may be different on each HW platform. Expose all the capabilities to
> > the guest can't make live migration safe. Do you have any new
> > proposals?
> 
> To support live migration, we need the set of features seen by the guest be
> determined only by the input given to QEMU, not host capabilities.  It can be
> via:
> (1) explicit "-cpu ...,+feat,feat=..." flags;
> (2) through data in the CPU model table; or
> (3) by hardcoding the same value for all configurations.
> 
> The current solution is (3).  (2) is probably the best solution, with the
> assumption that the host can always emulate features from an older CPU in a
> newer CPU.  If there are features that can't be emulated if migrating to a
> newer CPU, a more explicit configuration mechanism (1) might be better,
> because not being able to migrate a VM to newer hardware is inconvenient.
> 
> None of those approaches prevent us from implementing passthrough mode
> for "-cpu host".  Wouldn't that be preferred instead of removing support for
> live migration?

Thanks for the comments and suggestions. I think the newer CPU includes all the features of the older CPU, but no document have such statement. To make sure it can work in all the cases, the solution (1) might be better.
The Intel PT virtualization first supported on Icelake and we can use this CPUID as basic CPUID information. Any new feature which supports on the newer CPUs can be added by "-cpu ...,+feat,feat=...". What is your opinion?

Thanks,
Luwei Kang

Re: [PATCH v1 0/3] Remove the limitation of Intel PT CPUID info

[Eduardo Habkost]

On Thu, Sep 24, 2020 at 12:47:04PM +0000, Kang, Luwei wrote:
> > > > > Hi Eduardo,
> > > > >     This patch set will remove some limitations of Intel PT CPUID
> > information.
> > > > >     1. The "IP payloads" feature will disable the Intel PT in
> > > > > guests and it will be
> > > > coming soon.
> > > > >     2. To make the live migration safe, we set the Intel PT CPUID
> > > > > as a constant
> > > > value(Icelake server CPUID). It will mask off the new feature of Intel PT.
> > > >
> > > > Isn't this series doing the opposite of 2?  It replaces all constant
> > > > CPUID values with kvm_arch_get_supported_cpuid(), making the feature
> > > > unavailable in migration-safe mode.
> > >
> > > Yes, This series will expose all the HW capabilities to KVM guest if
> > > the Intel PT is supported in the guest.
> > >
> > > >
> > > > Does it mean the plan is to drop intel-pt migration support entirely?
> > >
> > > I don't want to drop intel-pt live migration feature. As discussed
> > > with you before, the Intel PT feature includes some sub-features and
> > > may be different on each HW platform. Expose all the capabilities to
> > > the guest can't make live migration safe. Do you have any new
> > > proposals?
> > 
> > To support live migration, we need the set of features seen by the guest be
> > determined only by the input given to QEMU, not host capabilities.  It can be
> > via:
> > (1) explicit "-cpu ...,+feat,feat=..." flags;
> > (2) through data in the CPU model table; or
> > (3) by hardcoding the same value for all configurations.
> > 
> > The current solution is (3).  (2) is probably the best solution, with the
> > assumption that the host can always emulate features from an older CPU in a
> > newer CPU.  If there are features that can't be emulated if migrating to a
> > newer CPU, a more explicit configuration mechanism (1) might be better,
> > because not being able to migrate a VM to newer hardware is inconvenient.
> > 
> > None of those approaches prevent us from implementing passthrough mode
> > for "-cpu host".  Wouldn't that be preferred instead of removing support for
> > live migration?
> 
> Thanks for the comments and suggestions. I think the newer CPU
> includes all the features of the older CPU, but no document
> have such statement. To make sure it can work in all the cases,
> the solution (1) might be better.

Maybe (2) is still viable, as long as there are no expectations
that features will be removed in future hardware.  Compatibility
with future hardware in (2) is more about convenience, not a hard
a requirement.

In either case, (1) is the building block for making (2) work.
This means we can start by implementing (1), and see if we can
include features in new CPU models later.

> The Intel PT virtualization first supported on Icelake and we
> can use this CPUID as basic CPUID information. Any new feature
> which supports on the newer CPUs can be added by "-cpu
> ...,+feat,feat=...". What is your opinion?

Sounds good to me.  I would also make intel-pt passthrough work
if using "-cpu host" and/or an explicit "intel-pt-passthrough=on"
option.

-- 
Eduardo

RE: [PATCH v1 0/3] Remove the limitation of Intel PT CPUID info

[Kang, Luwei]

> > > > > > Hi Eduardo,
> > > > > >     This patch set will remove some limitations of Intel PT
> > > > > > CPUID
> > > information.
> > > > > >     1. The "IP payloads" feature will disable the Intel PT in
> > > > > > guests and it will be
> > > > > coming soon.
> > > > > >     2. To make the live migration safe, we set the Intel PT
> > > > > > CPUID as a constant
> > > > > value(Icelake server CPUID). It will mask off the new feature of Intel PT.
> > > > >
> > > > > Isn't this series doing the opposite of 2?  It replaces all
> > > > > constant CPUID values with kvm_arch_get_supported_cpuid(),
> > > > > making the feature unavailable in migration-safe mode.
> > > >
> > > > Yes, This series will expose all the HW capabilities to KVM guest
> > > > if the Intel PT is supported in the guest.
> > > >
> > > > >
> > > > > Does it mean the plan is to drop intel-pt migration support entirely?
> > > >
> > > > I don't want to drop intel-pt live migration feature. As discussed
> > > > with you before, the Intel PT feature includes some sub-features
> > > > and may be different on each HW platform. Expose all the
> > > > capabilities to the guest can't make live migration safe. Do you
> > > > have any new proposals?
> > >
> > > To support live migration, we need the set of features seen by the
> > > guest be determined only by the input given to QEMU, not host
> > > capabilities.  It can be
> > > via:
> > > (1) explicit "-cpu ...,+feat,feat=..." flags;
> > > (2) through data in the CPU model table; or
> > > (3) by hardcoding the same value for all configurations.
> > >
> > > The current solution is (3).  (2) is probably the best solution,
> > > with the assumption that the host can always emulate features from
> > > an older CPU in a newer CPU.  If there are features that can't be
> > > emulated if migrating to a newer CPU, a more explicit configuration
> > > mechanism (1) might be better, because not being able to migrate a VM to
> newer hardware is inconvenient.
> > >
> > > None of those approaches prevent us from implementing passthrough
> > > mode for "-cpu host".  Wouldn't that be preferred instead of
> > > removing support for live migration?
> >
> > Thanks for the comments and suggestions. I think the newer CPU
> > includes all the features of the older CPU, but no document have such
> > statement. To make sure it can work in all the cases, the solution (1)
> > might be better.
> 
> Maybe (2) is still viable, as long as there are no expectations that features will
> be removed in future hardware.  Compatibility with future hardware in (2) is
> more about convenience, not a hard a requirement.
> 
> In either case, (1) is the building block for making (2) work.
> This means we can start by implementing (1), and see if we can include
> features in new CPU models later.

OK, I will add the new sub-feature of Intel PT base on the solution (1).

Regarding the limitation of IP payload, may I remove this limitation directly because LIP == RIP for most/all real code, and it will be set after ICX CPUs.
[PATCH v1 1/3] i386: Remove the limitation of IP payloads for Intel PT

Thanks,
Luwei Kang

> 
> > The Intel PT virtualization first supported on Icelake and we can use
> > this CPUID as basic CPUID information. Any new feature which supports
> > on the newer CPUs can be added by "-cpu ...,+feat,feat=...". What is
> > your opinion?
> 
> Sounds good to me.  I would also make intel-pt passthrough work if using "-cpu
> host" and/or an explicit "intel-pt-passthrough=on"
> option.
> 
> --
> Eduardo

Re: [PATCH v1 1/3] i386: Remove the limitation of IP payloads for Intel PT

[Eduardo Habkost]

On Tue, Feb 25, 2020 at 05:38:30AM +0800, Luwei Kang wrote:
> The Intel PT packets which contain IP payloads will have LIP values, and it
> will include the CS base component if the CPUID.(EAX=14H,ECX=0H).ECX.[bit31]
> is set. But it will disabled the Intel PT in kvm guest because of the need
> of live migration safe(c078ca9 i386: Disable Intel PT if packets IP payloads
> have LIP values).
> 
> This patch will revert the previous limitation because the Intel new hardware
> will set this bit and LIP == RIP for most/all real code.

"works most of the time" might be good enough if it's a conscious
user choice, but not for something we might be enabling by
default.  Under which conditions this wouldn't work?  Can we
detect those conditions somehow?

To allow live migration between LIP=0 and LIP=1 hosts, we need
KVM to be able to properly emulate LIP=0 on LIP=1 hosts.  Does
the hardware make this possible?

If KVM can't emulate LIP=0 on a LIP=1 host, what you can do here
is to make the flag configurable and check if the configured
value matches the one in the host.  This way we can support both
types of hosts, just not allow live migration between them.


> 
> Signed-off-by: Luwei Kang <luwei.kang@intel.com>
> ---
>  target/i386/cpu.c | 5 +----
>  1 file changed, 1 insertion(+), 4 deletions(-)
> 
> diff --git a/target/i386/cpu.c b/target/i386/cpu.c
> index 69f518a..8c0d1e4 100644
> --- a/target/i386/cpu.c
> +++ b/target/i386/cpu.c
> @@ -688,8 +688,6 @@ static CPUCacheInfo legacy_l3_cache = {
>   * bit[02]: Support Single-Range Output scheme;
>   */
>  #define INTEL_PT_MINIMAL_ECX     0x7
> -/* generated packets which contain IP payloads have LIP values */
> -#define INTEL_PT_IP_LIP          (1 << 31)
>  #define INTEL_PT_ADDR_RANGES_NUM 0x2 /* Number of configurable address ranges */
>  #define INTEL_PT_ADDR_RANGES_NUM_MASK 0x3
>  #define INTEL_PT_MTC_BITMAP      (0x0249 << 16) /* Support ART(0,3,6,9) */
> @@ -6281,8 +6279,7 @@ static void x86_cpu_filter_features(X86CPU *cpu, bool verbose)
>             ((eax_1 & INTEL_PT_ADDR_RANGES_NUM_MASK) <
>                                             INTEL_PT_ADDR_RANGES_NUM) ||
>             ((ebx_1 & (INTEL_PT_PSB_BITMAP | INTEL_PT_CYCLE_BITMAP)) !=
> -                (INTEL_PT_PSB_BITMAP | INTEL_PT_CYCLE_BITMAP)) ||
> -           (ecx_0 & INTEL_PT_IP_LIP)) {
> +                (INTEL_PT_PSB_BITMAP | INTEL_PT_CYCLE_BITMAP))) {
>              /*
>               * Processor Trace capabilities aren't configurable, so if the
>               * host can't emulate the capabilities we report on
> -- 
> 1.8.3.1
> 

-- 
Eduardo

RE: [PATCH v1 1/3] i386: Remove the limitation of IP payloads for Intel PT

[Strong, Beeman]

LIP=0 will differ from LIP=1 behavior only when CSbase is non-zero, which requires 32-bit code.  In that case a LIP=0 implementation will provide only the EIP offset from CSbase in IP packets (like TIP or FUP), while LIP=1 implementation will provide the full LIP (CSbase + EIP offset).

-----Original Message-----
From: Eduardo Habkost <ehabkost@redhat.com> 
Sent: Friday, September 25, 2020 9:16 AM
To: Kang, Luwei <luwei.kang@intel.com>
Cc: pbonzini@redhat.com; rth@twiddle.net; qemu-devel@nongnu.org; Strong, Beeman <beeman.strong@intel.com>
Subject: Re: [PATCH v1 1/3] i386: Remove the limitation of IP payloads for Intel PT

On Tue, Feb 25, 2020 at 05:38:30AM +0800, Luwei Kang wrote:
> The Intel PT packets which contain IP payloads will have LIP values, 
> and it will include the CS base component if the 
> CPUID.(EAX=14H,ECX=0H).ECX.[bit31]
> is set. But it will disabled the Intel PT in kvm guest because of the 
> need of live migration safe(c078ca9 i386: Disable Intel PT if packets 
> IP payloads have LIP values).
> 
> This patch will revert the previous limitation because the Intel new 
> hardware will set this bit and LIP == RIP for most/all real code.

"works most of the time" might be good enough if it's a conscious user choice, but not for something we might be enabling by default.  Under which conditions this wouldn't work?  Can we detect those conditions somehow?

To allow live migration between LIP=0 and LIP=1 hosts, we need KVM to be able to properly emulate LIP=0 on LIP=1 hosts.  Does the hardware make this possible?

If KVM can't emulate LIP=0 on a LIP=1 host, what you can do here is to make the flag configurable and check if the configured value matches the one in the host.  This way we can support both types of hosts, just not allow live migration between them.


> 
> Signed-off-by: Luwei Kang <luwei.kang@intel.com>
> ---
>  target/i386/cpu.c | 5 +----
>  1 file changed, 1 insertion(+), 4 deletions(-)
> 
> diff --git a/target/i386/cpu.c b/target/i386/cpu.c index 
> 69f518a..8c0d1e4 100644
> --- a/target/i386/cpu.c
> +++ b/target/i386/cpu.c
> @@ -688,8 +688,6 @@ static CPUCacheInfo legacy_l3_cache = {
>   * bit[02]: Support Single-Range Output scheme;
>   */
>  #define INTEL_PT_MINIMAL_ECX     0x7
> -/* generated packets which contain IP payloads have LIP values */
> -#define INTEL_PT_IP_LIP          (1 << 31)
>  #define INTEL_PT_ADDR_RANGES_NUM 0x2 /* Number of configurable 
> address ranges */  #define INTEL_PT_ADDR_RANGES_NUM_MASK 0x3
>  #define INTEL_PT_MTC_BITMAP      (0x0249 << 16) /* Support ART(0,3,6,9) */
> @@ -6281,8 +6279,7 @@ static void x86_cpu_filter_features(X86CPU *cpu, bool verbose)
>             ((eax_1 & INTEL_PT_ADDR_RANGES_NUM_MASK) <
>                                             INTEL_PT_ADDR_RANGES_NUM) ||
>             ((ebx_1 & (INTEL_PT_PSB_BITMAP | INTEL_PT_CYCLE_BITMAP)) !=
> -                (INTEL_PT_PSB_BITMAP | INTEL_PT_CYCLE_BITMAP)) ||
> -           (ecx_0 & INTEL_PT_IP_LIP)) {
> +                (INTEL_PT_PSB_BITMAP | INTEL_PT_CYCLE_BITMAP))) {
>              /*
>               * Processor Trace capabilities aren't configurable, so if the
>               * host can't emulate the capabilities we report on
> --
> 1.8.3.1
> 

-- 
Eduardo

Re: [PATCH v1 1/3] i386: Remove the limitation of IP payloads for Intel PT

[Eduardo Habkost]

On Fri, Sep 25, 2020 at 04:42:26PM +0000, Strong, Beeman wrote:
> LIP=0 will differ from LIP=1 behavior only when CSbase is non-zero, which requires 32-bit code.  In that case a LIP=0 implementation will provide only the EIP offset from CSbase in IP packets (like TIP or FUP), while LIP=1 implementation will provide the full LIP (CSbase + EIP offset).
> 

Thanks.  Is it possible to make KVM emulate LIP=0 behavior
correctly on LIP=1 hosts, or vice versa?


> -----Original Message-----
> From: Eduardo Habkost <ehabkost@redhat.com> 
> Sent: Friday, September 25, 2020 9:16 AM
> To: Kang, Luwei <luwei.kang@intel.com>
> Cc: pbonzini@redhat.com; rth@twiddle.net; qemu-devel@nongnu.org; Strong, Beeman <beeman.strong@intel.com>
> Subject: Re: [PATCH v1 1/3] i386: Remove the limitation of IP payloads for Intel PT
> 
> On Tue, Feb 25, 2020 at 05:38:30AM +0800, Luwei Kang wrote:
> > The Intel PT packets which contain IP payloads will have LIP values, 
> > and it will include the CS base component if the 
> > CPUID.(EAX=14H,ECX=0H).ECX.[bit31]
> > is set. But it will disabled the Intel PT in kvm guest because of the 
> > need of live migration safe(c078ca9 i386: Disable Intel PT if packets 
> > IP payloads have LIP values).
> > 
> > This patch will revert the previous limitation because the Intel new 
> > hardware will set this bit and LIP == RIP for most/all real code.
> 
> "works most of the time" might be good enough if it's a conscious user choice, but not for something we might be enabling by default.  Under which conditions this wouldn't work?  Can we detect those conditions somehow?
> 
> To allow live migration between LIP=0 and LIP=1 hosts, we need KVM to be able to properly emulate LIP=0 on LIP=1 hosts.  Does the hardware make this possible?
> 
> If KVM can't emulate LIP=0 on a LIP=1 host, what you can do here is to make the flag configurable and check if the configured value matches the one in the host.  This way we can support both types of hosts, just not allow live migration between them.
> 
> 
> > 
> > Signed-off-by: Luwei Kang <luwei.kang@intel.com>
> > ---
> >  target/i386/cpu.c | 5 +----
> >  1 file changed, 1 insertion(+), 4 deletions(-)
> > 
> > diff --git a/target/i386/cpu.c b/target/i386/cpu.c index 
> > 69f518a..8c0d1e4 100644
> > --- a/target/i386/cpu.c
> > +++ b/target/i386/cpu.c
> > @@ -688,8 +688,6 @@ static CPUCacheInfo legacy_l3_cache = {
> >   * bit[02]: Support Single-Range Output scheme;
> >   */
> >  #define INTEL_PT_MINIMAL_ECX     0x7
> > -/* generated packets which contain IP payloads have LIP values */
> > -#define INTEL_PT_IP_LIP          (1 << 31)
> >  #define INTEL_PT_ADDR_RANGES_NUM 0x2 /* Number of configurable 
> > address ranges */  #define INTEL_PT_ADDR_RANGES_NUM_MASK 0x3
> >  #define INTEL_PT_MTC_BITMAP      (0x0249 << 16) /* Support ART(0,3,6,9) */
> > @@ -6281,8 +6279,7 @@ static void x86_cpu_filter_features(X86CPU *cpu, bool verbose)
> >             ((eax_1 & INTEL_PT_ADDR_RANGES_NUM_MASK) <
> >                                             INTEL_PT_ADDR_RANGES_NUM) ||
> >             ((ebx_1 & (INTEL_PT_PSB_BITMAP | INTEL_PT_CYCLE_BITMAP)) !=
> > -                (INTEL_PT_PSB_BITMAP | INTEL_PT_CYCLE_BITMAP)) ||
> > -           (ecx_0 & INTEL_PT_IP_LIP)) {
> > +                (INTEL_PT_PSB_BITMAP | INTEL_PT_CYCLE_BITMAP))) {
> >              /*
> >               * Processor Trace capabilities aren't configurable, so if the
> >               * host can't emulate the capabilities we report on
> > --
> > 1.8.3.1
> > 
> 
> -- 
> Eduardo
> 

-- 
Eduardo

Re: [PATCH v1 1/3] i386: Remove the limitation of IP payloads for Intel PT

[Paolo Bonzini]

On 25/09/20 18:54, Eduardo Habkost wrote:
> On Fri, Sep 25, 2020 at 04:42:26PM +0000, Strong, Beeman wrote:
>> LIP=0 will differ from LIP=1 behavior only when CSbase is non-zero, which requires 32-bit code.  In that case a LIP=0 implementation will provide only the EIP offset from CSbase in IP packets (like TIP or FUP), while LIP=1 implementation will provide the full LIP (CSbase + EIP offset).
>>
> Thanks.  Is it possible to make KVM emulate LIP=0 behavior
> correctly on LIP=1 hosts, or vice versa?

No, it's not possible.  KVM doesn't have a say on what the processor
writes in the tracing packets.

Paolo

Re: [PATCH v1 1/3] i386: Remove the limitation of IP payloads for Intel PT

[Eduardo Habkost]

On Fri, Sep 25, 2020 at 10:23:49PM +0200, Paolo Bonzini wrote:
> On 25/09/20 18:54, Eduardo Habkost wrote:
> > On Fri, Sep 25, 2020 at 04:42:26PM +0000, Strong, Beeman wrote:
> >> LIP=0 will differ from LIP=1 behavior only when CSbase is non-zero, which requires 32-bit code.  In that case a LIP=0 implementation will provide only the EIP offset from CSbase in IP packets (like TIP or FUP), while LIP=1 implementation will provide the full LIP (CSbase + EIP offset).
> >>
> > Thanks.  Is it possible to make KVM emulate LIP=0 behavior
> > correctly on LIP=1 hosts, or vice versa?
> 
> No, it's not possible.  KVM doesn't have a say on what the processor
> writes in the tracing packets.

Can KVM refuse to enable packet generation if CSbase is not zero
and CPUID.(EAX=14H,ECX=0)[bit 31] seen by guest is different from
host?

-- 
Eduardo

Re: [PATCH v1 1/3] i386: Remove the limitation of IP payloads for Intel PT

[Paolo Bonzini]

On 25/09/20 22:29, Eduardo Habkost wrote:
>> No, it's not possible.  KVM doesn't have a say on what the processor
>> writes in the tracing packets.
> Can KVM refuse to enable packet generation if CSbase is not zero
> and CPUID.(EAX=14H,ECX=0)[bit 31] seen by guest is different from
> host?

Yes, but the processor could change operating mode (and hence CSbase)
while tracing is active.  This is very unlikely, since it would require
nonzero CS-base and a 32-bit host, but in principle not impossible
(could be a firmware call, for example).

The only solution is for KVM to accept both, and for QEMU to refuse a
setting that does not match the host.

Paolo

RE: [PATCH v1 1/3] i386: Remove the limitation of IP payloads for Intel PT

[Kang, Luwei]

> >> No, it's not possible.  KVM doesn't have a say on what the processor
> >> writes in the tracing packets.
> > Can KVM refuse to enable packet generation if CSbase is not zero and
> > CPUID.(EAX=14H,ECX=0)[bit 31] seen by guest is different from host?
> 
> Yes, but the processor could change operating mode (and hence CSbase) while
> tracing is active.  This is very unlikely, since it would require nonzero CS-base
> and a 32-bit host, but in principle not impossible (could be a firmware call, for
> example).
> 
> The only solution is for KVM to accept both, and for QEMU to refuse a setting
> that does not match the host.
> 

So I need to add a patch in KVM to disabled the Intel PT when the CSbase is not zero and the guest LIP different from the host. And this limitation in qemu (disabled the PT when LIP is enabled in the host) can be remove. Is that right?

Thanks,
Luwei Kang 

Re: [PATCH v1 1/3] i386: Remove the limitation of IP payloads for Intel PT

[Paolo Bonzini]

On 28/09/20 07:19, Kang, Luwei wrote:
>>>> No, it's not possible.  KVM doesn't have a say on what the processor
>>>> writes in the tracing packets.
>>> Can KVM refuse to enable packet generation if CSbase is not zero and
>>> CPUID.(EAX=14H,ECX=0)[bit 31] seen by guest is different from host?
>>
>> Yes, but the processor could change operating mode (and hence CSbase) while
>> tracing is active.  This is very unlikely, since it would require nonzero CS-base
>> and a 32-bit host, but in principle not impossible (could be a firmware call, for
>> example).
>>
>> The only solution is for KVM to accept both, and for QEMU to refuse a setting
>> that does not match the host.
>>
> 
> So I need to add a patch in KVM to disabled the Intel PT when the
> CSbase is not zero and the guest LIP different from the host. And this
> limitation in qemu (disabled the PT when LIP is enabled in the host) can
> be remove. Is that right?

No, if a feature cannot be emulated, that means it cannot be enabled
unless it matches the host.  That's generally not a problem since Intel
PT is usually used only with "-cpu host".

Paolo

RE: [PATCH v1 1/3] i386: Remove the limitation of IP payloads for Intel PT

[Kang, Luwei]

> >>>> No, it's not possible.  KVM doesn't have a say on what the
> >>>> processor writes in the tracing packets.
> >>> Can KVM refuse to enable packet generation if CSbase is not zero and
> >>> CPUID.(EAX=14H,ECX=0)[bit 31] seen by guest is different from host?
> >>
> >> Yes, but the processor could change operating mode (and hence CSbase)
> >> while tracing is active.  This is very unlikely, since it would
> >> require nonzero CS-base and a 32-bit host, but in principle not
> >> impossible (could be a firmware call, for example).
> >>
> >> The only solution is for KVM to accept both, and for QEMU to refuse a
> >> setting that does not match the host.
> >>
> >
> > So I need to add a patch in KVM to disabled the Intel PT when the
> > CSbase is not zero and the guest LIP different from the host. And this
> > limitation in qemu (disabled the PT when LIP is enabled in the host)
> > can be remove. Is that right?
> 
> No, if a feature cannot be emulated, that means it cannot be enabled unless it
> matches the host.  That's generally not a problem since Intel PT is usually used
> only with "-cpu host".
> 

The limitation of LIP in qemu will mask off the Intel PT in KVM guest even with "-cpu host". e.g. This bit will be set in SnowRidge HW and later.

How about "-cpu cpu_model, +intel-pt" use case? The current value of Intel PT CPUID is a constant. Can we make the ICX CPUID as basic inforation(LIP is 0) and using "+intel-pt-lip" to make Intel PT work on the CPU which LIP is 1 on the host? As you mentioned before, Intel PT cannot be enabled in guest unless it matches the host.

Thanks,
Luwei Kang

Re: [PATCH v1 1/3] i386: Remove the limitation of IP payloads for Intel PT

[Eduardo Habkost]

On Mon, Sep 28, 2020 at 12:42:37PM +0000, Kang, Luwei wrote:
> > >>>> No, it's not possible.  KVM doesn't have a say on what the
> > >>>> processor writes in the tracing packets.
> > >>> Can KVM refuse to enable packet generation if CSbase is not zero and
> > >>> CPUID.(EAX=14H,ECX=0)[bit 31] seen by guest is different from host?
> > >>
> > >> Yes, but the processor could change operating mode (and hence CSbase)
> > >> while tracing is active.  This is very unlikely, since it would
> > >> require nonzero CS-base and a 32-bit host, but in principle not
> > >> impossible (could be a firmware call, for example).
> > >>
> > >> The only solution is for KVM to accept both, and for QEMU to refuse a
> > >> setting that does not match the host.
> > >>
> > >
> > > So I need to add a patch in KVM to disabled the Intel PT when the
> > > CSbase is not zero and the guest LIP different from the host. And this
> > > limitation in qemu (disabled the PT when LIP is enabled in the host)
> > > can be remove. Is that right?
> > 
> > No, if a feature cannot be emulated, that means it cannot be enabled unless it
> > matches the host.  That's generally not a problem since Intel PT is usually used
> > only with "-cpu host".
> > 
> 
> The limitation of LIP in qemu will mask off the Intel PT in KVM
> guest even with "-cpu host". e.g. This bit will be set in
> SnowRidge HW and later.

This behavior can and should be changed.

> 
> How about "-cpu cpu_model, +intel-pt" use case? The current
> value of Intel PT CPUID is a constant. Can we make the ICX
> CPUID as basic inforation(LIP is 0) and using "+intel-pt-lip"
> to make Intel PT work on the CPU which LIP is 1 on the host? As
> you mentioned before, Intel PT cannot be enabled in guest
> unless it matches the host.

This makes sense, but you can also make each CPU model set a
default for the LIP bit.  "-cpu SnowRidge,+intel-pt" could set
LIP=1 by default.

-- 
Eduardo

Re: [PATCH v1 1/3] i386: Remove the limitation of IP payloads for Intel PT

[Paolo Bonzini]

On 28/09/20 14:42, Kang, Luwei wrote:
>> No, if a feature cannot be emulated, that means it cannot be
>> enabled unless it matches the host.  That's generally not a problem
>> since Intel PT is usually used only with "-cpu host".
>> 
> The limitation of LIP in qemu will mask off the Intel PT in KVM guest
> even with "-cpu host". e.g. This bit will be set in SnowRidge HW and
> later.

I agree that QEMU would have to learn about LIP.  Unlike this patch,
however, x86_cpu_filter_features would have to fail if host LIP != guest
LIP.  That is, something like

           (ecx_0 & INTEL_PT_IP_LIP)) !=
		(env->features[INTEL_PT_ECX_0] & INTEL_PT_IP_LIP)

where "intel-pt-lip" would be a feature in env->features[INTEL_PT_ECX_0].

> How about "-cpu cpu_model, +intel-pt" use case? The current value of
> Intel PT CPUID is a constant. Can we make the ICX CPUID as basic
> inforation(LIP is 0) and using "+intel-pt-lip" to make Intel PT work
> on the CPU which LIP is 1 on the host? As you mentioned before, Intel
> PT cannot be enabled in guest unless it matches the host.

Yes, this would work.

Paolo

RE: [PATCH v1 1/3] i386: Remove the limitation of IP payloads for Intel PT

[Kang, Luwei]

> > > >>>> No, it's not possible.  KVM doesn't have a say on what the
> > > >>>> processor writes in the tracing packets.
> > > >>> Can KVM refuse to enable packet generation if CSbase is not zero
> > > >>> and CPUID.(EAX=14H,ECX=0)[bit 31] seen by guest is different from
> host?
> > > >>
> > > >> Yes, but the processor could change operating mode (and hence
> > > >> CSbase) while tracing is active.  This is very unlikely, since it
> > > >> would require nonzero CS-base and a 32-bit host, but in principle
> > > >> not impossible (could be a firmware call, for example).
> > > >>
> > > >> The only solution is for KVM to accept both, and for QEMU to
> > > >> refuse a setting that does not match the host.
> > > >>
> > > >
> > > > So I need to add a patch in KVM to disabled the Intel PT when the
> > > > CSbase is not zero and the guest LIP different from the host. And
> > > > this limitation in qemu (disabled the PT when LIP is enabled in
> > > > the host) can be remove. Is that right?
> > >
> > > No, if a feature cannot be emulated, that means it cannot be enabled
> > > unless it matches the host.  That's generally not a problem since
> > > Intel PT is usually used only with "-cpu host".
> > >
> >
> > The limitation of LIP in qemu will mask off the Intel PT in KVM guest
> > even with "-cpu host". e.g. This bit will be set in SnowRidge HW and
> > later.
> 
> This behavior can and should be changed.
> 
> >
> > How about "-cpu cpu_model, +intel-pt" use case? The current value of
> > Intel PT CPUID is a constant. Can we make the ICX CPUID as basic
> > inforation(LIP is 0) and using "+intel-pt-lip"
> > to make Intel PT work on the CPU which LIP is 1 on the host? As you
> > mentioned before, Intel PT cannot be enabled in guest unless it
> > matches the host.
> 
> This makes sense, but you can also make each CPU model set a default for the
> LIP bit.  "-cpu SnowRidge,+intel-pt" could set
> LIP=1 by default.

I have a question on how to set LIP=1 in SnowRidge by default. 
1. Set LIP in "builtin_x86_defs[]" SnowRidge CPU model. The LIP included in CPUID.(eax=14x,ecx=0)ecx[bit31] and a new leaf needs to be added.
2. Checking the CPU model in the later software flow and set the LIP bit if the CPU model is Snowridge. And we also need to add more CPU model's checking for new CPUs.

What is your opinion?

Thanks,
Luwei Kang

> 
> --
> Eduardo

RE: [PATCH v1 1/3] i386: Remove the limitation of IP payloads for Intel PT

[Kang, Luwei]

> >> No, if a feature cannot be emulated, that means it cannot be enabled
> >> unless it matches the host.  That's generally not a problem since
> >> Intel PT is usually used only with "-cpu host".
> >>
> > The limitation of LIP in qemu will mask off the Intel PT in KVM guest
> > even with "-cpu host". e.g. This bit will be set in SnowRidge HW and
> > later.
> 
> I agree that QEMU would have to learn about LIP.  Unlike this patch, however,
> x86_cpu_filter_features would have to fail if host LIP != guest LIP.  That is,
> something like
> 
>            (ecx_0 & INTEL_PT_IP_LIP)) !=
> 		(env->features[INTEL_PT_ECX_0] & INTEL_PT_IP_LIP)
> 
> where "intel-pt-lip" would be a feature in env->features[INTEL_PT_ECX_0].

Got it. Thanks.

Luwei Kang

> 
> > How about "-cpu cpu_model, +intel-pt" use case? The current value of
> > Intel PT CPUID is a constant. Can we make the ICX CPUID as basic
> > inforation(LIP is 0) and using "+intel-pt-lip" to make Intel PT work
> > on the CPU which LIP is 1 on the host? As you mentioned before, Intel
> > PT cannot be enabled in guest unless it matches the host.
> 
> Yes, this would work.
> 
> Paolo

Re: [PATCH v1 1/3] i386: Remove the limitation of IP payloads for Intel PT

[Eduardo Habkost]

On Tue, Sep 29, 2020 at 02:28:48AM +0000, Kang, Luwei wrote:
> > > > >>>> No, it's not possible.  KVM doesn't have a say on what the
> > > > >>>> processor writes in the tracing packets.
> > > > >>> Can KVM refuse to enable packet generation if CSbase is not zero
> > > > >>> and CPUID.(EAX=14H,ECX=0)[bit 31] seen by guest is different from
> > host?
> > > > >>
> > > > >> Yes, but the processor could change operating mode (and hence
> > > > >> CSbase) while tracing is active.  This is very unlikely, since it
> > > > >> would require nonzero CS-base and a 32-bit host, but in principle
> > > > >> not impossible (could be a firmware call, for example).
> > > > >>
> > > > >> The only solution is for KVM to accept both, and for QEMU to
> > > > >> refuse a setting that does not match the host.
> > > > >>
> > > > >
> > > > > So I need to add a patch in KVM to disabled the Intel PT when the
> > > > > CSbase is not zero and the guest LIP different from the host. And
> > > > > this limitation in qemu (disabled the PT when LIP is enabled in
> > > > > the host) can be remove. Is that right?
> > > >
> > > > No, if a feature cannot be emulated, that means it cannot be enabled
> > > > unless it matches the host.  That's generally not a problem since
> > > > Intel PT is usually used only with "-cpu host".
> > > >
> > >
> > > The limitation of LIP in qemu will mask off the Intel PT in KVM guest
> > > even with "-cpu host". e.g. This bit will be set in SnowRidge HW and
> > > later.
> > 
> > This behavior can and should be changed.
> > 
> > >
> > > How about "-cpu cpu_model, +intel-pt" use case? The current value of
> > > Intel PT CPUID is a constant. Can we make the ICX CPUID as basic
> > > inforation(LIP is 0) and using "+intel-pt-lip"
> > > to make Intel PT work on the CPU which LIP is 1 on the host? As you
> > > mentioned before, Intel PT cannot be enabled in guest unless it
> > > matches the host.
> > 
> > This makes sense, but you can also make each CPU model set a default for the
> > LIP bit.  "-cpu SnowRidge,+intel-pt" could set
> > LIP=1 by default.
> 
> I have a question on how to set LIP=1 in SnowRidge by default. 
> 1. Set LIP in "builtin_x86_defs[]" SnowRidge CPU model. The LIP included in CPUID.(eax=14x,ecx=0)ecx[bit31] and a new leaf needs to be added.
> 2. Checking the CPU model in the later software flow and set the LIP bit if the CPU model is Snowridge. And we also need to add more CPU model's checking for new CPUs.
> 
> What is your opinion?
> 

1 is preferred.  Any CPU-model-specific data should be
represented as data inside builtin_x86_defs, not code.

-- 
Eduardo