[PATCH 0/9] Intel Processor Trace virtulization enabling

[PATCH 0/9] Intel Processor Trace virtulization enabling

[Luwei Kang]

From: Chao Peng <chao.p.peng@linux.intel.com>

Hi All,

Here is a patch-series which adding Processor Trace enabling in KVM guest. You can get It's software developer manuals from:
https://software.intel.com/sites/default/files/managed/c5/15/architecture-instruction-set-extensions-programming-reference.pdf
In Chapter 5 INTEL PROCESSOR TRACE: VMX IMPROVEMENTS.

Introduction:
Intel Processor Trace (Intel PT) is an extension of Intel Architecture that captures information about software execution using dedicated hardware facilities that cause only minimal performance perturbation to the software being traced. Details on the Intel PT infrastructure and trace capabilities can be found in the Intel 64 and IA-32 Architectures Software Developer’s Manual, Volume 3C.

The suite of architecture changes serve to simplify the process of virtualizing Intel PT for use by a guest software. There are two primary elements to this new architecture support for VMX support improvements made for Intel PT.
1. Addition of a new guest IA32_RTIT_CTL value field to the VMCS.
  — This serves to speed and simplify the process of disabling trace on VM exit, and restoring it on VM entry.
2. Enabling use of EPT to redirect PT output. 
  — This enables the VMM to elect to virtualize the PT output buffer using EPT. In this mode, the CPU will treat PT output addresses as Guest Physical Addresses (GPAs) and translate them using EPT. This means that Intel PT output reads (of the ToPA table) and writes (of trace output) can cause EPT violations, and other output events.

Intel Processor Trace virtualization can be work in one of 4 possible modes by set new option "pt_mode". Default is host guest mode.
 a. system-wide: trace both host/guest and output to host buffer;
 b. host-only: only trace host and output to host buffer;
 c. guest-only: only trace guest and output to guest buffer;
 d. host-guest: trace host/guest simultaneous and output to their respective buffer.

Chao Peng (9):
  perf/x86/intel/pt: Move Intel-PT MSR bit definitions to a public
    header
  perf/x86/intel/pt: change pt_cap_get() to a public function
  KVM: x86: add Intel processor trace virtualization mode
  perf/x86/intel/pt: add Intel processor trace virtualization call backs
  KVM: x86: implement intel processor trace virtualization callbacks
  KVM: x86: add Intel processor trace cpuid emulataion
  KVM: x86: add Intel processor trace context for each vcpu
  KVM: x86: Implement Intel processor trace context switch
  KVM: x86: Disable intercept for Intel processor trace MSRs

 arch/x86/events/intel/pt.c       |  24 +++-
 arch/x86/events/intel/pt.h       |  55 ---------
 arch/x86/include/asm/intel_pt.h  |  36 ++++++
 arch/x86/include/asm/kvm_host.h  |   1 +
 arch/x86/include/asm/msr-index.h |  35 ++++++
 arch/x86/include/asm/vmx.h       |   8 ++
 arch/x86/kvm/cpuid.c             |  23 +++-
 arch/x86/kvm/svm.c               |   6 +
 arch/x86/kvm/vmx.c               | 247 ++++++++++++++++++++++++++++++++++++++-
 9 files changed, 373 insertions(+), 62 deletions(-)

-- 
1.8.3.1

Re: [PATCH 0/9] Intel Processor Trace virtulization enabling

[Paolo Bonzini]

On 16/10/2017 14:09, Luwei Kang wrote:
> 2. Enabling use of EPT to redirect PT output. 
>   — This enables the VMM to elect to virtualize the PT output buffer using EPT. In this mode, the CPU will treat PT output addresses as Guest Physical Addresses (GPAs) and translate them using EPT. This means that Intel PT output reads (of the ToPA table) and writes (of trace output) can cause EPT violations, and other output events.

Yay yay yay! ;)

> Intel Processor Trace virtualization can be work in one of 4 possible modes by set new option "pt_mode". Default is host guest mode.
>  a. system-wide: trace both host/guest and output to host buffer;
>  b. host-only: only trace host and output to host buffer;
>  c. guest-only: only trace guest and output to guest buffer;
>  d. host-guest: trace host/guest simultaneous and output to their respective buffer.

I think the guest-only mode isn't particularly useful, and I would
remove it.

Nested virtualization is interesting.  We would like the nested
hypervisor to be forced to set the "use GPA for processor tracing"
secondary execution control whenever "enable EPT" is set and RTIT_CTL is
nonzero.  There is no way to encode that in IA32_VMX_PROCBASED_CTLS2,
however.  It would be nice if Intel could reserve a bit in
IA32_VMX_EPT_VPID_CAP for KVM to express that constraint.

Paolo

RE: [PATCH 0/9] Intel Processor Trace virtulization enabling

[Kang, Luwei]

> On 16/10/2017 14:09, Luwei Kang wrote:
> > 2. Enabling use of EPT to redirect PT output.
> >   — This enables the VMM to elect to virtualize the PT output buffer using EPT. In this mode, the CPU will treat PT output
> addresses as Guest Physical Addresses (GPAs) and translate them using EPT. This means that Intel PT output reads (of the ToPA
> table) and writes (of trace output) can cause EPT violations, and other output events.
> 
> Yay yay yay! ;)
> 
> > Intel Processor Trace virtualization can be work in one of 4 possible modes by set new option "pt_mode". Default is host guest
> mode.
> >  a. system-wide: trace both host/guest and output to host buffer;  b.
> > host-only: only trace host and output to host buffer;  c. guest-only:
> > only trace guest and output to guest buffer;  d. host-guest: trace
> > host/guest simultaneous and output to their respective buffer.
> 
> I think the guest-only mode isn't particularly useful, and I would remove it.

Agree, will remove it.

> 
> Nested virtualization is interesting.  We would like the nested hypervisor to be forced to set the "use GPA for processor tracing"
> secondary execution control whenever "enable EPT" is set and RTIT_CTL is nonzero.  There is no way to encode that in
> IA32_VMX_PROCBASED_CTLS2, however.  It would be nice if Intel could reserve a bit in IA32_VMX_EPT_VPID_CAP for KVM to
> express that constraint.

Do you mean if nested hypervisor get the capability of "Guest PT use GPA" and EPT has enable. Highly recommend nested hypervisor set " Guest PT use GPA "  as well. 
If nested hypervisor is also KVM, "use GPA for processor tracing" will be set for sure. But other hypervisor may not do that. So, we'd better add a flag in IA32_VMX_EPT_VPID_CAP to express that constraint.  Is that right?

Thanks,
Luwei Kang

Re: [PATCH 0/9] Intel Processor Trace virtulization enabling

[Paolo Bonzini]

On 18/10/2017 12:18, Kang, Luwei wrote:
>> Nested virtualization is interesting.  We would like the nested
>> hypervisor to be forced to set the "use GPA for processor tracing" 
>> secondary execution control whenever "enable EPT" is set and
>> RTIT_CTL is nonzero.  There is no way to encode that in 
>> IA32_VMX_PROCBASED_CTLS2, however.  It would be nice if Intel could
>> reserve a bit in IA32_VMX_EPT_VPID_CAP for KVM to express that
>> constraint.
>
> Do you mean if nested hypervisor get the capability of "Guest PT use 
> GPA" and EPT has enable. Highly recommend nested hypervisor set "
> Guest PT use GPA " as well.

Well, it's required more than recommended.  However, it's only required
if "enable EPT" is set and RTIT_CTL is nonzero.

> If nested hypervisor is also KVM, "use GPA for processor tracing"
> will be set for sure. But other hypervisor may not do that. So, we'd
> better add a flag in IA32_VMX_EPT_VPID_CAP to express that
> constraint.

Correct.  The constraint would be:

* RTIT_CTL on entry is zero if EPT is disabled

* RTIT_CTL on entry is zero if EPT is enabled and "Guest PT uses GPA" is
zero

Maybe IA32_VMX_EPT_VPID_CAP is not the best place.  I'll let Intel
decide that.

Paolo

RE: [PATCH 0/9] Intel Processor Trace virtulization enabling

[Kang, Luwei]

> >> Nested virtualization is interesting.  We would like the nested
> >> hypervisor to be forced to set the "use GPA for processor tracing"
> >> secondary execution control whenever "enable EPT" is set and RTIT_CTL
> >> is nonzero.  There is no way to encode that in
> >> IA32_VMX_PROCBASED_CTLS2, however.  It would be nice if Intel could
> >> reserve a bit in IA32_VMX_EPT_VPID_CAP for KVM to express that
> >> constraint.
> >
> > Do you mean if nested hypervisor get the capability of "Guest PT use
> > GPA" and EPT has enable. Highly recommend nested hypervisor set "
> > Guest PT use GPA " as well.
> 
> Well, it's required more than recommended.  However, it's only required if "enable EPT" is set and RTIT_CTL is nonzero.
> 
> > If nested hypervisor is also KVM, "use GPA for processor tracing"
> > will be set for sure. But other hypervisor may not do that. So, we'd
> > better add a flag in IA32_VMX_EPT_VPID_CAP to express that constraint.
> 
> Correct.  The constraint would be:
> 
> * RTIT_CTL on entry is zero if EPT is disabled
> 
> * RTIT_CTL on entry is zero if EPT is enabled and "Guest PT uses GPA" is zero
> 
> Maybe IA32_VMX_EPT_VPID_CAP is not the best place.  I'll let Intel decide that.
> 

Get it. I have feedback to hardware architect. I hope it can be applied but it may need wait a long time.

Thanks,
Luwei Kang

Re: [PATCH 0/9] Intel Processor Trace virtulization enabling

[Paolo Bonzini]

On 18/10/2017 13:06, Kang, Luwei wrote:
>>>> Nested virtualization is interesting.  We would like the nested
>>>> hypervisor to be forced to set the "use GPA for processor tracing"
>>>> secondary execution control whenever "enable EPT" is set and RTIT_CTL
>>>> is nonzero.  There is no way to encode that in
>>>> IA32_VMX_PROCBASED_CTLS2, however.  It would be nice if Intel could
>>>> reserve a bit in IA32_VMX_EPT_VPID_CAP for KVM to express that
>>>> constraint.
>>>
>>> Do you mean if nested hypervisor get the capability of "Guest PT use
>>> GPA" and EPT has enable. Highly recommend nested hypervisor set "
>>> Guest PT use GPA " as well.
>>
>> Well, it's required more than recommended.  However, it's only required if "enable EPT" is set and RTIT_CTL is nonzero.
>>
>>> If nested hypervisor is also KVM, "use GPA for processor tracing"
>>> will be set for sure. But other hypervisor may not do that. So, we'd
>>> better add a flag in IA32_VMX_EPT_VPID_CAP to express that constraint.
>>
>> Correct.  The constraint would be:
>>
>> * RTIT_CTL on entry is zero if EPT is disabled
>>
>> * RTIT_CTL on entry is zero if EPT is enabled and "Guest PT uses GPA" is zero
>>
>> Maybe IA32_VMX_EPT_VPID_CAP is not the best place.  I'll let Intel decide that.
> 
> Get it. I have feedback to hardware architect. I hope it can be applied but it may need wait a long time.

Note that the hardware need not do anything.  However it would be nice
if the SDM can define a bit _for the hypervisors_ to enforce the above
constraint and fail vmentry if they are not respected.

Paolo

Re: [PATCH 0/9] Intel Processor Trace virtulization enabling

[Stefan Hajnoczi]

Please send patch series with git's sendemail.thread=shallow and
sendemail.chainReplyTo=false settings (see the man git-send-email
--thread option for an explanation of the different settings).

That way email clients treat the entire series as a single email thread.
Patch management and continuous integration tools may have trouble with
patch series that are not sent in this standard format.

Thanks,
Stefan

RE: [PATCH 0/9] Intel Processor Trace virtulization enabling

[Kang, Luwei]

> >>>> Nested virtualization is interesting.  We would like the nested
> >>>> hypervisor to be forced to set the "use GPA for processor tracing"
> >>>> secondary execution control whenever "enable EPT" is set and
> >>>> RTIT_CTL is nonzero.  There is no way to encode that in
> >>>> IA32_VMX_PROCBASED_CTLS2, however.  It would be nice if Intel could
> >>>> reserve a bit in IA32_VMX_EPT_VPID_CAP for KVM to express that
> >>>> constraint.
> >>>
> >>> Do you mean if nested hypervisor get the capability of "Guest PT use
> >>> GPA" and EPT has enable. Highly recommend nested hypervisor set "
> >>> Guest PT use GPA " as well.
> >>
> >> Well, it's required more than recommended.  However, it's only required if "enable EPT" is set and RTIT_CTL is nonzero.
> >>
> >>> If nested hypervisor is also KVM, "use GPA for processor tracing"
> >>> will be set for sure. But other hypervisor may not do that. So, we'd
> >>> better add a flag in IA32_VMX_EPT_VPID_CAP to express that constraint.
> >>
> >> Correct.  The constraint would be:
> >>
> >> * RTIT_CTL on entry is zero if EPT is disabled
> >>
> >> * RTIT_CTL on entry is zero if EPT is enabled and "Guest PT uses GPA"
> >> is zero
> >>
> >> Maybe IA32_VMX_EPT_VPID_CAP is not the best place.  I'll let Intel decide that.
> >
> > Get it. I have feedback to hardware architect. I hope it can be applied but it may need wait a long time.
> 
> Note that the hardware need not do anything.  However it would be nice if the SDM can define a bit _for the hypervisors_ to
> enforce the above constraint and fail vmentry if they are not respected.
> 

Hi Paolo,
    Thanks for your response. I have a question want to ask for you. As you mentioned in previous mail " We would like the nested hypervisor to be forced to set the "use GPA for processor tracing"". 
    Is there have any problem if we don't set "use GPA for processor tracing" in nested hypervisor? If yes, what should we do? In patch 9, I pass though PT MSRs ( IA32_RTIT_* ) to guest.

Thanks,
Luwei Kang

Re: [PATCH 0/9] Intel Processor Trace virtulization enabling

[Paolo Bonzini]

On 19/10/2017 07:54, Kang, Luwei wrote:
>>> Get it. I have feedback to hardware architect. I hope it can be applied but it may need wait a long time.
>> Note that the hardware need not do anything.  However it would be nice if the SDM can define a bit _for the hypervisors_ to
>> enforce the above constraint and fail vmentry if they are not respected.
>
> Hi Paolo,
> Thanks for your response. I have a question want to ask for you. As
> you mentioned in previous mail " We would like the nested hypervisor
> to be forced to set the "use GPA for processor tracing"". Is there
> have any problem if we don't set "use GPA for processor tracing" in
> nested hypervisor?

If the nested hypervisor doesn't set "use GPA for processor tracing",
the processor should use L1 addresses for processor tracing.  This
however is not possible without shadowing the ToPA, same as in
non-nested virtualization for <=Skylake processors.

So we have

            |          mode
            |-----------------------------------------------------------
nested?     |          system-wide          |  host-guest
------------+-------------------------------+---------------------------
no          |          use HPA for tracing  |  use GPA for tracing
            |          (no EPT)             |  (EPT is GPA->HPA)
------------+-------------------------------+---------------------------
yes         |          use GPA for tracing  |  use nGPA for tracing
            |          (EPT is nGPA->HPA!!) |  (EPT is nGPA->HPA, so ok)

(for nested, L0 mode of course must be host-guest).  If the nested
hypervisor wants to use system-wide tracing, it cannot use "use GPA for
tracing" because the EPT table doesn't have the right mapping of L1->L0
physical address.

So if you want to do system-wide L1 tracing you have to disable EPT for
L1, and if you want to do host-guest L1 tracing you have to enable it.

Paolo

> If yes, what should we do? In patch 9, I pass
> though PT MSRs ( IA32_RTIT_* ) to guest.

RE: [PATCH 0/9] Intel Processor Trace virtulization enabling

[Kang, Luwei]

> On 19/10/2017 07:54, Kang, Luwei wrote:
> >>> Get it. I have feedback to hardware architect. I hope it can be applied but it may need wait a long time.
> >> Note that the hardware need not do anything.  However it would be
> >> nice if the SDM can define a bit _for the hypervisors_ to enforce the above constraint and fail vmentry if they are not respected.
> >
> > Hi Paolo,
> > Thanks for your response. I have a question want to ask for you. As
> > you mentioned in previous mail " We would like the nested hypervisor
> > to be forced to set the "use GPA for processor tracing"". Is there
> > have any problem if we don't set "use GPA for processor tracing" in
> > nested hypervisor?
> 
> If the nested hypervisor doesn't set "use GPA for processor tracing", the processor should use L1 addresses for processor tracing.
> This however is not possible without shadowing the ToPA, same as in non-nested virtualization for <=Skylake processors.
> 
> So we have
> 
>             |          mode
>             |-----------------------------------------------------------
> nested?     |          system-wide          |  host-guest
> ------------+-------------------------------+---------------------------
> no          |          use HPA for tracing  |  use GPA for tracing
>             |          (no EPT)             |  (EPT is GPA->HPA)
> ------------+-------------------------------+---------------------------
> yes         |          use GPA for tracing  |  use nGPA for tracing
>             |          (EPT is nGPA->HPA!!) |  (EPT is nGPA->HPA, so ok)
> 
> (for nested, L0 mode of course must be host-guest).  If the nested hypervisor wants to use system-wide tracing, it cannot use "use
> GPA for tracing" because the EPT table doesn't have the right mapping of L1->L0 physical address.
> 
> So if you want to do system-wide L1 tracing you have to disable EPT for L1, and if you want to do host-guest L1 tracing you have to
> enable it.
> 
HI Paolo,
    Thanks for your clarify. Have understood. So, we should set "use GPA for processor tracing" in any way( if we can do it) even in system mode. There don't have problem in no nested but have problem in nested if not set this bit.
    Still talking with  hardware designer but please don't expect it can be change in SDM or hardware(fail vmentry if they are not respected) soon. So, can we enable it in L1 guest only first?  I think it is not worth to disable EPT for L1 to enable intel PT. what is your opinion?

Thanks,
Luwei Kang

Re: [PATCH 0/9] Intel Processor Trace virtulization enabling

[Paolo Bonzini]

On 20/10/2017 02:22, Kang, Luwei wrote:
> HI Paolo, Thanks for your clarify. Have understood. So, we should set
> "use GPA for processor tracing" in any way( if we can do it) even in
> system mode. There don't have problem in no nested but have problem
> in nested if not set this bit. Still talking with  hardware designer
> but please don't expect it can be change in SDM or hardware(fail
> vmentry if they are not respected) soon.

No change in hardware is needed.

What I'm asking for is to define a bit in some architectural MSR such
that, _if the bit is 1_, you must have one of:

- RTIT_CTL = 0

- enable EPT = 0

- enable EPT = use GPA for processor tracing = 1, RTIT_CTL != 0

or vmentry would fail.

If the bit is 1 and RTIT_CTL = 0 and enable EPT = 1 and use GPA for
processor tracing = 0, the hypervisor must trap RTIT_CTL writes or
behavior is undefined.

Processors would just set it to 0 and have absolutely no change in behavior.

> So, can we enable it in L1
> guest only first?  I think it is not worth to disable EPT for L1 to
> enable intel PT. what is your opinion?

Yes, we can enable it.  But since KVM sets IA32_VMX_MISC[14]=0, your
patches must forbid enabling processor trace during VMX operation.

(In fact, another source of complexity is that we'd have to write the
VMPTRLD packet ourselves to the guest's processor trace buffer).

Paolo

RE: [PATCH 0/9] Intel Processor Trace virtulization enabling

[Kang, Luwei]

> > HI Paolo, Thanks for your clarify. Have understood. So, we should set
> > "use GPA for processor tracing" in any way( if we can do it) even in
> > system mode. There don't have problem in no nested but have problem in
> > nested if not set this bit. Still talking with  hardware designer but
> > please don't expect it can be change in SDM or hardware(fail vmentry
> > if they are not respected) soon.
> 
> No change in hardware is needed.
> 
> What I'm asking for is to define a bit in some architectural MSR such that, _if the bit is 1_, you must have one of:
> 
> - RTIT_CTL = 0
> 
> - enable EPT = 0
> 
> - enable EPT = use GPA for processor tracing = 1, RTIT_CTL != 0
> 
> or vmentry would fail.
> 
> If the bit is 1 and RTIT_CTL = 0 and enable EPT = 1 and use GPA for processor tracing = 0, the hypervisor must trap RTIT_CTL writes
> or behavior is undefined.
> 
> Processors would just set it to 0 and have absolutely no change in behavior.
> 

Get it. Will update with you when hardware designer have any response.

> > So, can we enable it in L1
> > guest only first?  I think it is not worth to disable EPT for L1 to
> > enable intel PT. what is your opinion?
> 
> Yes, we can enable it.  But since KVM sets IA32_VMX_MISC[14]=0, your patches must forbid enabling processor trace during VMX
> operation.

L1 hypervisor can't  get the capability of " TraceEn can be set in VMX operation (IA32_VMX_MISC[bit 14] is 0)" and set it to 0.
We need to trap whether L1 hypervisor have enable VMXON, and forbid enable PT when vmxon. Is that right? Or have something else?

Thanks,
Luwei Kang

> 
> (In fact, another source of complexity is that we'd have to write the VMPTRLD packet ourselves to the guest's processor trace
> buffer).
> 
> Paolo

Re: [PATCH 0/9] Intel Processor Trace virtulization enabling

[Paolo Bonzini]

On 23/10/2017 10:01, Kang, Luwei wrote:
> 
>>> So, can we enable it in L1
>>> guest only first?  I think it is not worth to disable EPT for L1 to
>>> enable intel PT. what is your opinion?
>> Yes, we can enable it.  But since KVM sets IA32_VMX_MISC[14]=0, your patches must forbid enabling processor trace during VMX
>> operation.
>
> L1 hypervisor can't  get the capability of " TraceEn can be set in
> VMX operation (IA32_VMX_MISC[bit 14] is 0)" and set it to 0. We need
> to trap whether L1 hypervisor have enable VMXON, and forbid enable PT
> when vmxon. Is that right? Or have something else?

Correct.  I don't have the SDM at hand, so I'm not sure what happens if
you do a VMXON while TraceEn is 1.  However, that has to be handled as well.

Paolo