RE: [PATCH v1 0/7] Implement support for external IPT monitoring

["Kang, Luwei" <luwei.kang@intel.com>]

> -----Original Message-----
> From: Tian, Kevin <kevin.tian@intel.com>
> Sent: Wednesday, June 17, 2020 9:35 AM
> To: Michał Leszczyński <michal.leszczynski@cert.pl>; Andrew Cooper
> <andrew.cooper3@citrix.com>
> Cc: Xen-devel <xen-devel@lists.xenproject.org>; Jan Beulich
> <jbeulich@suse.com>; Wei Liu <wl@xen.org>; Roger Pau Monné
> <roger.pau@citrix.com>; Nakajima, Jun <jun.nakajima@intel.com>; George
> Dunlap <george.dunlap@citrix.com>; Ian Jackson <ian.jackson@eu.citrix.com>;
> Julien Grall <julien@xen.org>; Stefano Stabellini <sstabellini@kernel.org>;
> Kang, Luwei <luwei.kang@intel.com>
> Subject: RE: [PATCH v1 0/7] Implement support for external IPT monitoring
> 
> +Luwei, who developed PT for KVM and is the best one who can help
> review VMX changes from Intel side. Please include him in future post or
> discussion.
> 
> > -----Original Message-----
> > From: Michał Leszczyński <michal.leszczynski@cert.pl>
> > Sent: Wednesday, June 17, 2020 2:48 AM
> > To: Andrew Cooper <andrew.cooper3@citrix.com>
> > Cc: Xen-devel <xen-devel@lists.xenproject.org>; Jan Beulich
> > <jbeulich@suse.com>; Wei Liu <wl@xen.org>; Roger Pau Monné
> > <roger.pau@citrix.com>; Nakajima, Jun <jun.nakajima@intel.com>; Tian,
> > Kevin <kevin.tian@intel.com>; George Dunlap
> > <george.dunlap@citrix.com>; Ian Jackson <ian.jackson@eu.citrix.com>;
> > Julien Grall <julien@xen.org>; Stefano Stabellini
> > <sstabellini@kernel.org>
> > Subject: Re: [PATCH v1 0/7] Implement support for external IPT
> > monitoring
> >
> > ----- 16 cze 2020 o 20:17, Andrew Cooper andrew.cooper3@citrix.com
> > napisał(a):
> >
> > > On 16/06/2020 16:16, Michał Leszczyński wrote:
> > >> Intel Processor Trace is an architectural extension available in
> > >> modern
> > Intel
> > >> family CPUs. It allows recording the detailed trace of activity
> > >> while the processor executes the code. One might use the recorded
> > >> trace to
> > reconstruct
> > >> the code flow. It means, to find out the executed code paths,
> > >> determine branches taken, and so forth.
> > >>
> > >> The abovementioned feature is described in Intel(R) 64 and IA-32
> > Architectures
> > >> Software Developer's Manual Volume 3C: System Programming Guide,
> > Part 3,
> > >> Chapter 36: "Intel Processor Trace."
> > >>
> > >> This patch series implements an interface that Dom0 could use in
> > >> order to
> > enable
> > >> IPT for particular vCPUs in DomU, allowing for external monitoring.
> > >> Such a feature has numerous applications like malware monitoring,
> > >> fuzzing, or performance testing.
> > >
> > > Hello,
> > >
> > > I'm very excited to see support like this appearing.  However, be
> > > aware that we're currently in code freeze for the 4.14 release, so
> > > in-depth reviews will probably be delayed somewhat due to our bug
> > > queue and release activities.
> >
> > Sure, take your time :)
> >
> >
> > >
> > > That said, I've had a very quick look through the series, and have a
> > > few general questions first.
> > >
> > > AFAICT, this is strictly for external monitoring of the VM, not for
> > > the VM to use itself?  If so, it shouldn't have the H tag here:
> > >
> > > XEN_CPUFEATURE(IPT,           5*32+25) /*H  Intel Processor Trace */
> > >
> > > because that exposes the feature to the guest, with the implication
> > > that all other parts of the feature work as advertised.
> >
> > Ok, I will remove the H tag.
> >
> >
> > >
> > >
> > > Are there any restrictions on EPT being enabled in the first place?
> > > I'm not aware of any, and in principle we could use this
> > > functionality for PV guests as well (using the CPL filter).
> > > Therefore, I think it would be helpful to not tie the functionality
> > > to HVM guests, even if that is the only option enabled to start with.
> >
> > I think at the moment it's not required to have EPT. This patch series
> > doesn't use any translation feature flags, so the output address is
> > always a machine physical address, regardless of context. I will check
> > if it could be easily used with PV.
> >
> >
> > >
> > > The buffer mapping and creation logic is fairly problematic.
> > > Instead of fighting with another opencoded example, take a look at
> > > the IOREQ server's use of "acquire resource" which is a mapping
> > > interface which supports allocating memory on behalf of the guest,
> > > outside of the guest memory, for use by control tools.
> > >
> > > I think what this wants is a bit somewhere in domain_create to
> > > indicate that external tracing is used for this domain (and allocate
> > > whatever structures/buffers are necessary), acquire resource to map
> > > the buffers themselves, and a domctl for any necessary runtime controls.
> > >
> >
> > I will check this out, this sounds like a good option as it would
> > remove lots of complexity from the existing ipt_enable domctl.
> >
> > >
> > > What semantics do you want for the buffer becoming full?  Given that
> > > debugging/tracing is the goal, I presume "pause vcpu on full" is the
> > > preferred behaviour, rather than drop packets on full?
> > >
> >
> > Right now this is a ring-style buffer and when it would become full it
> > would simply wrap and override the old data.
> >
> > >
> > > When this subject was broached on xen-devel before, one issue was
> > > the fact that all actions which are intercepted don't end up writing
> > > any appropriate packets.  This is perhaps less of an issue for this
> > > example, where the external agent can see VMExits in the trace, but
> > > it still results in missing information.  (It is a major problem for
> > > PT within the guest, and needs Xen's intercept/emulation framework
> > > being updated to be PT-aware so it can fill in the same packets
> > > which hardware would have done for equivalent actions.)
> >
> > Ok, this sounds like a hard issue. Could you point out what could be
> > the particular problematic cases? For instance, if something would
> > alter EIP/RIP or CR3 then I belive it would still be recorded in PT
> > trace (i.e. these values will be logged on VM entry).

e.g. If a VM exit is taken on a guest write to CR3 (including “MOV CR3” as well as task switches), the PIP packet
normally generated on the CR3 write will be missing. The PIP packet needs to be written to the PT buffer by software. Another example is VM-exit taken on RDTSC. 

For VM introspection, all the Intel PT packets may need to emulated by software. Some description in SDM as below:
If a VMM emulates an element of processor state by taking a VM exit on reads and/or writes to that piece of state, and the state element impacts Intel PT packet generation or values, it may be incumbent upon the VMM to insert or modify the output trace data.

Thanks,
Luwei Kang

> >
> > >
> > >
> > > Thanks,
> > >
> > > ~Andrew
> >
> >
> > Best regards,
> > Michał Leszczyński
> > CERT Polska