Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Luiz Capitulino <lcapitulino@redhat.com>]

On Thu, 18 Aug 2016 14:53:27 +0100
Stefan Hajnoczi <stefanha@redhat.com> wrote:

> On Thu, Aug 18, 2016 at 12:22:18PM +0200, Lluís Vilanova wrote:
> > Stefan Hajnoczi writes:
> >   
> > > On Fri, Aug 05, 2016 at 06:59:23PM +0200, Lluís Vilanova wrote:  
> > >> The hypertrace channel allows guest code to emit events in QEMU (the host) using
> > >> its tracing infrastructure (see "docs/trace.txt"). This works in both 'system'
> > >> and 'user' modes. That is, hypertrace is to tracing, what hypercalls are to
> > >> system calls.
> > >> 
> > >> You can use this to emit an event on both guest and QEMU (host) traces to easily
> > >> synchronize or correlate them. You could also modify you guest's tracing system
> > >> to emit all events through the hypertrace channel, providing a unified and fully
> > >> synchronized trace log. Another use case is timing the performance of guest code
> > >> when optimizing TCG (QEMU traces have a timestamp).
> > >> 
> > >> See first commit for a full description.
> > >> 
> > >> Signed-off-by: Lluís Vilanova <vilanova@ac.upc.edu>
> > >> ---  
> >   
> > > CCing Steven Rostedt, Masami Hiramatsu, Luiz Capitulino, and LTTng folks
> > > who have all looked into host/guest tracing solutions.  
> > [...]
> > 
> > Oh, I wasn't aware of that. I'm certainly interested in collaborating.  
> 
> They are working on or have worked on different approaches to host/guest
> tracing.  Unfortunately there isn't an out-of-the-box solution as far as
> I know.

The ftrace solution is documented here:

 https://lists.nongnu.org/archive/html/qemu-devel/2016-03/msg00887.html

This traces the guest and host kernels. It supports merging the guest
and host traces. It's extremely low latency and has helped us to
find several spikes for real-time KVM (we're talking a few to
a dozen microseconds at most).

Now, our stack actually is:

 - Guest app
 - Guest kernel
 - Host kernel
 - QEMU

QEMU already has its own tracing (which I don't know how it works).
If I had to trace the guest app, I'd certainly start off by using
LTTng. Although, we'd have to write a tool to merge and orchestrate
(wooo, cloud buzzword!) all those traces (if that's what one wants).

> It would be nice if there was a documented host/guest tracing approach
> that didn't involve much manual setup and handled most use cases.

I'd volunteer to do it, although it will take me weeks to do it.