Re: [Qemu-devel] [PATCH 00/13] instrument: Add basic event instrumentation

From: Stefan Hajnoczi <stefanha@redhat.com>
To: "Daniel P. Berrange" <berrange@redhat.com>
Cc: "Peter Maydell" <peter.maydell@linaro.org>,
	"Stefan Hajnoczi" <stefanha@gmail.com>,
	"Emilio G. Cota" <cota@braap.org>,
	"Lluís Vilanova" <vilanova@ac.upc.edu>,
	"QEMU Developers" <qemu-devel@nongnu.org>
Subject: Re: [Qemu-devel] [PATCH 00/13] instrument: Add basic event instrumentation
Date: Fri, 28 Jul 2017 14:34:30 +0100	[thread overview]
Message-ID: <20170728133430.GS12364@stefanha-x1.localdomain> (raw)
In-Reply-To: <20170727154535.GY2555@redhat.com>

[-- Attachment #1: Type: text/plain, Size: 4086 bytes --]

On Thu, Jul 27, 2017 at 04:45:35PM +0100, Daniel P. Berrange wrote:
> On Thu, Jul 27, 2017 at 04:33:01PM +0100, Peter Maydell wrote:
> > On 27 July 2017 at 16:21, Daniel P. Berrange <berrange@redhat.com> wrote:
> > > On Thu, Jul 27, 2017 at 11:54:29AM +0100, Peter Maydell wrote:
> > >> That said, yes, I was going to ask if we could do this via
> > >> leveraging the tracepoint infrastructure and whatever scripting
> > >> facilities it provides. Are there any good worked examples of
> > >> this sort of thing? Can you do it as an ordinary non-root user?
> > >
> > > Do you have a particular thing you'd like to see an example of ?
> > >
> > > To dynamically probe a function which doesn't have a tracepoint
> > > defined you can do:
> > >
> > > probe process("/usr/bin/qemu-x86_64").function("helper_syscall") {
> > >   printf("syscall stasrt\n")
> > > }
> > >
> > > but getting access to the function args is not as easy as with
> > > pre-defined tracepoints.
> > 
> > How do I go about actually running that script? What I
> > have in mind by "worked example" is something like a blog
> > post that says "ok, here's a problem, we want to find out
> > what QEMU is doing in situation X, here's how you do this
> > with $TRACING_THINGY" and generally steps you through how
> > it works assuming you know nothing at all about whatever
> > the tracing facility you're using is.
> 
> Ok, so something like this example that I wrote for libvirt a
> while back then
> 
>   https://www.berrange.com/posts/2011/11/30/watching-the-libvirt-rpc-protocol-using-systemtap/
> 
> 
> > > You can't typically run this as root,
> > 
> > Do you mean "non-root" ?
> 
> Sigh, yes, of course.
> 
> > > however, I don't think that's a
> > > huge issue, because most QEMU deployments are not running as your own
> > > user account anyway, so you can't directly interact with them no
> > > matter what.
> > 
> > It is important, because almost all uses of TCG QEMU are
> > running it from the command line as non-root normal users,
> > especially if they're trying to debug what's going on with a
> > guest binary. So any tracing solution for this kind of usecase
> > must work without requiring root access, I think.
> 
> None of the Linux integrated tracing tools allow direct non-root access
> afaik. systemtap has ability to launch probes as non-root, via a privileged
> daemon, but it is restricted to probe scripts that the administrator has
> pre-defined.

One exception is gdb's static userspace probes support.  If you can run
gdb on QEMU then you can trace the same events as SystemTap.  I have
never tried this GDB feature:

  https://sourceware.org/gdb/onlinedocs/gdb/Static-Probe-Points.html

It should work out of the box if your distro builds QEMU with the
'dtrace' backend enabled.

> That pretty much leaves re-building QEMU, LD_PRELOADS, or something
> ptrace(), or qemu's built-in simpletrace feature, as the remaining
> options.  We have a scripts/simpletrace.py that lets you load a
> trace file into python and process it, but as written that's aimed
> as post-processing a tracefile you've previously collected.
> 
> It would be desirable to write a more advanced simpletrace python
> module that could collect & process the trace data live, and also
> interact with the qemu monitor to change what events are enabled
> dynamically.  Basically we'd need a way for the simpletrace backend
> to output its data to a fifo, instead of creating an file on disk,
> then you could dynanically consume it.

That would be interesting, I know Alex Bennee has wrangled with large
(~10 GB?) simpletrace files and it's not a pleasant experience :).

Lluís/Peter: What are the requirements for instrumentation code
interacting with the running QEMU instance?  simpletrace is
asynchronous, meaning it does not wait for anyone handle the trace event
before continuing execution, and is therefore not suitable for
SystemTap-style scripts that can interact with the program while
handling a trace event.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]