Re: [Qemu-devel] [PATCH 00/13] instrument: Add basic event instrumentation

From: Stefan Hajnoczi <stefanha@gmail.com>
To: "Daniel P. Berrange" <berrange@redhat.com>,
	Stefan Hajnoczi <stefanha@redhat.com>,
	QEMU Developers <qemu-devel@nongnu.org>,
	Peter Maydell <peter.maydell@linaro.org>,
	"Emilio G. Cota" <cota@braap.org>
Subject: Re: [Qemu-devel] [PATCH 00/13] instrument: Add basic event instrumentation
Date: Tue, 1 Aug 2017 14:13:56 +0100	[thread overview]
Message-ID: <20170801131356.GB22017@stefanha-x1.localdomain> (raw)
In-Reply-To: <87pockles6.fsf@frigg.lan>

[-- Attachment #1: Type: text/plain, Size: 4457 bytes --]

On Fri, Jul 28, 2017 at 07:14:33PM +0300, Lluís Vilanova wrote:
> Daniel P Berrange writes:
> 
> > On Fri, Jul 28, 2017 at 02:34:30PM +0100, Stefan Hajnoczi wrote:
> >> On Thu, Jul 27, 2017 at 04:45:35PM +0100, Daniel P. Berrange wrote:
> >> > On Thu, Jul 27, 2017 at 04:33:01PM +0100, Peter Maydell wrote:
> >> > > On 27 July 2017 at 16:21, Daniel P. Berrange <berrange@redhat.com> wrote:
> >> > > > On Thu, Jul 27, 2017 at 11:54:29AM +0100, Peter Maydell wrote:
> >> > > >> That said, yes, I was going to ask if we could do this via
> >> > > >> leveraging the tracepoint infrastructure and whatever scripting
> >> > > >> facilities it provides. Are there any good worked examples of
> >> > > >> this sort of thing? Can you do it as an ordinary non-root user?
> >> > > >
> >> > > > Do you have a particular thing you'd like to see an example of ?
> >> > > >
> >> > > > To dynamically probe a function which doesn't have a tracepoint
> >> > > > defined you can do:
> >> > > >
> >> > > > probe process("/usr/bin/qemu-x86_64").function("helper_syscall") {
> >> > > >   printf("syscall stasrt\n")
> >> > > > }
> >> > > >
> >> > > > but getting access to the function args is not as easy as with
> >> > > > pre-defined tracepoints.
> >> > > 
> >> > > How do I go about actually running that script? What I
> >> > > have in mind by "worked example" is something like a blog
> >> > > post that says "ok, here's a problem, we want to find out
> >> > > what QEMU is doing in situation X, here's how you do this
> >> > > with $TRACING_THINGY" and generally steps you through how
> >> > > it works assuming you know nothing at all about whatever
> >> > > the tracing facility you're using is.
> >> > 
> >> > Ok, so something like this example that I wrote for libvirt a
> >> > while back then
> >> > 
> >> >   https://www.berrange.com/posts/2011/11/30/watching-the-libvirt-rpc-protocol-using-systemtap/
> >> > 
> >> > 
> >> > > > You can't typically run this as root,
> >> > > 
> >> > > Do you mean "non-root" ?
> >> > 
> >> > Sigh, yes, of course.
> >> > 
> >> > > > however, I don't think that's a
> >> > > > huge issue, because most QEMU deployments are not running as your own
> >> > > > user account anyway, so you can't directly interact with them no
> >> > > > matter what.
> >> > > 
> >> > > It is important, because almost all uses of TCG QEMU are
> >> > > running it from the command line as non-root normal users,
> >> > > especially if they're trying to debug what's going on with a
> >> > > guest binary. So any tracing solution for this kind of usecase
> >> > > must work without requiring root access, I think.
> >> > 
> >> > None of the Linux integrated tracing tools allow direct non-root access
> >> > afaik. systemtap has ability to launch probes as non-root, via a privileged
> >> > daemon, but it is restricted to probe scripts that the administrator has
> >> > pre-defined.
> >> 
> >> One exception is gdb's static userspace probes support.  If you can run
> >> gdb on QEMU then you can trace the same events as SystemTap.  I have
> >> never tried this GDB feature:
> >> 
> >> https://sourceware.org/gdb/onlinedocs/gdb/Static-Probe-Points.html
> >> 
> >> It should work out of the box if your distro builds QEMU with the
> >> 'dtrace' backend enabled.
> 
> > Wow, that's great to learn about. It does indeed work !
> 
> > If you knew alot about ptrace() you could probably build something
> > that use ptrace() and these probe points to call your dynamic
> > instrumentation code with reasonable low overheads.
> 
> I don't think so. Ptrace traps into the kernel and stops the process while a
> separate process decides what to do. That's between 3 and 4 orders of magnitude
> slower than calling an instrumentor function.

Dan might be referring to dynamic patching a jump to the instrumentation
function.

A static userspace probe is a single nop instruction (plus metadata
stored in a separate ELF section).  Using ptrace you can binary patch
the nop instruction.

Unfortunately a single nop instruction cannot hold most x86
instructions.  uprobes places a breakpoint instruction (INT $3 - 0xcc)
there.  That works because it's just one byte.

This technique would be way out of scope for qemu.git but perhaps
perf(1) or a stand-alone tool could implement it.  There are libraries
for binary patching like http://www.dyninst.org/dyninst.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]