[Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Lluís Vilanova]

The hypertrace channel allows guest code to emit events in QEMU (the host) using
its tracing infrastructure (see "docs/trace.txt"). This works in both 'system'
and 'user' modes. That is, hypertrace is to tracing, what hypercalls are to
system calls.

You can use this to emit an event on both guest and QEMU (host) traces to easily
synchronize or correlate them. You could also modify you guest's tracing system
to emit all events through the hypertrace channel, providing a unified and fully
synchronized trace log. Another use case is timing the performance of guest code
when optimizing TCG (QEMU traces have a timestamp).

See first commit for a full description.

Signed-off-by: Lluís Vilanova <vilanova@ac.upc.edu>
---

Lluís Vilanova (6):
      hypertrace: Add documentation
      hypertrace: Add tracing event "guest_hypertrace"
      hypertrace: [*-user] Add QEMU-side proxy to "guest_hypertrace" event
      hypertrace: [softmmu] Add QEMU-side proxy to "guest_hypertrace" event
      hypertrace: Add guest-side user-level library
      hypertrace: Add guest-side Linux module


 Makefile                                           |    8 -
 Makefile.objs                                      |    6 
 bsd-user/main.c                                    |   16 +
 bsd-user/mmap.c                                    |    2 
 bsd-user/syscall.c                                 |    4 
 configure                                          |   42 +++
 docs/hypertrace.txt                                |  141 ++++++++++
 docs/tracing.txt                                   |    3 
 hypertrace/Makefile.objs                           |   20 +
 hypertrace/guest/linux-module/Kbuild.in            |    5 
 hypertrace/guest/linux-module/Makefile             |   24 ++
 .../include/linux/qemu-hypertrace-internal.h       |   46 +++
 .../linux-module/include/linux/qemu-hypertrace.h   |   71 +++++
 hypertrace/guest/linux-module/qemu-hypertrace.c    |  133 +++++++++
 hypertrace/guest/user/Makefile                     |   28 ++
 hypertrace/guest/user/common.c                     |  221 +++++++++++++++
 hypertrace/guest/user/qemu-hypertrace.h            |   77 +++++
 hypertrace/softmmu.c                               |  243 +++++++++++++++++
 hypertrace/user.c                                  |  292 ++++++++++++++++++++
 hypertrace/user.h                                  |   52 ++++
 include/hw/pci/pci.h                               |    2 
 linux-user/main.c                                  |   19 +
 linux-user/mmap.c                                  |    2 
 linux-user/syscall.c                               |    3 
 trace/Makefile.objs                                |    2 
 25 files changed, 1460 insertions(+), 2 deletions(-)
 create mode 100644 docs/hypertrace.txt
 create mode 100644 hypertrace/Makefile.objs
 create mode 100644 hypertrace/guest/linux-module/Kbuild.in
 create mode 100644 hypertrace/guest/linux-module/Makefile
 create mode 100644 hypertrace/guest/linux-module/include/linux/qemu-hypertrace-internal.h
 create mode 100644 hypertrace/guest/linux-module/include/linux/qemu-hypertrace.h
 create mode 100644 hypertrace/guest/linux-module/qemu-hypertrace.c
 create mode 100644 hypertrace/guest/user/Makefile
 create mode 100644 hypertrace/guest/user/common.c
 create mode 100644 hypertrace/guest/user/qemu-hypertrace.h
 create mode 100644 hypertrace/softmmu.c
 create mode 100644 hypertrace/user.c
 create mode 100644 hypertrace/user.h


To: qemu-devel@nongnu.org
Cc: Stefan Hajnoczi <stefanha@redhat.com>

[Qemu-devel] [PATCH 1/6] hypertrace: Add documentation

[Lluís Vilanova]

Signed-off-by: Lluís Vilanova <vilanova@ac.upc.edu>
---
 docs/hypertrace.txt |  141 +++++++++++++++++++++++++++++++++++++++++++++++++++
 docs/tracing.txt    |    3 +
 2 files changed, 144 insertions(+)
 create mode 100644 docs/hypertrace.txt

diff --git a/docs/hypertrace.txt b/docs/hypertrace.txt
new file mode 100644
index 0000000..4a31bcd
--- /dev/null
+++ b/docs/hypertrace.txt
@@ -0,0 +1,141 @@
+= Hypertrace channel =
+
+The hypertrace channel allows guest code to emit events in QEMU (the host) using
+its tracing infrastructure (see "docs/trace.txt"). This works in both 'system'
+and 'user' modes. That is, hypertrace is to tracing, what hypercalls are to
+system calls.
+
+You can use this to emit an event on both guest and QEMU (host) traces to easily
+synchronize or correlate them. You could also modify you guest's tracing system
+to emit all events through the hypertrace channel, providing a unified and fully
+synchronized trace log. Another use case is timing the performance of guest code
+when optimizing TCG (QEMU traces have a timestamp).
+
+QEMU provides an example library and Linux kernel module that guest code can use
+to interact with the hypertrace channel.
+
+Hypertrace highlights:
+
+* Works with 'system' and 'user' mode.
+
+* Minimal setup for the guest (e.g., 'system' mode with Linux and 'user' mode
+  work out of the box).
+
+* Independent of guest architecture; the guest code uses accesses to special
+  memory regions, as opposed to redefining instruction semantics.
+
+* Negligible guest overhead; guest operations do not go through any OS
+  abstraction, except during the setup of the communication channel.
+
+Warning: The hypertrace channel in 'system' mode is presented as a PCI device,
+and thus will only be available on systems with support for PCI. You can get the
+list of guests with PCI support with 'grep pci.mak default-configs/*'.
+
+
+== Quick guide ==
+
+1. Set the number of arguments for the hypertrace events:
+
+    mkdir /tmp/qemu-build
+    cd /tmp/qemu-build
+    /path/to/qemu-source/configure              \
+        --with-hypertrace-args=1                \
+        --prefix=/tmp/qemu-install
+    make -j install
+
+2. Compile the corresponding guest support code:
+
+    make -C /tmp/qemu-build/x86_64-linux-user/hypertrace/guest/user
+    make -C /tmp/qemu-build/x86_64-softmmu/hypertrace/guest/user
+    make -C /tmp/qemu-build/x86_64-softmmu/hypertrace/guest/linux-module
+
+   If you need to cross-compile the guest library, set the 'CC' variable (e.g.,
+   for mipsel):
+
+    make -C /tmp/qemu-build/mipsel-linux-user/hypertrace/guest/user CC=mipsel-gnu-linux-gcc
+
+3. Create a guest application using "qemu-hypertrace.h":
+
+    cat > /tmp/my-hypertrace.c <<EOF
+    #include <stdio.h>
+    #include <errno.h>
+    #include <stdlib.h>
+    #include <string.h>
+    #include <qemu-hypertrace.h>
+    
+    
+    int main(int argc, char **argv)
+    {
+        char *base = NULL;
+        if (argc > 1) {
+            base = argv[1];
+        }
+
+        /* In 'user' mode this path must be the same we will use to start QEMU. */
+        if (qemu_hypertrace_init(base) != 0) {
+            fprintf(stderr, "error: qemu_hypertrace_init: %s\n", strerror(errno));
+            abort();
+        }
+    
+        /* Set event arguments */
+        uint64_t voffset  = 0;
+        uint64_t *data = qemu_hypertrace_data(voffset);
+        data[0] = 0xcafe;
+        data[1] = 0xbabe;
+        data[2] = 0xdead;
+        data[3] = 0xbeef;
+    
+        /* Emit event */
+        printf("emitting hypertrace event\n");
+        qemu_hypertrace(voffset);
+    }
+    EOF
+
+    gcc -o /tmp/my-hypertrace-user /tmp/my-hypertrace.c                                    \
+        /tmp/qemu-build/x86_64-linux-user/hypertrace/guest/user/libqemu-hypertrace-guest.a \
+        -I/tmp/qemu-install/include
+
+    gcc -o /tmp/my-hypertrace-softmmu /tmp/my-hypertrace.c                              \
+        /tmp/qemu-build/x86_64-softmmu/hypertrace/guest/user/libqemu-hypertrace-guest.a \
+        -I/tmp/qemu-install/include
+
+4. Run a guest with access to QEMU's hypertrace:
+
+    /tmp/qemu-install/bin/qemu-x86_64          \
+      -hypertrace /tmp/hypertrace              \
+      -trace enable=guest* -D /dev/stdout      \
+      /tmp/my-hypertrace-user /tmp/hypertrace
+
+   Or, to run in 'system' mode:
+
+    /tmp/qemu-install/x86_64-softmmu/qemu-system-x86_64 \
+      -device hypertrace                                \
+      -trace enable=guest* -D /dev/stdout               \
+      ...
+
+   And inside the VM:
+
+    sudo /tmp/my-hypertrace-softmmu
+
+   You can also use hypertrace from Linux's kernel code with the provided module
+   (see the header in "/tmp/qemu-install/include/linux/qemu-hypertrace.h"):
+
+    sudo insmod /tmp/qemu-build/x86_64-softmmu/hypertrace/guest/linux-module/qemu-hypertrace.ko
+
+== Details ==
+
+To make it more efficient in terms of guest and host time, hypertrace provides
+two different memory areas (channels).
+
+The control channel is used by the guest to tell QEMU that new data is ready to
+be processed in the data channel. Writes to the control channel are intercepted
+by QEMU, which emits the "hypertrace" tracing event.
+
+The data channel is a regular memory buffer used by the guest to write the event
+arguments before raising the event through the control channel.
+
+The data channel is a physical memory region used by all virtual CPUs. To allow
+multiple guest threads or virtual CPUs to use hypertrace concurrently, the value
+passed on the control channel is used as an index to the data channel (i.e.,
+each guest thread or virtual CPU must write on a different portion of the data
+channel).
diff --git a/docs/tracing.txt b/docs/tracing.txt
index 29f2f9a..f312596 100644
--- a/docs/tracing.txt
+++ b/docs/tracing.txt
@@ -5,6 +5,9 @@
 This document describes the tracing infrastructure in QEMU and how to use it
 for debugging, profiling, and observing execution.
 
+See "docs/hypertrace.txt" to correlate guest tracing events with those in the
+QEMU host.
+
 == Quickstart ==
 
 1. Build with the 'simple' trace backend:

Re: [Qemu-devel] [PATCH 1/6] hypertrace: Add documentation

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 2590 bytes --]

On 08/05/2016 10:59 AM, Lluís Vilanova wrote:
> Signed-off-by: Lluís Vilanova <vilanova@ac.upc.edu>
> ---
>  docs/hypertrace.txt |  141 +++++++++++++++++++++++++++++++++++++++++++++++++++
>  docs/tracing.txt    |    3 +
>  2 files changed, 144 insertions(+)
>  create mode 100644 docs/hypertrace.txt
> 
> diff --git a/docs/hypertrace.txt b/docs/hypertrace.txt
> new file mode 100644
> index 0000000..4a31bcd
> --- /dev/null
> +++ b/docs/hypertrace.txt
> @@ -0,0 +1,141 @@
> += Hypertrace channel =

No explicit copyright means that this document inherits the project
default of GPLv2+. If that's not what you intended (or to be clear that
it IS what you intended), you may want to add a copyright and license blurb.

> +
> +The hypertrace channel allows guest code to emit events in QEMU (the host) using
> +its tracing infrastructure (see "docs/trace.txt"). This works in both 'system'
> +and 'user' modes. That is, hypertrace is to tracing, what hypercalls are to
> +system calls.
> +
> +You can use this to emit an event on both guest and QEMU (host) traces to easily
> +synchronize or correlate them. You could also modify you guest's tracing system
> +to emit all events through the hypertrace channel, providing a unified and fully
> +synchronized trace log. Another use case is timing the performance of guest code
> +when optimizing TCG (QEMU traces have a timestamp).

Do you need to state that this channel should only be opened up to
trusted guests, as otherwise it represents a security hole that a guest
can cause a host denial of service by emitting events as fast as possible?


> +3. Create a guest application using "qemu-hypertrace.h":
> +
> +    cat > /tmp/my-hypertrace.c <<EOF

I'd suggest <<\EOF here, to make it obvious that we don't want any $foo
parameter expansion in the heredoc.

> +    #include <stdio.h>
> +    #include <errno.h>
> +    #include <stdlib.h>
> +    #include <string.h>
> +    #include <qemu-hypertrace.h>
> +    
> +    
> +    int main(int argc, char **argv)
> +    {
> +        char *base = NULL;
> +        if (argc > 1) {
> +            base = argv[1];
> +        }
> +
> +        /* In 'user' mode this path must be the same we will use to start QEMU. */
> +        if (qemu_hypertrace_init(base) != 0) {
> +            fprintf(stderr, "error: qemu_hypertrace_init: %s\n", strerror(errno));

Worth using perror() in this example code?

Overall looks reasonable.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH 1/6] hypertrace: Add documentation

[Lluís Vilanova]

Eric Blake writes:

> On 08/05/2016 10:59 AM, Lluís Vilanova wrote:
>> Signed-off-by: Lluís Vilanova <vilanova@ac.upc.edu>
>> ---
>> docs/hypertrace.txt |  141 +++++++++++++++++++++++++++++++++++++++++++++++++++
>> docs/tracing.txt    |    3 +
>> 2 files changed, 144 insertions(+)
>> create mode 100644 docs/hypertrace.txt
>> 
>> diff --git a/docs/hypertrace.txt b/docs/hypertrace.txt
>> new file mode 100644
>> index 0000000..4a31bcd
>> --- /dev/null
>> +++ b/docs/hypertrace.txt
>> @@ -0,0 +1,141 @@
>> += Hypertrace channel =

> No explicit copyright means that this document inherits the project
> default of GPLv2+. If that's not what you intended (or to be clear that
> it IS what you intended), you may want to add a copyright and license blurb.

Oh, I just followed what's on other docs. I can add a copyright notice on top.


>> +
>> +The hypertrace channel allows guest code to emit events in QEMU (the host) using
>> +its tracing infrastructure (see "docs/trace.txt"). This works in both 'system'
>> +and 'user' modes. That is, hypertrace is to tracing, what hypercalls are to
>> +system calls.
>> +
>> +You can use this to emit an event on both guest and QEMU (host) traces to easily
>> +synchronize or correlate them. You could also modify you guest's tracing system
>> +to emit all events through the hypertrace channel, providing a unified and fully
>> +synchronized trace log. Another use case is timing the performance of guest code
>> +when optimizing TCG (QEMU traces have a timestamp).

> Do you need to state that this channel should only be opened up to
> trusted guests, as otherwise it represents a security hole that a guest
> can cause a host denial of service by emitting events as fast as possible?

Mmmm, for user-mode ({linux,bsd}-user), that is not a problem, since there is a
single application running. In the softmmu case, it is already "protected" by
the OS filesystem permissions, which won't give you write access to a device by
default (unless you're root). I can certainly clarify that if that's what you
meant.


>> +3. Create a guest application using "qemu-hypertrace.h":
>> +
>> +    cat > /tmp/my-hypertrace.c <<EOF

> I'd suggest <<\EOF here, to make it obvious that we don't want any $foo
> parameter expansion in the heredoc.

Got it.


>> +    #include <stdio.h>
>> +    #include <errno.h>
>> +    #include <stdlib.h>
>> +    #include <string.h>
>> +    #include <qemu-hypertrace.h>
>> +    
>> +    
>> +    int main(int argc, char **argv)
>> +    {
>> +        char *base = NULL;
>> +        if (argc > 1) {
>> +            base = argv[1];
>> +        }
>> +
>> +        /* In 'user' mode this path must be the same we will use to start QEMU. */
>> +        if (qemu_hypertrace_init(base) != 0) {
>> +            fprintf(stderr, "error: qemu_hypertrace_init: %s\n", strerror(errno));

> Worth using perror() in this example code?

Yes, thanks.


> Overall looks reasonable.

Thanks,
  Lluis

[Qemu-devel] [PATCH 2/6] hypertrace: Add tracing event "guest_hypertrace"

[Lluís Vilanova]

Generates the "guest_hypertrace" event with a user-configurable number
of arguments.

Signed-off-by: Lluís Vilanova <vilanova@ac.upc.edu>
---
 Makefile.objs       |    2 ++
 configure           |   36 ++++++++++++++++++++++++++++++++++++
 trace/Makefile.objs |    2 +-
 3 files changed, 39 insertions(+), 1 deletion(-)

diff --git a/Makefile.objs b/Makefile.objs
index 7f1f0a3..1c1b03c 100644
--- a/Makefile.objs
+++ b/Makefile.objs
@@ -158,3 +158,5 @@ trace-events-y += target-s390x/trace-events
 trace-events-y += target-ppc/trace-events
 trace-events-y += qom/trace-events
 trace-events-y += linux-user/trace-events
+
+trace-events-gen-y = hypertrace/trace-events
diff --git a/configure b/configure
index 5ada56d..e80fde4 100755
--- a/configure
+++ b/configure
@@ -273,6 +273,7 @@ pie=""
 qom_cast_debug="yes"
 trace_backends="log"
 trace_file="trace"
+hypertrace="disabled"
 spice=""
 rbd=""
 smartcard=""
@@ -782,6 +783,8 @@ for opt do
   ;;
   --with-trace-file=*) trace_file="$optarg"
   ;;
+  --with-hypertrace-args=*) hypertrace="$optarg"
+  ;;
   --enable-gprof) gprof="yes"
   ;;
   --enable-gcov) gcov="yes"
@@ -1300,6 +1303,8 @@ Advanced options (experts only):
                            Available backends: $($python $source_path/scripts/tracetool.py --list-backends)
   --with-trace-file=NAME   Full PATH,NAME of file to store traces
                            Default:trace-<pid>
+  --with-hypertrace-args=NUMBER
+                           number of hypertrace arguments (default: disabled)
   --disable-slirp          disable SLIRP userspace network connectivity
   --enable-tcg-interpreter enable TCG with bytecode interpreter (TCI)
   --oss-lib                path to OSS library
@@ -4197,6 +4202,14 @@ if test "$?" -ne 0 ; then
 fi
 
 ##########################################
+# check hypertrace arguments
+case "$hypertrace" in
+    disabled) ;;
+    ''|*[!0-9]*) error_exit "invalid number of hypertrace arguments" ;;
+    *) ;;
+esac
+
+##########################################
 # For 'ust' backend, test if ust headers are present
 if have_backend "ust"; then
   cat > $TMPC << EOF
@@ -4862,6 +4875,7 @@ echo "Trace backends    $trace_backends"
 if have_backend "simple"; then
 echo "Trace output file $trace_file-<pid>"
 fi
+echo "Hypertrace arguments  $hypertrace"
 echo "spice support     $spice $(echo_version $spice $spice_protocol_version/$spice_server_version)"
 echo "rbd support       $rbd"
 echo "xfsctl support    $xfs"
@@ -5490,6 +5504,28 @@ else
 fi
 QEMU_INCLUDES="-I\$(SRC_PATH)/tcg $QEMU_INCLUDES"
 
+# hypertrace
+hyperargs=$hypertrace
+if test $hypertrace = "disabled"; then
+    hyperargs=0
+fi
+echo "CONFIG_HYPERTRACE_ARGS=$hyperargs" >> $config_host_mak
+hypertrace_events=hypertrace/trace-events
+mkdir -p $(dirname $hypertrace_events)
+echo "# See docs/trace-events.txt for syntax documentation." >$hypertrace_events
+echo -n 'vcpu guest_hypertrace(' >>$hypertrace_events
+for i in `seq $hypertrace`; do
+    if test $i != 1; then
+        echo -n ", " >>$hypertrace_events
+    fi
+    echo -n "uint64_t arg$i" >>$hypertrace_events
+done
+echo -n ') ' >>$hypertrace_events
+for i in `seq $hypertrace`; do
+    echo -n "\" arg$i=0x%016\"PRIx64" >>$hypertrace_events
+done
+echo >>$hypertrace_events
+
 echo "TOOLS=$tools" >> $config_host_mak
 echo "ROMS=$roms" >> $config_host_mak
 echo "MAKE=$make" >> $config_host_mak
diff --git a/trace/Makefile.objs b/trace/Makefile.objs
index 4d91b3b..b71ec54 100644
--- a/trace/Makefile.objs
+++ b/trace/Makefile.objs
@@ -8,7 +8,7 @@
 tracetool-y = $(SRC_PATH)/scripts/tracetool.py
 tracetool-y += $(shell find $(SRC_PATH)/scripts/tracetool -name "*.py")
 
-$(BUILD_DIR)/trace-events-all: $(trace-events-y:%=$(SRC_PATH)/%)
+$(BUILD_DIR)/trace-events-all: $(trace-events-y:%=$(SRC_PATH)/%) $(trace-events-gen-y:%=$(BUILD_DIR)/%)
 	$(call quiet-command,cat $^ > $@)
 
 ######################################################################

Re: [Qemu-devel] [PATCH 2/6] hypertrace: Add tracing event "guest_hypertrace"

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1427 bytes --]

On Fri, Aug 05, 2016 at 06:59:34PM +0200, Lluís Vilanova wrote:
> +# hypertrace
> +hyperargs=$hypertrace
> +if test $hypertrace = "disabled"; then
> +    hyperargs=0
> +fi
> +echo "CONFIG_HYPERTRACE_ARGS=$hyperargs" >> $config_host_mak
> +hypertrace_events=hypertrace/trace-events
> +mkdir -p $(dirname $hypertrace_events)
> +echo "# See docs/trace-events.txt for syntax documentation." >$hypertrace_events
> +echo -n 'vcpu guest_hypertrace(' >>$hypertrace_events
> +for i in `seq $hypertrace`; do
> +    if test $i != 1; then
> +        echo -n ", " >>$hypertrace_events
> +    fi
> +    echo -n "uint64_t arg$i" >>$hypertrace_events
> +done
> +echo -n ') ' >>$hypertrace_events
> +for i in `seq $hypertrace`; do
> +    echo -n "\" arg$i=0x%016\"PRIx64" >>$hypertrace_events
> +done
> +echo >>$hypertrace_events

This reminds me of the first versions of "simpletrace" where the number
of arguments was fixed and argument size was fixed.

This meant strings cannot be traced, number of arguments is limited, and
you pay an space overhead for unused arguments.

Later on the format was changed to header (including .length field) and
binary data payload.  This reduced the space overhead, elminated the
argument count limit, and allowed strings to be traced.

I think these are desirable qualities for any tracing mechanism and
would reconsider a fixed number of uint64_t arguments.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [PATCH 2/6] hypertrace: Add tracing event "guest_hypertrace"

[Lluís Vilanova]

Stefan Hajnoczi writes:

> On Fri, Aug 05, 2016 at 06:59:34PM +0200, Lluís Vilanova wrote:
>> +# hypertrace
>> +hyperargs=$hypertrace
>> +if test $hypertrace = "disabled"; then
>> +    hyperargs=0
>> +fi
>> +echo "CONFIG_HYPERTRACE_ARGS=$hyperargs" >> $config_host_mak
>> +hypertrace_events=hypertrace/trace-events
>> +mkdir -p $(dirname $hypertrace_events)
>> +echo "# See docs/trace-events.txt for syntax documentation." >$hypertrace_events
>> +echo -n 'vcpu guest_hypertrace(' >>$hypertrace_events
>> +for i in `seq $hypertrace`; do
>> +    if test $i != 1; then
>> +        echo -n ", " >>$hypertrace_events
>> +    fi
>> +    echo -n "uint64_t arg$i" >>$hypertrace_events
>> +done
>> +echo -n ') ' >>$hypertrace_events
>> +for i in `seq $hypertrace`; do
>> +    echo -n "\" arg$i=0x%016\"PRIx64" >>$hypertrace_events
>> +done
>> +echo >>$hypertrace_events

> This reminds me of the first versions of "simpletrace" where the number
> of arguments was fixed and argument size was fixed.

> This meant strings cannot be traced, number of arguments is limited, and
> you pay an space overhead for unused arguments.

> Later on the format was changed to header (including .length field) and
> binary data payload.  This reduced the space overhead, elminated the
> argument count limit, and allowed strings to be traced.

> I think these are desirable qualities for any tracing mechanism and
> would reconsider a fixed number of uint64_t arguments.

The number of arguments is fixed at *compilation time*. What is not supported
out of the box is multiplexing multiple guest events over the hypertrace
event. That is, you need to do it yourself when post-processing the traces, but
I don't think that's too bad.


Cheers,
  Lluis

[Qemu-devel] [PATCH 3/6] hypertrace: [*-user] Add QEMU-side proxy to "guest_hypertrace" event

[Lluís Vilanova]

QEMU detects when the guest uses 'mmap' on hypertrace's control channel
file, and then uses 'mprotect' to detect accesses to it, which are used
to trigger traceing event "guest_hypertrace".

Signed-off-by: Lluís Vilanova <vilanova@ac.upc.edu>
---
 Makefile.objs            |    4 +
 bsd-user/main.c          |   16 +++
 bsd-user/mmap.c          |    2 
 bsd-user/syscall.c       |    4 +
 hypertrace/Makefile.objs |   17 +++
 hypertrace/user.c        |  292 ++++++++++++++++++++++++++++++++++++++++++++++
 hypertrace/user.h        |   52 ++++++++
 linux-user/main.c        |   19 +++
 linux-user/mmap.c        |    2 
 linux-user/syscall.c     |    3 
 10 files changed, 411 insertions(+)
 create mode 100644 hypertrace/Makefile.objs
 create mode 100644 hypertrace/user.c
 create mode 100644 hypertrace/user.h

diff --git a/Makefile.objs b/Makefile.objs
index 1c1b03c..f598e0e 100644
--- a/Makefile.objs
+++ b/Makefile.objs
@@ -104,6 +104,10 @@ util-obj-y +=  trace/
 target-obj-y += trace/
 
 ######################################################################
+# hypertrace
+target-obj-y += hypertrace/
+
+######################################################################
 # guest agent
 
 # FIXME: a few definitions from qapi-types.o/qapi-visit.o are needed
diff --git a/bsd-user/main.c b/bsd-user/main.c
index 315ba1d..9721240 100644
--- a/bsd-user/main.c
+++ b/bsd-user/main.c
@@ -30,9 +30,12 @@
 #include "tcg.h"
 #include "qemu/timer.h"
 #include "qemu/envlist.h"
+#include "qemu/error-report.h"
 #include "exec/log.h"
 #include "trace/control.h"
 #include "glib-compat.h"
+#include "hypertrace/user.h"
+
 
 int singlestep;
 unsigned long mmap_min_addr;
@@ -692,6 +695,8 @@ static void usage(void)
            "-strace           log system calls\n"
            "-trace            [[enable=]<pattern>][,events=<file>][,file=<file>]\n"
            "                  specify tracing options\n"
+           "-hypertrace       [[base=]<path>][,pages=<int>]\n"
+           "                  specify hypertrace options\n"
            "\n"
            "Environment variables:\n"
            "QEMU_STRACE       Print system calls and arguments similar to the\n"
@@ -742,6 +747,8 @@ int main(int argc, char **argv)
     envlist_t *envlist = NULL;
     char *trace_file = NULL;
     bsd_type = target_openbsd;
+    char *hypertrace_base = NULL;
+    uint64_t hypertrace_size = 0;
 
     if (argc <= 1)
         usage();
@@ -761,6 +768,7 @@ int main(int argc, char **argv)
     cpu_model = NULL;
 
     qemu_add_opts(&qemu_trace_opts);
+    qemu_add_opts(&qemu_hypertrace_opts);
 
     optind = 1;
     for (;;) {
@@ -851,6 +859,9 @@ int main(int argc, char **argv)
         } else if (!strcmp(r, "trace")) {
             g_free(trace_file);
             trace_file = trace_opt_parse(optarg);
+        } else if (!strcmp(r, "hypertrace")) {
+            g_free(hypertrace_file);
+            hypertrace_opt_parse(optarg, &hypertrace_base, &hypertrace_size);
         } else {
             usage();
         }
@@ -985,6 +996,11 @@ int main(int argc, char **argv)
     target_set_brk(info->brk);
     syscall_init();
     signal_init();
+    if (atexit(hypertrace_fini) != 0) {
+        fprintf(stderr, "error: atexit: %s\n", strerror(errno));
+        abort();
+    }
+    hypertrace_init(hypertrace_base, hypertrace_size);
 
     /* Now that we've loaded the binary, GUEST_BASE is fixed.  Delay
        generating the prologue until now so that the prologue can take
diff --git a/bsd-user/mmap.c b/bsd-user/mmap.c
index 610f91b..faf255f 100644
--- a/bsd-user/mmap.c
+++ b/bsd-user/mmap.c
@@ -21,6 +21,7 @@
 #include "qemu.h"
 #include "qemu-common.h"
 #include "bsd-mman.h"
+#include "hypertrace/user.h"
 
 //#define DEBUG_MMAP
 
@@ -407,6 +408,7 @@ abi_long target_mmap(abi_ulong start, abi_ulong len, int prot,
         }
     }
  the_end1:
+    hypertrace_guest_mmap(fd, (void *)g2h(start));
     page_set_flags(start, start + len, prot | PAGE_VALID);
  the_end:
 #ifdef DEBUG_MMAP
diff --git a/bsd-user/syscall.c b/bsd-user/syscall.c
index 66492aa..275ef61 100644
--- a/bsd-user/syscall.c
+++ b/bsd-user/syscall.c
@@ -26,6 +26,7 @@
 
 #include "qemu.h"
 #include "qemu-common.h"
+#include "hypertrace/user.h"
 
 //#define DEBUG
 
@@ -332,6 +333,7 @@ abi_long do_freebsd_syscall(void *cpu_env, int num, abi_long arg1,
         _mcleanup();
 #endif
         gdb_exit(cpu_env, arg1);
+        hypertrace_fini();
         /* XXX: should free thread stack and CPU env */
         _exit(arg1);
         ret = 0; /* avoid warning */
@@ -430,6 +432,7 @@ abi_long do_netbsd_syscall(void *cpu_env, int num, abi_long arg1,
         _mcleanup();
 #endif
         gdb_exit(cpu_env, arg1);
+        hypertrace_fini();
         /* XXX: should free thread stack and CPU env */
         _exit(arg1);
         ret = 0; /* avoid warning */
@@ -505,6 +508,7 @@ abi_long do_openbsd_syscall(void *cpu_env, int num, abi_long arg1,
         _mcleanup();
 #endif
         gdb_exit(cpu_env, arg1);
+        hypertrace_fini();
         /* XXX: should free thread stack and CPU env */
         _exit(arg1);
         ret = 0; /* avoid warning */
diff --git a/hypertrace/Makefile.objs b/hypertrace/Makefile.objs
new file mode 100644
index 0000000..6eb5acf
--- /dev/null
+++ b/hypertrace/Makefile.objs
@@ -0,0 +1,17 @@
+# -*- mode: makefile -*-
+
+target-obj-$(CONFIG_USER_ONLY) += user.o
+
+$(obj)/user.o: $(obj)/emit.c
+
+$(obj)/emit.c: $(obj)/emit.c-timestamp $(BUILD_DIR)/config-host.mak
+	@cmp $< $@ >/dev/null 2>&1 || cp $< $@
+$(obj)/emit.c-timestamp: $(BUILD_DIR)/config-host.mak
+	@echo "static void hypertrace_emit(CPUState *cpu, uint64_t *data)" >$@
+	@echo "{" >>$@
+	@echo -n "    trace_guest_hypertrace(cpu" >>$@
+	@for i in `seq $(CONFIG_HYPERTRACE_ARGS)`; do \
+	    echo -n ", data[$$i-1]" >>$@; \
+	done
+	@echo ");" >>$@
+	@echo "}" >>$@
diff --git a/hypertrace/user.c b/hypertrace/user.c
new file mode 100644
index 0000000..55df79d
--- /dev/null
+++ b/hypertrace/user.c
@@ -0,0 +1,292 @@
+/*
+ * QEMU-side management of hypertrace in user-level emulation.
+ *
+ * Copyright (C) 2016 Lluís Vilanova <vilanova@ac.upc.edu>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+/*
+ * Implementation details
+ * ======================
+ *
+ * Both channels are provided as regular files in the host system, which must be
+ * mmap'ed by the guest application.
+ *
+ * Data channel
+ * ------------
+ *
+ * The guest must mmap a file named <base>-data, where base is the argument
+ * given to hypertrace_init.
+ *
+ * Regular memory accesses are used on the data channel.
+ *
+ * Control channel
+ * ---------------
+ *
+ * The guest must mmap a file named <base>-control, where base is the argument
+ * given to hypertrace_init.
+ *
+ * The first 64 bits of that memory contain the size of the data channel.
+ *
+ * The control channel is mprotect'ed by QEMU so that guest writes can be
+ * intercepted by QEMU in order to raise the "guest_hypertrace" tracing
+ * event. The guest must perform writes twice, one on each of two consecutive
+ * pages, so that the written data can be both read by QEMU and the access
+ * intercepted.
+ */
+
+#include <assert.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <stdint.h>
+#include <string.h>
+#include <sys/mman.h>
+
+#include "qemu/osdep.h"
+#include "cpu.h"
+
+#include "hypertrace/user.h"
+#include "qemu/config-file.h"
+#include "qemu/error-report.h"
+#include "trace.h"
+
+
+static char *data_path = NULL;
+static char *control_path = NULL;
+static int data_fd = -1;
+static int control_fd = -1;
+
+static uint64_t *qemu_data = NULL;
+static void *qemu_control_0 = NULL;
+static void *qemu_control_1 = NULL;
+
+static struct stat control_fd_stat;
+
+struct sigaction segv_next;
+static void segv_handler(int signum, siginfo_t *siginfo, void *sigctxt);
+
+
+QemuOptsList qemu_hypertrace_opts = {
+    .name = "hypertrace",
+    .implied_opt_name = "path",
+    .head = QTAILQ_HEAD_INITIALIZER(qemu_hypertrace_opts.head),
+    .desc = {
+        {
+            .name = "path",
+            .type = QEMU_OPT_STRING,
+        },
+        {
+            .name = "pages",
+            .type = QEMU_OPT_NUMBER,
+            .def_value_str = "1",
+        },
+        { /* end of list */ }
+    },
+};
+
+void hypertrace_opt_parse(const char *optarg, char **base, size_t *size)
+{
+    int pages;
+    QemuOpts *opts = qemu_opts_parse_noisily(qemu_find_opts("hypertrace"),
+                                             optarg, true);
+    if (!opts) {
+        exit(1);
+    }
+    if (qemu_opt_get(opts, "path")) {
+        *base = g_strdup(qemu_opt_get(opts, "path"));
+    } else {
+        *base = NULL;
+    }
+    pages = qemu_opt_get_number(opts, "pages", 1);
+    if (pages <= 0) {
+        error_report("Parameter 'pages' expects a positive number");
+        exit(EXIT_FAILURE);
+    }
+    *size = pages * TARGET_PAGE_SIZE;
+}
+
+static void init_channel(const char *base, const char *suffix, size_t size,
+                         char ** path, int *fd, uint64_t **addr)
+{
+    *path = g_malloc(strlen(base) + strlen(suffix) + 1);
+    sprintf(*path, "%s%s", base, suffix);
+
+    *fd = open(*path, O_CREAT | O_EXCL | O_RDWR, S_IRUSR | S_IWUSR);
+    if (*fd == -1) {
+        error_report("error: open(%s): %s", *path, strerror(errno));
+        abort();
+    }
+
+    off_t lres = lseek(*fd, size - 1, SEEK_SET);
+    if (lres == (off_t)-1) {
+        error_report("error: lseek(%s): %s", *path, strerror(errno));
+        abort();
+    }
+
+    char tmp;
+    ssize_t wres = write(*fd, &tmp, 1);
+    if (wres == -1) {
+        error_report("error: write(%s): %s", *path, strerror(errno));
+        abort();
+    }
+
+    if (addr) {
+        *addr = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, *fd, 0);
+        if (*addr == MAP_FAILED) {
+            error_report("error: mmap(%s): %s", *path, strerror(errno));
+            abort();
+        }
+    }
+}
+
+static void fini_handler(int signum, siginfo_t *siginfo, void *sigctxt)
+{
+    hypertrace_fini();
+}
+
+void hypertrace_init(const char *base, uint64_t data_size)
+{
+    if (base == NULL) {
+        return;
+    }
+
+    struct sigaction sigint;
+    memset(&sigint, 0, sizeof(sigint));
+    sigint.sa_sigaction = fini_handler;
+    sigint.sa_flags = SA_SIGINFO | SA_RESTART;
+    if (sigaction(SIGINT, &sigint, NULL) != 0) {
+        error_report("error: sigaction(SIGINT): %s", strerror(errno));
+        abort();
+    }
+    if (sigaction(SIGABRT, &sigint, NULL) != 0) {
+        error_report("error: sigaction(SIGABRT): %s", strerror(errno));
+        abort();
+    }
+
+    init_channel(base, "-data", data_size, &data_path, &data_fd, &qemu_data);
+    uint64_t *control;
+    init_channel(base, "-control", getpagesize() * 2, &control_path, &control_fd, &control);
+
+    control[0] = tswap64(data_size / (CONFIG_HYPERTRACE_ARGS * sizeof(uint64_t)));
+
+    if (fstat(control_fd, &control_fd_stat) == -1) {
+        error_report("error: fstat(hypertrace_control): %s", strerror(errno));
+        abort();
+    }
+
+    struct sigaction segv;
+    memset(&segv, 0, sizeof(segv));
+    segv.sa_sigaction = segv_handler;
+    segv.sa_flags = SA_SIGINFO | SA_RESTART;
+    sigemptyset(&segv.sa_mask);
+
+    if (sigaction(SIGSEGV, &segv, &segv_next) != 0) {
+        error_report("error: sigaction(SIGSEGV): %s", strerror(errno));
+        abort();
+    }
+}
+
+
+static void fini_channel(int *fd, char **path)
+{
+    if (*fd != -1) {
+        if (close(*fd) == -1) {
+            error_report("error: close: %s", strerror(errno));
+            abort();
+        }
+        if (unlink(*path) == -1) {
+            error_report("error: unlink(%s): %s", *path, strerror(errno));
+            abort();
+        }
+        *fd = -1;
+    }
+    if (*path != NULL) {
+        g_free(*path);
+        *path =  NULL;
+    }
+}
+
+void hypertrace_fini(void)
+{
+    static bool atexit_in = false;
+    if (atexit_in) {
+        return;
+    }
+    atexit_in = true;
+
+    if (sigaction(SIGSEGV, &segv_next, NULL) != 0) {
+        error_report("error: sigaction(SIGSEGV): %s", strerror(errno));
+        abort();
+    }
+    fini_channel(&data_fd, &data_path);
+    fini_channel(&control_fd, &control_path);
+}
+
+
+void hypertrace_guest_mmap(int fd, void *qemu_addr)
+{
+    struct stat s;
+    if (fstat(fd, &s) != 0) {
+        return;
+    }
+
+    if (s.st_dev != control_fd_stat.st_dev ||
+        s.st_ino != control_fd_stat.st_ino) {
+        return;
+    }
+
+    /* it's an mmap of the control channel; split it in two and mprotect it to
+     * detect writes (cmd is written once on each part)
+     */
+    qemu_control_0 = qemu_addr;
+    qemu_control_1 = qemu_control_0 + getpagesize();
+    if (mprotect(qemu_control_0, getpagesize(), PROT_READ) == -1) {
+        error_report("error: mprotect(hypertrace_control): %s", strerror(errno));
+        abort();
+    }
+}
+
+static void swap_control(void *from, void *to)
+{
+    if (mprotect(from, getpagesize(), PROT_READ | PROT_WRITE) == -1) {
+        error_report("error: mprotect(from): %s", strerror(errno));
+        abort();
+    }
+    if (mprotect(to, getpagesize(), PROT_READ) == -1) {
+        error_report("error: mprotect(to): %s", strerror(errno));
+        abort();
+    }
+}
+
+#include "hypertrace/emit.c"
+
+static void segv_handler(int signum, siginfo_t *siginfo, void *sigctxt)
+{
+    if (qemu_control_0 <= siginfo->si_addr &&
+        siginfo->si_addr < qemu_control_1) {
+
+        /* 1st fault (guest will write cmd) */
+        assert(((unsigned long)siginfo->si_addr % getpagesize()) == sizeof(uint64_t));
+        swap_control(qemu_control_0, qemu_control_1);
+
+    } else if (qemu_control_1 <= siginfo->si_addr &&
+               siginfo->si_addr < qemu_control_1 + getpagesize()) {
+        uint64_t vcontrol = ((uint64_t*)qemu_control_0)[2];
+        uint64_t *data_ptr = &qemu_data[vcontrol * CONFIG_HYPERTRACE_ARGS * sizeof(uint64_t)];
+
+        /* 2nd fault (invoke) */
+        assert(((unsigned long)siginfo->si_addr % getpagesize()) == sizeof(uint64_t));
+        hypertrace_emit(current_cpu, data_ptr);
+        swap_control(qemu_control_1, qemu_control_0);
+
+    } else {
+        /* proxy to next handler */
+        if (segv_next.sa_sigaction != NULL) {
+            segv_next.sa_sigaction(signum, siginfo, sigctxt);
+        } else if (segv_next.sa_handler != NULL) {
+            segv_next.sa_handler(signum);
+        }
+    }
+}
diff --git a/hypertrace/user.h b/hypertrace/user.h
new file mode 100644
index 0000000..a13bae4
--- /dev/null
+++ b/hypertrace/user.h
@@ -0,0 +1,52 @@
+/*
+ * QEMU-side management of hypertrace in user-level emulation.
+ *
+ * Copyright (C) 2016 Lluís Vilanova <vilanova@ac.upc.edu>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include <stdint.h>
+#include <sys/types.h>
+
+
+/**
+ * Definition of QEMU options describing hypertrace subsystem configuration
+ */
+extern QemuOptsList qemu_hypertrace_opts;
+
+/**
+ * hypertrace_opt_parse:
+ * @optarg: Input arguments.
+ * @base: Output base path for the hypertrace channel files.
+ * @data_size: Output length in bytes for the data channel.
+ *
+ * Parse the commandline arguments for hypertrace.
+ */
+void hypertrace_opt_parse(const char *optarg, char **base, size_t *size);
+
+/**
+ * hypertrace_init:
+ * @base: Base path for the hypertrace channel files.
+ * @data_size: Length in bytes for the data channel.
+ *
+ * Initialize the backing files for the hypertrace channel.
+ */
+void hypertrace_init(const char *base, uint64_t data_size);
+
+/**
+ * hypertrace_guest_mmap:
+ *
+ * Check if this mmap is for the control channel and act accordingly.
+ *
+ * Precondition: defined(CONFIG_USER_ONLY)
+ */
+void hypertrace_guest_mmap(int fd, void *qemu_addr);
+
+/**
+ * hypertrace_fini:
+ *
+ * Remove the backing files for the hypertrace channel.
+ */
+void hypertrace_fini(void);
diff --git a/linux-user/main.c b/linux-user/main.c
index 462e820..8f3d9cf 100644
--- a/linux-user/main.c
+++ b/linux-user/main.c
@@ -32,10 +32,12 @@
 #include "tcg.h"
 #include "qemu/timer.h"
 #include "qemu/envlist.h"
+#include "qemu/error-report.h"
 #include "elf.h"
 #include "exec/log.h"
 #include "trace/control.h"
 #include "glib-compat.h"
+#include "hypertrace/user.h"
 
 char *exec_path;
 
@@ -4011,6 +4013,14 @@ static void handle_arg_trace(const char *arg)
     trace_file = trace_opt_parse(arg);
 }
 
+static char *hypertrace_base;
+static size_t hypertrace_size;
+static void handle_arg_hypertrace(const char *arg)
+{
+    g_free(hypertrace_base);
+    hypertrace_opt_parse(arg, &hypertrace_base, &hypertrace_size);
+}
+
 struct qemu_argument {
     const char *argv;
     const char *env;
@@ -4060,6 +4070,8 @@ static const struct qemu_argument arg_table[] = {
      "",           "Seed for pseudo-random number generator"},
     {"trace",      "QEMU_TRACE",       true,  handle_arg_trace,
      "",           "[[enable=]<pattern>][,events=<file>][,file=<file>]"},
+    {"hypertrace", "QEMU_HYPERTRACE",  true,  handle_arg_hypertrace,
+     "",           "[[base=]<path>][,pages=<int>]"},
     {"version",    "QEMU_VERSION",     false, handle_arg_version,
      "",           "display version information and exit"},
     {NULL, NULL, false, NULL, NULL, NULL}
@@ -4250,6 +4262,7 @@ int main(int argc, char **argv, char **envp)
     srand(time(NULL));
 
     qemu_add_opts(&qemu_trace_opts);
+    qemu_add_opts(&qemu_hypertrace_opts);
 
     optind = parse_args(argc, argv);
 
@@ -4448,6 +4461,12 @@ int main(int argc, char **argv, char **envp)
     syscall_init();
     signal_init();
 
+    if (atexit(hypertrace_fini)) {
+        fprintf(stderr, "error: atexit: %s\n", strerror(errno));
+        abort();
+    }
+    hypertrace_init(hypertrace_base, hypertrace_size);
+
     /* Now that we've loaded the binary, GUEST_BASE is fixed.  Delay
        generating the prologue until now so that the prologue can take
        the real value of GUEST_BASE into account.  */
diff --git a/linux-user/mmap.c b/linux-user/mmap.c
index c4371d9..3207d98 100644
--- a/linux-user/mmap.c
+++ b/linux-user/mmap.c
@@ -23,6 +23,7 @@
 #include "qemu.h"
 #include "qemu-common.h"
 #include "translate-all.h"
+#include "hypertrace/user.h"
 
 //#define DEBUG_MMAP
 
@@ -553,6 +554,7 @@ abi_long target_mmap(abi_ulong start, abi_ulong len, int prot,
         }
     }
  the_end1:
+    hypertrace_guest_mmap(fd, (void *)g2h(start));
     page_set_flags(start, start + len, prot | PAGE_VALID);
  the_end:
 #ifdef DEBUG_MMAP
diff --git a/linux-user/syscall.c b/linux-user/syscall.c
index ca6a2b4..e73ec5d 100644
--- a/linux-user/syscall.c
+++ b/linux-user/syscall.c
@@ -111,6 +111,7 @@ int __clone2(int (*fn)(void *), void *child_stack_base,
 #include "uname.h"
 
 #include "qemu.h"
+#include "hypertrace/user.h"
 
 #define CLONE_NPTL_FLAGS2 (CLONE_SETTLS | \
     CLONE_PARENT_SETTID | CLONE_CHILD_SETTID | CLONE_CHILD_CLEARTID)
@@ -7214,6 +7215,7 @@ abi_long do_syscall(void *cpu_env, int num, abi_long arg1,
         _mcleanup();
 #endif
         gdb_exit(cpu_env, arg1);
+        hypertrace_fini();
         _exit(arg1);
         ret = 0; /* avoid warning */
         break;
@@ -9219,6 +9221,7 @@ abi_long do_syscall(void *cpu_env, int num, abi_long arg1,
         _mcleanup();
 #endif
         gdb_exit(cpu_env, arg1);
+        hypertrace_fini();
         ret = get_errno(exit_group(arg1));
         break;
 #endif

Re: [Qemu-devel] [PATCH 3/6] hypertrace: [*-user] Add QEMU-side proxy to "guest_hypertrace" event

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 860 bytes --]

On 08/05/2016 10:59 AM, Lluís Vilanova wrote:
> QEMU detects when the guest uses 'mmap' on hypertrace's control channel
> file, and then uses 'mprotect' to detect accesses to it, which are used
> to trigger traceing event "guest_hypertrace".

s/traceing/tracing/

I'll probably leave the technical review to others, though



> +++ b/bsd-user/mmap.c
> @@ -21,6 +21,7 @@
>  #include "qemu.h"
>  #include "qemu-common.h"
>  #include "bsd-mman.h"
> +#include "hypertrace/user.h"
>  
>  //#define DEBUG_MMAP
>  
> @@ -407,6 +408,7 @@ abi_long target_mmap(abi_ulong start, abi_ulong len, int prot,
>          }
>      }
>   the_end1:
> +    hypertrace_guest_mmap(fd, (void *)g2h(start));

Why is the cast to void* needed?



-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH 3/6] hypertrace: [*-user] Add QEMU-side proxy to "guest_hypertrace" event

[Lluís Vilanova]

Eric Blake writes:

> On 08/05/2016 10:59 AM, Lluís Vilanova wrote:
>> QEMU detects when the guest uses 'mmap' on hypertrace's control channel
>> file, and then uses 'mprotect' to detect accesses to it, which are used
>> to trigger traceing event "guest_hypertrace".

> s/traceing/tracing/

> I'll probably leave the technical review to others, though

Thanks.


>> +++ b/bsd-user/mmap.c
>> @@ -21,6 +21,7 @@
>> #include "qemu.h"
>> #include "qemu-common.h"
>> #include "bsd-mman.h"
>> +#include "hypertrace/user.h"
>> 
>> //#define DEBUG_MMAP
>> 
>> @@ -407,6 +408,7 @@ abi_long target_mmap(abi_ulong start, abi_ulong len, int prot,
>> }
>> }
>> the_end1:
>> +    hypertrace_guest_mmap(fd, (void *)g2h(start));

> Why is the cast to void* needed?

That's unnecessary, my bad.


Thanks,
  Lluis

Re: [Qemu-devel] [PATCH 3/6] hypertrace: [*-user] Add QEMU-side proxy to "guest_hypertrace" event

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2246 bytes --]

On Fri, Aug 05, 2016 at 06:59:39PM +0200, Lluís Vilanova wrote:
> +static void init_channel(const char *base, const char *suffix, size_t size,
> +                         char ** path, int *fd, uint64_t **addr)
> +{
> +    *path = g_malloc(strlen(base) + strlen(suffix) + 1);
> +    sprintf(*path, "%s%s", base, suffix);

Use g_strdup_printf() instead.

> +static void swap_control(void *from, void *to)
> +{
> +    if (mprotect(from, getpagesize(), PROT_READ | PROT_WRITE) == -1) {
> +        error_report("error: mprotect(from): %s", strerror(errno));
> +        abort();
> +    }
> +    if (mprotect(to, getpagesize(), PROT_READ) == -1) {
> +        error_report("error: mprotect(to): %s", strerror(errno));
> +        abort();
> +    }
> +}
> +
> +#include "hypertrace/emit.c"
> +
> +static void segv_handler(int signum, siginfo_t *siginfo, void *sigctxt)
> +{
> +    if (qemu_control_0 <= siginfo->si_addr &&
> +        siginfo->si_addr < qemu_control_1) {
> +
> +        /* 1st fault (guest will write cmd) */
> +        assert(((unsigned long)siginfo->si_addr % getpagesize()) == sizeof(uint64_t));
> +        swap_control(qemu_control_0, qemu_control_1);
> +
> +    } else if (qemu_control_1 <= siginfo->si_addr &&
> +               siginfo->si_addr < qemu_control_1 + getpagesize()) {
> +        uint64_t vcontrol = ((uint64_t*)qemu_control_0)[2];
> +        uint64_t *data_ptr = &qemu_data[vcontrol * CONFIG_HYPERTRACE_ARGS * sizeof(uint64_t)];
> +
> +        /* 2nd fault (invoke) */
> +        assert(((unsigned long)siginfo->si_addr % getpagesize()) == sizeof(uint64_t));
> +        hypertrace_emit(current_cpu, data_ptr);
> +        swap_control(qemu_control_1, qemu_control_0);
> +
> +    } else {
> +        /* proxy to next handler */
> +        if (segv_next.sa_sigaction != NULL) {
> +            segv_next.sa_sigaction(signum, siginfo, sigctxt);
> +        } else if (segv_next.sa_handler != NULL) {
> +            segv_next.sa_handler(signum);
> +        }
> +    }
> +}

Can this approach be made thread-safe?

If not then it would be good to consider the problem right away and
switch to something that is thread-safe, even if it depends on the
target architecture.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [PATCH 3/6] hypertrace: [*-user] Add QEMU-side proxy to "guest_hypertrace" event

[Lluís Vilanova]

Stefan Hajnoczi writes:

> On Fri, Aug 05, 2016 at 06:59:39PM +0200, Lluís Vilanova wrote:
>> +static void init_channel(const char *base, const char *suffix, size_t size,
>> +                         char ** path, int *fd, uint64_t **addr)
>> +{
>> +    *path = g_malloc(strlen(base) + strlen(suffix) + 1);
>> +    sprintf(*path, "%s%s", base, suffix);

> Use g_strdup_printf() instead.

>> +static void swap_control(void *from, void *to)
>> +{
>> +    if (mprotect(from, getpagesize(), PROT_READ | PROT_WRITE) == -1) {
>> +        error_report("error: mprotect(from): %s", strerror(errno));
>> +        abort();
>> +    }
>> +    if (mprotect(to, getpagesize(), PROT_READ) == -1) {
>> +        error_report("error: mprotect(to): %s", strerror(errno));
>> +        abort();
>> +    }
>> +}
>> +
>> +#include "hypertrace/emit.c"
>> +
>> +static void segv_handler(int signum, siginfo_t *siginfo, void *sigctxt)
>> +{
>> +    if (qemu_control_0 <= siginfo->si_addr &&
>> +        siginfo->si_addr < qemu_control_1) {
>> +
>> +        /* 1st fault (guest will write cmd) */
>> +        assert(((unsigned long)siginfo->si_addr % getpagesize()) == sizeof(uint64_t));
>> +        swap_control(qemu_control_0, qemu_control_1);
>> +
>> +    } else if (qemu_control_1 <= siginfo->si_addr &&
>> +               siginfo->si_addr < qemu_control_1 + getpagesize()) {
>> +        uint64_t vcontrol = ((uint64_t*)qemu_control_0)[2];
>> +        uint64_t *data_ptr = &qemu_data[vcontrol * CONFIG_HYPERTRACE_ARGS * sizeof(uint64_t)];
>> +
>> +        /* 2nd fault (invoke) */
>> +        assert(((unsigned long)siginfo->si_addr % getpagesize()) == sizeof(uint64_t));
>> +        hypertrace_emit(current_cpu, data_ptr);
>> +        swap_control(qemu_control_1, qemu_control_0);
>> +
>> +    } else {
>> +        /* proxy to next handler */
>> +        if (segv_next.sa_sigaction != NULL) {
>> +            segv_next.sa_sigaction(signum, siginfo, sigctxt);
>> +        } else if (segv_next.sa_handler != NULL) {
>> +            segv_next.sa_handler(signum);
>> +        }
>> +    }
>> +}

> Can this approach be made thread-safe?

> If not then it would be good to consider the problem right away and
> switch to something that is thread-safe, even if it depends on the
> target architecture.

Kind of. The easiest solution is to have each thread have an mmap of its own of
the device (and moving qemu_control_{0,1} into CPUState). This would be
completely explicit and easy to understand.

Cheers,
  Lluis

Re: [Qemu-devel] [PATCH 3/6] hypertrace: [*-user] Add QEMU-side proxy to "guest_hypertrace" event

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 675 bytes --]

On Sun, Aug 21, 2016 at 02:15:31PM +0200, Lluís Vilanova wrote:
> > Can this approach be made thread-safe?
> 
> > If not then it would be good to consider the problem right away and
> > switch to something that is thread-safe, even if it depends on the
> > target architecture.
> 
> Kind of. The easiest solution is to have each thread have an mmap of its own of
> the device (and moving qemu_control_{0,1} into CPUState). This would be
> completely explicit and easy to understand.

Yes.  I don't know if the TCG and -user folks will accept the SIGSEGV
approach, but I'd be okay with it if the multi-threading issue is solved
by 1 device per thread.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

[Qemu-devel] [PATCH 4/6] hypertrace: [softmmu] Add QEMU-side proxy to "guest_hypertrace" event

[Lluís Vilanova]

Uses a virtual device to trigger the hypertrace channel event.

Signed-off-by: Lluís Vilanova <vilanova@ac.upc.edu>
---
 hypertrace/Makefile.objs |    5 +
 hypertrace/softmmu.c     |  243 ++++++++++++++++++++++++++++++++++++++++++++++
 include/hw/pci/pci.h     |    2 
 3 files changed, 249 insertions(+), 1 deletion(-)
 create mode 100644 hypertrace/softmmu.c

diff --git a/hypertrace/Makefile.objs b/hypertrace/Makefile.objs
index 6eb5acf..2a120b4 100644
--- a/hypertrace/Makefile.objs
+++ b/hypertrace/Makefile.objs
@@ -1,8 +1,11 @@
 # -*- mode: makefile -*-
 
 target-obj-$(CONFIG_USER_ONLY) += user.o
+ifdef CONFIG_PCI
+target-obj-$(CONFIG_SOFTMMU) += softmmu.o
+endif
 
-$(obj)/user.o: $(obj)/emit.c
+$(obj)/user.o $(obj)/softmmu.o: $(obj)/emit.c
 
 $(obj)/emit.c: $(obj)/emit.c-timestamp $(BUILD_DIR)/config-host.mak
 	@cmp $< $@ >/dev/null 2>&1 || cp $< $@
diff --git a/hypertrace/softmmu.c b/hypertrace/softmmu.c
new file mode 100644
index 0000000..4927197
--- /dev/null
+++ b/hypertrace/softmmu.c
@@ -0,0 +1,243 @@
+/*
+ * QEMU-side management of hypertrace in softmmu emulation.
+ *
+ * Copyright (C) 2016 Lluís Vilanova <vilanova@ac.upc.edu>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+/*
+ * Implementation details
+ * ======================
+ *
+ * Both channels are provided as a virtual device with two BARs that can be used
+ * through MMIO.
+ *
+ * Data channel
+ * ------------
+ *
+ * The guest must mmap BAR 1 of the hypertrace virtual device, which will act as
+ * regular writable device memory.
+ *
+ * Regular memory accesses pass data through the data channel.
+ *
+ * Control channel
+ * ---------------
+ *
+ * The guest must mmap BAR 0 of the hypertrace virtual device.
+ *
+ * Guest reads from that memory are intercepted by QEMU in order to return the
+ * size of the data channel.
+ *
+ * Guest writes into that memory are intercepted by QEMU in order to raise the
+ * "guest_hypertrace" tracing event.
+ */
+
+#include "qemu/osdep.h"
+#include "cpu.h"
+#include "exec/ram_addr.h"
+#include "hw/pci/pci.h"
+#include "migration/migration.h"
+#include "qapi/error.h"
+#include "qemu/error-report.h"
+#include "trace.h"
+
+
+#define PAGE_SIZE TARGET_PAGE_SIZE
+
+
+typedef struct HypertraceState
+{
+    PCIDevice dev;
+
+    uint8_t pages;
+    uint64_t size;
+
+    union
+    {
+        uint64_t v;
+        char     a[8];
+    } c_max_offset;
+    union
+    {
+        uint64_t v;
+        char     a[8];
+    } c_cmd;
+
+    void *data_ptr;
+    MemoryRegion data;
+    MemoryRegion control;
+
+    Error *migration_blocker;
+} HypertraceState;
+
+
+static uint64_t hypertrace_control_io_read(void *opaque, hwaddr addr,
+                                           unsigned size)
+{
+    HypertraceState *s = opaque;
+    char *mem = &s->c_max_offset.a[addr % sizeof(uint64_t)];
+
+    if (addr + size > sizeof(uint64_t)) {
+        error_report("error: hypertrace: Unexpected read to address %lu\n", addr);
+        return 0;
+    }
+
+    /* control values already have target endianess */
+
+    switch (size) {
+    case 1:
+    {
+        uint8_t *res = (uint8_t*)mem;
+        return *res;
+    }
+    case 2:
+    {
+        uint16_t *res = (uint16_t*)mem;
+        return *res;
+    }
+    case 4:
+    {
+        uint32_t *res = (uint32_t*)mem;
+        return *res;
+    }
+    case 8:
+    {
+        uint64_t *res = (uint64_t*)mem;
+        return *res;
+    }
+    default:
+        error_report("error: hypertrace: Unexpected read of size %d\n", size);
+        return 0;
+    }
+}
+
+#include "hypertrace/emit.c"
+
+static void hypertrace_control_io_write(void *opaque, hwaddr addr,
+                                        uint64_t data, unsigned size)
+{
+    HypertraceState *s = opaque;
+    char *mem = &s->c_cmd.a[addr % sizeof(uint64_t)];
+
+    if (addr < sizeof(uint64_t) || addr + size > sizeof(uint64_t) * 2) {
+        error_report("error: hypertrace: Unexpected write to address %lu\n", addr);
+        return;
+    }
+
+    /* c_cmd will have target endianess (left up to the user) */
+
+    switch (size) {
+    case 1:
+    {
+        uint8_t *res = (uint8_t*)mem;
+        *res = (uint8_t)data;
+        break;
+    }
+    case 2:
+    {
+        uint16_t *res = (uint16_t*)mem;
+        *res = (uint16_t)data;
+        break;
+    }
+    case 4:
+    {
+        uint32_t *res = (uint32_t*)mem;
+        *res = (uint32_t)data;
+        break;
+    }
+    case 8:
+    {
+        uint64_t *res = (uint64_t*)mem;
+        *res = (uint64_t)data;
+        break;
+    }
+    default:
+        error_report("error: hypertrace: Unexpected write of size %d\n", size);
+    }
+
+    if ((addr + size) % sizeof(s->c_cmd.v) == 0) {
+        uint64_t vcontrol = s->c_cmd.v;
+        uint64_t *data_ptr = (uint64_t*)s->data_ptr;
+        data_ptr = &data_ptr[vcontrol * CONFIG_HYPERTRACE_ARGS * sizeof(uint64_t)];
+        hypertrace_emit(current_cpu, data_ptr);
+    }
+}
+
+static const MemoryRegionOps hypertrace_control_ops = {
+    .read = &hypertrace_control_io_read,
+    .write = &hypertrace_control_io_write,
+    .endianness = DEVICE_NATIVE_ENDIAN,
+    .impl = {
+        .min_access_size = 1,
+        .max_access_size = 8,
+    },
+};
+
+
+static void hypertrace_realize(PCIDevice *dev, Error **errp)
+{
+    HypertraceState *s = DO_UPCAST(HypertraceState, dev, dev);
+    Error *err = NULL;
+    size_t size = s->pages * TARGET_PAGE_SIZE;
+
+    if (s->pages < 1) {
+        error_setg(errp, "hypertrace: the data channel must have one or more pages\n");
+        return;
+    }
+    s->c_max_offset.v = tswap64(s->size / (CONFIG_HYPERTRACE_ARGS * sizeof(uint64_t)));
+
+    error_setg(&s->migration_blocker, "The 'hypertrace' device cannot be migrated");
+    migrate_add_blocker(s->migration_blocker);
+
+    pci_set_word(s->dev.config + PCI_COMMAND,
+                 PCI_COMMAND_IO | PCI_COMMAND_MEMORY);
+
+    /* control channel */
+    memory_region_init_io(&s->control, OBJECT(s), &hypertrace_control_ops, s,
+                          "hypertrace.control", PAGE_SIZE);
+    pci_register_bar(&s->dev, 0, PCI_BASE_ADDRESS_SPACE_MEMORY, &s->control);
+
+    /* data channel */
+    memory_region_init_ram(&s->data, OBJECT(s), "hypertrace.data", size, &err);
+    if (err) {
+        error_propagate(errp, err);
+        return;
+    }
+    pci_register_bar(&s->dev, 1, PCI_BASE_ADDRESS_SPACE_MEMORY, &s->data);
+    s->data_ptr = qemu_map_ram_ptr(s->data.ram_block, 0);
+}
+
+
+static Property hypertrace_properties[] = {
+    DEFINE_PROP_UINT8("pages", HypertraceState, pages, 1),
+    DEFINE_PROP_END_OF_LIST(),
+};
+
+static void hypertrace_class_init(ObjectClass *klass, void *data)
+{
+    DeviceClass *dc = DEVICE_CLASS(klass);
+    PCIDeviceClass *k = PCI_DEVICE_CLASS(klass);
+
+    k->realize = hypertrace_realize;
+    k->vendor_id = PCI_VENDOR_ID_REDHAT_QUMRANET;
+    k->device_id = PCI_DEVICE_ID_HYPERTRACE;
+    k->class_id = PCI_CLASS_MEMORY_RAM;
+    dc->desc  = "Hypertrace communication channel",
+    dc->props = hypertrace_properties;
+}
+
+static TypeInfo hypertrace_info = {
+    .name          = "hypertrace",
+    .parent        = TYPE_PCI_DEVICE,
+    .instance_size = sizeof(HypertraceState),
+    .class_init    = hypertrace_class_init,
+};
+
+static void hypertrace_register_types(void)
+{
+    type_register_static(&hypertrace_info);
+}
+
+type_init(hypertrace_register_types)
diff --git a/include/hw/pci/pci.h b/include/hw/pci/pci.h
index 9ed1624..af0772f 100644
--- a/include/hw/pci/pci.h
+++ b/include/hw/pci/pci.h
@@ -80,6 +80,8 @@
 #define PCI_DEVICE_ID_VIRTIO_RNG         0x1005
 #define PCI_DEVICE_ID_VIRTIO_9P          0x1009
 
+#define PCI_DEVICE_ID_HYPERTRACE         0x10f0
+
 #define PCI_VENDOR_ID_REDHAT             0x1b36
 #define PCI_DEVICE_ID_REDHAT_BRIDGE      0x0001
 #define PCI_DEVICE_ID_REDHAT_SERIAL      0x0002

[Qemu-devel] [PATCH 5/6] hypertrace: Add guest-side user-level library

[Lluís Vilanova]

Provides guest library "libqemu-hypertrace-guest.a" to abstract access
to the hypertrace channel.

Signed-off-by: Lluís Vilanova <vilanova@ac.upc.edu>
---
 Makefile                                |    6 +
 configure                               |    2 
 hypertrace/guest/user/Makefile          |   28 ++++
 hypertrace/guest/user/common.c          |  221 +++++++++++++++++++++++++++++++
 hypertrace/guest/user/qemu-hypertrace.h |   77 +++++++++++
 5 files changed, 333 insertions(+), 1 deletion(-)
 create mode 100644 hypertrace/guest/user/Makefile
 create mode 100644 hypertrace/guest/user/common.c
 create mode 100644 hypertrace/guest/user/qemu-hypertrace.h

diff --git a/Makefile b/Makefile
index 0d7647f..bebd6e6 100644
--- a/Makefile
+++ b/Makefile
@@ -459,9 +459,13 @@ ifneq (,$(findstring qemu-ga,$(TOOLS)))
 endif
 endif
 
+install-hypertrace:
+	$(INSTALL_DIR) "$(DESTDIR)$(includedir)"
+	$(INSTALL_DATA) "$(SRC_PATH)/hypertrace/guest/user/qemu-hypertrace.h" "$(DESTDIR)$(includedir)/"
+
 
 install: all $(if $(BUILD_DOCS),install-doc) \
-install-datadir install-localstatedir
+install-datadir install-localstatedir install-hypertrace
 ifneq ($(TOOLS),)
 	$(call install-prog,$(subst qemu-ga,qemu-ga$(EXESUF),$(TOOLS)),$(DESTDIR)$(bindir))
 endif
diff --git a/configure b/configure
index e80fde4..076eb8b 100755
--- a/configure
+++ b/configure
@@ -5780,6 +5780,8 @@ if [ "$TARGET_BASE_ARCH" = "" ]; then
 fi
 
 symlink "$source_path/Makefile.target" "$target_dir/Makefile"
+mkdir -p $target_dir/hypertrace/guest/user
+symlink $source_path/hypertrace/guest/user/Makefile $target_dir/hypertrace/guest/user/Makefile
 
 upper() {
     echo "$@"| LC_ALL=C tr '[a-z]' '[A-Z]'
diff --git a/hypertrace/guest/user/Makefile b/hypertrace/guest/user/Makefile
new file mode 100644
index 0000000..bcc9dc9
--- /dev/null
+++ b/hypertrace/guest/user/Makefile
@@ -0,0 +1,28 @@
+include ../../../../config-host.mak
+include ../../../config-target.mak
+include $(SRC_PATH)/rules.mak
+
+vpath % $(SRC_PATH)/hypertrace/guest/user
+
+# do not use QEMU's per-host cflags when building guest code
+QEMU_CFLAGS  = -Werror -Wall
+
+QEMU_CFLAGS += $(GLIB_CFLAGS)
+QEMU_CFLAGS += -I$(SRC_PATH)/include
+QEMU_CFLAGS += -I../../../../linux-headers
+QEMU_CFLAGS += -I../../../../
+QEMU_CFLAGS += -I../../../
+
+ifdef CONFIG_SOFTMMU
+QEMU_CFLAGS += -DNEED_CPU_H
+QEMU_CFLAGS += -I$(SRC_PATH)/target-$(TARGET_BASE_ARCH)
+endif
+
+obj-y = common.o
+
+libqemu-hypertrace-guest.a: $(obj-y)
+
+all: libqemu-hypertrace-guest.a
+
+clean:
+	rm -f $(obj-y) libqemu-hypertrace-guest.a
diff --git a/hypertrace/guest/user/common.c b/hypertrace/guest/user/common.c
new file mode 100644
index 0000000..1429b2a
--- /dev/null
+++ b/hypertrace/guest/user/common.c
@@ -0,0 +1,221 @@
+/*
+ * Guest-side management of hypertrace.
+ *
+ * Copyright (C) 2016 Lluís Vilanova <vilanova@ac.upc.edu>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu-hypertrace.h"
+
+#include <errno.h>
+#include <fcntl.h>
+#include <stdio.h>
+#include <unistd.h>
+#include <stdlib.h>
+#include <string.h>
+#include <stdint.h>
+#include <stdbool.h>
+#include <sys/stat.h>
+#include <sys/mman.h>
+#include <glob.h>
+
+#include "config-host.h"
+#include "config-target.h"
+#if defined(CONFIG_SOFTMMU)
+#include "qemu/osdep.h"
+#include "hw/pci/pci.h"
+#endif
+
+
+static char *data_path = NULL;
+static char *control_path = NULL;
+static int data_fd = -1;
+static int control_fd = -1;
+
+static uint64_t *data_addr = NULL;
+static uint64_t *control_addr = NULL;
+
+
+static int init_channel_file(const char *base, const char *suffix, size_t size,
+                             char ** path, int *fd, uint64_t **addr)
+{
+    *path = malloc(strlen(base) + strlen(suffix) + 1);
+    sprintf(*path, "%s%s", base, suffix);
+
+    *fd = open(*path, O_RDWR);
+    if (*fd == -1) {
+        return -1;
+    }
+
+    *addr = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, *fd, 0);
+    if (*addr == MAP_FAILED) {
+        return -1;
+    }
+    return 0;
+}
+
+#if !defined(CONFIG_USER_ONLY) && defined(__linux__)
+static int check_device_id (const char *base, const char *name, uint64_t value)
+{
+    char tmp[1024];
+    sprintf(tmp, "%s/%s", base, name);
+
+    int fd = open(tmp, O_RDONLY);
+    if (fd < 0) {
+        return -1;
+    }
+
+    char v[1024];
+    ssize_t s = read(fd, v, sizeof(v));
+    if (s < 0) {
+        close(fd);
+        return -1;
+    }
+    v[s] = '\0';
+
+    char *end;
+    uint64_t vv = strtoull(v, &end, 16);
+    if (*end == '\n' && vv == value) {
+        return 0;
+    }
+    else {
+        return -1;
+    }
+}
+
+static char* find_device(void)
+{
+    static char tmp[1024];
+    char *res = NULL;
+
+    glob_t g;
+    if (glob("/sys/devices/pci*/*", GLOB_NOSORT, NULL, &g) != 0) {
+        return NULL;
+    }
+
+
+    int i;
+    for (i = 0; i < g.gl_pathc; i++) {
+        char *path = g.gl_pathv[i];
+
+        if (check_device_id(path, "vendor", PCI_VENDOR_ID_REDHAT_QUMRANET) < 0) {
+            continue;
+        }
+        if (check_device_id(path, "device", PCI_DEVICE_ID_HYPERTRACE) < 0) {
+            continue;
+        }
+
+        sprintf(tmp, "%s", path);
+        res = tmp;
+        break;
+    }
+
+    globfree(&g);
+
+    return res;
+}
+#endif
+
+int qemu_hypertrace_init(const char *base)
+{
+#if defined(CONFIG_USER_ONLY)
+    const char *control_suff = "-control";
+    const size_t control_size = getpagesize() * 2;
+    const char *data_suff = "-data";
+#elif defined(__linux__)
+    const char *control_suff = "/resource0";
+    const size_t control_size = getpagesize();
+    const char *data_suff = "/resource1";
+#else
+#error Unsupported OS
+#endif
+
+#if defined(CONFIG_USER_ONLY)
+    if (base == NULL) {
+        errno = ENOENT;
+        return -1;
+    }
+#elif defined(__linux__)
+    if (base == NULL) {
+        /* try to guess the base path */
+        base = find_device();
+        if (base == NULL) {
+            errno = ENOENT;
+            return -1;
+        }
+    }
+#endif
+
+    int res;
+    res = init_channel_file(base, control_suff, control_size,
+                            &control_path, &control_fd, &control_addr);
+    if (res != 0) {
+        return res;
+    }
+
+    size_t data_size = qemu_hypertrace_num_args() * sizeof(uint64_t);
+    data_size *= qemu_hypertrace_max_offset() + 1;
+    res = init_channel_file(base, data_suff, data_size,
+                            &data_path, &data_fd, &data_addr);
+    if (res != 0) {
+        return res;
+    }
+    return 0;
+}
+
+
+static int fini_channel(int *fd, char **path)
+{
+    if (*fd != -1) {
+        if (close(*fd) == -1) {
+            return -1;
+        }
+        *fd = -1;
+    }
+    if (*path != NULL) {
+        free(*path);
+        *path =  NULL;
+    }
+    return 0;
+}
+
+int qemu_hypertrace_fini(void)
+{
+    if (fini_channel(&data_fd, &data_path) != 0) {
+        return -1;
+    }
+    if (fini_channel(&control_fd, &control_path) != 0) {
+        return -1;
+    }
+    return 0;
+}
+
+
+uint64_t qemu_hypertrace_num_args(void)
+{
+    return CONFIG_HYPERTRACE_ARGS;
+}
+
+uint64_t qemu_hypertrace_max_offset(void)
+{
+    return control_addr[0];
+}
+
+uint64_t *qemu_hypertrace_data(uint64_t data_offset)
+{
+    return &data_addr[data_offset * CONFIG_HYPERTRACE_ARGS * sizeof(uint64_t)];
+}
+
+void qemu_hypertrace (uint64_t data_offset)
+{
+    uint64_t *ctrl;
+    ctrl = control_addr;
+    ctrl[1] = data_offset;
+#if defined(CONFIG_USER_ONLY)
+    /* QEMU in 'user' mode uses two faulting pages to detect invocations */
+    ctrl = (uint64_t*)((char*)control_addr + getpagesize());
+    ctrl[1] = data_offset;
+#endif
+}
diff --git a/hypertrace/guest/user/qemu-hypertrace.h b/hypertrace/guest/user/qemu-hypertrace.h
new file mode 100644
index 0000000..ba58e49
--- /dev/null
+++ b/hypertrace/guest/user/qemu-hypertrace.h
@@ -0,0 +1,77 @@
+/*
+ * Guest-side management of hypertrace.
+ *
+ * Copyright (C) 2016 Lluís Vilanova <vilanova@ac.upc.edu>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include <stdint.h>
+#include <sys/types.h>
+
+
+/**
+ * qemu_hypertrace_init:
+ * @base: Base path to the hypertrace channel.
+ *
+ * Initialize the hypertrace channel.
+ *
+ * The base path to the hypertrace channel depends on the type of QEMU target:
+ *
+ * - User (single-application)
+ *   The base path provided when starting QEMU ("-hypertrace" commandline
+ *   option).
+ *
+ * - System (OS-dependant)
+ *   + Linux
+ *     The base path to the hypertrace channel virtual device; on a default QEMU
+ *     device setup for x86 this is "/sys/devices/pci0000:00/0000:00:04.0". If
+ *     NULL is provided, the hypertrace device will be automatically detected.
+ *
+ * Returns: Zero on success.
+ */
+int qemu_hypertrace_init(const char *base);
+
+/**
+ * qemu_hypertrace_fini:
+ *
+ * Deinitialize the hypertrace channel.
+ *
+ * Returns: Zero on success.
+ */
+int qemu_hypertrace_fini(void);
+
+/**
+ * qemu_hypertrace_num_args:
+ *
+ * Number of uint64_t values read by each call to qemu_hypertrace().
+ */
+uint64_t qemu_hypertrace_num_args(void);
+
+/**
+ * qemu_hypertrace_data_size:
+ *
+ * Maximum data offset value accepted by other calls.
+ */
+uint64_t qemu_hypertrace_max_offset(void);
+
+/**
+ * qemu_hypertrace_data:
+ * @data_offset: Offset in multiples of argument packs.
+ *
+ * Pointer to the start of the data channel.
+ */
+uint64_t *qemu_hypertrace_data(uint64_t data_offset);
+
+/**
+ * qemu_hypertrace:
+ * @data_offset: Offset in multiples of argument packs.
+ *
+ * Invoke the control channel.
+ *
+ * Each of the users (e.g., thread) of the hypertrace channel can use a
+ * different data offset to ensure they can work concurrently without using
+ * locks (i.e., each uses a different portion of the data channel).
+ */
+void qemu_hypertrace(uint64_t data_offset);

[Qemu-devel] [PATCH 6/6] hypertrace: Add guest-side Linux module

[Lluís Vilanova]

Provides guest Linux kernel module "qemu-hypertrace.ko" to abstract
access to the hypertrace channel.

Signed-off-by: Lluís Vilanova <vilanova@ac.upc.edu>
---
 Makefile                                           |    4 -
 configure                                          |    4 +
 hypertrace/guest/linux-module/Kbuild.in            |    5 +
 hypertrace/guest/linux-module/Makefile             |   24 ++++
 .../include/linux/qemu-hypertrace-internal.h       |   46 +++++++
 .../linux-module/include/linux/qemu-hypertrace.h   |   71 +++++++++++
 hypertrace/guest/linux-module/qemu-hypertrace.c    |  133 ++++++++++++++++++++
 7 files changed, 286 insertions(+), 1 deletion(-)
 create mode 100644 hypertrace/guest/linux-module/Kbuild.in
 create mode 100644 hypertrace/guest/linux-module/Makefile
 create mode 100644 hypertrace/guest/linux-module/include/linux/qemu-hypertrace-internal.h
 create mode 100644 hypertrace/guest/linux-module/include/linux/qemu-hypertrace.h
 create mode 100644 hypertrace/guest/linux-module/qemu-hypertrace.c

diff --git a/Makefile b/Makefile
index bebd6e6..26fb418 100644
--- a/Makefile
+++ b/Makefile
@@ -460,8 +460,10 @@ endif
 endif
 
 install-hypertrace:
-	$(INSTALL_DIR) "$(DESTDIR)$(includedir)"
+	$(INSTALL_DIR) "$(DESTDIR)$(includedir)/linux"
 	$(INSTALL_DATA) "$(SRC_PATH)/hypertrace/guest/user/qemu-hypertrace.h" "$(DESTDIR)$(includedir)/"
+	$(INSTALL_DATA) "$(SRC_PATH)/hypertrace/guest/linux-module/include/linux/qemu-hypertrace.h" "$(DESTDIR)$(includedir)/linux"
+	$(INSTALL_DATA) "$(SRC_PATH)/hypertrace/guest/linux-module/include/linux/qemu-hypertrace-internal.h" "$(DESTDIR)$(includedir)/linux"
 
 
 install: all $(if $(BUILD_DOCS),install-doc) \
diff --git a/configure b/configure
index 076eb8b..b809f69 100755
--- a/configure
+++ b/configure
@@ -5782,6 +5782,10 @@ fi
 symlink "$source_path/Makefile.target" "$target_dir/Makefile"
 mkdir -p $target_dir/hypertrace/guest/user
 symlink $source_path/hypertrace/guest/user/Makefile $target_dir/hypertrace/guest/user/Makefile
+if test "$target_softmmu" = "yes"; then
+    mkdir -p $target_dir/hypertrace/guest/linux-module
+    symlink $source_path/hypertrace/guest/linux-module/Makefile $target_dir/hypertrace/guest/linux-module/Makefile
+fi
 
 upper() {
     echo "$@"| LC_ALL=C tr '[a-z]' '[A-Z]'
diff --git a/hypertrace/guest/linux-module/Kbuild.in b/hypertrace/guest/linux-module/Kbuild.in
new file mode 100644
index 0000000..87656d4
--- /dev/null
+++ b/hypertrace/guest/linux-module/Kbuild.in
@@ -0,0 +1,5 @@
+# -*- mode: makefile -*-
+
+src = $(MOD_SRC_PATH)
+ccflags-y := -I$(src)/include/ -DHYPERTRACE_NUM_ARGS=$(HYPERTRACE_NUM_ARGS)
+obj-m := qemu-hypertrace.o
diff --git a/hypertrace/guest/linux-module/Makefile b/hypertrace/guest/linux-module/Makefile
new file mode 100644
index 0000000..d9247a0
--- /dev/null
+++ b/hypertrace/guest/linux-module/Makefile
@@ -0,0 +1,24 @@
+include ../../../../config-host.mak
+include ../../../config-target.mak
+include $(SRC_PATH)/rules.mak
+
+MOD_SRC_PATH = $(SRC_PATH)/hypertrace/guest/linux-module
+
+LINUX_BUILD_PATH = /lib/modules/$(shell uname -r)/build
+
+vpath % $(MOD_SRC_PATH)
+
+
+all: Kbuild
+	$(MAKE) -C $(LINUX_BUILD_PATH) M=$(shell pwd) \
+	  MOD_SRC_PATH=$(MOD_SRC_PATH) \
+	  HYPERTRACE_NUM_ARGS=$(CONFIG_HYPERTRACE_ARGS) \
+	  modules
+
+clean: Kbuild
+	$(MAKE) -C $(LINUX_BUILD_PATH) M=$(shell pwd) clean
+	rm -f $<
+
+Kbuild: $(MOD_SRC_PATH)/Kbuild.in Makefile
+	rm -f $@
+	cp $< $@
diff --git a/hypertrace/guest/linux-module/include/linux/qemu-hypertrace-internal.h b/hypertrace/guest/linux-module/include/linux/qemu-hypertrace-internal.h
new file mode 100644
index 0000000..465c003
--- /dev/null
+++ b/hypertrace/guest/linux-module/include/linux/qemu-hypertrace-internal.h
@@ -0,0 +1,46 @@
+/* -*- C -*-
+ *
+ * Guest-side management of hypertrace.
+ *
+ * Copyright (C) 2016 Lluís Vilanova <vilanova@ac.upc.edu>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see http://www.gnu.org/licenses/.
+ */
+
+extern uint64_t _qemu_hypertrace_channel_num_args;
+extern uint64_t _qemu_hypertrace_channel_max_offset;
+extern uint64_t *_qemu_hypertrace_channel_data;
+extern uint64_t *_qemu_hypertrace_channel_control;
+
+static inline uint64_t qemu_hypertrace_num_args(void)
+{
+    return _qemu_hypertrace_channel_num_args;
+}
+
+static inline uint64_t qemu_hypertrace_max_offset(void)
+{
+    return _qemu_hypertrace_channel_max_offset;
+}
+
+static inline uint64_t *qemu_hypertrace_data(uint64_t data_offset)
+{
+    size_t args_size = qemu_hypertrace_num_args() * sizeof(uint64_t);
+    return &_qemu_hypertrace_channel_data[data_offset * args_size];
+}
+
+static inline void qemu_hypertrace (uint64_t data_offset)
+{
+    uint64_t *ctrlp = _qemu_hypertrace_channel_control;
+    ctrlp[1] = data_offset;
+}
diff --git a/hypertrace/guest/linux-module/include/linux/qemu-hypertrace.h b/hypertrace/guest/linux-module/include/linux/qemu-hypertrace.h
new file mode 100644
index 0000000..bb6ad7f
--- /dev/null
+++ b/hypertrace/guest/linux-module/include/linux/qemu-hypertrace.h
@@ -0,0 +1,71 @@
+/* -*- C -*-
+ *
+ * Guest-side management of hypertrace.
+ *
+ * Copyright (C) 2016 Lluís Vilanova <vilanova@ac.upc.edu>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see http://www.gnu.org/licenses/.
+ */
+
+#ifndef QEMU_HYPERTRACE_H
+#define QEMU_HYPERTRACE_H
+
+#include <linux/types.h>
+
+
+/**
+ * qemu_hypertrace_num_args:
+ *
+ * Number of uint64_t values read by each call to qemu_hypertrace().
+ */
+static uint64_t qemu_hypertrace_num_args(void);
+
+/**
+ * qemu_hypertrace_data_size:
+ *
+ * Maximum data offset value accepted by other calls.
+ */
+static uint64_t qemu_hypertrace_max_offset(void);
+
+/**
+ * qemu_hypertrace_data:
+ * @data_offset: Offset in multiples of argument packs.
+ *
+ * Pointer to the start of the data channel.
+ */
+static uint64_t *qemu_hypertrace_data(uint64_t data_offset);
+
+/**
+ * qemu_hypertrace:
+ * @data_offset: Offset in multiples of argument packs.
+ *
+ * Invoke the control channel.
+ *
+ * Each of the users (e.g., CPU) of the hypertrace channel can use a different
+ * data offset to ensure they can work concurrently without using locks (i.e.,
+ * each uses a different portion of the data channel).
+ *
+ * Note: You should use wmb() before writing into the control channel iff you
+ * have written into the data channel.
+ *
+ * Note: Use preempt_disable() and preempt_enable() if you're using data offsets
+ * based on the CPU identifiers (or else data might be mixed if a task is
+ * re-scheduled).
+ */
+static void qemu_hypertrace(uint64_t data_offset);
+
+
+#include <linux/qemu-hypertrace-internal.h>
+
+#endif  /* QEMU_HYPERTRACE_H */
diff --git a/hypertrace/guest/linux-module/qemu-hypertrace.c b/hypertrace/guest/linux-module/qemu-hypertrace.c
new file mode 100644
index 0000000..43d35d7
--- /dev/null
+++ b/hypertrace/guest/linux-module/qemu-hypertrace.c
@@ -0,0 +1,133 @@
+/*
+ * Guest-side management of hypertrace.
+ *
+ * Copyright (C) 2016 Lluís Vilanova <vilanova@ac.upc.edu>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see http://www.gnu.org/licenses/.
+ */
+
+#include <linux/qemu-hypertrace.h>
+
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/uaccess.h>
+
+
+#define VERSION_STR "0.1"
+#define PCI_VENDOR_ID_REDHAT_QUMRANET    0x1af4
+#define PCI_DEVICE_ID_HYPERTRACE         0x10f0
+
+
+MODULE_DESCRIPTION("Kernel interface to QEMU's hypertrace device");
+MODULE_LICENSE("GPL");
+MODULE_AUTHOR("Lluís Vilanova");
+MODULE_VERSION(VERSION_STR);
+
+
+//////////////////////////////////////////////////////////////////////
+// Kernel interface
+
+uint64_t _qemu_hypertrace_channel_num_args = HYPERTRACE_NUM_ARGS;
+uint64_t _qemu_hypertrace_channel_max_offset = 0;
+uint64_t *_qemu_hypertrace_channel_control = NULL;
+uint64_t *_qemu_hypertrace_channel_data = NULL;
+
+EXPORT_SYMBOL(_qemu_hypertrace_channel_num_args);
+EXPORT_SYMBOL(_qemu_hypertrace_channel_max_offset);
+EXPORT_SYMBOL(_qemu_hypertrace_channel_data);
+EXPORT_SYMBOL(_qemu_hypertrace_channel_control);
+
+
+//////////////////////////////////////////////////////////////////////
+// Channel initialization
+
+static
+int
+init_channel (uint64_t **vaddr, struct pci_dev *dev, int bar)
+{
+    void * res;
+    resource_size_t start, size;
+
+    start = pci_resource_start(dev, bar);
+    size = pci_resource_len(dev, bar);
+
+    if (start == 0 || size == 0) {
+        return -ENOENT;
+    }
+
+    res = ioremap(start, size);
+    if (res == 0) {
+        return -EINVAL;
+    }
+
+    *vaddr = res;
+    return 0;
+}
+
+//////////////////////////////////////////////////////////////////////
+// Module (de)initialization
+
+int init_module(void)
+{
+    int res = 0;
+    struct pci_dev *dev = NULL;
+    size_t args_size;
+    size_t data_size;
+
+    printk(KERN_NOTICE "Loading QEMU hypertrace module (version %s)\n",
+           VERSION_STR);
+
+    dev = pci_get_device(PCI_VENDOR_ID_REDHAT_QUMRANET, PCI_DEVICE_ID_HYPERTRACE, NULL);
+    if (dev == NULL) {
+        res = -ENOENT;
+        printk(KERN_ERR "Unable to find hypertrace device\n");
+        goto error;
+    }
+
+    res = init_channel(&_qemu_hypertrace_channel_control, dev, 0);
+    if (res != 0) {
+        printk(KERN_ERR "Unable to find hypertrace control channel\n");
+        goto error;
+    }
+
+    res = init_channel(&_qemu_hypertrace_channel_data, dev, 1);
+    if (res != 0) {
+        printk(KERN_ERR "Unable to find hypertrace data channel\n");
+        goto error_data;
+    }
+
+    args_size = _qemu_hypertrace_channel_num_args * sizeof(uint64_t);
+    data_size = pci_resource_len(dev, 1);
+    _qemu_hypertrace_channel_max_offset = data_size / args_size;
+
+    goto ok;
+
+error_data:
+    iounmap(_qemu_hypertrace_channel_control);
+
+error:
+    _qemu_hypertrace_channel_control = NULL;
+    _qemu_hypertrace_channel_data = NULL;
+
+ok:
+    return res;
+}
+
+void cleanup_module(void)
+{
+    printk(KERN_NOTICE "Unloading QEMU hypertrace module\n");
+
+    iounmap(_qemu_hypertrace_channel_control);
+    iounmap(_qemu_hypertrace_channel_data);
+}

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 4072 bytes --]

On Fri, Aug 05, 2016 at 06:59:23PM +0200, Lluís Vilanova wrote:
> The hypertrace channel allows guest code to emit events in QEMU (the host) using
> its tracing infrastructure (see "docs/trace.txt"). This works in both 'system'
> and 'user' modes. That is, hypertrace is to tracing, what hypercalls are to
> system calls.
> 
> You can use this to emit an event on both guest and QEMU (host) traces to easily
> synchronize or correlate them. You could also modify you guest's tracing system
> to emit all events through the hypertrace channel, providing a unified and fully
> synchronized trace log. Another use case is timing the performance of guest code
> when optimizing TCG (QEMU traces have a timestamp).
> 
> See first commit for a full description.
> 
> Signed-off-by: Lluís Vilanova <vilanova@ac.upc.edu>
> ---

CCing Steven Rostedt, Masami Hiramatsu, Luiz Capitulino, and LTTng folks
who have all looked into host/guest tracing solutions.

Here is a mailing list archive link to this email thread:
https://lists.gnu.org/archive/html/qemu-devel/2016-08/msg01081.html

> 
> Lluís Vilanova (6):
>       hypertrace: Add documentation
>       hypertrace: Add tracing event "guest_hypertrace"
>       hypertrace: [*-user] Add QEMU-side proxy to "guest_hypertrace" event
>       hypertrace: [softmmu] Add QEMU-side proxy to "guest_hypertrace" event
>       hypertrace: Add guest-side user-level library
>       hypertrace: Add guest-side Linux module
> 
> 
>  Makefile                                           |    8 -
>  Makefile.objs                                      |    6 
>  bsd-user/main.c                                    |   16 +
>  bsd-user/mmap.c                                    |    2 
>  bsd-user/syscall.c                                 |    4 
>  configure                                          |   42 +++
>  docs/hypertrace.txt                                |  141 ++++++++++
>  docs/tracing.txt                                   |    3 
>  hypertrace/Makefile.objs                           |   20 +
>  hypertrace/guest/linux-module/Kbuild.in            |    5 
>  hypertrace/guest/linux-module/Makefile             |   24 ++
>  .../include/linux/qemu-hypertrace-internal.h       |   46 +++
>  .../linux-module/include/linux/qemu-hypertrace.h   |   71 +++++
>  hypertrace/guest/linux-module/qemu-hypertrace.c    |  133 +++++++++
>  hypertrace/guest/user/Makefile                     |   28 ++
>  hypertrace/guest/user/common.c                     |  221 +++++++++++++++
>  hypertrace/guest/user/qemu-hypertrace.h            |   77 +++++
>  hypertrace/softmmu.c                               |  243 +++++++++++++++++
>  hypertrace/user.c                                  |  292 ++++++++++++++++++++
>  hypertrace/user.h                                  |   52 ++++
>  include/hw/pci/pci.h                               |    2 
>  linux-user/main.c                                  |   19 +
>  linux-user/mmap.c                                  |    2 
>  linux-user/syscall.c                               |    3 
>  trace/Makefile.objs                                |    2 
>  25 files changed, 1460 insertions(+), 2 deletions(-)
>  create mode 100644 docs/hypertrace.txt
>  create mode 100644 hypertrace/Makefile.objs
>  create mode 100644 hypertrace/guest/linux-module/Kbuild.in
>  create mode 100644 hypertrace/guest/linux-module/Makefile
>  create mode 100644 hypertrace/guest/linux-module/include/linux/qemu-hypertrace-internal.h
>  create mode 100644 hypertrace/guest/linux-module/include/linux/qemu-hypertrace.h
>  create mode 100644 hypertrace/guest/linux-module/qemu-hypertrace.c
>  create mode 100644 hypertrace/guest/user/Makefile
>  create mode 100644 hypertrace/guest/user/common.c
>  create mode 100644 hypertrace/guest/user/qemu-hypertrace.h
>  create mode 100644 hypertrace/softmmu.c
>  create mode 100644 hypertrace/user.c
>  create mode 100644 hypertrace/user.h
> 
> 
> To: qemu-devel@nongnu.org
> Cc: Stefan Hajnoczi <stefanha@redhat.com>
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 4072 bytes --]

On Fri, Aug 05, 2016 at 06:59:23PM +0200, Lluís Vilanova wrote:
> The hypertrace channel allows guest code to emit events in QEMU (the host) using
> its tracing infrastructure (see "docs/trace.txt"). This works in both 'system'
> and 'user' modes. That is, hypertrace is to tracing, what hypercalls are to
> system calls.
> 
> You can use this to emit an event on both guest and QEMU (host) traces to easily
> synchronize or correlate them. You could also modify you guest's tracing system
> to emit all events through the hypertrace channel, providing a unified and fully
> synchronized trace log. Another use case is timing the performance of guest code
> when optimizing TCG (QEMU traces have a timestamp).
> 
> See first commit for a full description.
> 
> Signed-off-by: Lluís Vilanova <vilanova@ac.upc.edu>
> ---

CCing Steven Rostedt, Masami Hiramatsu, Luiz Capitulino, and LTTng folks
who have all looked into host/guest tracing solutions.

Here is a mailing list archive link to this email thread:
https://lists.gnu.org/archive/html/qemu-devel/2016-08/msg01081.html

> 
> Lluís Vilanova (6):
>       hypertrace: Add documentation
>       hypertrace: Add tracing event "guest_hypertrace"
>       hypertrace: [*-user] Add QEMU-side proxy to "guest_hypertrace" event
>       hypertrace: [softmmu] Add QEMU-side proxy to "guest_hypertrace" event
>       hypertrace: Add guest-side user-level library
>       hypertrace: Add guest-side Linux module
> 
> 
>  Makefile                                           |    8 -
>  Makefile.objs                                      |    6 
>  bsd-user/main.c                                    |   16 +
>  bsd-user/mmap.c                                    |    2 
>  bsd-user/syscall.c                                 |    4 
>  configure                                          |   42 +++
>  docs/hypertrace.txt                                |  141 ++++++++++
>  docs/tracing.txt                                   |    3 
>  hypertrace/Makefile.objs                           |   20 +
>  hypertrace/guest/linux-module/Kbuild.in            |    5 
>  hypertrace/guest/linux-module/Makefile             |   24 ++
>  .../include/linux/qemu-hypertrace-internal.h       |   46 +++
>  .../linux-module/include/linux/qemu-hypertrace.h   |   71 +++++
>  hypertrace/guest/linux-module/qemu-hypertrace.c    |  133 +++++++++
>  hypertrace/guest/user/Makefile                     |   28 ++
>  hypertrace/guest/user/common.c                     |  221 +++++++++++++++
>  hypertrace/guest/user/qemu-hypertrace.h            |   77 +++++
>  hypertrace/softmmu.c                               |  243 +++++++++++++++++
>  hypertrace/user.c                                  |  292 ++++++++++++++++++++
>  hypertrace/user.h                                  |   52 ++++
>  include/hw/pci/pci.h                               |    2 
>  linux-user/main.c                                  |   19 +
>  linux-user/mmap.c                                  |    2 
>  linux-user/syscall.c                               |    3 
>  trace/Makefile.objs                                |    2 
>  25 files changed, 1460 insertions(+), 2 deletions(-)
>  create mode 100644 docs/hypertrace.txt
>  create mode 100644 hypertrace/Makefile.objs
>  create mode 100644 hypertrace/guest/linux-module/Kbuild.in
>  create mode 100644 hypertrace/guest/linux-module/Makefile
>  create mode 100644 hypertrace/guest/linux-module/include/linux/qemu-hypertrace-internal.h
>  create mode 100644 hypertrace/guest/linux-module/include/linux/qemu-hypertrace.h
>  create mode 100644 hypertrace/guest/linux-module/qemu-hypertrace.c
>  create mode 100644 hypertrace/guest/user/Makefile
>  create mode 100644 hypertrace/guest/user/common.c
>  create mode 100644 hypertrace/guest/user/qemu-hypertrace.h
>  create mode 100644 hypertrace/softmmu.c
>  create mode 100644 hypertrace/user.c
>  create mode 100644 hypertrace/user.h
> 
> 
> To: qemu-devel@nongnu.org
> Cc: Stefan Hajnoczi <stefanha@redhat.com>
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Lluís Vilanova]

Stefan Hajnoczi writes:

> On Fri, Aug 05, 2016 at 06:59:23PM +0200, Lluís Vilanova wrote:
>> The hypertrace channel allows guest code to emit events in QEMU (the host) using
>> its tracing infrastructure (see "docs/trace.txt"). This works in both 'system'
>> and 'user' modes. That is, hypertrace is to tracing, what hypercalls are to
>> system calls.
>> 
>> You can use this to emit an event on both guest and QEMU (host) traces to easily
>> synchronize or correlate them. You could also modify you guest's tracing system
>> to emit all events through the hypertrace channel, providing a unified and fully
>> synchronized trace log. Another use case is timing the performance of guest code
>> when optimizing TCG (QEMU traces have a timestamp).
>> 
>> See first commit for a full description.
>> 
>> Signed-off-by: Lluís Vilanova <vilanova@ac.upc.edu>
>> ---

> CCing Steven Rostedt, Masami Hiramatsu, Luiz Capitulino, and LTTng folks
> who have all looked into host/guest tracing solutions.
[...]

Oh, I wasn't aware of that. I'm certainly interested in collaborating.


Cheers,
  Lluis

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1519 bytes --]

On Thu, Aug 18, 2016 at 12:22:18PM +0200, Lluís Vilanova wrote:
> Stefan Hajnoczi writes:
> 
> > On Fri, Aug 05, 2016 at 06:59:23PM +0200, Lluís Vilanova wrote:
> >> The hypertrace channel allows guest code to emit events in QEMU (the host) using
> >> its tracing infrastructure (see "docs/trace.txt"). This works in both 'system'
> >> and 'user' modes. That is, hypertrace is to tracing, what hypercalls are to
> >> system calls.
> >> 
> >> You can use this to emit an event on both guest and QEMU (host) traces to easily
> >> synchronize or correlate them. You could also modify you guest's tracing system
> >> to emit all events through the hypertrace channel, providing a unified and fully
> >> synchronized trace log. Another use case is timing the performance of guest code
> >> when optimizing TCG (QEMU traces have a timestamp).
> >> 
> >> See first commit for a full description.
> >> 
> >> Signed-off-by: Lluís Vilanova <vilanova@ac.upc.edu>
> >> ---
> 
> > CCing Steven Rostedt, Masami Hiramatsu, Luiz Capitulino, and LTTng folks
> > who have all looked into host/guest tracing solutions.
> [...]
> 
> Oh, I wasn't aware of that. I'm certainly interested in collaborating.

They are working on or have worked on different approaches to host/guest
tracing.  Unfortunately there isn't an out-of-the-box solution as far as
I know.

It would be nice if there was a documented host/guest tracing approach
that didn't involve much manual setup and handled most use cases.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Luiz Capitulino]

On Thu, 18 Aug 2016 14:53:27 +0100
Stefan Hajnoczi <stefanha@redhat.com> wrote:

> On Thu, Aug 18, 2016 at 12:22:18PM +0200, Lluís Vilanova wrote:
> > Stefan Hajnoczi writes:
> >   
> > > On Fri, Aug 05, 2016 at 06:59:23PM +0200, Lluís Vilanova wrote:  
> > >> The hypertrace channel allows guest code to emit events in QEMU (the host) using
> > >> its tracing infrastructure (see "docs/trace.txt"). This works in both 'system'
> > >> and 'user' modes. That is, hypertrace is to tracing, what hypercalls are to
> > >> system calls.
> > >> 
> > >> You can use this to emit an event on both guest and QEMU (host) traces to easily
> > >> synchronize or correlate them. You could also modify you guest's tracing system
> > >> to emit all events through the hypertrace channel, providing a unified and fully
> > >> synchronized trace log. Another use case is timing the performance of guest code
> > >> when optimizing TCG (QEMU traces have a timestamp).
> > >> 
> > >> See first commit for a full description.
> > >> 
> > >> Signed-off-by: Lluís Vilanova <vilanova@ac.upc.edu>
> > >> ---  
> >   
> > > CCing Steven Rostedt, Masami Hiramatsu, Luiz Capitulino, and LTTng folks
> > > who have all looked into host/guest tracing solutions.  
> > [...]
> > 
> > Oh, I wasn't aware of that. I'm certainly interested in collaborating.  
> 
> They are working on or have worked on different approaches to host/guest
> tracing.  Unfortunately there isn't an out-of-the-box solution as far as
> I know.

The ftrace solution is documented here:

 https://lists.nongnu.org/archive/html/qemu-devel/2016-03/msg00887.html

This traces the guest and host kernels. It supports merging the guest
and host traces. It's extremely low latency and has helped us to
find several spikes for real-time KVM (we're talking a few to
a dozen microseconds at most).

Now, our stack actually is:

 - Guest app
 - Guest kernel
 - Host kernel
 - QEMU

QEMU already has its own tracing (which I don't know how it works).
If I had to trace the guest app, I'd certainly start off by using
LTTng. Although, we'd have to write a tool to merge and orchestrate
(wooo, cloud buzzword!) all those traces (if that's what one wants).

> It would be nice if there was a documented host/guest tracing approach
> that didn't involve much manual setup and handled most use cases.

I'd volunteer to do it, although it will take me weeks to do it.

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Lluís Vilanova]

Luiz Capitulino writes:

> On Thu, 18 Aug 2016 14:53:27 +0100
> Stefan Hajnoczi <stefanha@redhat.com> wrote:

>> On Thu, Aug 18, 2016 at 12:22:18PM +0200, Lluís Vilanova wrote:
>> > Stefan Hajnoczi writes:
>> >   
>> > > On Fri, Aug 05, 2016 at 06:59:23PM +0200, Lluís Vilanova wrote:  
>> > >> The hypertrace channel allows guest code to emit events in QEMU (the host) using
>> > >> its tracing infrastructure (see "docs/trace.txt"). This works in both 'system'
>> > >> and 'user' modes. That is, hypertrace is to tracing, what hypercalls are to
>> > >> system calls.
>> > >> 
>> > >> You can use this to emit an event on both guest and QEMU (host) traces to easily
>> > >> synchronize or correlate them. You could also modify you guest's tracing system
>> > >> to emit all events through the hypertrace channel, providing a unified and fully
>> > >> synchronized trace log. Another use case is timing the performance of guest code
>> > >> when optimizing TCG (QEMU traces have a timestamp).
>> > >> 
>> > >> See first commit for a full description.
>> > >> 
>> > >> Signed-off-by: Lluís Vilanova <vilanova@ac.upc.edu>
>> > >> ---  
>> >   
>> > > CCing Steven Rostedt, Masami Hiramatsu, Luiz Capitulino, and LTTng folks
>> > > who have all looked into host/guest tracing solutions.  
>> > [...]
>> > 
>> > Oh, I wasn't aware of that. I'm certainly interested in collaborating.  
>> 
>> They are working on or have worked on different approaches to host/guest
>> tracing.  Unfortunately there isn't an out-of-the-box solution as far as
>> I know.

> The ftrace solution is documented here:

>  https://lists.nongnu.org/archive/html/qemu-devel/2016-03/msg00887.html

> This traces the guest and host kernels. It supports merging the guest
> and host traces. It's extremely low latency and has helped us to
> find several spikes for real-time KVM (we're talking a few to
> a dozen microseconds at most).

> Now, our stack actually is:

>  - Guest app
>  - Guest kernel
>  - Host kernel
>  - QEMU

> QEMU already has its own tracing (which I don't know how it works).
> If I had to trace the guest app, I'd certainly start off by using
> LTTng. Although, we'd have to write a tool to merge and orchestrate
> (wooo, cloud buzzword!) all those traces (if that's what one wants).
[...]

One of my targets was to simplify the merge by providing known reference points
between guest and host traces.


Cheers,
  Lluis
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Lluís Vilanova]

Luiz Capitulino writes:

> On Thu, 18 Aug 2016 14:53:27 +0100
> Stefan Hajnoczi <stefanha@redhat.com> wrote:

>> On Thu, Aug 18, 2016 at 12:22:18PM +0200, Lluís Vilanova wrote:
>> > Stefan Hajnoczi writes:
>> >   
>> > > On Fri, Aug 05, 2016 at 06:59:23PM +0200, Lluís Vilanova wrote:  
>> > >> The hypertrace channel allows guest code to emit events in QEMU (the host) using
>> > >> its tracing infrastructure (see "docs/trace.txt"). This works in both 'system'
>> > >> and 'user' modes. That is, hypertrace is to tracing, what hypercalls are to
>> > >> system calls.
>> > >> 
>> > >> You can use this to emit an event on both guest and QEMU (host) traces to easily
>> > >> synchronize or correlate them. You could also modify you guest's tracing system
>> > >> to emit all events through the hypertrace channel, providing a unified and fully
>> > >> synchronized trace log. Another use case is timing the performance of guest code
>> > >> when optimizing TCG (QEMU traces have a timestamp).
>> > >> 
>> > >> See first commit for a full description.
>> > >> 
>> > >> Signed-off-by: Lluís Vilanova <vilanova@ac.upc.edu>
>> > >> ---  
>> >   
>> > > CCing Steven Rostedt, Masami Hiramatsu, Luiz Capitulino, and LTTng folks
>> > > who have all looked into host/guest tracing solutions.  
>> > [...]
>> > 
>> > Oh, I wasn't aware of that. I'm certainly interested in collaborating.  
>> 
>> They are working on or have worked on different approaches to host/guest
>> tracing.  Unfortunately there isn't an out-of-the-box solution as far as
>> I know.

> The ftrace solution is documented here:

>  https://lists.nongnu.org/archive/html/qemu-devel/2016-03/msg00887.html

> This traces the guest and host kernels. It supports merging the guest
> and host traces. It's extremely low latency and has helped us to
> find several spikes for real-time KVM (we're talking a few to
> a dozen microseconds at most).

> Now, our stack actually is:

>  - Guest app
>  - Guest kernel
>  - Host kernel
>  - QEMU

> QEMU already has its own tracing (which I don't know how it works).
> If I had to trace the guest app, I'd certainly start off by using
> LTTng. Although, we'd have to write a tool to merge and orchestrate
> (wooo, cloud buzzword!) all those traces (if that's what one wants).
[...]

One of my targets was to simplify the merge by providing known reference points
between guest and host traces.


Cheers,
  Lluis

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Luiz Capitulino]

On Thu, 18 Aug 2016 14:53:27 +0100
Stefan Hajnoczi <stefanha@redhat.com> wrote:

> On Thu, Aug 18, 2016 at 12:22:18PM +0200, Lluís Vilanova wrote:
> > Stefan Hajnoczi writes:
> >   
> > > On Fri, Aug 05, 2016 at 06:59:23PM +0200, Lluís Vilanova wrote:  
> > >> The hypertrace channel allows guest code to emit events in QEMU (the host) using
> > >> its tracing infrastructure (see "docs/trace.txt"). This works in both 'system'
> > >> and 'user' modes. That is, hypertrace is to tracing, what hypercalls are to
> > >> system calls.
> > >> 
> > >> You can use this to emit an event on both guest and QEMU (host) traces to easily
> > >> synchronize or correlate them. You could also modify you guest's tracing system
> > >> to emit all events through the hypertrace channel, providing a unified and fully
> > >> synchronized trace log. Another use case is timing the performance of guest code
> > >> when optimizing TCG (QEMU traces have a timestamp).
> > >> 
> > >> See first commit for a full description.
> > >> 
> > >> Signed-off-by: Lluís Vilanova <vilanova@ac.upc.edu>
> > >> ---  
> >   
> > > CCing Steven Rostedt, Masami Hiramatsu, Luiz Capitulino, and LTTng folks
> > > who have all looked into host/guest tracing solutions.  
> > [...]
> > 
> > Oh, I wasn't aware of that. I'm certainly interested in collaborating.  
> 
> They are working on or have worked on different approaches to host/guest
> tracing.  Unfortunately there isn't an out-of-the-box solution as far as
> I know.

The ftrace solution is documented here:

 https://lists.nongnu.org/archive/html/qemu-devel/2016-03/msg00887.html

This traces the guest and host kernels. It supports merging the guest
and host traces. It's extremely low latency and has helped us to
find several spikes for real-time KVM (we're talking a few to
a dozen microseconds at most).

Now, our stack actually is:

 - Guest app
 - Guest kernel
 - Host kernel
 - QEMU

QEMU already has its own tracing (which I don't know how it works).
If I had to trace the guest app, I'd certainly start off by using
LTTng. Although, we'd have to write a tool to merge and orchestrate
(wooo, cloud buzzword!) all those traces (if that's what one wants).

> It would be nice if there was a documented host/guest tracing approach
> that didn't involve much manual setup and handled most use cases.

I'd volunteer to do it, although it will take me weeks to do it.
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Stefan Hajnoczi]

[-- Attachment #1.1: Type: text/plain, Size: 1519 bytes --]

On Thu, Aug 18, 2016 at 12:22:18PM +0200, Lluís Vilanova wrote:
> Stefan Hajnoczi writes:
> 
> > On Fri, Aug 05, 2016 at 06:59:23PM +0200, Lluís Vilanova wrote:
> >> The hypertrace channel allows guest code to emit events in QEMU (the host) using
> >> its tracing infrastructure (see "docs/trace.txt"). This works in both 'system'
> >> and 'user' modes. That is, hypertrace is to tracing, what hypercalls are to
> >> system calls.
> >> 
> >> You can use this to emit an event on both guest and QEMU (host) traces to easily
> >> synchronize or correlate them. You could also modify you guest's tracing system
> >> to emit all events through the hypertrace channel, providing a unified and fully
> >> synchronized trace log. Another use case is timing the performance of guest code
> >> when optimizing TCG (QEMU traces have a timestamp).
> >> 
> >> See first commit for a full description.
> >> 
> >> Signed-off-by: Lluís Vilanova <vilanova@ac.upc.edu>
> >> ---
> 
> > CCing Steven Rostedt, Masami Hiramatsu, Luiz Capitulino, and LTTng folks
> > who have all looked into host/guest tracing solutions.
> [...]
> 
> Oh, I wasn't aware of that. I'm certainly interested in collaborating.

They are working on or have worked on different approaches to host/guest
tracing.  Unfortunately there isn't an out-of-the-box solution as far as
I know.

It would be nice if there was a documented host/guest tracing approach
that didn't involve much manual setup and handled most use cases.

Stefan

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Lluís Vilanova]

Stefan Hajnoczi writes:

> On Fri, Aug 05, 2016 at 06:59:23PM +0200, Lluís Vilanova wrote:
>> The hypertrace channel allows guest code to emit events in QEMU (the host) using
>> its tracing infrastructure (see "docs/trace.txt"). This works in both 'system'
>> and 'user' modes. That is, hypertrace is to tracing, what hypercalls are to
>> system calls.
>> 
>> You can use this to emit an event on both guest and QEMU (host) traces to easily
>> synchronize or correlate them. You could also modify you guest's tracing system
>> to emit all events through the hypertrace channel, providing a unified and fully
>> synchronized trace log. Another use case is timing the performance of guest code
>> when optimizing TCG (QEMU traces have a timestamp).
>> 
>> See first commit for a full description.
>> 
>> Signed-off-by: Lluís Vilanova <vilanova@ac.upc.edu>
>> ---

> CCing Steven Rostedt, Masami Hiramatsu, Luiz Capitulino, and LTTng folks
> who have all looked into host/guest tracing solutions.
[...]

Oh, I wasn't aware of that. I'm certainly interested in collaborating.


Cheers,
  Lluis
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2360 bytes --]

On Fri, Aug 05, 2016 at 06:59:23PM +0200, Lluís Vilanova wrote:
> The hypertrace channel allows guest code to emit events in QEMU (the host) using
> its tracing infrastructure (see "docs/trace.txt"). This works in both 'system'
> and 'user' modes. That is, hypertrace is to tracing, what hypercalls are to
> system calls.
> 
> You can use this to emit an event on both guest and QEMU (host) traces to easily
> synchronize or correlate them. You could also modify you guest's tracing system
> to emit all events through the hypertrace channel, providing a unified and fully
> synchronized trace log. Another use case is timing the performance of guest code
> when optimizing TCG (QEMU traces have a timestamp).
> 
> See first commit for a full description.

This tracing approach has a high performance overhead, particularly for
SMP guests where each trace event requires writing to the global control
register.  All CPUs will be hammering this register (heavyweight vmexit)
for each trace event.

I think the folks CCed on this email all take an asynchronous approach
to avoid this performance overhead.  Synchronous means taking a VM exit
for every event.  Asynchronous means writing trace data to a buffer and
later interleaving guest data with host trace data.

LTTng Userspace Tracer is an example of the asynchronous approach.  The
trace data buffers are in shared memory.  The LTTng process can grab
buffers at appropriate times.

The ftrace virtio-serial approach has been to splice() the ftrace
buffers, resulting in efficient I/O.

Steven is working on a host/guest solution for trace-cmd.  It is also
asynchronous.  No new paravirt hardware is needed and it makes me wonder
whether the hypertrace PCI device is trying to solve the problem at the
wrong layer.

If you want to play around with asynchronous tracing, you could start
with trace/simple.c.  It has a trace buffer that is asynchronously
written out to file by a dedicated "writer" thread.

The one case where hypertrace makes sense to me is for -user tracing.
There QEMU can efficiently interleave guest and QEMU traces, although as
mentioned in the patch, I don't think the SIGSEGV approach should be
used.

I suggest stripping this series down to focus on -user.  Synchronous
tracing is not a good approach for -system emulation.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2360 bytes --]

On Fri, Aug 05, 2016 at 06:59:23PM +0200, Lluís Vilanova wrote:
> The hypertrace channel allows guest code to emit events in QEMU (the host) using
> its tracing infrastructure (see "docs/trace.txt"). This works in both 'system'
> and 'user' modes. That is, hypertrace is to tracing, what hypercalls are to
> system calls.
> 
> You can use this to emit an event on both guest and QEMU (host) traces to easily
> synchronize or correlate them. You could also modify you guest's tracing system
> to emit all events through the hypertrace channel, providing a unified and fully
> synchronized trace log. Another use case is timing the performance of guest code
> when optimizing TCG (QEMU traces have a timestamp).
> 
> See first commit for a full description.

This tracing approach has a high performance overhead, particularly for
SMP guests where each trace event requires writing to the global control
register.  All CPUs will be hammering this register (heavyweight vmexit)
for each trace event.

I think the folks CCed on this email all take an asynchronous approach
to avoid this performance overhead.  Synchronous means taking a VM exit
for every event.  Asynchronous means writing trace data to a buffer and
later interleaving guest data with host trace data.

LTTng Userspace Tracer is an example of the asynchronous approach.  The
trace data buffers are in shared memory.  The LTTng process can grab
buffers at appropriate times.

The ftrace virtio-serial approach has been to splice() the ftrace
buffers, resulting in efficient I/O.

Steven is working on a host/guest solution for trace-cmd.  It is also
asynchronous.  No new paravirt hardware is needed and it makes me wonder
whether the hypertrace PCI device is trying to solve the problem at the
wrong layer.

If you want to play around with asynchronous tracing, you could start
with trace/simple.c.  It has a trace buffer that is asynchronously
written out to file by a dedicated "writer" thread.

The one case where hypertrace makes sense to me is for -user tracing.
There QEMU can efficiently interleave guest and QEMU traces, although as
mentioned in the patch, I don't think the SIGSEGV approach should be
used.

I suggest stripping this series down to focus on -user.  Synchronous
tracing is not a good approach for -system emulation.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Luiz Capitulino]

On Thu, 18 Aug 2016 11:54:24 +0100
Stefan Hajnoczi <stefanha@gmail.com> wrote:

> On Fri, Aug 05, 2016 at 06:59:23PM +0200, Lluís Vilanova wrote:
> > The hypertrace channel allows guest code to emit events in QEMU (the host) using
> > its tracing infrastructure (see "docs/trace.txt"). This works in both 'system'
> > and 'user' modes. That is, hypertrace is to tracing, what hypercalls are to
> > system calls.
> > 
> > You can use this to emit an event on both guest and QEMU (host) traces to easily
> > synchronize or correlate them. You could also modify you guest's tracing system
> > to emit all events through the hypertrace channel, providing a unified and fully
> > synchronized trace log. Another use case is timing the performance of guest code
> > when optimizing TCG (QEMU traces have a timestamp).
> > 
> > See first commit for a full description.  

I hate the be the one asking the stupid questions, but what's
the problem this series solves? What are the use-cases? Why
existing solutions don't solve this problem? How does this
compares to existing solutions?

> This tracing approach has a high performance overhead, particularly for
> SMP guests where each trace event requires writing to the global control
> register.  All CPUs will be hammering this register (heavyweight vmexit)
> for each trace event.
> 
> I think the folks CCed on this email all take an asynchronous approach
> to avoid this performance overhead.  Synchronous means taking a VM exit
> for every event.  Asynchronous means writing trace data to a buffer and
> later interleaving guest data with host trace data.
> 
> LTTng Userspace Tracer is an example of the asynchronous approach.  The
> trace data buffers are in shared memory.  The LTTng process can grab
> buffers at appropriate times.
> 
> The ftrace virtio-serial approach has been to splice() the ftrace
> buffers, resulting in efficient I/O.

Yes. However, note that the virtio-serial device is only used to
transfer the tracing data to the host. It has no role in the
tracing process. Also, it's not required. I've been using the
networking and it works OK as long as your tracing data is not big.

> 
> Steven is working on a host/guest solution for trace-cmd.  It is also
> asynchronous.  No new paravirt hardware is needed and it makes me wonder
> whether the hypertrace PCI device is trying to solve the problem at the
> wrong layer.
> 
> If you want to play around with asynchronous tracing, you could start
> with trace/simple.c.  It has a trace buffer that is asynchronously
> written out to file by a dedicated "writer" thread.
> 
> The one case where hypertrace makes sense to me is for -user tracing.
> There QEMU can efficiently interleave guest and QEMU traces, although as
> mentioned in the patch, I don't think the SIGSEGV approach should be
> used.
> 
> I suggest stripping this series down to focus on -user.  Synchronous
> tracing is not a good approach for -system emulation.
> 
> Stefan

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Masami Hiramatsu]

On Thu, 18 Aug 2016 09:37:11 -0400
Luiz Capitulino <lcapitulino@redhat.com> wrote:

> On Thu, 18 Aug 2016 11:54:24 +0100
> Stefan Hajnoczi <stefanha@gmail.com> wrote:
> 
> > On Fri, Aug 05, 2016 at 06:59:23PM +0200, Lluís Vilanova wrote:
> > > The hypertrace channel allows guest code to emit events in QEMU (the host) using
> > > its tracing infrastructure (see "docs/trace.txt"). This works in both 'system'
> > > and 'user' modes. That is, hypertrace is to tracing, what hypercalls are to
> > > system calls.
> > > 
> > > You can use this to emit an event on both guest and QEMU (host) traces to easily
> > > synchronize or correlate them. You could also modify you guest's tracing system
> > > to emit all events through the hypertrace channel, providing a unified and fully
> > > synchronized trace log. Another use case is timing the performance of guest code
> > > when optimizing TCG (QEMU traces have a timestamp).
> > > 
> > > See first commit for a full description.  
> 
> I hate the be the one asking the stupid questions, but what's
> the problem this series solves? What are the use-cases? Why
> existing solutions don't solve this problem? How does this
> compares to existing solutions?
> 
> > This tracing approach has a high performance overhead, particularly for
> > SMP guests where each trace event requires writing to the global control
> > register.  All CPUs will be hammering this register (heavyweight vmexit)
> > for each trace event.

That reminds me xenprobes.
https://www.usenix.org/legacy/event/usenix07/tech/full_papers/quynh/quynh.pdf

The paper also tried to implement kprobe-like feature for tracing events in
guest. Synchronous tracing maybe easy to read the raw trace data, but cause
big slowdown. Moreover, we can merge and synchronize the data easily after
traced.

> > 
> > I think the folks CCed on this email all take an asynchronous approach
> > to avoid this performance overhead.  Synchronous means taking a VM exit
> > for every event.  Asynchronous means writing trace data to a buffer and
> > later interleaving guest data with host trace data.
> > 
> > LTTng Userspace Tracer is an example of the asynchronous approach.  The
> > trace data buffers are in shared memory.  The LTTng process can grab
> > buffers at appropriate times.
> > 
> > The ftrace virtio-serial approach has been to splice() the ftrace
> > buffers, resulting in efficient I/O.
> 
> Yes. However, note that the virtio-serial device is only used to
> transfer the tracing data to the host. It has no role in the
> tracing process. Also, it's not required. I've been using the
> networking and it works OK as long as your tracing data is not big.

Right, the no-copy virtio-serial is for reducing the overhead as small as
possible but it also requires a special programs in both of guest and host.
(since it uses unix domain socket to transfer data without copy)
In normal case, you can just trace it as like as tracing on a remote machine
via network. That's more scalable and flexible. 


> > Steven is working on a host/guest solution for trace-cmd.  It is also
> > asynchronous.  No new paravirt hardware is needed and it makes me wonder
> > whether the hypertrace PCI device is trying to solve the problem at the
> > wrong layer.

Thanks Steven to continue that!

Regards,

> > 
> > If you want to play around with asynchronous tracing, you could start
> > with trace/simple.c.  It has a trace buffer that is asynchronously
> > written out to file by a dedicated "writer" thread.
> > 
> > The one case where hypertrace makes sense to me is for -user tracing.
> > There QEMU can efficiently interleave guest and QEMU traces, although as
> > mentioned in the patch, I don't think the SIGSEGV approach should be
> > used.
> > 
> > I suggest stripping this series down to focus on -user.  Synchronous
> > tracing is not a good approach for -system emulation.
> > 
> > Stefan
> 


-- 
Masami Hiramatsu <mhiramat@kernel.org>

Re: [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Masami Hiramatsu]

On Thu, 18 Aug 2016 09:37:11 -0400
Luiz Capitulino <lcapitulino@redhat.com> wrote:

> On Thu, 18 Aug 2016 11:54:24 +0100
> Stefan Hajnoczi <stefanha@gmail.com> wrote:
> 
> > On Fri, Aug 05, 2016 at 06:59:23PM +0200, Lluís Vilanova wrote:
> > > The hypertrace channel allows guest code to emit events in QEMU (the host) using
> > > its tracing infrastructure (see "docs/trace.txt"). This works in both 'system'
> > > and 'user' modes. That is, hypertrace is to tracing, what hypercalls are to
> > > system calls.
> > > 
> > > You can use this to emit an event on both guest and QEMU (host) traces to easily
> > > synchronize or correlate them. You could also modify you guest's tracing system
> > > to emit all events through the hypertrace channel, providing a unified and fully
> > > synchronized trace log. Another use case is timing the performance of guest code
> > > when optimizing TCG (QEMU traces have a timestamp).
> > > 
> > > See first commit for a full description.  
> 
> I hate the be the one asking the stupid questions, but what's
> the problem this series solves? What are the use-cases? Why
> existing solutions don't solve this problem? How does this
> compares to existing solutions?
> 
> > This tracing approach has a high performance overhead, particularly for
> > SMP guests where each trace event requires writing to the global control
> > register.  All CPUs will be hammering this register (heavyweight vmexit)
> > for each trace event.

That reminds me xenprobes.
https://www.usenix.org/legacy/event/usenix07/tech/full_papers/quynh/quynh.pdf

The paper also tried to implement kprobe-like feature for tracing events in
guest. Synchronous tracing maybe easy to read the raw trace data, but cause
big slowdown. Moreover, we can merge and synchronize the data easily after
traced.

> > 
> > I think the folks CCed on this email all take an asynchronous approach
> > to avoid this performance overhead.  Synchronous means taking a VM exit
> > for every event.  Asynchronous means writing trace data to a buffer and
> > later interleaving guest data with host trace data.
> > 
> > LTTng Userspace Tracer is an example of the asynchronous approach.  The
> > trace data buffers are in shared memory.  The LTTng process can grab
> > buffers at appropriate times.
> > 
> > The ftrace virtio-serial approach has been to splice() the ftrace
> > buffers, resulting in efficient I/O.
> 
> Yes. However, note that the virtio-serial device is only used to
> transfer the tracing data to the host. It has no role in the
> tracing process. Also, it's not required. I've been using the
> networking and it works OK as long as your tracing data is not big.

Right, the no-copy virtio-serial is for reducing the overhead as small as
possible but it also requires a special programs in both of guest and host.
(since it uses unix domain socket to transfer data without copy)
In normal case, you can just trace it as like as tracing on a remote machine
via network. That's more scalable and flexible. 


> > Steven is working on a host/guest solution for trace-cmd.  It is also
> > asynchronous.  No new paravirt hardware is needed and it makes me wonder
> > whether the hypertrace PCI device is trying to solve the problem at the
> > wrong layer.

Thanks Steven to continue that!

Regards,

> > 
> > If you want to play around with asynchronous tracing, you could start
> > with trace/simple.c.  It has a trace buffer that is asynchronously
> > written out to file by a dedicated "writer" thread.
> > 
> > The one case where hypertrace makes sense to me is for -user tracing.
> > There QEMU can efficiently interleave guest and QEMU traces, although as
> > mentioned in the patch, I don't think the SIGSEGV approach should be
> > used.
> > 
> > I suggest stripping this series down to focus on -user.  Synchronous
> > tracing is not a good approach for -system emulation.
> > 
> > Stefan
> 


-- 
Masami Hiramatsu <mhiramat@kernel.org>

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Luiz Capitulino]

On Thu, 18 Aug 2016 11:54:24 +0100
Stefan Hajnoczi <stefanha@gmail.com> wrote:

> On Fri, Aug 05, 2016 at 06:59:23PM +0200, Lluís Vilanova wrote:
> > The hypertrace channel allows guest code to emit events in QEMU (the host) using
> > its tracing infrastructure (see "docs/trace.txt"). This works in both 'system'
> > and 'user' modes. That is, hypertrace is to tracing, what hypercalls are to
> > system calls.
> > 
> > You can use this to emit an event on both guest and QEMU (host) traces to easily
> > synchronize or correlate them. You could also modify you guest's tracing system
> > to emit all events through the hypertrace channel, providing a unified and fully
> > synchronized trace log. Another use case is timing the performance of guest code
> > when optimizing TCG (QEMU traces have a timestamp).
> > 
> > See first commit for a full description.  

I hate the be the one asking the stupid questions, but what's
the problem this series solves? What are the use-cases? Why
existing solutions don't solve this problem? How does this
compares to existing solutions?

> This tracing approach has a high performance overhead, particularly for
> SMP guests where each trace event requires writing to the global control
> register.  All CPUs will be hammering this register (heavyweight vmexit)
> for each trace event.
> 
> I think the folks CCed on this email all take an asynchronous approach
> to avoid this performance overhead.  Synchronous means taking a VM exit
> for every event.  Asynchronous means writing trace data to a buffer and
> later interleaving guest data with host trace data.
> 
> LTTng Userspace Tracer is an example of the asynchronous approach.  The
> trace data buffers are in shared memory.  The LTTng process can grab
> buffers at appropriate times.
> 
> The ftrace virtio-serial approach has been to splice() the ftrace
> buffers, resulting in efficient I/O.

Yes. However, note that the virtio-serial device is only used to
transfer the tracing data to the host. It has no role in the
tracing process. Also, it's not required. I've been using the
networking and it works OK as long as your tracing data is not big.

> 
> Steven is working on a host/guest solution for trace-cmd.  It is also
> asynchronous.  No new paravirt hardware is needed and it makes me wonder
> whether the hypertrace PCI device is trying to solve the problem at the
> wrong layer.
> 
> If you want to play around with asynchronous tracing, you could start
> with trace/simple.c.  It has a trace buffer that is asynchronously
> written out to file by a dedicated "writer" thread.
> 
> The one case where hypertrace makes sense to me is for -user tracing.
> There QEMU can efficiently interleave guest and QEMU traces, although as
> mentioned in the patch, I don't think the SIGSEGV approach should be
> used.
> 
> I suggest stripping this series down to focus on -user.  Synchronous
> tracing is not a good approach for -system emulation.
> 
> Stefan

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Steven Rostedt]

On Thu, 18 Aug 2016 11:54:24 +0100
Stefan Hajnoczi <stefanha@gmail.com> wrote:

> Steven is working on a host/guest solution for trace-cmd.  It is also
> asynchronous.  No new paravirt hardware is needed and it makes me wonder
> whether the hypertrace PCI device is trying to solve the problem at the
> wrong layer.

Yes, and I'm currently working on it again. In fact, if any of you will
be at LinuxCon in Toronto next week, I'll be presenting[1] what I have
and what I plan to complete in the near future.

-- Steve

[1] https://lcccna2016.sched.org/event/7JWL/trace-cmd-virt-server-a-status-update-steven-rostedt-red-hat?iframe=no&w=i:100;&sidebar=yes&bg=no

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 997 bytes --]

On Thu, Aug 18, 2016 at 12:19:50PM -0400, Steven Rostedt wrote:
> On Thu, 18 Aug 2016 11:54:24 +0100
> Stefan Hajnoczi <stefanha@gmail.com> wrote:
> 
> > Steven is working on a host/guest solution for trace-cmd.  It is also
> > asynchronous.  No new paravirt hardware is needed and it makes me wonder
> > whether the hypertrace PCI device is trying to solve the problem at the
> > wrong layer.
> 
> Yes, and I'm currently working on it again. In fact, if any of you will
> be at LinuxCon in Toronto next week, I'll be presenting[1] what I have
> and what I plan to complete in the near future.
> 
> -- Steve
> 
> [1] https://lcccna2016.sched.org/event/7JWL/trace-cmd-virt-server-a-status-update-steven-rostedt-red-hat?iframe=no&w=i:100;&sidebar=yes&bg=no

Cool, that's very interesting!  I'll try to attend.

I guess LinuxCon will post a video of your talk and the slides will be
available after the talk so people not at LinuxCon will be able to check
it out too.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Steven Rostedt]

On Fri, 19 Aug 2016 11:02:07 +0100
Stefan Hajnoczi <stefanha@redhat.com> wrote:


> > [1] https://lcccna2016.sched.org/event/7JWL/trace-cmd-virt-server-a-status-update-steven-rostedt-red-hat?iframe=no&w=i:100;&sidebar=yes&bg=no  
> 
> Cool, that's very interesting!  I'll try to attend.

Good, see you then!

> 
> I guess LinuxCon will post a video of your talk and the slides will be
> available after the talk so people not at LinuxCon will be able to check
> it out too.

Hmm, IIRC the LinuxCon talks are seldom recorded. But the slides will be
available.

-- Steve

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Steven Rostedt]

On Fri, 19 Aug 2016 11:02:07 +0100
Stefan Hajnoczi <stefanha@redhat.com> wrote:


> > [1] https://lcccna2016.sched.org/event/7JWL/trace-cmd-virt-server-a-status-update-steven-rostedt-red-hat?iframe=no&w=i:100;&sidebar=yes&bg=no  
> 
> Cool, that's very interesting!  I'll try to attend.

Good, see you then!

> 
> I guess LinuxCon will post a video of your talk and the slides will be
> available after the talk so people not at LinuxCon will be able to check
> it out too.

Hmm, IIRC the LinuxCon talks are seldom recorded. But the slides will be
available.

-- Steve

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Stefan Hajnoczi]

[-- Attachment #1.1: Type: text/plain, Size: 997 bytes --]

On Thu, Aug 18, 2016 at 12:19:50PM -0400, Steven Rostedt wrote:
> On Thu, 18 Aug 2016 11:54:24 +0100
> Stefan Hajnoczi <stefanha@gmail.com> wrote:
> 
> > Steven is working on a host/guest solution for trace-cmd.  It is also
> > asynchronous.  No new paravirt hardware is needed and it makes me wonder
> > whether the hypertrace PCI device is trying to solve the problem at the
> > wrong layer.
> 
> Yes, and I'm currently working on it again. In fact, if any of you will
> be at LinuxCon in Toronto next week, I'll be presenting[1] what I have
> and what I plan to complete in the near future.
> 
> -- Steve
> 
> [1] https://lcccna2016.sched.org/event/7JWL/trace-cmd-virt-server-a-status-update-steven-rostedt-red-hat?iframe=no&w=i:100;&sidebar=yes&bg=no

Cool, that's very interesting!  I'll try to attend.

I guess LinuxCon will post a video of your talk and the slides will be
available after the talk so people not at LinuxCon will be able to check
it out too.

Stefan

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Steven Rostedt]

On Thu, 18 Aug 2016 11:54:24 +0100
Stefan Hajnoczi <stefanha@gmail.com> wrote:

> Steven is working on a host/guest solution for trace-cmd.  It is also
> asynchronous.  No new paravirt hardware is needed and it makes me wonder
> whether the hypertrace PCI device is trying to solve the problem at the
> wrong layer.

Yes, and I'm currently working on it again. In fact, if any of you will
be at LinuxCon in Toronto next week, I'll be presenting[1] what I have
and what I plan to complete in the near future.

-- Steve

[1] https://lcccna2016.sched.org/event/7JWL/trace-cmd-virt-server-a-status-update-steven-rostedt-red-hat?iframe=no&w=i:100;&sidebar=yes&bg=no
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Lluís Vilanova]

Stefan Hajnoczi writes:

> On Fri, Aug 05, 2016 at 06:59:23PM +0200, Lluís Vilanova wrote:
>> The hypertrace channel allows guest code to emit events in QEMU (the host) using
>> its tracing infrastructure (see "docs/trace.txt"). This works in both 'system'
>> and 'user' modes. That is, hypertrace is to tracing, what hypercalls are to
>> system calls.
>> 
>> You can use this to emit an event on both guest and QEMU (host) traces to easily
>> synchronize or correlate them. You could also modify you guest's tracing system
>> to emit all events through the hypertrace channel, providing a unified and fully
>> synchronized trace log. Another use case is timing the performance of guest code
>> when optimizing TCG (QEMU traces have a timestamp).
>> 
>> See first commit for a full description.

> This tracing approach has a high performance overhead, particularly for
> SMP guests where each trace event requires writing to the global control
> register.  All CPUs will be hammering this register (heavyweight vmexit)
> for each trace event.

> I think the folks CCed on this email all take an asynchronous approach
> to avoid this performance overhead.  Synchronous means taking a VM exit
> for every event.  Asynchronous means writing trace data to a buffer and
> later interleaving guest data with host trace data.

> LTTng Userspace Tracer is an example of the asynchronous approach.  The
> trace data buffers are in shared memory.  The LTTng process can grab
> buffers at appropriate times.

> The ftrace virtio-serial approach has been to splice() the ftrace
> buffers, resulting in efficient I/O.

> Steven is working on a host/guest solution for trace-cmd.  It is also
> asynchronous.  No new paravirt hardware is needed and it makes me wonder
> whether the hypertrace PCI device is trying to solve the problem at the
> wrong layer.

> If you want to play around with asynchronous tracing, you could start
> with trace/simple.c.  It has a trace buffer that is asynchronously
> written out to file by a dedicated "writer" thread.

> The one case where hypertrace makes sense to me is for -user tracing.
> There QEMU can efficiently interleave guest and QEMU traces, although as
> mentioned in the patch, I don't think the SIGSEGV approach should be
> used.

> I suggest stripping this series down to focus on -user.  Synchronous
> tracing is not a good approach for -system emulation.

As I said, I wanted to implement a simple way to provide common reference points
in guest and host traces. For that, a *synchronous* guest-to-host channel is the
simplest way to do it (otherwise you're back to square one).

Another reason to have this is that I wanted to attach a dtrace/systemtap hook
to the hypertrace event, and from there dynamically control QEMU's tracing. Then
you can insert simple calls to hypertrace on your guest code to tell your dtrace
hooks when to enable/disable tracing of certain QEMU events (like guest instr
traces).

Unfortuntely, I've been unable to to make dtrace recognise QEMU's events (I'm
only able to see the host kernel events). If someone with more experience on it
can help me use dtrace with QEMU's events, I'll also add the supporting library
to let dtrace do the callout to QEMU's moitor interface and control the events,
and add a prperly useful example of that on the hypertrace docs (which was my
original intention).


Thanks,
  Lluis

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Stefan Hajnoczi]

[-- Attachment #1.1: Type: text/plain, Size: 694 bytes --]

On Sun, Aug 21, 2016 at 02:32:34PM +0200, Lluís Vilanova wrote:
> Unfortuntely, I've been unable to to make dtrace recognise QEMU's events (I'm
> only able to see the host kernel events). If someone with more experience on it
> can help me use dtrace with QEMU's events, I'll also add the supporting library
> to let dtrace do the callout to QEMU's moitor interface and control the events,
> and add a prperly useful example of that on the hypertrace docs (which was my
> original intention).

Which "dtrace" and host OS are you using?

QEMU builds with static user-space probes.  You need to tell DTrace or
SystemTap to enable those probes in order to record trace data.

Stefan

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 694 bytes --]

On Sun, Aug 21, 2016 at 02:32:34PM +0200, Lluís Vilanova wrote:
> Unfortuntely, I've been unable to to make dtrace recognise QEMU's events (I'm
> only able to see the host kernel events). If someone with more experience on it
> can help me use dtrace with QEMU's events, I'll also add the supporting library
> to let dtrace do the callout to QEMU's moitor interface and control the events,
> and add a prperly useful example of that on the hypertrace docs (which was my
> original intention).

Which "dtrace" and host OS are you using?

QEMU builds with static user-space probes.  You need to tell DTrace or
SystemTap to enable those probes in order to record trace data.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Lluís Vilanova]

Stefan Hajnoczi writes:

> On Sun, Aug 21, 2016 at 02:32:34PM +0200, Lluís Vilanova wrote:
>> Unfortuntely, I've been unable to to make dtrace recognise QEMU's events (I'm
>> only able to see the host kernel events). If someone with more experience on it
>> can help me use dtrace with QEMU's events, I'll also add the supporting library
>> to let dtrace do the callout to QEMU's moitor interface and control the events,
>> and add a prperly useful example of that on the hypertrace docs (which was my
>> original intention).

> Which "dtrace" and host OS are you using?

> QEMU builds with static user-space probes.  You need to tell DTrace or
> SystemTap to enable those probes in order to record trace data.

I'm using debian on a 4.6.0-1-amd64 kernel with systemtap 3.0.6.

I just gave it another try, and works if I use probes like:

  process("<binary-path>").mark("<event-name>")

although they don't seem to appear on "stap -l" or anything like that (I cannot
find a "qemu" provider). But I'm still unable to print the event values. This:

  probe process("./install/vanilla/bin/qemu-system-i386").mark("guest_mem_before_exec")
  {
      printf("%p %lx %d\n", $arg1, $arg2, $arg3)
  }

always prints "0x0 0x0 0", which is clearly wrong (other backend on the same
build print the correct values).

Also, I'm still not sure how to interact with QEMU's monitor interface from
within the probe code (probes execute in kernel mode, including "guru mode"
code).

If anybody can shed some like into any of this, I'd appreaciate it.


Cheers,
  Lluis

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1932 bytes --]

On Wed, Aug 24, 2016 at 12:25:54PM +0200, Lluís Vilanova wrote:
> Stefan Hajnoczi writes:
> 
> > On Sun, Aug 21, 2016 at 02:32:34PM +0200, Lluís Vilanova wrote:
> >> Unfortuntely, I've been unable to to make dtrace recognise QEMU's events (I'm
> >> only able to see the host kernel events). If someone with more experience on it
> >> can help me use dtrace with QEMU's events, I'll also add the supporting library
> >> to let dtrace do the callout to QEMU's moitor interface and control the events,
> >> and add a prperly useful example of that on the hypertrace docs (which was my
> >> original intention).
> 
> > Which "dtrace" and host OS are you using?
> 
> > QEMU builds with static user-space probes.  You need to tell DTrace or
> > SystemTap to enable those probes in order to record trace data.
> 
> I'm using debian on a 4.6.0-1-amd64 kernel with systemtap 3.0.6.
> 
> I just gave it another try, and works if I use probes like:
> 
>   process("<binary-path>").mark("<event-name>")
> 
> although they don't seem to appear on "stap -l" or anything like that (I cannot
> find a "qemu" provider). But I'm still unable to print the event values. This:

The following enumerates events for me:

$ sudo stap -L 'process("/usr/bin/qemu-system-x86_64").mark("*")'
process("/usr/bin/qemu-system-x86_64").mark("alsa_no_frames") $arg1:long
process("/usr/bin/qemu-system-x86_64").mark("alsa_pollout") $arg1:long $arg2:long
process("/usr/bin/qemu-system-x86_64").mark("alsa_read_zero") $arg1:long
...

You can also use /usr/share/systemtap/tapset/qemu-system-x86_64.stp.
I'm running Fedora 24 with qemu-system-x86 2.6.0-5.fc24 and SystemTap
3.0-3.fc24.

> Also, I'm still not sure how to interact with QEMU's monitor interface from
> within the probe code (probes execute in kernel mode, including "guru mode"
> code).

When SystemTap is used the QEMU monitor interface does nothing.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1932 bytes --]

On Wed, Aug 24, 2016 at 12:25:54PM +0200, Lluís Vilanova wrote:
> Stefan Hajnoczi writes:
> 
> > On Sun, Aug 21, 2016 at 02:32:34PM +0200, Lluís Vilanova wrote:
> >> Unfortuntely, I've been unable to to make dtrace recognise QEMU's events (I'm
> >> only able to see the host kernel events). If someone with more experience on it
> >> can help me use dtrace with QEMU's events, I'll also add the supporting library
> >> to let dtrace do the callout to QEMU's moitor interface and control the events,
> >> and add a prperly useful example of that on the hypertrace docs (which was my
> >> original intention).
> 
> > Which "dtrace" and host OS are you using?
> 
> > QEMU builds with static user-space probes.  You need to tell DTrace or
> > SystemTap to enable those probes in order to record trace data.
> 
> I'm using debian on a 4.6.0-1-amd64 kernel with systemtap 3.0.6.
> 
> I just gave it another try, and works if I use probes like:
> 
>   process("<binary-path>").mark("<event-name>")
> 
> although they don't seem to appear on "stap -l" or anything like that (I cannot
> find a "qemu" provider). But I'm still unable to print the event values. This:

The following enumerates events for me:

$ sudo stap -L 'process("/usr/bin/qemu-system-x86_64").mark("*")'
process("/usr/bin/qemu-system-x86_64").mark("alsa_no_frames") $arg1:long
process("/usr/bin/qemu-system-x86_64").mark("alsa_pollout") $arg1:long $arg2:long
process("/usr/bin/qemu-system-x86_64").mark("alsa_read_zero") $arg1:long
...

You can also use /usr/share/systemtap/tapset/qemu-system-x86_64.stp.
I'm running Fedora 24 with qemu-system-x86 2.6.0-5.fc24 and SystemTap
3.0-3.fc24.

> Also, I'm still not sure how to interact with QEMU's monitor interface from
> within the probe code (probes execute in kernel mode, including "guru mode"
> code).

When SystemTap is used the QEMU monitor interface does nothing.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Lluís Vilanova]

Stefan Hajnoczi writes:

> On Wed, Aug 24, 2016 at 12:25:54PM +0200, Lluís Vilanova wrote:
>> Stefan Hajnoczi writes:
>> 
>> > On Sun, Aug 21, 2016 at 02:32:34PM +0200, Lluís Vilanova wrote:
>> >> Unfortuntely, I've been unable to to make dtrace recognise QEMU's events (I'm
>> >> only able to see the host kernel events). If someone with more experience on it
>> >> can help me use dtrace with QEMU's events, I'll also add the supporting library
>> >> to let dtrace do the callout to QEMU's moitor interface and control the events,
>> >> and add a prperly useful example of that on the hypertrace docs (which was my
>> >> original intention).
>> 
>> > Which "dtrace" and host OS are you using?
>> 
>> > QEMU builds with static user-space probes.  You need to tell DTrace or
>> > SystemTap to enable those probes in order to record trace data.
>> 
>> I'm using debian on a 4.6.0-1-amd64 kernel with systemtap 3.0.6.
>> 
>> I just gave it another try, and works if I use probes like:
>> 
>> process("<binary-path>").mark("<event-name>")
>> 
>> although they don't seem to appear on "stap -l" or anything like that (I cannot
>> find a "qemu" provider). But I'm still unable to print the event values. This:

> The following enumerates events for me:

> $ sudo stap -L 'process("/usr/bin/qemu-system-x86_64").mark("*")'
> process("/usr/bin/qemu-system-x86_64").mark("alsa_no_frames") $arg1:long
> process("/usr/bin/qemu-system-x86_64").mark("alsa_pollout") $arg1:long $arg2:long
> process("/usr/bin/qemu-system-x86_64").mark("alsa_read_zero") $arg1:long
> ...

> You can also use /usr/share/systemtap/tapset/qemu-system-x86_64.stp.
> I'm running Fedora 24 with qemu-system-x86 2.6.0-5.fc24 and SystemTap
> 3.0-3.fc24.

I meant I cannot just use "qemu" as a provider name. Later I discovered that
"process()" does work.


>> Also, I'm still not sure how to interact with QEMU's monitor interface from
>> within the probe code (probes execute in kernel mode, including "guru mode"
>> code).

> When SystemTap is used the QEMU monitor interface does nothing.

That's not what I've experienced. I was able to use a stap script to change the
tracing state of events:

   #!/usr/bin/env stap

   %{
   #include </home/lluis/Projects/qemu-dbi-test/test.h>
   %}

   function event:long(cpu:long, addr:long, info:long)
   %{
       char *argv[4] = {"/bin/sh", "-c", "echo 'trace-event * off' | telnet localhost 1234", NULL};
       call_usermodehelper(argv[0], argv, NULL, UMH_WAIT_EXEC);
       STAP_RETURN(0);
   %}

   probe begin {
       printf("hello\n")
   }
   probe process("./install/vanilla/bin/qemu-system-i386").mark("guest_mem_before_exec")
   {
       printf("%x %d %d\n", $arg1, $arg2, $arg3)
       event($arg1, $arg2, $arg3)
       exit()
   }

The only caveat is that you must pass the "-g" argument to stap.

Also, for some reason the printf in the probe always prints zeros, no matter
what the actual event receives (I've debugged QEMU down to the call to the
auto-generated stap functions). Could this be an error in systemtap?


Thanks,
  Lluis
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Lluís Vilanova]

Stefan Hajnoczi writes:

> On Wed, Aug 24, 2016 at 12:25:54PM +0200, Lluís Vilanova wrote:
>> Stefan Hajnoczi writes:
>> 
>> > On Sun, Aug 21, 2016 at 02:32:34PM +0200, Lluís Vilanova wrote:
>> >> Unfortuntely, I've been unable to to make dtrace recognise QEMU's events (I'm
>> >> only able to see the host kernel events). If someone with more experience on it
>> >> can help me use dtrace with QEMU's events, I'll also add the supporting library
>> >> to let dtrace do the callout to QEMU's moitor interface and control the events,
>> >> and add a prperly useful example of that on the hypertrace docs (which was my
>> >> original intention).
>> 
>> > Which "dtrace" and host OS are you using?
>> 
>> > QEMU builds with static user-space probes.  You need to tell DTrace or
>> > SystemTap to enable those probes in order to record trace data.
>> 
>> I'm using debian on a 4.6.0-1-amd64 kernel with systemtap 3.0.6.
>> 
>> I just gave it another try, and works if I use probes like:
>> 
>> process("<binary-path>").mark("<event-name>")
>> 
>> although they don't seem to appear on "stap -l" or anything like that (I cannot
>> find a "qemu" provider). But I'm still unable to print the event values. This:

> The following enumerates events for me:

> $ sudo stap -L 'process("/usr/bin/qemu-system-x86_64").mark("*")'
> process("/usr/bin/qemu-system-x86_64").mark("alsa_no_frames") $arg1:long
> process("/usr/bin/qemu-system-x86_64").mark("alsa_pollout") $arg1:long $arg2:long
> process("/usr/bin/qemu-system-x86_64").mark("alsa_read_zero") $arg1:long
> ...

> You can also use /usr/share/systemtap/tapset/qemu-system-x86_64.stp.
> I'm running Fedora 24 with qemu-system-x86 2.6.0-5.fc24 and SystemTap
> 3.0-3.fc24.

I meant I cannot just use "qemu" as a provider name. Later I discovered that
"process()" does work.


>> Also, I'm still not sure how to interact with QEMU's monitor interface from
>> within the probe code (probes execute in kernel mode, including "guru mode"
>> code).

> When SystemTap is used the QEMU monitor interface does nothing.

That's not what I've experienced. I was able to use a stap script to change the
tracing state of events:

   #!/usr/bin/env stap

   %{
   #include </home/lluis/Projects/qemu-dbi-test/test.h>
   %}

   function event:long(cpu:long, addr:long, info:long)
   %{
       char *argv[4] = {"/bin/sh", "-c", "echo 'trace-event * off' | telnet localhost 1234", NULL};
       call_usermodehelper(argv[0], argv, NULL, UMH_WAIT_EXEC);
       STAP_RETURN(0);
   %}

   probe begin {
       printf("hello\n")
   }
   probe process("./install/vanilla/bin/qemu-system-i386").mark("guest_mem_before_exec")
   {
       printf("%x %d %d\n", $arg1, $arg2, $arg3)
       event($arg1, $arg2, $arg3)
       exit()
   }

The only caveat is that you must pass the "-g" argument to stap.

Also, for some reason the printf in the probe always prints zeros, no matter
what the actual event receives (I've debugged QEMU down to the call to the
auto-generated stap functions). Could this be an error in systemtap?


Thanks,
  Lluis

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Stefan Hajnoczi]

[-- Attachment #1.1: Type: text/plain, Size: 1657 bytes --]

On Mon, Aug 29, 2016 at 08:46:02PM +0200, Lluís Vilanova wrote:
> >> Also, I'm still not sure how to interact with QEMU's monitor interface from
> >> within the probe code (probes execute in kernel mode, including "guru mode"
> >> code).
> 
> > When SystemTap is used the QEMU monitor interface does nothing.
> 
> That's not what I've experienced. I was able to use a stap script to change the
> tracing state of events:
> 
>    #!/usr/bin/env stap
> 
>    %{
>    #include </home/lluis/Projects/qemu-dbi-test/test.h>
>    %}
> 
>    function event:long(cpu:long, addr:long, info:long)
>    %{
>        char *argv[4] = {"/bin/sh", "-c", "echo 'trace-event * off' | telnet localhost 1234", NULL};
>        call_usermodehelper(argv[0], argv, NULL, UMH_WAIT_EXEC);
>        STAP_RETURN(0);
>    %}
> 
>    probe begin {
>        printf("hello\n")
>    }
>    probe process("./install/vanilla/bin/qemu-system-i386").mark("guest_mem_before_exec")
>    {
>        printf("%x %d %d\n", $arg1, $arg2, $arg3)
>        event($arg1, $arg2, $arg3)
>        exit()
>    }
> 
> The only caveat is that you must pass the "-g" argument to stap.
> 
> Also, for some reason the printf in the probe always prints zeros, no matter
> what the actual event receives (I've debugged QEMU down to the call to the
> auto-generated stap functions). Could this be an error in systemtap?

It's strange that arguments do not have valid values.  Debugging the
stap functions is the next step if you want to figure out what happened.
I've never had this issue before so maybe something with Debian
SystemTap userspace probes is broken.

Stefan

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1657 bytes --]

On Mon, Aug 29, 2016 at 08:46:02PM +0200, Lluís Vilanova wrote:
> >> Also, I'm still not sure how to interact with QEMU's monitor interface from
> >> within the probe code (probes execute in kernel mode, including "guru mode"
> >> code).
> 
> > When SystemTap is used the QEMU monitor interface does nothing.
> 
> That's not what I've experienced. I was able to use a stap script to change the
> tracing state of events:
> 
>    #!/usr/bin/env stap
> 
>    %{
>    #include </home/lluis/Projects/qemu-dbi-test/test.h>
>    %}
> 
>    function event:long(cpu:long, addr:long, info:long)
>    %{
>        char *argv[4] = {"/bin/sh", "-c", "echo 'trace-event * off' | telnet localhost 1234", NULL};
>        call_usermodehelper(argv[0], argv, NULL, UMH_WAIT_EXEC);
>        STAP_RETURN(0);
>    %}
> 
>    probe begin {
>        printf("hello\n")
>    }
>    probe process("./install/vanilla/bin/qemu-system-i386").mark("guest_mem_before_exec")
>    {
>        printf("%x %d %d\n", $arg1, $arg2, $arg3)
>        event($arg1, $arg2, $arg3)
>        exit()
>    }
> 
> The only caveat is that you must pass the "-g" argument to stap.
> 
> Also, for some reason the printf in the probe always prints zeros, no matter
> what the actual event receives (I've debugged QEMU down to the call to the
> auto-generated stap functions). Could this be an error in systemtap?

It's strange that arguments do not have valid values.  Debugging the
stap functions is the next step if you want to figure out what happened.
I've never had this issue before so maybe something with Debian
SystemTap userspace probes is broken.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Lluís Vilanova]

Stefan Hajnoczi writes:

> On Mon, Aug 29, 2016 at 08:46:02PM +0200, Lluís Vilanova wrote:
>> >> Also, I'm still not sure how to interact with QEMU's monitor interface from
>> >> within the probe code (probes execute in kernel mode, including "guru mode"
>> >> code).
>> 
>> > When SystemTap is used the QEMU monitor interface does nothing.
>> 
>> That's not what I've experienced. I was able to use a stap script to change the
>> tracing state of events:
>> 
>> #!/usr/bin/env stap
>> 
>> %{
>> #include </home/lluis/Projects/qemu-dbi-test/test.h>
>> %}
>> 
>> function event:long(cpu:long, addr:long, info:long)
>> %{
>> char *argv[4] = {"/bin/sh", "-c", "echo 'trace-event * off' | telnet localhost 1234", NULL};
>> call_usermodehelper(argv[0], argv, NULL, UMH_WAIT_EXEC);
>> STAP_RETURN(0);
>> %}
>> 
>> probe begin {
>> printf("hello\n")
>> }
>> probe process("./install/vanilla/bin/qemu-system-i386").mark("guest_mem_before_exec")
>> {
>> printf("%x %d %d\n", $arg1, $arg2, $arg3)
>> event($arg1, $arg2, $arg3)
>> exit()
>> }
>> 
>> The only caveat is that you must pass the "-g" argument to stap.
>> 
>> Also, for some reason the printf in the probe always prints zeros, no matter
>> what the actual event receives (I've debugged QEMU down to the call to the
>> auto-generated stap functions). Could this be an error in systemtap?

> It's strange that arguments do not have valid values.  Debugging the
> stap functions is the next step if you want to figure out what happened.
> I've never had this issue before so maybe something with Debian
> SystemTap userspace probes is broken.

I already debugged it, to the point where QEMU executes the trap injected by
systemtap, and the register values that were supposed to hold the arguments are
correct.

I suppose that if you execute the stap script I pasted it will show the proper
values. Then it's definitely a problem with Debian's userspace probes.


Thanks,
  Lluis
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Lluís Vilanova]

Stefan Hajnoczi writes:

> On Mon, Aug 29, 2016 at 08:46:02PM +0200, Lluís Vilanova wrote:
>> >> Also, I'm still not sure how to interact with QEMU's monitor interface from
>> >> within the probe code (probes execute in kernel mode, including "guru mode"
>> >> code).
>> 
>> > When SystemTap is used the QEMU monitor interface does nothing.
>> 
>> That's not what I've experienced. I was able to use a stap script to change the
>> tracing state of events:
>> 
>> #!/usr/bin/env stap
>> 
>> %{
>> #include </home/lluis/Projects/qemu-dbi-test/test.h>
>> %}
>> 
>> function event:long(cpu:long, addr:long, info:long)
>> %{
>> char *argv[4] = {"/bin/sh", "-c", "echo 'trace-event * off' | telnet localhost 1234", NULL};
>> call_usermodehelper(argv[0], argv, NULL, UMH_WAIT_EXEC);
>> STAP_RETURN(0);
>> %}
>> 
>> probe begin {
>> printf("hello\n")
>> }
>> probe process("./install/vanilla/bin/qemu-system-i386").mark("guest_mem_before_exec")
>> {
>> printf("%x %d %d\n", $arg1, $arg2, $arg3)
>> event($arg1, $arg2, $arg3)
>> exit()
>> }
>> 
>> The only caveat is that you must pass the "-g" argument to stap.
>> 
>> Also, for some reason the printf in the probe always prints zeros, no matter
>> what the actual event receives (I've debugged QEMU down to the call to the
>> auto-generated stap functions). Could this be an error in systemtap?

> It's strange that arguments do not have valid values.  Debugging the
> stap functions is the next step if you want to figure out what happened.
> I've never had this issue before so maybe something with Debian
> SystemTap userspace probes is broken.

I already debugged it, to the point where QEMU executes the trap injected by
systemtap, and the register values that were supposed to hold the arguments are
correct.

I suppose that if you execute the stap script I pasted it will show the proper
values. Then it's definitely a problem with Debian's userspace probes.


Thanks,
  Lluis

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Masami Hiramatsu]

On Mon, 05 Sep 2016 16:37:01 +0200
Lluís Vilanova <vilanova@ac.upc.edu> wrote:

> Stefan Hajnoczi writes:
> 
> > On Mon, Aug 29, 2016 at 08:46:02PM +0200, Lluís Vilanova wrote:
> >> >> Also, I'm still not sure how to interact with QEMU's monitor interface from
> >> >> within the probe code (probes execute in kernel mode, including "guru mode"
> >> >> code).
> >> 
> >> > When SystemTap is used the QEMU monitor interface does nothing.
> >> 
> >> That's not what I've experienced. I was able to use a stap script to change the
> >> tracing state of events:
> >> 
> >> #!/usr/bin/env stap
> >> 
> >> %{
> >> #include </home/lluis/Projects/qemu-dbi-test/test.h>
> >> %}
> >> 
> >> function event:long(cpu:long, addr:long, info:long)
> >> %{
> >> char *argv[4] = {"/bin/sh", "-c", "echo 'trace-event * off' | telnet localhost 1234", NULL};
> >> call_usermodehelper(argv[0], argv, NULL, UMH_WAIT_EXEC);
> >> STAP_RETURN(0);
> >> %}
> >> 
> >> probe begin {
> >> printf("hello\n")
> >> }
> >> probe process("./install/vanilla/bin/qemu-system-i386").mark("guest_mem_before_exec")
> >> {
> >> printf("%x %d %d\n", $arg1, $arg2, $arg3)
> >> event($arg1, $arg2, $arg3)
> >> exit()
> >> }
> >> 
> >> The only caveat is that you must pass the "-g" argument to stap.
> >> 
> >> Also, for some reason the printf in the probe always prints zeros, no matter
> >> what the actual event receives (I've debugged QEMU down to the call to the
> >> auto-generated stap functions). Could this be an error in systemtap?
> 
> > It's strange that arguments do not have valid values.  Debugging the
> > stap functions is the next step if you want to figure out what happened.
> > I've never had this issue before so maybe something with Debian
> > SystemTap userspace probes is broken.
> 
> I already debugged it, to the point where QEMU executes the trap injected by
> systemtap, and the register values that were supposed to hold the arguments are
> correct.
> 
> I suppose that if you execute the stap script I pasted it will show the proper
> values. Then it's definitely a problem with Debian's userspace probes.

Would you have tried to update your kernel to mainline and tested it ?
If it occurs, you also should try to use a raw uprobe via ftrace(uprobe_events)
and perftools.
If you have the latest perf (maybe you'll need checkout the latest tip tree),
you can use SDT as below (currently it doesn't support args, so you'll need
debuginfo.)

# perf buildid-cache --add ./install/vanilla/bin/qemu-system-i386
# perf probe -x ./install/vanilla/bin/qemu-system-i386 -a 'guest_mem_before_exec $vars'

And you'll see new event is registered which can be traced by ftrace or perf.

Thanks,

-- 
Masami Hiramatsu <mhiramat@kernel.org>

Re: [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Masami Hiramatsu]

On Mon, 05 Sep 2016 16:37:01 +0200
Lluís Vilanova <vilanova@ac.upc.edu> wrote:

> Stefan Hajnoczi writes:
> 
> > On Mon, Aug 29, 2016 at 08:46:02PM +0200, Lluís Vilanova wrote:
> >> >> Also, I'm still not sure how to interact with QEMU's monitor interface from
> >> >> within the probe code (probes execute in kernel mode, including "guru mode"
> >> >> code).
> >> 
> >> > When SystemTap is used the QEMU monitor interface does nothing.
> >> 
> >> That's not what I've experienced. I was able to use a stap script to change the
> >> tracing state of events:
> >> 
> >> #!/usr/bin/env stap
> >> 
> >> %{
> >> #include </home/lluis/Projects/qemu-dbi-test/test.h>
> >> %}
> >> 
> >> function event:long(cpu:long, addr:long, info:long)
> >> %{
> >> char *argv[4] = {"/bin/sh", "-c", "echo 'trace-event * off' | telnet localhost 1234", NULL};
> >> call_usermodehelper(argv[0], argv, NULL, UMH_WAIT_EXEC);
> >> STAP_RETURN(0);
> >> %}
> >> 
> >> probe begin {
> >> printf("hello\n")
> >> }
> >> probe process("./install/vanilla/bin/qemu-system-i386").mark("guest_mem_before_exec")
> >> {
> >> printf("%x %d %d\n", $arg1, $arg2, $arg3)
> >> event($arg1, $arg2, $arg3)
> >> exit()
> >> }
> >> 
> >> The only caveat is that you must pass the "-g" argument to stap.
> >> 
> >> Also, for some reason the printf in the probe always prints zeros, no matter
> >> what the actual event receives (I've debugged QEMU down to the call to the
> >> auto-generated stap functions). Could this be an error in systemtap?
> 
> > It's strange that arguments do not have valid values.  Debugging the
> > stap functions is the next step if you want to figure out what happened.
> > I've never had this issue before so maybe something with Debian
> > SystemTap userspace probes is broken.
> 
> I already debugged it, to the point where QEMU executes the trap injected by
> systemtap, and the register values that were supposed to hold the arguments are
> correct.
> 
> I suppose that if you execute the stap script I pasted it will show the proper
> values. Then it's definitely a problem with Debian's userspace probes.

Would you have tried to update your kernel to mainline and tested it ?
If it occurs, you also should try to use a raw uprobe via ftrace(uprobe_events)
and perftools.
If you have the latest perf (maybe you'll need checkout the latest tip tree),
you can use SDT as below (currently it doesn't support args, so you'll need
debuginfo.)

# perf buildid-cache --add ./install/vanilla/bin/qemu-system-i386
# perf probe -x ./install/vanilla/bin/qemu-system-i386 -a 'guest_mem_before_exec $vars'

And you'll see new event is registered which can be traced by ftrace or perf.

Thanks,

-- 
Masami Hiramatsu <mhiramat@kernel.org>

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Lluís Vilanova]

Masami Hiramatsu writes:

> On Mon, 05 Sep 2016 16:37:01 +0200
> Lluís Vilanova <vilanova@ac.upc.edu> wrote:

>> Stefan Hajnoczi writes:
>> 
>> > On Mon, Aug 29, 2016 at 08:46:02PM +0200, Lluís Vilanova wrote:
>> >> >> Also, I'm still not sure how to interact with QEMU's monitor interface from
>> >> >> within the probe code (probes execute in kernel mode, including "guru mode"
>> >> >> code).
>> >> 
>> >> > When SystemTap is used the QEMU monitor interface does nothing.
>> >> 
>> >> That's not what I've experienced. I was able to use a stap script to change the
>> >> tracing state of events:
>> >> 
>> >> #!/usr/bin/env stap
>> >> 
>> >> %{
>> >> #include </home/lluis/Projects/qemu-dbi-test/test.h>
>> >> %}
>> >> 
>> >> function event:long(cpu:long, addr:long, info:long)
>> >> %{
>> >> char *argv[4] = {"/bin/sh", "-c", "echo 'trace-event * off' | telnet localhost 1234", NULL};
>> >> call_usermodehelper(argv[0], argv, NULL, UMH_WAIT_EXEC);
>> >> STAP_RETURN(0);
>> >> %}
>> >> 
>> >> probe begin {
>> >> printf("hello\n")
>> >> }
>> >> probe process("./install/vanilla/bin/qemu-system-i386").mark("guest_mem_before_exec")
>> >> {
>> >> printf("%x %d %d\n", $arg1, $arg2, $arg3)
>> >> event($arg1, $arg2, $arg3)
>> >> exit()
>> >> }
>> >> 
>> >> The only caveat is that you must pass the "-g" argument to stap.
>> >> 
>> >> Also, for some reason the printf in the probe always prints zeros, no matter
>> >> what the actual event receives (I've debugged QEMU down to the call to the
>> >> auto-generated stap functions). Could this be an error in systemtap?
>> 
>> > It's strange that arguments do not have valid values.  Debugging the
>> > stap functions is the next step if you want to figure out what happened.
>> > I've never had this issue before so maybe something with Debian
>> > SystemTap userspace probes is broken.
>> 
>> I already debugged it, to the point where QEMU executes the trap injected by
>> systemtap, and the register values that were supposed to hold the arguments are
>> correct.
>> 
>> I suppose that if you execute the stap script I pasted it will show the proper
>> values. Then it's definitely a problem with Debian's userspace probes.

> Would you have tried to update your kernel to mainline and tested it ?

I've compiled the tarball for 4.8-rc5 (.config from "make localmodconfig") and a
printf of the probe arguments still shows zeroes. Also, I've had to add a small
patch to [1] to properly lock/unlock inodes in this kernel version (using
inode_lock/unlock instead of mutex_lock/unlock on inode->i_mutex).

[1] /usr/share/systemtap/runtime/transport/transport.c


> If it occurs, you also should try to use a raw uprobe via ftrace(uprobe_events)
> and perftools.
> If you have the latest perf (maybe you'll need checkout the latest tip tree),
> you can use SDT as below (currently it doesn't support args, so you'll need
> debuginfo.)

> # perf buildid-cache --add ./install/vanilla/bin/qemu-system-i386
> # perf probe -x ./install/vanilla/bin/qemu-system-i386 -a 'guest_mem_before_exec $vars'

> And you'll see new event is registered which can be traced by ftrace or perf.

It does show something (I'm interested in stap probe "guest_hypertrace", raised
on function "trace_guest_hypertrace"), but is incorrect:

    50.00%  (55e36d5ee32b) __cpu=0x55e370446fd0 arg1=0xcafe
    50.00%  (55e36d5ee32b) __cpu=0x7f2ee0a10b20 arg1=0x198

My test app calls "trace_guest_hypertrace" twice, always with the same "__cpu"
and "arg1" argument values. Just in case, this is runnign a QEMU compiled with
"-O0".

Running "record" multiple times shows different "random" values on the
arguments, and keeps changing which of the two trace elements shows incorrect
values.


Thanks,
  Lluis
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Lluís Vilanova]

Masami Hiramatsu writes:

> On Mon, 05 Sep 2016 16:37:01 +0200
> Lluís Vilanova <vilanova@ac.upc.edu> wrote:

>> Stefan Hajnoczi writes:
>> 
>> > On Mon, Aug 29, 2016 at 08:46:02PM +0200, Lluís Vilanova wrote:
>> >> >> Also, I'm still not sure how to interact with QEMU's monitor interface from
>> >> >> within the probe code (probes execute in kernel mode, including "guru mode"
>> >> >> code).
>> >> 
>> >> > When SystemTap is used the QEMU monitor interface does nothing.
>> >> 
>> >> That's not what I've experienced. I was able to use a stap script to change the
>> >> tracing state of events:
>> >> 
>> >> #!/usr/bin/env stap
>> >> 
>> >> %{
>> >> #include </home/lluis/Projects/qemu-dbi-test/test.h>
>> >> %}
>> >> 
>> >> function event:long(cpu:long, addr:long, info:long)
>> >> %{
>> >> char *argv[4] = {"/bin/sh", "-c", "echo 'trace-event * off' | telnet localhost 1234", NULL};
>> >> call_usermodehelper(argv[0], argv, NULL, UMH_WAIT_EXEC);
>> >> STAP_RETURN(0);
>> >> %}
>> >> 
>> >> probe begin {
>> >> printf("hello\n")
>> >> }
>> >> probe process("./install/vanilla/bin/qemu-system-i386").mark("guest_mem_before_exec")
>> >> {
>> >> printf("%x %d %d\n", $arg1, $arg2, $arg3)
>> >> event($arg1, $arg2, $arg3)
>> >> exit()
>> >> }
>> >> 
>> >> The only caveat is that you must pass the "-g" argument to stap.
>> >> 
>> >> Also, for some reason the printf in the probe always prints zeros, no matter
>> >> what the actual event receives (I've debugged QEMU down to the call to the
>> >> auto-generated stap functions). Could this be an error in systemtap?
>> 
>> > It's strange that arguments do not have valid values.  Debugging the
>> > stap functions is the next step if you want to figure out what happened.
>> > I've never had this issue before so maybe something with Debian
>> > SystemTap userspace probes is broken.
>> 
>> I already debugged it, to the point where QEMU executes the trap injected by
>> systemtap, and the register values that were supposed to hold the arguments are
>> correct.
>> 
>> I suppose that if you execute the stap script I pasted it will show the proper
>> values. Then it's definitely a problem with Debian's userspace probes.

> Would you have tried to update your kernel to mainline and tested it ?

I've compiled the tarball for 4.8-rc5 (.config from "make localmodconfig") and a
printf of the probe arguments still shows zeroes. Also, I've had to add a small
patch to [1] to properly lock/unlock inodes in this kernel version (using
inode_lock/unlock instead of mutex_lock/unlock on inode->i_mutex).

[1] /usr/share/systemtap/runtime/transport/transport.c


> If it occurs, you also should try to use a raw uprobe via ftrace(uprobe_events)
> and perftools.
> If you have the latest perf (maybe you'll need checkout the latest tip tree),
> you can use SDT as below (currently it doesn't support args, so you'll need
> debuginfo.)

> # perf buildid-cache --add ./install/vanilla/bin/qemu-system-i386
> # perf probe -x ./install/vanilla/bin/qemu-system-i386 -a 'guest_mem_before_exec $vars'

> And you'll see new event is registered which can be traced by ftrace or perf.

It does show something (I'm interested in stap probe "guest_hypertrace", raised
on function "trace_guest_hypertrace"), but is incorrect:

    50.00%  (55e36d5ee32b) __cpu=0x55e370446fd0 arg1=0xcafe
    50.00%  (55e36d5ee32b) __cpu=0x7f2ee0a10b20 arg1=0x198

My test app calls "trace_guest_hypertrace" twice, always with the same "__cpu"
and "arg1" argument values. Just in case, this is runnign a QEMU compiled with
"-O0".

Running "record" multiple times shows different "random" values on the
arguments, and keeps changing which of the two trace elements shows incorrect
values.


Thanks,
  Lluis

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1016 bytes --]

On Mon, Sep 05, 2016 at 04:37:01PM +0200, Lluís Vilanova wrote:
> I suppose that if you execute the stap script I pasted it will show the proper
> values. Then it's definitely a problem with Debian's userspace probes.

Sorry for the delay.  SystemTap static probes appear to work correctly on
Fedora 24.

I built qemu.git/master from source with "--enable-trace-backends=dtrace" and
tried the following:

$ rpm -qi systemtap kernel-devel | grep Source
Source RPM  : systemtap-3.0-3.fc24.src.rpm
Source RPM  : kernel-4.7.2-201.fc24.src.rpm

(By the way, I hit the same mutex_lock() vs inode_lock() issue in systemtap as you.)

$ cat test.stp
probe begin {
	printf("hello\n");
}

probe process("path/to/qemu-system-x86_64").mark("kvm_ioctl")
{
	printf("%x %p\n", $arg1, $arg2)
}

$ sudo stap test.stp -c 'path/to/qemu-system-x86_64 -enable-kvm -m 1024'
hello
ae00 0x0
ae03 0xa
ae03 0x9
ae03 0x42
ae01 0x0

These are valid argument values.  What happens on your Debian box?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1016 bytes --]

On Mon, Sep 05, 2016 at 04:37:01PM +0200, Lluís Vilanova wrote:
> I suppose that if you execute the stap script I pasted it will show the proper
> values. Then it's definitely a problem with Debian's userspace probes.

Sorry for the delay.  SystemTap static probes appear to work correctly on
Fedora 24.

I built qemu.git/master from source with "--enable-trace-backends=dtrace" and
tried the following:

$ rpm -qi systemtap kernel-devel | grep Source
Source RPM  : systemtap-3.0-3.fc24.src.rpm
Source RPM  : kernel-4.7.2-201.fc24.src.rpm

(By the way, I hit the same mutex_lock() vs inode_lock() issue in systemtap as you.)

$ cat test.stp
probe begin {
	printf("hello\n");
}

probe process("path/to/qemu-system-x86_64").mark("kvm_ioctl")
{
	printf("%x %p\n", $arg1, $arg2)
}

$ sudo stap test.stp -c 'path/to/qemu-system-x86_64 -enable-kvm -m 1024'
hello
ae00 0x0
ae03 0xa
ae03 0x9
ae03 0x42
ae01 0x0

These are valid argument values.  What happens on your Debian box?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Lluís Vilanova]

Stefan Hajnoczi writes:

> On Mon, Sep 05, 2016 at 04:37:01PM +0200, Lluís Vilanova wrote:
>> I suppose that if you execute the stap script I pasted it will show the proper
>> values. Then it's definitely a problem with Debian's userspace probes.

> Sorry for the delay.  SystemTap static probes appear to work correctly on
> Fedora 24.

> I built qemu.git/master from source with "--enable-trace-backends=dtrace" and
> tried the following:

> $ rpm -qi systemtap kernel-devel | grep Source
> Source RPM  : systemtap-3.0-3.fc24.src.rpm
> Source RPM  : kernel-4.7.2-201.fc24.src.rpm

> (By the way, I hit the same mutex_lock() vs inode_lock() issue in systemtap as you.)

> $ cat test.stp
> probe begin {
> 	printf("hello\n");
> }

> probe process("path/to/qemu-system-x86_64").mark("kvm_ioctl")
> {
> 	printf("%x %p\n", $arg1, $arg2)
> }

> $ sudo stap test.stp -c 'path/to/qemu-system-x86_64 -enable-kvm -m 1024'
> hello
> ae00 0x0
> ae03 0xa
> ae03 0x9
> ae03 0x42
> ae01 0x0

> These are valid argument values.  What happens on your Debian box?

Like in my example, I get all zeroes. So I'll open a bug on debian and assume my
example stap script shows the proper values.

Thanks,
  Lluis
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Lluís Vilanova]

Stefan Hajnoczi writes:

> On Mon, Sep 05, 2016 at 04:37:01PM +0200, Lluís Vilanova wrote:
>> I suppose that if you execute the stap script I pasted it will show the proper
>> values. Then it's definitely a problem with Debian's userspace probes.

> Sorry for the delay.  SystemTap static probes appear to work correctly on
> Fedora 24.

> I built qemu.git/master from source with "--enable-trace-backends=dtrace" and
> tried the following:

> $ rpm -qi systemtap kernel-devel | grep Source
> Source RPM  : systemtap-3.0-3.fc24.src.rpm
> Source RPM  : kernel-4.7.2-201.fc24.src.rpm

> (By the way, I hit the same mutex_lock() vs inode_lock() issue in systemtap as you.)

> $ cat test.stp
> probe begin {
> 	printf("hello\n");
> }

> probe process("path/to/qemu-system-x86_64").mark("kvm_ioctl")
> {
> 	printf("%x %p\n", $arg1, $arg2)
> }

> $ sudo stap test.stp -c 'path/to/qemu-system-x86_64 -enable-kvm -m 1024'
> hello
> ae00 0x0
> ae03 0xa
> ae03 0x9
> ae03 0x42
> ae01 0x0

> These are valid argument values.  What happens on your Debian box?

Like in my example, I get all zeroes. So I'll open a bug on debian and assume my
example stap script shows the proper values.

Thanks,
  Lluis

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Daniel P. Berrange]

On Mon, Aug 29, 2016 at 08:46:02PM +0200, Lluís Vilanova wrote:
> Stefan Hajnoczi writes:
> 
> > When SystemTap is used the QEMU monitor interface does nothing.
> 
> That's not what I've experienced. I was able to use a stap script to change the
> tracing state of events:
> 
>    #!/usr/bin/env stap
> 
>    %{
>    #include </home/lluis/Projects/qemu-dbi-test/test.h>
>    %}
> 
>    function event:long(cpu:long, addr:long, info:long)
>    %{
>        char *argv[4] = {"/bin/sh", "-c", "echo 'trace-event * off' | telnet localhost 1234", NULL};
>        call_usermodehelper(argv[0], argv, NULL, UMH_WAIT_EXEC);
>        STAP_RETURN(0);
>    %}

I don't know what you're trying to achieve here.  The trace-event state,
as changed/viewed via QEMU monitor, is irrelevant to the dtrace (systemtap)
backend. dtrace and ltt-ust are both fully dynamic trace event backends,
so the QEMU event state has no effect on them. The probe points in the
binary are dynamically enabled / disabled by the dtrace runtime. ie dtrace
will automatically enable an event if you write a dtrace script that uses
the event.


Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Daniel P. Berrange]

On Mon, Aug 29, 2016 at 08:46:02PM +0200, Lluís Vilanova wrote:
> Stefan Hajnoczi writes:
> 
> > When SystemTap is used the QEMU monitor interface does nothing.
> 
> That's not what I've experienced. I was able to use a stap script to change the
> tracing state of events:
> 
>    #!/usr/bin/env stap
> 
>    %{
>    #include </home/lluis/Projects/qemu-dbi-test/test.h>
>    %}
> 
>    function event:long(cpu:long, addr:long, info:long)
>    %{
>        char *argv[4] = {"/bin/sh", "-c", "echo 'trace-event * off' | telnet localhost 1234", NULL};
>        call_usermodehelper(argv[0], argv, NULL, UMH_WAIT_EXEC);
>        STAP_RETURN(0);
>    %}

I don't know what you're trying to achieve here.  The trace-event state,
as changed/viewed via QEMU monitor, is irrelevant to the dtrace (systemtap)
backend. dtrace and ltt-ust are both fully dynamic trace event backends,
so the QEMU event state has no effect on them. The probe points in the
binary are dynamically enabled / disabled by the dtrace runtime. ie dtrace
will automatically enable an event if you write a dtrace script that uses
the event.


Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Lluís Vilanova]

Daniel P Berrange writes:

> On Mon, Aug 29, 2016 at 08:46:02PM +0200, Lluís Vilanova wrote:
>> Stefan Hajnoczi writes:
>> 
>> > When SystemTap is used the QEMU monitor interface does nothing.
>> 
>> That's not what I've experienced. I was able to use a stap script to change the
>> tracing state of events:
>> 
>> #!/usr/bin/env stap
>> 
>> %{
>> #include </home/lluis/Projects/qemu-dbi-test/test.h>
>> %}
>> 
>> function event:long(cpu:long, addr:long, info:long)
>> %{
>> char *argv[4] = {"/bin/sh", "-c", "echo 'trace-event * off' | telnet localhost 1234", NULL};
>> call_usermodehelper(argv[0], argv, NULL, UMH_WAIT_EXEC);
>> STAP_RETURN(0);
>> %}

> I don't know what you're trying to achieve here.  The trace-event state,
> as changed/viewed via QEMU monitor, is irrelevant to the dtrace (systemtap)
> backend. dtrace and ltt-ust are both fully dynamic trace event backends,
> so the QEMU event state has no effect on them. The probe points in the
> binary are dynamically enabled / disabled by the dtrace runtime. ie dtrace
> will automatically enable an event if you write a dtrace script that uses
> the event.

Sorry, I did not properly explain the use case. This is an example of using
QEMU's tracing infrastructure to control itself. Here I'm using the "log"
backend to trace events to disk, and the "dtrace" backend (systemtap) to control
the tracing state of such events.

The guest code is something like:

  // some guest code executed *without* tracing enabled

  invoke_hypertrace_event(); // stap will enable some events here

  // some guest code executed *with* tracing enabled

  invoke_hypertrace_event(); // stap will disable some events here

  // some guest code executed *without* tracing enabled

So we can use it to mark regions of interest for tracing.


Cheers,
  Lluis

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Daniel P. Berrange]

On Mon, Sep 05, 2016 at 08:29:54PM +0200, Lluís Vilanova wrote:
> Daniel P Berrange writes:
> 
> > On Mon, Aug 29, 2016 at 08:46:02PM +0200, Lluís Vilanova wrote:
> >> Stefan Hajnoczi writes:
> >> 
> >> > When SystemTap is used the QEMU monitor interface does nothing.
> >> 
> >> That's not what I've experienced. I was able to use a stap script to change the
> >> tracing state of events:
> >> 
> >> #!/usr/bin/env stap
> >> 
> >> %{
> >> #include </home/lluis/Projects/qemu-dbi-test/test.h>
> >> %}
> >> 
> >> function event:long(cpu:long, addr:long, info:long)
> >> %{
> >> char *argv[4] = {"/bin/sh", "-c", "echo 'trace-event * off' | telnet localhost 1234", NULL};
> >> call_usermodehelper(argv[0], argv, NULL, UMH_WAIT_EXEC);
> >> STAP_RETURN(0);
> >> %}
> 
> > I don't know what you're trying to achieve here.  The trace-event state,
> > as changed/viewed via QEMU monitor, is irrelevant to the dtrace (systemtap)
> > backend. dtrace and ltt-ust are both fully dynamic trace event backends,
> > so the QEMU event state has no effect on them. The probe points in the
> > binary are dynamically enabled / disabled by the dtrace runtime. ie dtrace
> > will automatically enable an event if you write a dtrace script that uses
> > the event.
> 
> Sorry, I did not properly explain the use case. This is an example of using
> QEMU's tracing infrastructure to control itself. Here I'm using the "log"
> backend to trace events to disk, and the "dtrace" backend (systemtap) to control
> the tracing state of such events.

Ah, I see. I guess I'd personally just have systemtap/dtrace directly emit
the data to be recorded, and avoid the log backend entirely.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Lluís Vilanova]

Daniel P Berrange writes:

> On Mon, Sep 05, 2016 at 08:29:54PM +0200, Lluís Vilanova wrote:
>> Daniel P Berrange writes:
>> 
>> > On Mon, Aug 29, 2016 at 08:46:02PM +0200, Lluís Vilanova wrote:
>> >> Stefan Hajnoczi writes:
>> >> 
>> >> > When SystemTap is used the QEMU monitor interface does nothing.
>> >> 
>> >> That's not what I've experienced. I was able to use a stap script to change the
>> >> tracing state of events:
>> >> 
>> >> #!/usr/bin/env stap
>> >> 
>> >> %{
>> >> #include </home/lluis/Projects/qemu-dbi-test/test.h>
>> >> %}
>> >> 
>> >> function event:long(cpu:long, addr:long, info:long)
>> >> %{
>> >> char *argv[4] = {"/bin/sh", "-c", "echo 'trace-event * off' | telnet localhost 1234", NULL};
>> >> call_usermodehelper(argv[0], argv, NULL, UMH_WAIT_EXEC);
>> >> STAP_RETURN(0);
>> >> %}
>> 
>> > I don't know what you're trying to achieve here.  The trace-event state,
>> > as changed/viewed via QEMU monitor, is irrelevant to the dtrace (systemtap)
>> > backend. dtrace and ltt-ust are both fully dynamic trace event backends,
>> > so the QEMU event state has no effect on them. The probe points in the
>> > binary are dynamically enabled / disabled by the dtrace runtime. ie dtrace
>> > will automatically enable an event if you write a dtrace script that uses
>> > the event.
>> 
>> Sorry, I did not properly explain the use case. This is an example of using
>> QEMU's tracing infrastructure to control itself. Here I'm using the "log"
>> backend to trace events to disk, and the "dtrace" backend (systemtap) to control
>> the tracing state of such events.

> Ah, I see. I guess I'd personally just have systemtap/dtrace directly emit
> the data to be recorded, and avoid the log backend entirely.

Backends with a buffered output stream like "simple" should be faster than
"dtrace", which AFAIK injects a software trap. Also, you can take advantage of
existing QEMU infrastructure to control the tracing of events, instead of
implementing your own on each script.

Cheers,
  Lluis
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Lluís Vilanova]

Daniel P Berrange writes:

> On Mon, Sep 05, 2016 at 08:29:54PM +0200, Lluís Vilanova wrote:
>> Daniel P Berrange writes:
>> 
>> > On Mon, Aug 29, 2016 at 08:46:02PM +0200, Lluís Vilanova wrote:
>> >> Stefan Hajnoczi writes:
>> >> 
>> >> > When SystemTap is used the QEMU monitor interface does nothing.
>> >> 
>> >> That's not what I've experienced. I was able to use a stap script to change the
>> >> tracing state of events:
>> >> 
>> >> #!/usr/bin/env stap
>> >> 
>> >> %{
>> >> #include </home/lluis/Projects/qemu-dbi-test/test.h>
>> >> %}
>> >> 
>> >> function event:long(cpu:long, addr:long, info:long)
>> >> %{
>> >> char *argv[4] = {"/bin/sh", "-c", "echo 'trace-event * off' | telnet localhost 1234", NULL};
>> >> call_usermodehelper(argv[0], argv, NULL, UMH_WAIT_EXEC);
>> >> STAP_RETURN(0);
>> >> %}
>> 
>> > I don't know what you're trying to achieve here.  The trace-event state,
>> > as changed/viewed via QEMU monitor, is irrelevant to the dtrace (systemtap)
>> > backend. dtrace and ltt-ust are both fully dynamic trace event backends,
>> > so the QEMU event state has no effect on them. The probe points in the
>> > binary are dynamically enabled / disabled by the dtrace runtime. ie dtrace
>> > will automatically enable an event if you write a dtrace script that uses
>> > the event.
>> 
>> Sorry, I did not properly explain the use case. This is an example of using
>> QEMU's tracing infrastructure to control itself. Here I'm using the "log"
>> backend to trace events to disk, and the "dtrace" backend (systemtap) to control
>> the tracing state of such events.

> Ah, I see. I guess I'd personally just have systemtap/dtrace directly emit
> the data to be recorded, and avoid the log backend entirely.

Backends with a buffered output stream like "simple" should be faster than
"dtrace", which AFAIK injects a software trap. Also, you can take advantage of
existing QEMU infrastructure to control the tracing of events, instead of
implementing your own on each script.

Cheers,
  Lluis

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Daniel P. Berrange]

On Mon, Sep 05, 2016 at 08:29:54PM +0200, Lluís Vilanova wrote:
> Daniel P Berrange writes:
> 
> > On Mon, Aug 29, 2016 at 08:46:02PM +0200, Lluís Vilanova wrote:
> >> Stefan Hajnoczi writes:
> >> 
> >> > When SystemTap is used the QEMU monitor interface does nothing.
> >> 
> >> That's not what I've experienced. I was able to use a stap script to change the
> >> tracing state of events:
> >> 
> >> #!/usr/bin/env stap
> >> 
> >> %{
> >> #include </home/lluis/Projects/qemu-dbi-test/test.h>
> >> %}
> >> 
> >> function event:long(cpu:long, addr:long, info:long)
> >> %{
> >> char *argv[4] = {"/bin/sh", "-c", "echo 'trace-event * off' | telnet localhost 1234", NULL};
> >> call_usermodehelper(argv[0], argv, NULL, UMH_WAIT_EXEC);
> >> STAP_RETURN(0);
> >> %}
> 
> > I don't know what you're trying to achieve here.  The trace-event state,
> > as changed/viewed via QEMU monitor, is irrelevant to the dtrace (systemtap)
> > backend. dtrace and ltt-ust are both fully dynamic trace event backends,
> > so the QEMU event state has no effect on them. The probe points in the
> > binary are dynamically enabled / disabled by the dtrace runtime. ie dtrace
> > will automatically enable an event if you write a dtrace script that uses
> > the event.
> 
> Sorry, I did not properly explain the use case. This is an example of using
> QEMU's tracing infrastructure to control itself. Here I'm using the "log"
> backend to trace events to disk, and the "dtrace" backend (systemtap) to control
> the tracing state of such events.

Ah, I see. I guess I'd personally just have systemtap/dtrace directly emit
the data to be recorded, and avoid the log backend entirely.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Lluís Vilanova]

Daniel P Berrange writes:

> On Mon, Aug 29, 2016 at 08:46:02PM +0200, Lluís Vilanova wrote:
>> Stefan Hajnoczi writes:
>> 
>> > When SystemTap is used the QEMU monitor interface does nothing.
>> 
>> That's not what I've experienced. I was able to use a stap script to change the
>> tracing state of events:
>> 
>> #!/usr/bin/env stap
>> 
>> %{
>> #include </home/lluis/Projects/qemu-dbi-test/test.h>
>> %}
>> 
>> function event:long(cpu:long, addr:long, info:long)
>> %{
>> char *argv[4] = {"/bin/sh", "-c", "echo 'trace-event * off' | telnet localhost 1234", NULL};
>> call_usermodehelper(argv[0], argv, NULL, UMH_WAIT_EXEC);
>> STAP_RETURN(0);
>> %}

> I don't know what you're trying to achieve here.  The trace-event state,
> as changed/viewed via QEMU monitor, is irrelevant to the dtrace (systemtap)
> backend. dtrace and ltt-ust are both fully dynamic trace event backends,
> so the QEMU event state has no effect on them. The probe points in the
> binary are dynamically enabled / disabled by the dtrace runtime. ie dtrace
> will automatically enable an event if you write a dtrace script that uses
> the event.

Sorry, I did not properly explain the use case. This is an example of using
QEMU's tracing infrastructure to control itself. Here I'm using the "log"
backend to trace events to disk, and the "dtrace" backend (systemtap) to control
the tracing state of such events.

The guest code is something like:

  // some guest code executed *without* tracing enabled

  invoke_hypertrace_event(); // stap will enable some events here

  // some guest code executed *with* tracing enabled

  invoke_hypertrace_event(); // stap will disable some events here

  // some guest code executed *without* tracing enabled

So we can use it to mark regions of interest for tracing.


Cheers,
  Lluis
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Lluís Vilanova]

Stefan Hajnoczi writes:

> On Sun, Aug 21, 2016 at 02:32:34PM +0200, Lluís Vilanova wrote:
>> Unfortuntely, I've been unable to to make dtrace recognise QEMU's events (I'm
>> only able to see the host kernel events). If someone with more experience on it
>> can help me use dtrace with QEMU's events, I'll also add the supporting library
>> to let dtrace do the callout to QEMU's moitor interface and control the events,
>> and add a prperly useful example of that on the hypertrace docs (which was my
>> original intention).

> Which "dtrace" and host OS are you using?

> QEMU builds with static user-space probes.  You need to tell DTrace or
> SystemTap to enable those probes in order to record trace data.

I'm using debian on a 4.6.0-1-amd64 kernel with systemtap 3.0.6.

I just gave it another try, and works if I use probes like:

  process("<binary-path>").mark("<event-name>")

although they don't seem to appear on "stap -l" or anything like that (I cannot
find a "qemu" provider). But I'm still unable to print the event values. This:

  probe process("./install/vanilla/bin/qemu-system-i386").mark("guest_mem_before_exec")
  {
      printf("%p %lx %d\n", $arg1, $arg2, $arg3)
  }

always prints "0x0 0x0 0", which is clearly wrong (other backend on the same
build print the correct values).

Also, I'm still not sure how to interact with QEMU's monitor interface from
within the probe code (probes execute in kernel mode, including "guru mode"
code).

If anybody can shed some like into any of this, I'd appreaciate it.


Cheers,
  Lluis
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Lluís Vilanova]

Stefan Hajnoczi writes:

> On Fri, Aug 05, 2016 at 06:59:23PM +0200, Lluís Vilanova wrote:
>> The hypertrace channel allows guest code to emit events in QEMU (the host) using
>> its tracing infrastructure (see "docs/trace.txt"). This works in both 'system'
>> and 'user' modes. That is, hypertrace is to tracing, what hypercalls are to
>> system calls.
>> 
>> You can use this to emit an event on both guest and QEMU (host) traces to easily
>> synchronize or correlate them. You could also modify you guest's tracing system
>> to emit all events through the hypertrace channel, providing a unified and fully
>> synchronized trace log. Another use case is timing the performance of guest code
>> when optimizing TCG (QEMU traces have a timestamp).
>> 
>> See first commit for a full description.

> This tracing approach has a high performance overhead, particularly for
> SMP guests where each trace event requires writing to the global control
> register.  All CPUs will be hammering this register (heavyweight vmexit)
> for each trace event.

> I think the folks CCed on this email all take an asynchronous approach
> to avoid this performance overhead.  Synchronous means taking a VM exit
> for every event.  Asynchronous means writing trace data to a buffer and
> later interleaving guest data with host trace data.

> LTTng Userspace Tracer is an example of the asynchronous approach.  The
> trace data buffers are in shared memory.  The LTTng process can grab
> buffers at appropriate times.

> The ftrace virtio-serial approach has been to splice() the ftrace
> buffers, resulting in efficient I/O.

> Steven is working on a host/guest solution for trace-cmd.  It is also
> asynchronous.  No new paravirt hardware is needed and it makes me wonder
> whether the hypertrace PCI device is trying to solve the problem at the
> wrong layer.

> If you want to play around with asynchronous tracing, you could start
> with trace/simple.c.  It has a trace buffer that is asynchronously
> written out to file by a dedicated "writer" thread.

> The one case where hypertrace makes sense to me is for -user tracing.
> There QEMU can efficiently interleave guest and QEMU traces, although as
> mentioned in the patch, I don't think the SIGSEGV approach should be
> used.

> I suggest stripping this series down to focus on -user.  Synchronous
> tracing is not a good approach for -system emulation.

As I said, I wanted to implement a simple way to provide common reference points
in guest and host traces. For that, a *synchronous* guest-to-host channel is the
simplest way to do it (otherwise you're back to square one).

Another reason to have this is that I wanted to attach a dtrace/systemtap hook
to the hypertrace event, and from there dynamically control QEMU's tracing. Then
you can insert simple calls to hypertrace on your guest code to tell your dtrace
hooks when to enable/disable tracing of certain QEMU events (like guest instr
traces).

Unfortuntely, I've been unable to to make dtrace recognise QEMU's events (I'm
only able to see the host kernel events). If someone with more experience on it
can help me use dtrace with QEMU's events, I'll also add the supporting library
to let dtrace do the callout to QEMU's moitor interface and control the events,
and add a prperly useful example of that on the hypertrace docs (which was my
original intention).


Thanks,
  Lluis
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev