[PATCH 00/15] xen/tools: add tracing to various Xen subsystems

[Dario Faggioli <dario.faggioli@citrix.com>]

Hello,

While chasing and dealing with bugs, over this last period, I've found myself
augmenting Xen with quite a few new tracing capabilities, especially focusing
on:
 - IRQ being disabled and (re)enabled (in addition to the already existing
   tracing of IRQ related activity that we have);
 - RCU;
 - softirqs (I think I sent a preliminary version of this, long ago, but can't
   be sure);
 - tasklets;
 - timers;

And, apart from the first 4 patches (which are random, but still tracing
related, of course, improvements), this is what this patch series does: it adds
tracing to the Xen susystems listed above.

That happens, one subsystem after another, in patches 6 to 15.

Patch 5 deserves special mention. In fact, now that we have Kconfig, I thought
it could be a nice thing to make it possible to select, at build config time,
whether we want tracing or not, in the hypervisor (like, for instance, we do
for performance counters).

To be honest, my goal was to be able to compile tracing off, and run
benchmarks, to assess how much of a overhead tracing introduces, but then I
decided it was worth doing this properly, and now sending it.  I am still
running those benchmarks. Preliminary results seems to be showing that having
tracing support in the hypervisor does not (when it's disabled, of course)
introduce too much overhead. Still, I think it could be useful, to people that
wants a very specificly tailored version of Xen (embedded, small footprint,
etc.), to be able to rip it off nice and easily (e.g., like OpenXT guys did for
schedulers).

Of course, I will report here what I find, when benchmarks will finish running.
(In the meantime, patch 5 has some data about .text section shrinking in its
changelog.)

I also thought, for similar reasons, that it would have been good to be able to
also individually enable or disable the new tracing I'm introducing with this
series.  This potentially applies even to the tracing we already have in tree
(e.g., one may want to have tracing compiled in, for everything except that for
scheduling), and I'd be up for working on this. However, in this series, I am
touching really hot paths (with the exception, maybe, of RCU), and so I think
it is even more important to be able to disable tracing for them, for minimum
overhead.

For instance, the IRQ enabling and disabling tracing, I find it really really
usable for understanding certain class of behavior, and, with some scripting
and some more gnuplot "magic" (which I'll also share shortly) we can also use
it to automatically measure and graph for how long interrupt are kept disabled,
in various places within the hypervisor (Jennifer has done a similar analysis
for XenServer, a while back).  But it is very invasive, so you want to be able
to turn it on and off.
As said, I don't have all the result I need to present conclusions, but what I
see in preliminary data is that, although the tracing we currently have in Xen
is not too bad, performance wise, this new IRQs on/off tracing does have an
impact on performance, just for being there in the code (i.e., even when it is
there in the code but is *disabled*).

Of course, all this fine grain control of tracing options is hidden under
XEN_CONFIG_EXPERT, and all the new tracing is disabled by default.

There is a branch with this series here:
 git://xenbits.xen.org/people/dariof/xen.git  rel/tracing/xen-internals
 http://xenbits.xen.org/gitweb/?p=people/dariof/xen.git;a=shortlog;h=refs/heads/rel/tracing/xen-internals

And Travis is happy about it:
 https://travis-ci.org/fdario/xen/builds/238421024

Let me know what you think.

Thanks and Regards,
Dario
---
Dario Faggioli (15):
      xen: in do_softirq() sample smp_processor_id() once and for all.
      xen: tracing: avoid checking tb_init_done multiple times.
      xen/tools: tracing: several improvements on IRQs tracing
      tools: xenalyze: fix dumping of PM_IDLE events.
      xen: make it possible to disable tracing in Kconfig.
      xen: trace IRQ enabling/disabling
      tools: tracing: handle IRQs on/off events in xentrace and xenalyze
      xen: trace RCU behavior
      tools: tracing: handle RCU events in xentrace and xenalyze
      xen: trace softirqs
      tools: tracing: handle RCU events in xentrace and xenalyze
      xen: trace tasklets
      tools: tracing: handle tasklets events in xentrace and xenalyze
      xen: trace timers
      tools: tracing: handle timers events in xentrace and xenalyze

 tools/xentrace/analyze.h           |    1 
 tools/xentrace/formats             |   39 +++
 tools/xentrace/xenalyze.c          |  525 +++++++++++++++++++++++++++++++++++-
 xen/Kconfig.debug                  |   50 +++
 xen/arch/x86/hvm/svm/entry.S       |    2 
 xen/arch/x86/irq.c                 |   61 +++-
 xen/arch/x86/trace.c               |   23 ++
 xen/common/rcupdate.c              |   82 +++++-
 xen/common/softirq.c               |   66 ++++-
 xen/common/spinlock.c              |   16 +
 xen/common/tasklet.c               |   87 ++++++
 xen/common/timer.c                 |  158 ++++++++++-
 xen/common/trace.c                 |   57 +++-
 xen/drivers/cpufreq/utility.c      |    7 
 xen/include/asm-arm/arm32/system.h |   12 +
 xen/include/asm-arm/arm64/system.h |   12 +
 xen/include/asm-x86/system.h       |   85 +++++-
 xen/include/public/trace.h         |   48 +++
 xen/include/xen/rwlock.h           |   33 ++
 xen/include/xen/trace.h            |   30 ++
 20 files changed, 1320 insertions(+), 74 deletions(-)
--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel