Re: [PATCH 00/15] xen/tools: add tracing to various Xen subsystems

[George Dunlap <george.dunlap@citrix.com>]

On 01/06/17 18:33, Dario Faggioli wrote:
> Hello,
> 
> While chasing and dealing with bugs, over this last period, I've found myself
> augmenting Xen with quite a few new tracing capabilities, especially focusing
> on:
>  - IRQ being disabled and (re)enabled (in addition to the already existing
>    tracing of IRQ related activity that we have);
>  - RCU;
>  - softirqs (I think I sent a preliminary version of this, long ago, but can't
>    be sure);
>  - tasklets;
>  - timers;
> 
> And, apart from the first 4 patches (which are random, but still tracing
> related, of course, improvements), this is what this patch series does: it adds
> tracing to the Xen susystems listed above.
> 
> That happens, one subsystem after another, in patches 6 to 15.
> 
> Patch 5 deserves special mention. In fact, now that we have Kconfig, I thought
> it could be a nice thing to make it possible to select, at build config time,
> whether we want tracing or not, in the hypervisor (like, for instance, we do
> for performance counters).
> 
> To be honest, my goal was to be able to compile tracing off, and run
> benchmarks, to assess how much of a overhead tracing introduces, but then I
> decided it was worth doing this properly, and now sending it.  I am still
> running those benchmarks. Preliminary results seems to be showing that having
> tracing support in the hypervisor does not (when it's disabled, of course)
> introduce too much overhead. Still, I think it could be useful, to people that
> wants a very specificly tailored version of Xen (embedded, small footprint,
> etc.), to be able to rip it off nice and easily (e.g., like OpenXT guys did for
> schedulers).
> 
> Of course, I will report here what I find, when benchmarks will finish running.
> (In the meantime, patch 5 has some data about .text section shrinking in its
> changelog.)
> 
> I also thought, for similar reasons, that it would have been good to be able to
> also individually enable or disable the new tracing I'm introducing with this
> series.  This potentially applies even to the tracing we already have in tree
> (e.g., one may want to have tracing compiled in, for everything except that for
> scheduling), and I'd be up for working on this. However, in this series, I am
> touching really hot paths (with the exception, maybe, of RCU), and so I think
> it is even more important to be able to disable tracing for them, for minimum
> overhead.
> 
> For instance, the IRQ enabling and disabling tracing, I find it really really
> usable for understanding certain class of behavior, and, with some scripting
> and some more gnuplot "magic" (which I'll also share shortly) we can also use
> it to automatically measure and graph for how long interrupt are kept disabled,
> in various places within the hypervisor (Jennifer has done a similar analysis
> for XenServer, a while back).  But it is very invasive, so you want to be able
> to turn it on and off.
> As said, I don't have all the result I need to present conclusions, but what I
> see in preliminary data is that, although the tracing we currently have in Xen
> is not too bad, performance wise, this new IRQs on/off tracing does have an
> impact on performance, just for being there in the code (i.e., even when it is
> there in the code but is *disabled*).
> 
> Of course, all this fine grain control of tracing options is hidden under
> XEN_CONFIG_EXPERT, and all the new tracing is disabled by default.
> 
> There is a branch with this series here:
>  git://xenbits.xen.org/people/dariof/xen.git  rel/tracing/xen-internals
>  http://xenbits.xen.org/gitweb/?p=people/dariof/xen.git;a=shortlog;h=refs/heads/rel/tracing/xen-internals
> 
> And Travis is happy about it:
>  https://travis-ci.org/fdario/xen/builds/238421024
> 
> Let me know what you think.
> 
> Thanks and Regards,
> Dario
> ---
> Dario Faggioli (15):
>       xen: in do_softirq() sample smp_processor_id() once and for all.
>       xen: tracing: avoid checking tb_init_done multiple times.
>       xen/tools: tracing: several improvements on IRQs tracing
>       tools: xenalyze: fix dumping of PM_IDLE events.
>       xen: make it possible to disable tracing in Kconfig.
>       xen: trace IRQ enabling/disabling
>       tools: tracing: handle IRQs on/off events in xentrace and xenalyze
>       xen: trace RCU behavior
>       tools: tracing: handle RCU events in xentrace and xenalyze
>       xen: trace softirqs
>       tools: tracing: handle RCU events in xentrace and xenalyze
>       xen: trace tasklets
>       tools: tracing: handle tasklets events in xentrace and xenalyze
>       xen: trace timers
>       tools: tracing: handle timers events in xentrace and xenalyze

Patches 10-15 look good to me too:

Acked-by: George Dunlap <george.dunlap@citrix.com>

 -George


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel