* [PATCH 00/15] xen/tools: add tracing to various Xen subsystems
@ 2017-06-01 17:33 Dario Faggioli
2017-06-01 17:33 ` [PATCH 01/15] xen: in do_softirq() sample smp_processor_id() once and for all Dario Faggioli
` (16 more replies)
0 siblings, 17 replies; 68+ messages in thread
From: Dario Faggioli @ 2017-06-01 17:33 UTC (permalink / raw)
To: xen-devel
Cc: Stefano Stabellini, Wei Liu, George Dunlap, Andrew Cooper,
Ian Jackson, Tim Deegan, Julien Grall, Jan Beulich,
Jennifer Herbert, Doug Goldstein
Hello,
While chasing and dealing with bugs, over this last period, I've found myself
augmenting Xen with quite a few new tracing capabilities, especially focusing
on:
- IRQ being disabled and (re)enabled (in addition to the already existing
tracing of IRQ related activity that we have);
- RCU;
- softirqs (I think I sent a preliminary version of this, long ago, but can't
be sure);
- tasklets;
- timers;
And, apart from the first 4 patches (which are random, but still tracing
related, of course, improvements), this is what this patch series does: it adds
tracing to the Xen susystems listed above.
That happens, one subsystem after another, in patches 6 to 15.
Patch 5 deserves special mention. In fact, now that we have Kconfig, I thought
it could be a nice thing to make it possible to select, at build config time,
whether we want tracing or not, in the hypervisor (like, for instance, we do
for performance counters).
To be honest, my goal was to be able to compile tracing off, and run
benchmarks, to assess how much of a overhead tracing introduces, but then I
decided it was worth doing this properly, and now sending it. I am still
running those benchmarks. Preliminary results seems to be showing that having
tracing support in the hypervisor does not (when it's disabled, of course)
introduce too much overhead. Still, I think it could be useful, to people that
wants a very specificly tailored version of Xen (embedded, small footprint,
etc.), to be able to rip it off nice and easily (e.g., like OpenXT guys did for
schedulers).
Of course, I will report here what I find, when benchmarks will finish running.
(In the meantime, patch 5 has some data about .text section shrinking in its
changelog.)
I also thought, for similar reasons, that it would have been good to be able to
also individually enable or disable the new tracing I'm introducing with this
series. This potentially applies even to the tracing we already have in tree
(e.g., one may want to have tracing compiled in, for everything except that for
scheduling), and I'd be up for working on this. However, in this series, I am
touching really hot paths (with the exception, maybe, of RCU), and so I think
it is even more important to be able to disable tracing for them, for minimum
overhead.
For instance, the IRQ enabling and disabling tracing, I find it really really
usable for understanding certain class of behavior, and, with some scripting
and some more gnuplot "magic" (which I'll also share shortly) we can also use
it to automatically measure and graph for how long interrupt are kept disabled,
in various places within the hypervisor (Jennifer has done a similar analysis
for XenServer, a while back). But it is very invasive, so you want to be able
to turn it on and off.
As said, I don't have all the result I need to present conclusions, but what I
see in preliminary data is that, although the tracing we currently have in Xen
is not too bad, performance wise, this new IRQs on/off tracing does have an
impact on performance, just for being there in the code (i.e., even when it is
there in the code but is *disabled*).
Of course, all this fine grain control of tracing options is hidden under
XEN_CONFIG_EXPERT, and all the new tracing is disabled by default.
There is a branch with this series here:
git://xenbits.xen.org/people/dariof/xen.git rel/tracing/xen-internals
http://xenbits.xen.org/gitweb/?p=people/dariof/xen.git;a=shortlog;h=refs/heads/rel/tracing/xen-internals
And Travis is happy about it:
https://travis-ci.org/fdario/xen/builds/238421024
Let me know what you think.
Thanks and Regards,
Dario
---
Dario Faggioli (15):
xen: in do_softirq() sample smp_processor_id() once and for all.
xen: tracing: avoid checking tb_init_done multiple times.
xen/tools: tracing: several improvements on IRQs tracing
tools: xenalyze: fix dumping of PM_IDLE events.
xen: make it possible to disable tracing in Kconfig.
xen: trace IRQ enabling/disabling
tools: tracing: handle IRQs on/off events in xentrace and xenalyze
xen: trace RCU behavior
tools: tracing: handle RCU events in xentrace and xenalyze
xen: trace softirqs
tools: tracing: handle RCU events in xentrace and xenalyze
xen: trace tasklets
tools: tracing: handle tasklets events in xentrace and xenalyze
xen: trace timers
tools: tracing: handle timers events in xentrace and xenalyze
tools/xentrace/analyze.h | 1
tools/xentrace/formats | 39 +++
tools/xentrace/xenalyze.c | 525 +++++++++++++++++++++++++++++++++++-
xen/Kconfig.debug | 50 +++
xen/arch/x86/hvm/svm/entry.S | 2
xen/arch/x86/irq.c | 61 +++-
xen/arch/x86/trace.c | 23 ++
xen/common/rcupdate.c | 82 +++++-
xen/common/softirq.c | 66 ++++-
xen/common/spinlock.c | 16 +
xen/common/tasklet.c | 87 ++++++
xen/common/timer.c | 158 ++++++++++-
xen/common/trace.c | 57 +++-
xen/drivers/cpufreq/utility.c | 7
xen/include/asm-arm/arm32/system.h | 12 +
xen/include/asm-arm/arm64/system.h | 12 +
xen/include/asm-x86/system.h | 85 +++++-
xen/include/public/trace.h | 48 +++
xen/include/xen/rwlock.h | 33 ++
xen/include/xen/trace.h | 30 ++
20 files changed, 1320 insertions(+), 74 deletions(-)
--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* [PATCH 01/15] xen: in do_softirq() sample smp_processor_id() once and for all.
2017-06-01 17:33 [PATCH 00/15] xen/tools: add tracing to various Xen subsystems Dario Faggioli
@ 2017-06-01 17:33 ` Dario Faggioli
2017-06-07 14:38 ` Jan Beulich
2017-06-08 14:20 ` George Dunlap
2017-06-01 17:33 ` [PATCH 02/15] xen: tracing: avoid checking tb_init_done multiple times Dario Faggioli
` (15 subsequent siblings)
16 siblings, 2 replies; 68+ messages in thread
From: Dario Faggioli @ 2017-06-01 17:33 UTC (permalink / raw)
To: xen-devel
Cc: Stefano Stabellini, George Dunlap, Andrew Cooper, Tim Deegan,
Julien Grall, Jan Beulich
In fact, right now, we read it at every iteration of the loop.
The reason it's done like this is how context switch was handled
on IA64 (see commit ae9bfcdc, "[XEN] Various softirq cleanups" [1]).
However:
1) we don't have IA64 any longer, and all the achitectures that
we do support, are ok with sampling once and for all;
2) sampling at every iteration (slightly) affect performance;
3) sampling at every iteration is misleading, as it makes people
believe that it is currently possible that SCHEDULE_SOFTIRQ
moves the execution flow on another CPU (and the comment,
by reinforcing this belief, makes things even worse!).
Therefore, let's:
- do the sampling once and for all, and remove the comment;
- leave an ASSERT() around, so that, if context switching
logic changes (in current or new arches), we will notice.
[1] Some more (historical) information here:
http://old-list-archives.xenproject.org/archives/html/xen-devel/2006-06/msg01262.html
Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
---
Cc: Andrew Cooper <andrew.cooper3@citrix.com>
Cc: George Dunlap <George.Dunlap@eu.citrix.com>
Cc: Jan Beulich <jbeulich@suse.com>
Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: Stefano Stabellini <sstabellini@kernel.org>
Cc: Julien Grall <julien.grall@arm.com>
Cc: Tim Deegan <tim@xen.org>
---
xen/common/softirq.c | 8 ++------
1 file changed, 2 insertions(+), 6 deletions(-)
diff --git a/xen/common/softirq.c b/xen/common/softirq.c
index ac12cf8..67c84ba 100644
--- a/xen/common/softirq.c
+++ b/xen/common/softirq.c
@@ -27,16 +27,12 @@ static DEFINE_PER_CPU(unsigned int, batching);
static void __do_softirq(unsigned long ignore_mask)
{
- unsigned int i, cpu;
+ unsigned int i, cpu = smp_processor_id();
unsigned long pending;
for ( ; ; )
{
- /*
- * Initialise @cpu on every iteration: SCHEDULE_SOFTIRQ may move
- * us to another processor.
- */
- cpu = smp_processor_id();
+ ASSERT(cpu == smp_processor_id());
if ( rcu_pending(cpu) )
rcu_check_callbacks(cpu);
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 02/15] xen: tracing: avoid checking tb_init_done multiple times.
2017-06-01 17:33 [PATCH 00/15] xen/tools: add tracing to various Xen subsystems Dario Faggioli
2017-06-01 17:33 ` [PATCH 01/15] xen: in do_softirq() sample smp_processor_id() once and for all Dario Faggioli
@ 2017-06-01 17:33 ` Dario Faggioli
2017-06-01 17:53 ` Andrew Cooper
` (2 more replies)
2017-06-01 17:33 ` [PATCH 03/15] xen/tools: tracing: several improvements on IRQs tracing Dario Faggioli
` (14 subsequent siblings)
16 siblings, 3 replies; 68+ messages in thread
From: Dario Faggioli @ 2017-06-01 17:33 UTC (permalink / raw)
To: xen-devel; +Cc: George Dunlap
In fact, when calling __trace_var() directly, we can
assume that tb_init_done has been checked to be true,
and the if is hence redundant.
While there, also:
- still in __trace_var(), move the check that the event
is actually being traced up a little bit (to bail as
soon as possible, if it is not);
- make it explicit that tb_init_done is likely 0 in
trace_will_trace_event(), as it is almost everywhere
in the code.
Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
---
Cc: George Dunlap <george.dunlap@eu.citrix.com>
---
xen/common/trace.c | 8 +++-----
1 file changed, 3 insertions(+), 5 deletions(-)
diff --git a/xen/common/trace.c b/xen/common/trace.c
index 4fedc26..f29cd4c 100644
--- a/xen/common/trace.c
+++ b/xen/common/trace.c
@@ -311,7 +311,7 @@ static int tb_set_size(unsigned int pages)
int trace_will_trace_event(u32 event)
{
- if ( !tb_init_done )
+ if ( likely(!tb_init_done) )
return 0;
/*
@@ -691,7 +691,8 @@ void __trace_var(u32 event, bool_t cycles, unsigned int extra,
unsigned int extra_word;
bool_t started_below_highwater;
- if( !tb_init_done )
+ /* If the event is not interesting, bail, as early as possible */
+ if ( (tb_event_mask & event) == 0 )
return;
/* Convert byte count into word count, rounding up */
@@ -705,9 +706,6 @@ void __trace_var(u32 event, bool_t cycles, unsigned int extra,
/* Round size up to nearest word */
extra = extra_word * sizeof(u32);
- if ( (tb_event_mask & event) == 0 )
- return;
-
/* match class */
if ( ((tb_event_mask >> TRC_CLS_SHIFT) & (event >> TRC_CLS_SHIFT)) == 0 )
return;
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 03/15] xen/tools: tracing: several improvements on IRQs tracing
2017-06-01 17:33 [PATCH 00/15] xen/tools: add tracing to various Xen subsystems Dario Faggioli
2017-06-01 17:33 ` [PATCH 01/15] xen: in do_softirq() sample smp_processor_id() once and for all Dario Faggioli
2017-06-01 17:33 ` [PATCH 02/15] xen: tracing: avoid checking tb_init_done multiple times Dario Faggioli
@ 2017-06-01 17:33 ` Dario Faggioli
2017-06-01 18:02 ` Andrew Cooper
` (2 more replies)
2017-06-01 17:34 ` [PATCH 04/15] tools: xenalyze: fix dumping of PM_IDLE events Dario Faggioli
` (13 subsequent siblings)
16 siblings, 3 replies; 68+ messages in thread
From: Dario Faggioli @ 2017-06-01 17:33 UTC (permalink / raw)
To: xen-devel; +Cc: George Dunlap, Wei Liu, Ian Jackson, Jan Beulich, Andrew Cooper
More specifically:
- the handling of the TRC_HW_IRQ_HANDLED is fixed, both in
xentrace_format and in xenalyze;
- simple events for recording when we enter and exit the
do_IRQ function, as well as when we deal with a guest
IRQ, are added;
- tracing of IRQs handled with direct vectors is also
added.
With all the above, a trace will now look like this (using
xenalyze):
0.001299072 .x- d32768v5 irq_enter, irq 80000000
0.001299072 .x- d32768v5 irq_direct, vec fa, handler = 0xffff82d08026d7e4
0.001300014 .x- d32768v5 raise_softirq nr 0
0.001300487 .x- d32768v5 irq_exit irq 80000000, in_irq = 0
Or like this:
0.049437892 -|- d32767v12 irq_enter, irq 4
0.049437892 -|- d32767v12 irq_handled irq 4, 85428 cycles
0.049474336 -|- d32767v12 irq_exit irq 4, status = 0x0, in_irq = 0
Making it much easier to figure out when interrupt
processing start, what it does and when it ends.
Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
---
Cc: George Dunlap <george.dunlap@eu.citrix.com>
Cc: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Wei Liu <wei.liu2@citrix.com>
Cc: Jan Beulich <jbeulich@suse.com>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>
---
tools/xentrace/formats | 6 ++++
tools/xentrace/xenalyze.c | 56 +++++++++++++++++++++++++++++++++++-----
xen/arch/x86/irq.c | 61 +++++++++++++++++++++++++++++++++++++-------
xen/include/public/trace.h | 4 +++
4 files changed, 109 insertions(+), 18 deletions(-)
diff --git a/tools/xentrace/formats b/tools/xentrace/formats
index 8b31780..00c29b8 100644
--- a/tools/xentrace/formats
+++ b/tools/xentrace/formats
@@ -197,7 +197,11 @@
0x00802005 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) move_vector [ irq = %(1)d had vector 0x%(2)x on CPU%(3)d ]
0x00802006 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) assign_vector [ irq = %(1)d = vector 0x%(2)x, CPU mask: 0x%(3)08x ]
0x00802007 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) bogus_vector [ 0x%(1)x ]
-0x00802008 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) do_irq [ irq = %(1)d, began = %(2)dus, ended = %(3)dus ]
+0x00802008 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) irq_handled [ irq = %(1)d, dur = 0x%(3)08x%(2)08x ]
+0x00802009 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) direct_vector [ vector 0x%(1)x, handler = 0x%(3)08x%(2)08x ]
+0x0080200a CPU%(cpu)d %(tsc)d (+%(reltsc)8d) do_irq [ irq = %(1)d ]
+0x0080200b CPU%(cpu)d %(tsc)d (+%(reltsc)8d) do_guest_irq [ irq = %(1)d ]
+0x0080200c CPU%(cpu)d %(tsc)d (+%(reltsc)8d) irq_exit [ irq = %(1)d, status = 0x%(2)x, in_irq = %(3)d ]
0x00084001 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) hpet create [ tn = %(1)d, irq = %(2)d, delta = 0x%(4)08x%(3)08x, period = 0x%(6)08x%(5)08x ]
0x00084002 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) pit create [ delta = 0x%(1)016x, period = 0x%(2)016x ]
diff --git a/tools/xentrace/xenalyze.c b/tools/xentrace/xenalyze.c
index fa608ad..063eee7 100644
--- a/tools/xentrace/xenalyze.c
+++ b/tools/xentrace/xenalyze.c
@@ -8380,19 +8380,45 @@ void irq_process(struct pcpu_info *p) {
}
break;
}
- case TRC_HW_IRQ_HANDLED:
+ case TRC_HW_IRQ_ENTER:
+ case TRC_HW_IRQ_GUEST:
{
struct {
- int irq, start_tsc, end_tsc;
+ int32_t irq;
} *r = (typeof(r))ri->d;
- int arctime;
- arctime = r->end_tsc - r->start_tsc;
if ( opt.dump_all )
{
- printf(" %s irq_handled irq %x %d (%d,%d)\n",
- ri->dump_header,
- r->irq, arctime, r->start_tsc, r->end_tsc);
+ printf(" %s irq_%s, irq %x\n", ri->dump_header,
+ ri->event == TRC_HW_IRQ_ENTER ? "enter" : "guest", r->irq);
+ }
+ break;
+ }
+ case TRC_HW_IRQ_DIRECT_VECTOR:
+ {
+ struct {
+ uint32_t vec;
+ uint64_t handler;
+ } __attribute__((packed)) *r = (typeof(r))ri->d;
+
+ if ( opt.dump_all )
+ {
+ printf(" %s irq_direct, vec %x, handler = %p\n",
+ ri->dump_header, r->vec, (void*)r->handler);
+ }
+ break;
+ }
+ case TRC_HW_IRQ_HANDLED:
+ {
+ struct {
+ int32_t irq;
+ uint64_t arctime;
+ } __attribute__((packed)) *r = (typeof(r))ri->d;
+
+ if ( opt.dump_all )
+ {
+ printf(" %s irq_handled irq %x, %"PRIu64" cycles\n",
+ ri->dump_header, r->irq, r->arctime);
}
if ( opt.scatterplot_irq )
{
@@ -8407,6 +8433,22 @@ void irq_process(struct pcpu_info *p) {
}
break;
}
+ case TRC_HW_IRQ_EXIT:
+ {
+ struct {
+ int32_t irq, status;
+ uint32_t in_irq;
+ } *r = (typeof(r))ri->d;
+
+ if ( opt.dump_all )
+ {
+ printf(" %s irq_exit irq %x", ri->dump_header, r->irq);
+ if ( r->status != -1 )
+ printf(", status = 0x%x", r->status);
+ printf(", in_irq = %d\n", r->in_irq);
+ }
+ break;
+ }
case TRC_HW_IRQ_ASSIGN_VECTOR:
{
struct {
diff --git a/xen/arch/x86/irq.c b/xen/arch/x86/irq.c
index 676ba52..f5d9302 100644
--- a/xen/arch/x86/irq.c
+++ b/xen/arch/x86/irq.c
@@ -100,11 +100,15 @@ static void trace_irq_mask(u32 event, int irq, int vector, cpumask_t *mask)
unsigned int irq:16, vec:16;
unsigned int mask[6];
} d;
+
+ if ( likely(!tb_init_done) )
+ return;
+
d.irq = irq;
d.vec = vector;
memset(d.mask, 0, sizeof(d.mask));
memcpy(d.mask, mask, min(sizeof(d.mask), sizeof(cpumask_t)));
- trace_var(event, 1, sizeof(d), &d);
+ __trace_var(event, 1, sizeof(d), &d);
}
static int __init __bind_irq_vector(int irq, int vector, const cpumask_t *cpu_mask)
@@ -804,23 +808,54 @@ void alloc_direct_apic_vector(
spin_unlock(&lock);
}
+static inline void trace_irq_handled(int irq, uint64_t time)
+{
+ struct __packed {
+ int32_t irq;
+ uint64_t time;
+ } d;
+
+ if ( likely(!tb_init_done) )
+ return;
+
+ d.irq = irq;
+ d.time = time;
+ __trace_var(TRC_HW_IRQ_HANDLED, 0, sizeof(d), &d);
+}
+
void do_IRQ(struct cpu_user_regs *regs)
{
struct irqaction *action;
- uint32_t tsc_in;
- struct irq_desc *desc;
+ uint64_t tsc_in = 0;
+ struct irq_desc *desc = NULL;
unsigned int vector = (u8)regs->entry_vector;
int irq = __get_cpu_var(vector_irq[vector]);
struct cpu_user_regs *old_regs = set_irq_regs(regs);
perfc_incr(irqs);
this_cpu(irq_count)++;
+ TRACE_1D(TRC_HW_IRQ_ENTER, irq);
irq_enter();
- if (irq < 0) {
- if (direct_apic_vector[vector] != NULL) {
+ if (irq < 0)
+ {
+ if (direct_apic_vector[vector] != NULL)
+ {
+ if ( unlikely(tb_init_done) )
+ {
+ struct __packed {
+ uint32_t vec;
+ uint64_t handler;
+ } d;
+
+ d.vec = vector;
+ d.handler = (uint64_t)direct_apic_vector[vector];
+ __trace_var(TRC_HW_IRQ_DIRECT_VECTOR, 0, sizeof(d), &d);
+ }
(*direct_apic_vector[vector])(regs);
- } else {
+ }
+ else
+ {
const char *kind = ", LAPIC";
if ( apic_isr_read(vector) )
@@ -884,9 +919,13 @@ void do_IRQ(struct cpu_user_regs *regs)
desc->rl_quantum_start = now;
}
- tsc_in = tb_init_done ? get_cycles() : 0;
+ if ( unlikely(tb_init_done) )
+ {
+ __trace_var(TRC_HW_IRQ_GUEST, 0, sizeof(irq), &irq);
+ tsc_in = get_cycles();
+ }
__do_IRQ_guest(irq);
- TRACE_3D(TRC_HW_IRQ_HANDLED, irq, tsc_in, get_cycles());
+ trace_irq_handled(irq, get_cycles() - tsc_in);
goto out_no_end;
}
@@ -907,9 +946,10 @@ void do_IRQ(struct cpu_user_regs *regs)
{
desc->status &= ~IRQ_PENDING;
spin_unlock_irq(&desc->lock);
- tsc_in = tb_init_done ? get_cycles() : 0;
+ if ( unlikely(tb_init_done) )
+ tsc_in = get_cycles();
action->handler(irq, action->dev_id, regs);
- TRACE_3D(TRC_HW_IRQ_HANDLED, irq, tsc_in, get_cycles());
+ trace_irq_handled(irq, get_cycles() - tsc_in);
spin_lock_irq(&desc->lock);
}
@@ -922,6 +962,7 @@ void do_IRQ(struct cpu_user_regs *regs)
spin_unlock(&desc->lock);
out_no_unlock:
irq_exit();
+ TRACE_3D(TRC_HW_IRQ_EXIT, irq, desc == NULL ? -1 : desc->status, in_irq());
set_irq_regs(old_regs);
}
diff --git a/xen/include/public/trace.h b/xen/include/public/trace.h
index 7f2e891..f66a7af 100644
--- a/xen/include/public/trace.h
+++ b/xen/include/public/trace.h
@@ -271,6 +271,10 @@
#define TRC_HW_IRQ_ASSIGN_VECTOR (TRC_HW_IRQ + 0x6)
#define TRC_HW_IRQ_UNMAPPED_VECTOR (TRC_HW_IRQ + 0x7)
#define TRC_HW_IRQ_HANDLED (TRC_HW_IRQ + 0x8)
+#define TRC_HW_IRQ_DIRECT_VECTOR (TRC_HW_IRQ + 0x9)
+#define TRC_HW_IRQ_ENTER (TRC_HW_IRQ + 0xA)
+#define TRC_HW_IRQ_GUEST (TRC_HW_IRQ + 0xB)
+#define TRC_HW_IRQ_EXIT (TRC_HW_IRQ + 0xC)
/*
* Event Flags
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 04/15] tools: xenalyze: fix dumping of PM_IDLE events.
2017-06-01 17:33 [PATCH 00/15] xen/tools: add tracing to various Xen subsystems Dario Faggioli
` (2 preceding siblings ...)
2017-06-01 17:33 ` [PATCH 03/15] xen/tools: tracing: several improvements on IRQs tracing Dario Faggioli
@ 2017-06-01 17:34 ` Dario Faggioli
2017-06-08 15:06 ` George Dunlap
2017-06-01 17:34 ` [PATCH 05/15] xen: make it possible to disable tracing in Kconfig Dario Faggioli
` (12 subsequent siblings)
16 siblings, 1 reply; 68+ messages in thread
From: Dario Faggioli @ 2017-06-01 17:34 UTC (permalink / raw)
To: xen-devel
In fact, not all the information present in the trace
record were used and printed.
Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
---
George Dunlap <george.dunlap@eu.citrix.com>
Ian Jackson <ian.jackson@eu.citrix.com>
Wei Liu <wei.liu2@citrix.com>
---
tools/xentrace/xenalyze.c | 24 +++++++++++++++++++-----
1 file changed, 19 insertions(+), 5 deletions(-)
diff --git a/tools/xentrace/xenalyze.c b/tools/xentrace/xenalyze.c
index 063eee7..3c2f600 100644
--- a/tools/xentrace/xenalyze.c
+++ b/tools/xentrace/xenalyze.c
@@ -8269,9 +8269,10 @@ void pm_process(struct pcpu_info *p) {
break;
case TRC_PM_IDLE_ENTRY:
if (opt.dump_all )
- printf(" %s pm_idle_start c%d\n",
+ printf(" %s pm_idle_start c%d, pm_tick=%u, exp=%dus, pred=%dus\n",
ri->dump_header,
- ri->d[0]);
+ ri->d[0], ri->d[1],
+ ri->d[2], ri->d[3]);
if ( ri->d[0] <= CSTATE_MAX )
{
p->power_state=ri->d[0];
@@ -8280,9 +8281,22 @@ void pm_process(struct pcpu_info *p) {
break;
case TRC_PM_IDLE_EXIT:
if (opt.dump_all )
- printf(" %s pm_idle_end c%d\n",
- ri->dump_header,
- ri->d[0]);
+ {
+ int i;
+
+ printf(" %s pm_idle_end c%d pm_tick=%u",
+ ri->dump_header, ri->d[0], ri->d[1]);
+ printf(", irr_status=%d", ri->d[2]);
+ for (i = 3; i <= 5; i++)
+ {
+ if (ri->d[i] == 0)
+ break;
+ printf(",%d", ri->d[i]);
+ }
+ if (i == 6)
+ printf(",...");
+ printf("\n");
+ }
if ( p->power_state != ri->d[0]
&& p->power_state != CSTATE_INVALID )
printf("Strange, pm_idle_end %d, power_state %d!\n",
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 05/15] xen: make it possible to disable tracing in Kconfig.
2017-06-01 17:33 [PATCH 00/15] xen/tools: add tracing to various Xen subsystems Dario Faggioli
` (3 preceding siblings ...)
2017-06-01 17:34 ` [PATCH 04/15] tools: xenalyze: fix dumping of PM_IDLE events Dario Faggioli
@ 2017-06-01 17:34 ` Dario Faggioli
2017-06-01 18:43 ` Andrew Cooper
` (2 more replies)
2017-06-01 17:34 ` [PATCH 06/15] xen: trace IRQ enabling/disabling Dario Faggioli
` (11 subsequent siblings)
16 siblings, 3 replies; 68+ messages in thread
From: Dario Faggioli @ 2017-06-01 17:34 UTC (permalink / raw)
To: xen-devel
Cc: Stefano Stabellini, Wei Liu, George Dunlap, Andrew Cooper,
Ian Jackson, Julien Grall, Jan Beulich, Doug Goldstein
And compile it out of the hypervisor entirely.
Code and other sections' sizes change as follows.
Output of `size`:
vanilla patched-Y patched-N
text 1929007 1929007 1902783
data 337784 337784 337688
bss 1310464 1310464 1310336
Output of `size -A`:
vanilla patched-Y patched-N
.text 1372602 1372602 1348026
.rodata 312152 312152 310680
.init.text 244209 244209 244033
.init.data 224576 224576 224576
.data 57472 57472 57376
.bss 1310464 1310464 1310336
Total 23026516 23027008 22858069
No functional change intended.
Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
---
Cc: George Dunlap <George.Dunlap@eu.citrix.com>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>
Cc: Jan Beulich <jbeulich@suse.com>
Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: Stefano Stabellini <sstabellini@kernel.org>
Cc: Julien Grall <julien.grall@arm.com>
Cc: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Wei Liu <wei.liu2@citrix.com>
Cc: Doug Goldstein <cardoe@cardoe.com>
---
xen/Kconfig.debug | 8 ++++++++
xen/arch/x86/hvm/svm/entry.S | 2 ++
xen/arch/x86/trace.c | 23 +++++++++++++++++++++++
xen/common/trace.c | 39 +++++++++++++++++++++++++++++++++++----
xen/drivers/cpufreq/utility.c | 7 +++++--
xen/include/xen/trace.h | 30 +++++++++++++++++++++++++++++-
6 files changed, 102 insertions(+), 7 deletions(-)
diff --git a/xen/Kconfig.debug b/xen/Kconfig.debug
index 689f297..374c1c0 100644
--- a/xen/Kconfig.debug
+++ b/xen/Kconfig.debug
@@ -98,6 +98,14 @@ config PERF_ARRAYS
---help---
Enables software performance counter array histograms.
+config TRACING
+ bool "Tracing"
+ default y
+ ---help---
+ Enables collecting traces of events occurring in the hypervisor
+ in per-CPU ring buffers. The 'xentrace' tool can be used to read
+ the buffers and dump the content on the disk.
+
config VERBOSE_DEBUG
bool "Verbose debug messages"
diff --git a/xen/arch/x86/hvm/svm/entry.S b/xen/arch/x86/hvm/svm/entry.S
index a4ab40a..ea4a106 100644
--- a/xen/arch/x86/hvm/svm/entry.S
+++ b/xen/arch/x86/hvm/svm/entry.S
@@ -61,10 +61,12 @@ __UNLIKELY_END(nsvm_hap)
call svm_asid_handle_vmrun
+#ifdef CONFIG_TRACING
cmpb $0,tb_init_done(%rip)
UNLIKELY_START(nz, svm_trace)
call svm_trace_vmentry
UNLIKELY_END(svm_trace)
+#endif
mov VCPU_svm_vmcb(%rbx),%rcx
mov UREGS_rax(%rsp),%rax
diff --git a/xen/arch/x86/trace.c b/xen/arch/x86/trace.c
index 4a953c5..411f798 100644
--- a/xen/arch/x86/trace.c
+++ b/xen/arch/x86/trace.c
@@ -1,3 +1,4 @@
+#ifdef CONFIG_TRACING
#include <xen/init.h>
#include <xen/kernel.h>
#include <xen/lib.h>
@@ -157,3 +158,25 @@ void __trace_ptwr_emulation(unsigned long addr, l1_pgentry_t npte)
__trace_var(event, 1/*tsc*/, sizeof(d), &d);
}
}
+#else /* !CONFIG_TRACING */
+#include <xen/kernel.h>
+#include <xen/trace.h>
+
+void __trace_pv_trap(int trapnr, unsigned long eip,
+ int use_error_code, unsigned error_code)
+{
+}
+void __trace_pv_page_fault(unsigned long addr, unsigned error_code)
+{
+}
+void __trace_trap_one_addr(unsigned event, unsigned long va)
+{
+}
+void __trace_trap_two_addr(unsigned event, unsigned long va1,
+ unsigned long va2)
+{
+}
+void __trace_ptwr_emulation(unsigned long addr, l1_pgentry_t npte)
+{
+}
+#endif /* CONFIG_TRACING */
diff --git a/xen/common/trace.c b/xen/common/trace.c
index f29cd4c..2c18462 100644
--- a/xen/common/trace.c
+++ b/xen/common/trace.c
@@ -48,6 +48,11 @@ static unsigned int opt_tevt_mask;
integer_param("tbuf_size", opt_tbuf_size);
integer_param("tevt_mask", opt_tevt_mask);
+#ifdef CONFIG_TRACING
+/* a flag recording whether initialization has been done */
+/* or more properly, if the tbuf subsystem is enabled right now */
+int tb_init_done __read_mostly;
+
/* Pointers to the meta-data objects for all system trace buffers */
static struct t_info *t_info;
static unsigned int t_info_pages;
@@ -64,10 +69,6 @@ static u32 t_buf_highwater;
static DEFINE_PER_CPU(unsigned long, lost_records);
static DEFINE_PER_CPU(unsigned long, lost_records_first_tsc);
-/* a flag recording whether initialization has been done */
-/* or more properly, if the tbuf subsystem is enabled right now */
-int tb_init_done __read_mostly;
-
/* which CPUs tracing is enabled on */
static cpumask_t tb_cpu_mask;
@@ -868,6 +869,36 @@ void __trace_hypercall(uint32_t event, unsigned long op,
__trace_var(event, 1, sizeof(uint32_t) * (1 + (a - d.args)), &d);
}
+#else /* !CONFIG_TRACING */
+void __init init_trace_bufs(void)
+{
+ opt_tbuf_size = 0;
+}
+
+int tb_control(xen_sysctl_tbuf_op_t *tbc)
+{
+ static DEFINE_SPINLOCK(lock);
+ int rc = 0;
+
+ spin_lock(&lock);
+
+ switch ( tbc->cmd )
+ {
+ case XEN_SYSCTL_TBUFOP_get_info:
+ tbc->evt_mask = 0;
+ tbc->buffer_mfn = 0;
+ tbc->size = 0;
+ break;
+ default:
+ rc = -ENOSYS;
+ break;
+ }
+
+ spin_unlock(&lock);
+
+ return rc;
+}
+#endif /* CONFIG_TRACING */
/*
* Local variables:
diff --git a/xen/drivers/cpufreq/utility.c b/xen/drivers/cpufreq/utility.c
index 53879fe..b686a9d 100644
--- a/xen/drivers/cpufreq/utility.c
+++ b/xen/drivers/cpufreq/utility.c
@@ -362,11 +362,14 @@ int __cpufreq_driver_target(struct cpufreq_policy *policy,
if (cpu_online(policy->cpu) && cpufreq_driver->target)
{
- unsigned int prev_freq = policy->cur;
+ uint32_t d[2] = { policy->cur, 0 };
retval = cpufreq_driver->target(policy, target_freq, relation);
if ( retval == 0 )
- TRACE_2D(TRC_PM_FREQ_CHANGE, prev_freq/1000, policy->cur/1000);
+ {
+ d[1] = policy->cur/1000;
+ trace_var(TRC_PM_FREQ_CHANGE, 1, sizeof(d), d);
+ }
}
return retval;
diff --git a/xen/include/xen/trace.h b/xen/include/xen/trace.h
index 12966ea..d1b6c70 100644
--- a/xen/include/xen/trace.h
+++ b/xen/include/xen/trace.h
@@ -21,7 +21,11 @@
#ifndef __XEN_TRACE_H__
#define __XEN_TRACE_H__
+#ifdef CONFIG_TRACING
extern int tb_init_done;
+#else
+#define tb_init_done 0
+#endif
#include <public/sysctl.h>
#include <public/trace.h>
@@ -33,6 +37,7 @@ void init_trace_bufs(void);
/* used to retrieve the physical address of the trace buffers */
int tb_control(struct xen_sysctl_tbuf_op *tbc);
+#ifdef CONFIG_TRACING
int trace_will_trace_event(u32 event);
void __trace_var(u32 event, bool_t cycles, unsigned int extra, const void *);
@@ -113,7 +118,7 @@ void __trace_hypercall(uint32_t event, unsigned long op,
} \
} while ( 0 )
-#define TRACE_6D(_e,d1,d2,d3,d4,d5,d6) \
+#define TRACE_6D(_e,d1,d2,d3,d4,d5,d6) \
do { \
if ( unlikely(tb_init_done) ) \
{ \
@@ -127,5 +132,28 @@ void __trace_hypercall(uint32_t event, unsigned long op,
__trace_var(_e, 1, sizeof(_d), _d); \
} \
} while ( 0 )
+#else /* !CONFIG_TRACING */
+#define trace_will_trace_event(u) (0)
+static inline void __trace_var(u32 event, bool_t cycles, unsigned int extra,
+ const void *extra_data)
+{
+}
+static inline void trace_var(u32 event, int cycles, int extra,
+ const void *extra_data)
+{
+}
+static inline void __trace_hypercall(uint32_t event, unsigned long op,
+ const xen_ulong_t *args)
+{
+}
+
+#define TRACE_0D(e) do {} while ( 0 )
+#define TRACE_1D(e,d1) do {} while ( 0 )
+#define TRACE_2D(e,d1,d2) do {} while ( 0 )
+#define TRACE_3D(e,d1,d2,d3) do {} while ( 0 )
+#define TRACE_4D(e,d1,d2,d3,d4) do {} while ( 0 )
+#define TRACE_5D(e,d1,d2,d3,d4,d5) do {} while ( 0 )
+#define TRACE_6D(e,d1,d2,d3,d4,d5,d6) do {} while ( 0 )
+#endif /* CONFIG_TRACING */
#endif /* __XEN_TRACE_H__ */
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 06/15] xen: trace IRQ enabling/disabling
2017-06-01 17:33 [PATCH 00/15] xen/tools: add tracing to various Xen subsystems Dario Faggioli
` (4 preceding siblings ...)
2017-06-01 17:34 ` [PATCH 05/15] xen: make it possible to disable tracing in Kconfig Dario Faggioli
@ 2017-06-01 17:34 ` Dario Faggioli
2017-06-01 19:08 ` Andrew Cooper
` (3 more replies)
2017-06-01 17:34 ` [PATCH 07/15] tools: tracing: handle IRQs on/off events in xentrace and xenalyze Dario Faggioli
` (10 subsequent siblings)
16 siblings, 4 replies; 68+ messages in thread
From: Dario Faggioli @ 2017-06-01 17:34 UTC (permalink / raw)
To: xen-devel
Cc: Stefano Stabellini, George Dunlap, Andrew Cooper,
Jennifer Herbert, Julien Grall, Jan Beulich
Trace when interrupts are disabled and (re)enabled.
Basically, we replace the IRQ disabling and enabling
functions with helpers that does the same, but also
output the proper trace record.
For putting in the record something that will let
us identify _where_ in the code (i.e., in what function)
the IRQ manipulation is happening, use either:
- current_text_addr(),
- or __builtin_return_address(0).
In fact, depending on whether the disabling/enabling
happens in macros (like for local_irq_disable() and
local_irq_enable()) or in actual functions (like in
spin_lock_irq*()), it is either:
- the actual content of the instruction pointer when
IRQ are disabled/enabled,
- or the return address of the utility function where
IRQ are disabled/enabled,
that will tell us what it is the actual piece of code
that is asking for the IRQ manipulation operation.
Gate this with its specific Kconfig option, and keep
it in disabled state by default (i.e., don't build it,
if not explicitly specified), as the impact on
performance may be non negligible.
Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
---
Cc: George Dunlap <george.dunlap@eu.citrix.com>
Cc: Stefano Stabellini <sstabellini@kernel.org>
Cc: Julien Grall <julien.grall@arm.com>
Cc: Jan Beulich <jbeulich@suse.com>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>
Cc: Jennifer Herbert <jennifer.herbert@citrix.com>
---
xen/Kconfig.debug | 11 ++++-
xen/common/spinlock.c | 16 +++++--
xen/common/trace.c | 10 +++-
xen/include/asm-arm/arm32/system.h | 12 +++++
xen/include/asm-arm/arm64/system.h | 12 +++++
xen/include/asm-x86/system.h | 85 ++++++++++++++++++++++++++++++++++--
xen/include/public/trace.h | 2 +
xen/include/xen/rwlock.h | 33 +++++++++++---
8 files changed, 161 insertions(+), 20 deletions(-)
diff --git a/xen/Kconfig.debug b/xen/Kconfig.debug
index 374c1c0..81910c9 100644
--- a/xen/Kconfig.debug
+++ b/xen/Kconfig.debug
@@ -98,7 +98,7 @@ config PERF_ARRAYS
---help---
Enables software performance counter array histograms.
-config TRACING
+menuconfig TRACING
bool "Tracing"
default y
---help---
@@ -106,6 +106,15 @@ config TRACING
in per-CPU ring buffers. The 'xentrace' tool can be used to read
the buffers and dump the content on the disk.
+config TRACE_IRQSOFF
+ bool "Trace when IRQs are disabled and (re)enabled" if EXPERT = "y"
+ default n
+ depends on TRACING
+ ---help---
+ Makes it possible to generate events _every_ time IRQs are disabled
+ and (re)enabled, with also an indication of where that happened.
+ Note that this comes with high overead and produces huge amount of
+ tracing data.
config VERBOSE_DEBUG
bool "Verbose debug messages"
diff --git a/xen/common/spinlock.c b/xen/common/spinlock.c
index 2a06406..33b903e 100644
--- a/xen/common/spinlock.c
+++ b/xen/common/spinlock.c
@@ -150,7 +150,9 @@ void _spin_lock(spinlock_t *lock)
void _spin_lock_irq(spinlock_t *lock)
{
ASSERT(local_irq_is_enabled());
- local_irq_disable();
+ _local_irq_disable();
+ if ( unlikely(tb_init_done) )
+ trace_irq_disable_ret();
_spin_lock(lock);
}
@@ -158,7 +160,9 @@ unsigned long _spin_lock_irqsave(spinlock_t *lock)
{
unsigned long flags;
- local_irq_save(flags);
+ _local_irq_save(flags);
+ if ( unlikely(tb_init_done) )
+ trace_irq_save_ret(flags);
_spin_lock(lock);
return flags;
}
@@ -175,13 +179,17 @@ void _spin_unlock(spinlock_t *lock)
void _spin_unlock_irq(spinlock_t *lock)
{
_spin_unlock(lock);
- local_irq_enable();
+ if ( unlikely(tb_init_done) )
+ trace_irq_enable_ret();
+ _local_irq_enable();
}
void _spin_unlock_irqrestore(spinlock_t *lock, unsigned long flags)
{
_spin_unlock(lock);
- local_irq_restore(flags);
+ if ( unlikely(tb_init_done) )
+ trace_irq_restore_ret(flags);
+ _local_irq_restore(flags);
}
int _spin_is_locked(spinlock_t *lock)
diff --git a/xen/common/trace.c b/xen/common/trace.c
index 2c18462..71202df 100644
--- a/xen/common/trace.c
+++ b/xen/common/trace.c
@@ -722,7 +722,12 @@ void __trace_var(u32 event, bool_t cycles, unsigned int extra,
/* Read tb_init_done /before/ t_bufs. */
smp_rmb();
- spin_lock_irqsave(&this_cpu(t_lock), flags);
+ /*
+ * spin_lock_irqsave() would call local_irq_save(), which (may)
+ * call __trace_var(). Open code it to avoid recursing.
+ */
+ _local_irq_save(flags);
+ spin_lock(&this_cpu(t_lock));
buf = this_cpu(t_bufs);
@@ -809,7 +814,8 @@ void __trace_var(u32 event, bool_t cycles, unsigned int extra,
__insert_record(buf, event, extra, cycles, rec_size, extra_data);
unlock:
- spin_unlock_irqrestore(&this_cpu(t_lock), flags);
+ spin_unlock(&this_cpu(t_lock));
+ _local_irq_restore(flags);
/* Notify trace buffer consumer that we've crossed the high water mark. */
if ( likely(buf!=NULL)
diff --git a/xen/include/asm-arm/arm32/system.h b/xen/include/asm-arm/arm32/system.h
index c617b40..20871ad 100644
--- a/xen/include/asm-arm/arm32/system.h
+++ b/xen/include/asm-arm/arm32/system.h
@@ -4,6 +4,8 @@
#include <asm/arm32/cmpxchg.h>
+#include <xen/trace.h>
+
#define local_irq_disable() asm volatile ( "cpsid i @ local_irq_disable\n" : : : "cc" )
#define local_irq_enable() asm volatile ( "cpsie i @ local_irq_enable\n" : : : "cc" )
@@ -41,6 +43,16 @@ static inline int local_irq_is_enabled(void)
#define local_abort_enable() __asm__("cpsie a @ __sta\n" : : : "memory", "cc")
#define local_abort_disable() __asm__("cpsid a @ __sta\n" : : : "memory", "cc")
+/* We do not support tracing (at all) yet */
+#define trace_irq_disable_ret() do { } while ( 0 )
+#define trace_irq_enable_ret() do { } while ( 0 )
+#define trace_irq_save_ret(_x) do { } while ( 0 )
+#define trace_irq_restore_ret(_x) do { } while ( 0 )
+#define _local_irq_disable() local_irq_disable()
+#define _local_irq_enable() local_irq_enable()
+#define _local_irq_save(_x) local_irq_save(_x)
+#define _local_irq_restore(_x) local_irq_restore(_x)
+
static inline int local_fiq_is_enabled(void)
{
unsigned long flags;
diff --git a/xen/include/asm-arm/arm64/system.h b/xen/include/asm-arm/arm64/system.h
index 2e2ee21..6603b0c 100644
--- a/xen/include/asm-arm/arm64/system.h
+++ b/xen/include/asm-arm/arm64/system.h
@@ -4,6 +4,8 @@
#include <asm/arm64/cmpxchg.h>
+#include <xen/trace.h>
+
/* Uses uimm4 as a bitmask to select the clearing of one or more of
* the DAIF exception mask bits:
* bit 3 selects the D mask,
@@ -44,6 +46,16 @@
: "memory"); \
})
+/* We do not support tracing (at all) yet */
+#define trace_irq_disable_ret() do { } while ( 0 )
+#define trace_irq_enable_ret() do { } while ( 0 )
+#define trace_irq_save_ret(_x) do { } while ( 0 )
+#define trace_irq_restore_ret(_x) do { } while ( 0 )
+#define _local_irq_disable() local_irq_disable()
+#define _local_irq_enable() local_irq_enable()
+#define _local_irq_save(_x) local_irq_save(_x)
+#define _local_irq_restore(_x) local_irq_restore(_x)
+
static inline int local_irq_is_enabled(void)
{
unsigned long flags;
diff --git a/xen/include/asm-x86/system.h b/xen/include/asm-x86/system.h
index eb498f5..0e7bf01 100644
--- a/xen/include/asm-x86/system.h
+++ b/xen/include/asm-x86/system.h
@@ -5,6 +5,8 @@
#include <xen/bitops.h>
#include <asm/processor.h>
+#include <xen/trace.h>
+
#define read_sreg(name) \
({ unsigned int __sel; \
asm volatile ( "mov %%" STR(name) ",%0" : "=r" (__sel) ); \
@@ -185,8 +187,8 @@ static always_inline unsigned long __xadd(
#define set_mb(var, value) do { xchg(&var, value); } while (0)
#define set_wmb(var, value) do { var = value; wmb(); } while (0)
-#define local_irq_disable() asm volatile ( "cli" : : : "memory" )
-#define local_irq_enable() asm volatile ( "sti" : : : "memory" )
+#define _local_irq_disable() asm volatile ( "cli" : : : "memory" )
+#define _local_irq_enable() asm volatile ( "sti" : : : "memory" )
/* used in the idle loop; sti takes one instruction cycle to complete */
#define safe_halt() asm volatile ( "sti; hlt" : : : "memory" )
@@ -198,12 +200,12 @@ static always_inline unsigned long __xadd(
BUILD_BUG_ON(sizeof(x) != sizeof(long)); \
asm volatile ( "pushf" __OS " ; pop" __OS " %0" : "=g" (x)); \
})
-#define local_irq_save(x) \
+#define _local_irq_save(x) \
({ \
local_save_flags(x); \
- local_irq_disable(); \
+ _local_irq_disable(); \
})
-#define local_irq_restore(x) \
+#define _local_irq_restore(x) \
({ \
BUILD_BUG_ON(sizeof(x) != sizeof(long)); \
asm volatile ( "pushfq\n\t" \
@@ -214,6 +216,79 @@ static always_inline unsigned long __xadd(
"ri" ( (x) & X86_EFLAGS_IF ) ); \
})
+#ifdef CONFIG_TRACE_IRQSOFF
+
+#define TRACE_LOCAL_ADDR ((uint64_t) current_text_addr())
+#define TRACE_RET_ADDR ((uint64_t) __builtin_return_address(0))
+
+#define trace_irq_disable(_a) \
+({ \
+ uint64_t addr = _a; \
+ __trace_var(TRC_HW_IRQ_DISABLE, 1, sizeof(addr), &addr); \
+})
+#define trace_irq_enable(_a) \
+({ \
+ uint64_t addr = _a; \
+ __trace_var(TRC_HW_IRQ_ENABLE, 1, sizeof(addr), &addr); \
+})
+#define trace_irq_save(_x, _a) \
+({ \
+ uint64_t addr = _a; \
+ if ( _x & X86_EFLAGS_IF ) \
+ __trace_var(TRC_HW_IRQ_DISABLE, 1, sizeof(addr), &addr); \
+})
+#define trace_irq_restore(_x, _a) \
+({ \
+ uint64_t addr = _a; \
+ if ( _x & X86_EFLAGS_IF ) \
+ __trace_var(TRC_HW_IRQ_ENABLE, 1, sizeof(addr), &addr); \
+})
+
+#define trace_irq_disable_ret() trace_irq_disable(TRACE_RET_ADDR)
+#define trace_irq_enable_ret() trace_irq_enable(TRACE_RET_ADDR)
+#define trace_irq_save_ret(_x) trace_irq_save(_x, TRACE_RET_ADDR)
+#define trace_irq_restore_ret(_x) trace_irq_restore(_x, TRACE_RET_ADDR)
+
+#define local_irq_disable() \
+({ \
+ bool_t irqon = local_irq_is_enabled(); \
+ _local_irq_disable(); \
+ if ( unlikely(tb_init_done && irqon) ) \
+ trace_irq_disable(TRACE_LOCAL_ADDR); \
+})
+
+#define local_irq_enable() \
+({ \
+ if ( unlikely(tb_init_done) ) \
+ trace_irq_enable(TRACE_LOCAL_ADDR); \
+ _local_irq_enable(); \
+})
+
+#define local_irq_save(_x) \
+({ \
+ local_save_flags(_x); \
+ _local_irq_disable(); \
+ if ( unlikely(tb_init_done) ) \
+ trace_irq_save(_x, TRACE_LOCAL_ADDR); \
+})
+
+#define local_irq_restore(_x) \
+({ \
+ if ( unlikely(tb_init_done) ) \
+ trace_irq_restore(_x, TRACE_LOCAL_ADDR); \
+ _local_irq_restore(_x); \
+})
+#else /* !TRACE_IRQSOFF */
+#define trace_irq_disable_ret() do { } while ( 0 )
+#define trace_irq_enable_ret() do { } while ( 0 )
+#define trace_irq_save_ret(_x) do { } while ( 0 )
+#define trace_irq_restore_ret(_x) do { } while ( 0 )
+#define local_irq_disable() _local_irq_disable()
+#define local_irq_enable() _local_irq_enable()
+#define local_irq_save(_x) _local_irq_save(_x)
+#define local_irq_restore(_x) _local_irq_restore(_x)
+#endif /* TRACE_IRQSOFF */
+
static inline int local_irq_is_enabled(void)
{
unsigned long flags;
diff --git a/xen/include/public/trace.h b/xen/include/public/trace.h
index f66a7af..1692a79 100644
--- a/xen/include/public/trace.h
+++ b/xen/include/public/trace.h
@@ -275,6 +275,8 @@
#define TRC_HW_IRQ_ENTER (TRC_HW_IRQ + 0xA)
#define TRC_HW_IRQ_GUEST (TRC_HW_IRQ + 0xB)
#define TRC_HW_IRQ_EXIT (TRC_HW_IRQ + 0xC)
+#define TRC_HW_IRQ_DISABLE (TRC_HW_IRQ + 0xD)
+#define TRC_HW_IRQ_ENABLE (TRC_HW_IRQ + 0xE)
/*
* Event Flags
diff --git a/xen/include/xen/rwlock.h b/xen/include/xen/rwlock.h
index 35657c5..04f50e5 100644
--- a/xen/include/xen/rwlock.h
+++ b/xen/include/xen/rwlock.h
@@ -73,14 +73,19 @@ static inline void _read_lock(rwlock_t *lock)
static inline void _read_lock_irq(rwlock_t *lock)
{
ASSERT(local_irq_is_enabled());
- local_irq_disable();
+ _local_irq_disable();
+ if ( unlikely(tb_init_done) )
+ trace_irq_disable_ret();
_read_lock(lock);
}
static inline unsigned long _read_lock_irqsave(rwlock_t *lock)
{
unsigned long flags;
- local_irq_save(flags);
+
+ _local_irq_save(flags);
+ if ( unlikely(tb_init_done) )
+ trace_irq_save_ret(flags);
_read_lock(lock);
return flags;
}
@@ -100,13 +105,17 @@ static inline void _read_unlock(rwlock_t *lock)
static inline void _read_unlock_irq(rwlock_t *lock)
{
_read_unlock(lock);
- local_irq_enable();
+ if ( unlikely(tb_init_done) )
+ trace_irq_enable_ret();
+ _local_irq_enable();
}
static inline void _read_unlock_irqrestore(rwlock_t *lock, unsigned long flags)
{
_read_unlock(lock);
- local_irq_restore(flags);
+ if ( unlikely(tb_init_done) )
+ trace_irq_restore_ret(flags);
+ _local_irq_restore(flags);
}
static inline int _rw_is_locked(rwlock_t *lock)
@@ -130,7 +139,9 @@ static inline void _write_lock(rwlock_t *lock)
static inline void _write_lock_irq(rwlock_t *lock)
{
ASSERT(local_irq_is_enabled());
- local_irq_disable();
+ _local_irq_disable();
+ if ( unlikely(tb_init_done) )
+ trace_irq_disable_ret();
_write_lock(lock);
}
@@ -138,7 +149,9 @@ static inline unsigned long _write_lock_irqsave(rwlock_t *lock)
{
unsigned long flags;
- local_irq_save(flags);
+ _local_irq_save(flags);
+ if ( unlikely(tb_init_done) )
+ trace_irq_save_ret(flags);
_write_lock(lock);
return flags;
}
@@ -171,13 +184,17 @@ static inline void _write_unlock(rwlock_t *lock)
static inline void _write_unlock_irq(rwlock_t *lock)
{
_write_unlock(lock);
- local_irq_enable();
+ if ( unlikely(tb_init_done) )
+ trace_irq_enable_ret();
+ _local_irq_enable();
}
static inline void _write_unlock_irqrestore(rwlock_t *lock, unsigned long flags)
{
_write_unlock(lock);
- local_irq_restore(flags);
+ if ( unlikely(tb_init_done) )
+ trace_irq_restore_ret(flags);
+ _local_irq_restore(flags);
}
static inline int _rw_is_write_locked(rwlock_t *lock)
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 07/15] tools: tracing: handle IRQs on/off events in xentrace and xenalyze
2017-06-01 17:33 [PATCH 00/15] xen/tools: add tracing to various Xen subsystems Dario Faggioli
` (5 preceding siblings ...)
2017-06-01 17:34 ` [PATCH 06/15] xen: trace IRQ enabling/disabling Dario Faggioli
@ 2017-06-01 17:34 ` Dario Faggioli
2017-06-13 15:58 ` George Dunlap
2017-06-01 17:34 ` [PATCH 08/15] xen: trace RCU behavior Dario Faggioli
` (9 subsequent siblings)
16 siblings, 1 reply; 68+ messages in thread
From: Dario Faggioli @ 2017-06-01 17:34 UTC (permalink / raw)
To: xen-devel
Cc: George Dunlap, Andrew Cooper, Ian Jackson, Wei Liu, Jennifer Herbert
so the trace will show properly decoded info,
rather than just a bunch of hex codes.
Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
---
Cc: George Dunlap <george.dunlap@eu.citrix.com>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>
Cc: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Wei Liu <wei.liu2@citrix.com>
Cc: Jennifer Herbert <jennifer.herbert@citrix.com>
---
tools/xentrace/formats | 2 ++
tools/xentrace/xenalyze.c | 16 ++++++++++++++++
2 files changed, 18 insertions(+)
diff --git a/tools/xentrace/formats b/tools/xentrace/formats
index 00c29b8..2e653ca 100644
--- a/tools/xentrace/formats
+++ b/tools/xentrace/formats
@@ -202,6 +202,8 @@
0x0080200a CPU%(cpu)d %(tsc)d (+%(reltsc)8d) do_irq [ irq = %(1)d ]
0x0080200b CPU%(cpu)d %(tsc)d (+%(reltsc)8d) do_guest_irq [ irq = %(1)d ]
0x0080200c CPU%(cpu)d %(tsc)d (+%(reltsc)8d) irq_exit [ irq = %(1)d, status = 0x%(2)x, in_irq = %(3)d ]
+0x0080200d CPU%(cpu)d %(tsc)d (+%(reltsc)8d) irq_disable [ from = 0x%(2)08x%(1)08x ]
+0x0080200e CPU%(cpu)d %(tsc)d (+%(reltsc)8d) irq_enable [ from = 0x%(2)08x%(1)08x ]
0x00084001 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) hpet create [ tn = %(1)d, irq = %(2)d, delta = 0x%(4)08x%(3)08x, period = 0x%(6)08x%(5)08x ]
0x00084002 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) pit create [ delta = 0x%(1)016x, period = 0x%(2)016x ]
diff --git a/tools/xentrace/xenalyze.c b/tools/xentrace/xenalyze.c
index 3c2f600..c4a8340 100644
--- a/tools/xentrace/xenalyze.c
+++ b/tools/xentrace/xenalyze.c
@@ -8562,6 +8562,22 @@ void irq_process(struct pcpu_info *p) {
}
break;
}
+ case TRC_HW_IRQ_DISABLE:
+ case TRC_HW_IRQ_ENABLE:
+ {
+ struct {
+ uint64_t addr;
+ } *r = (typeof(r))ri->d;
+
+ if ( opt.dump_all )
+ {
+ printf(" %s irq_%s, from %p\n",
+ ri->dump_header,
+ ri->event == TRC_HW_IRQ_DISABLE ? "disable" : "enable",
+ (void*)r->addr);
+ }
+ break;
+ }
case TRC_HW_IRQ_CLEAR_VECTOR:
case TRC_HW_IRQ_MOVE_FINISH :
default:
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 08/15] xen: trace RCU behavior
2017-06-01 17:33 [PATCH 00/15] xen/tools: add tracing to various Xen subsystems Dario Faggioli
` (6 preceding siblings ...)
2017-06-01 17:34 ` [PATCH 07/15] tools: tracing: handle IRQs on/off events in xentrace and xenalyze Dario Faggioli
@ 2017-06-01 17:34 ` Dario Faggioli
2017-06-09 10:48 ` Jan Beulich
2017-06-13 16:05 ` George Dunlap
2017-06-01 17:34 ` [PATCH 09/15] tools: tracing: handle RCU events in xentrace and xenalyze Dario Faggioli
` (8 subsequent siblings)
16 siblings, 2 replies; 68+ messages in thread
From: Dario Faggioli @ 2017-06-01 17:34 UTC (permalink / raw)
To: xen-devel
Cc: Stefano Stabellini, George Dunlap, Andrew Cooper, Tim Deegan,
Jan Beulich
Making it possible generate events showing the
activity and the behavior of the RCU subsystem.
Gate this with its specific Kconfig option (under
CONFIG_TRACING), and keep it in disabled state by
default.
---
Cc: George Dunlap <George.Dunlap@eu.citrix.com>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>
Cc: Jan Beulich <jbeulich@suse.com>
Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: Stefano Stabellini <sstabellini@kernel.org>
Cc: Tim Deegan <tim@xen.org>
---
xen/Kconfig.debug | 8 ++++
xen/common/rcupdate.c | 82 ++++++++++++++++++++++++++++++++++++++++----
xen/include/public/trace.h | 14 ++++++++
3 files changed, 97 insertions(+), 7 deletions(-)
diff --git a/xen/Kconfig.debug b/xen/Kconfig.debug
index 81910c9..fd5cccc 100644
--- a/xen/Kconfig.debug
+++ b/xen/Kconfig.debug
@@ -116,6 +116,14 @@ config TRACE_IRQSOFF
Note that this comes with high overead and produces huge amount of
tracing data.
+config TRACE_RCU
+ bool "Trace RCU behavior" if EXPERT = "y"
+ default n
+ depends on TRACING
+ ---help---
+ Makes it possible generate events showing the activity and the
+ behavior of the RCU subsystem.
+
config VERBOSE_DEBUG
bool "Verbose debug messages"
default DEBUG
diff --git a/xen/common/rcupdate.c b/xen/common/rcupdate.c
index 8cc5a82..f160582 100644
--- a/xen/common/rcupdate.c
+++ b/xen/common/rcupdate.c
@@ -92,6 +92,57 @@ static int qhimark = 10000;
static int qlowmark = 100;
static int rsinterval = 1000;
+#ifdef CONFIG_TRACE_RCU
+static inline void trace_call_rcu(void *func)
+{
+ uint64_t addr = (uint64_t)func;
+
+ if ( likely(!tb_init_done) )
+ return;
+
+ __trace_var(TRC_XEN_RCU_CALL_RCU, 0, sizeof(addr), &addr);
+}
+static inline void trace_start_batch(const cpumask_t *m)
+{
+ uint32_t mask[6];
+
+ if ( likely(!tb_init_done) )
+ return;
+
+ memset(mask, 0, sizeof(mask));
+ memcpy(mask, m, min(sizeof(mask), sizeof(cpumask_t)));
+ __trace_var(TRC_XEN_RCU_START_BATCH, 0, sizeof(mask), &mask);
+}
+static inline void trace_do_batch(void *func, long int qlen)
+{
+ struct {
+ uint64_t addr;
+ int64_t qlen;
+ } d;
+
+ if ( likely(!tb_init_done) )
+ return;
+
+ d.addr = (uint64_t)func;
+ d.qlen = qlen;
+ __trace_var(TRC_XEN_RCU_DO_BATCH, 0, sizeof(d), &d);
+}
+#define trace_force_qstate() TRACE_0D(TRC_XEN_RCU_FORCE_QSTATE)
+#define trace_cpu_quiet() TRACE_0D(TRC_XEN_RCU_CPU_QUIET)
+#define trace_check_qstate(p) TRACE_1D(TRC_XEN_RCU_CHECK_QSTATE, p)
+#define trace_do_callbacks() TRACE_0D(TRC_XEN_RCU_DO_CALLBKS)
+#define trace_pending(p) TRACE_1D(TRC_XEN_RCU_PENDING, p)
+#else /* !TRACE_RCU */
+#define trace_call_rcu(f) do {} while ( 0 )
+#define trace_start_batch(m) do {} while ( 0 )
+#define trace_do_batch(f, q) do {} while ( 0 )
+#define trace_force_qstate() do {} while ( 0 )
+#define trace_cpu_quiet() do {} while ( 0 )
+#define trace_check_qstate(p) do {} while ( 0 )
+#define trace_do_callbacks() do {} while ( 0 )
+#define trace_pending(p) do {} while ( 0 )
+#endif /* TRACE_RCU */
+
struct rcu_barrier_data {
struct rcu_head head;
atomic_t *cpu_count;
@@ -145,6 +196,9 @@ static void force_quiescent_state(struct rcu_data *rdp,
struct rcu_ctrlblk *rcp)
{
cpumask_t cpumask;
+
+ trace_force_qstate();
+
raise_softirq(SCHEDULE_SOFTIRQ);
if (unlikely(rdp->qlen - rdp->last_rs_qlen > rsinterval)) {
rdp->last_rs_qlen = rdp->qlen;
@@ -177,6 +231,7 @@ void call_rcu(struct rcu_head *head,
head->func = func;
head->next = NULL;
local_irq_save(flags);
+ trace_call_rcu(func);
rdp = &__get_cpu_var(rcu_data);
*rdp->nxttail = head;
rdp->nxttail = &head->next;
@@ -199,12 +254,14 @@ static void rcu_do_batch(struct rcu_data *rdp)
list = rdp->donelist;
while (list) {
next = rdp->donelist = list->next;
+ trace_do_batch(list->func, rdp->qlen);
list->func(list);
list = next;
rdp->qlen--;
if (++count >= rdp->blimit)
break;
}
+
if (rdp->blimit == INT_MAX && rdp->qlen <= qlowmark)
rdp->blimit = blimit;
if (!rdp->donelist)
@@ -249,6 +306,7 @@ static void rcu_start_batch(struct rcu_ctrlblk *rcp)
rcp->cur++;
cpumask_copy(&rcp->cpumask, &cpu_online_map);
+ trace_start_batch(&rcp->cpumask);
}
}
@@ -259,6 +317,7 @@ static void rcu_start_batch(struct rcu_ctrlblk *rcp)
*/
static void cpu_quiet(int cpu, struct rcu_ctrlblk *rcp)
{
+ trace_cpu_quiet();
cpumask_clear_cpu(cpu, &rcp->cpumask);
if (cpumask_empty(&rcp->cpumask)) {
/* batch completed ! */
@@ -279,7 +338,7 @@ static void rcu_check_quiescent_state(struct rcu_ctrlblk *rcp,
/* start new grace period: */
rdp->qs_pending = 1;
rdp->quiescbatch = rcp->cur;
- return;
+ goto out;
}
/* Grace period already completed for this cpu?
@@ -287,7 +346,7 @@ static void rcu_check_quiescent_state(struct rcu_ctrlblk *rcp,
* cacheline trashing.
*/
if (!rdp->qs_pending)
- return;
+ goto out;
rdp->qs_pending = 0;
@@ -300,6 +359,8 @@ static void rcu_check_quiescent_state(struct rcu_ctrlblk *rcp,
cpu_quiet(rdp->cpu, rcp);
spin_unlock(&rcp->lock);
+ out:
+ trace_check_qstate(rdp->qs_pending);
}
@@ -309,6 +370,8 @@ static void rcu_check_quiescent_state(struct rcu_ctrlblk *rcp,
static void __rcu_process_callbacks(struct rcu_ctrlblk *rcp,
struct rcu_data *rdp)
{
+ trace_do_callbacks();
+
if (rdp->curlist && !rcu_batch_before(rcp->completed, rdp->batch)) {
*rdp->donetail = rdp->curlist;
rdp->donetail = rdp->curtail;
@@ -357,26 +420,31 @@ static void rcu_process_callbacks(void)
static int __rcu_pending(struct rcu_ctrlblk *rcp, struct rcu_data *rdp)
{
+ bool ret = true;
+
/* This cpu has pending rcu entries and the grace period
* for them has completed.
*/
if (rdp->curlist && !rcu_batch_before(rcp->completed, rdp->batch))
- return 1;
+ goto out;
/* This cpu has no pending entries, but there are new entries */
if (!rdp->curlist && rdp->nxtlist)
- return 1;
+ goto out;
/* This cpu has finished callbacks to invoke */
if (rdp->donelist)
- return 1;
+ goto out;
/* The rcu core waits for a quiescent state from the cpu */
if (rdp->quiescbatch != rcp->cur || rdp->qs_pending)
- return 1;
+ goto out;
/* nothing to do */
- return 0;
+ ret = false;
+ out:
+ trace_pending(ret);
+ return ret;
}
int rcu_pending(int cpu)
diff --git a/xen/include/public/trace.h b/xen/include/public/trace.h
index 1692a79..50b4fcc 100644
--- a/xen/include/public/trace.h
+++ b/xen/include/public/trace.h
@@ -39,6 +39,7 @@
#define TRC_PV 0x0020f000 /* Xen PV traces */
#define TRC_SHADOW 0x0040f000 /* Xen shadow tracing */
#define TRC_HW 0x0080f000 /* Xen hardware-related traces */
+#define TRC_XEN 0x0100f000 /* Xen misc traces (RCU, softirq, etc) */
#define TRC_GUEST 0x0800f000 /* Guest-generated traces */
#define TRC_ALL 0x0ffff000
#define TRC_HD_TO_EVENT(x) ((x)&0x0fffffff)
@@ -92,6 +93,9 @@
#define TRC_HW_PM 0x00801000 /* Power management traces */
#define TRC_HW_IRQ 0x00802000 /* Traces relating to the handling of IRQs */
+/* Trace subclasses for Xen internals */
+#define TRC_XEN_RCU 0x01001000 /* RCU traces */
+
/* Trace events per class */
#define TRC_LOST_RECORDS (TRC_GEN + 1)
#define TRC_TRACE_WRAP_BUFFER (TRC_GEN + 2)
@@ -278,6 +282,16 @@
#define TRC_HW_IRQ_DISABLE (TRC_HW_IRQ + 0xD)
#define TRC_HW_IRQ_ENABLE (TRC_HW_IRQ + 0xE)
+/* Trace events for RCU */
+#define TRC_XEN_RCU_FORCE_QSTATE (TRC_XEN_RCU + 0x1)
+#define TRC_XEN_RCU_CALL_RCU (TRC_XEN_RCU + 0x2)
+#define TRC_XEN_RCU_START_BATCH (TRC_XEN_RCU + 0x3)
+#define TRC_XEN_RCU_DO_BATCH (TRC_XEN_RCU + 0x4)
+#define TRC_XEN_RCU_CPU_QUIET (TRC_XEN_RCU + 0x5)
+#define TRC_XEN_RCU_CHECK_QSTATE (TRC_XEN_RCU + 0x6)
+#define TRC_XEN_RCU_DO_CALLBKS (TRC_XEN_RCU + 0x7)
+#define TRC_XEN_RCU_PENDING (TRC_XEN_RCU + 0x8)
+
/*
* Event Flags
*
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 09/15] tools: tracing: handle RCU events in xentrace and xenalyze
2017-06-01 17:33 [PATCH 00/15] xen/tools: add tracing to various Xen subsystems Dario Faggioli
` (7 preceding siblings ...)
2017-06-01 17:34 ` [PATCH 08/15] xen: trace RCU behavior Dario Faggioli
@ 2017-06-01 17:34 ` Dario Faggioli
2017-06-13 16:12 ` George Dunlap
2017-06-01 17:34 ` [PATCH 10/15] xen: trace softirqs Dario Faggioli
` (7 subsequent siblings)
16 siblings, 1 reply; 68+ messages in thread
From: Dario Faggioli @ 2017-06-01 17:34 UTC (permalink / raw)
To: xen-devel; +Cc: George Dunlap, Wei Liu, Ian Jackson
Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
---
Cc: George Dunlap <george.dunlap@eu.citrix.com>
Cc: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Wei Liu <wei.liu2@citrix.com>
---
tools/xentrace/analyze.h | 1
tools/xentrace/formats | 9 ++++
tools/xentrace/xenalyze.c | 114 +++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 123 insertions(+), 1 deletion(-)
diff --git a/tools/xentrace/analyze.h b/tools/xentrace/analyze.h
index 40ee551..29e74fd 100644
--- a/tools/xentrace/analyze.h
+++ b/tools/xentrace/analyze.h
@@ -13,6 +13,7 @@
#define TRC_PV_MAIN 5
#define TRC_SHADOW_MAIN 6
#define TRC_HW_MAIN 7
+#define TRC_XEN_MAIN 8
#define TRC_LOST_RECORDS_END (TRC_GEN + 50)
diff --git a/tools/xentrace/formats b/tools/xentrace/formats
index 2e653ca..77dbd93 100644
--- a/tools/xentrace/formats
+++ b/tools/xentrace/formats
@@ -205,6 +205,15 @@
0x0080200d CPU%(cpu)d %(tsc)d (+%(reltsc)8d) irq_disable [ from = 0x%(2)08x%(1)08x ]
0x0080200e CPU%(cpu)d %(tsc)d (+%(reltsc)8d) irq_enable [ from = 0x%(2)08x%(1)08x ]
+0x01001001 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) rcu_force_qstate [ ]
+0x01001002 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) rcu_call [ fn = 0x%(2)08x%(1)08x ]
+0x01001003 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) rcu_start_batch [ CPU mask: 0x%(6)08x%(5)08x%(4)08x%(3)08x%(2)08x%(1)08x ]
+0x01001004 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) rcu_do_batch [ fn = 0x%(2)08x%(1)08x, qlen = 0x%(4)08x%(3)08x ]
+0x01001005 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) rcu_cpu_quiet [ ]
+0x01001006 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) rcu_check_qstate [ pending = %(d)d ]
+0x01001007 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) rcu_do_callbacks [ ]
+0x01001008 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) rcu_pending [ %(1)d ]
+
0x00084001 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) hpet create [ tn = %(1)d, irq = %(2)d, delta = 0x%(4)08x%(3)08x, period = 0x%(6)08x%(5)08x ]
0x00084002 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) pit create [ delta = 0x%(1)016x, period = 0x%(2)016x ]
0x00084003 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) rtc create [ delta = 0x%(1)016x , period = 0x%(2)016x ]
diff --git a/tools/xentrace/xenalyze.c b/tools/xentrace/xenalyze.c
index c4a8340..e98596a 100644
--- a/tools/xentrace/xenalyze.c
+++ b/tools/xentrace/xenalyze.c
@@ -1752,7 +1752,8 @@ enum {
TOPLEVEL_PV,
TOPLEVEL_SHADOW,
TOPLEVEL_HW,
- TOPLEVEL_MAX=TOPLEVEL_HW+1,
+ TOPLEVEL_XEN,
+ TOPLEVEL_MAX=TOPLEVEL_XEN+1,
};
char * toplevel_name[TOPLEVEL_MAX] = {
@@ -1764,6 +1765,7 @@ char * toplevel_name[TOPLEVEL_MAX] = {
[TOPLEVEL_PV]="pv",
[TOPLEVEL_SHADOW]="shadow",
[TOPLEVEL_HW]="hw",
+ [TOPLEVEL_XEN]="xen",
};
struct trace_volume {
@@ -8588,6 +8590,100 @@ void irq_process(struct pcpu_info *p) {
}
}
+void rcu_process(struct pcpu_info *p) {
+ struct record_info *ri = &p->ri;
+
+ switch ( ri->event )
+ {
+ case TRC_XEN_RCU_FORCE_QSTATE:
+ {
+ if ( opt.dump_all )
+ printf(" %s rcu_force_quiescent_state\n",
+ ri->dump_header);
+ break;
+ }
+ case TRC_XEN_RCU_CALL_RCU:
+ {
+ struct {
+ uint64_t addr;
+ } *r = (typeof(r))ri->d;
+
+ if ( opt.dump_all )
+ printf(" %s rcu_call fn=%p\n",
+ ri->dump_header, (void*)r->addr);
+ break;
+ }
+ case TRC_XEN_RCU_DO_BATCH:
+ {
+ struct {
+ uint64_t addr;
+ int64_t qlen;
+ } *r = (typeof(r))ri->d;
+
+ if ( opt.dump_all )
+ printf(" %s rcu_do_batch, fn=%p, qlen=%ld\n",
+ ri->dump_header, (void*)r->addr, r->qlen);
+ break;
+ }
+ case TRC_XEN_RCU_START_BATCH:
+ {
+ struct {
+ uint32_t mask[6];
+ } *r = (typeof(r))ri->d;
+
+ if ( opt.dump_all )
+ {
+ int i = 5;
+
+ while ( i >= 0 && !r->mask[i] ) i--;
+ printf(" %s rcu_start_batch, cpumask 0x", ri->dump_header);
+ for ( ; i >= 0 ; i-- )
+ printf("%08x", r->mask[i]);
+ printf("\n");
+ }
+ }
+ case TRC_XEN_RCU_CPU_QUIET:
+ {
+ if ( opt.dump_all )
+ printf(" %s rcu_cpu_quiet\n", ri->dump_header);
+ break;
+ }
+ case TRC_XEN_RCU_CHECK_QSTATE:
+ {
+ struct {
+ uint32_t qs_pending;
+ } *r = (typeof(r))ri->d;
+
+ if ( opt.dump_all )
+ printf(" %s rcu_check_quiesc_state, qs_pending=%u\n",
+ ri->dump_header, r->qs_pending);
+ break;
+ }
+ case TRC_XEN_RCU_DO_CALLBKS:
+ {
+ if ( opt.dump_all )
+ printf(" %s rcu_process_callbacks\n", ri->dump_header);
+ break;
+ }
+ case TRC_XEN_RCU_PENDING:
+ {
+ struct {
+ uint32_t pending;
+ } *r = (typeof(r))ri->d;
+
+ if ( opt.dump_all )
+ printf(" %s rcu_pending? %s\n",
+ ri->dump_header,
+ r->pending ? "yes" : "no");
+ break;
+ }
+ default:
+ if( opt.dump_all )
+ dump_generic(stdout, ri);
+ break;
+ }
+}
+
#define TRC_HW_SUB_PM 1
#define TRC_HW_SUB_IRQ 2
void hw_process(struct pcpu_info *p)
@@ -8606,6 +8702,19 @@ void hw_process(struct pcpu_info *p)
}
+#define TRC_XEN_SUB_RCU 1
+void xen_process(struct pcpu_info *p)
+{
+ struct record_info *ri = &p->ri;
+
+ switch(ri->evt.sub)
+ {
+ case TRC_XEN_SUB_RCU:
+ rcu_process(p);
+ break;
+ }
+}
+
#define TRC_DOM0_SUB_DOMOPS 1
void dom0_process(struct pcpu_info *p)
{
@@ -9464,6 +9573,9 @@ void process_record(struct pcpu_info *p) {
case TRC_HW_MAIN:
hw_process(p);
break;
+ case TRC_XEN_MAIN:
+ xen_process(p);
+ break;
case TRC_DOM0OP_MAIN:
dom0_process(p);
break;
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 10/15] xen: trace softirqs
2017-06-01 17:33 [PATCH 00/15] xen/tools: add tracing to various Xen subsystems Dario Faggioli
` (8 preceding siblings ...)
2017-06-01 17:34 ` [PATCH 09/15] tools: tracing: handle RCU events in xentrace and xenalyze Dario Faggioli
@ 2017-06-01 17:34 ` Dario Faggioli
2017-06-09 10:51 ` Jan Beulich
2017-06-01 17:34 ` [PATCH 11/15] tools: tracing: handle RCU events in xentrace and xenalyze Dario Faggioli
` (6 subsequent siblings)
16 siblings, 1 reply; 68+ messages in thread
From: Dario Faggioli @ 2017-06-01 17:34 UTC (permalink / raw)
To: xen-devel
Cc: Stefano Stabellini, George Dunlap, Andrew Cooper, Tim Deegan,
Jan Beulich
Making it possible generate events showing the
activity and the behavior of the softirq subsystem.
Gate this with its specific Kconfig option (under
CONFIG_TRACING), and keep it in disabled state by
default.
---
Cc: George Dunlap <George.Dunlap@eu.citrix.com>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>
Cc: Jan Beulich <jbeulich@suse.com>
Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: Stefano Stabellini <sstabellini@kernel.org>
Cc: Tim Deegan <tim@xen.org>
---
xen/Kconfig.debug | 8 ++++++
xen/common/softirq.c | 58 ++++++++++++++++++++++++++++++++++++++++++++
xen/include/public/trace.h | 7 +++++
3 files changed, 73 insertions(+)
diff --git a/xen/Kconfig.debug b/xen/Kconfig.debug
index fd5cccc..4979e83 100644
--- a/xen/Kconfig.debug
+++ b/xen/Kconfig.debug
@@ -124,6 +124,14 @@ config TRACE_RCU
Makes it possible generate events showing the activity and the
behavior of the RCU subsystem.
+config TRACE_SOFTIRQS
+ bool "Trace when softirqs are raised and handled" if EXPERT = "y"
+ default n
+ depends on TRACING
+ ---help---
+ Makes it possible to generate events related to raising and
+ handling of softirqs within Xen.
+
config VERBOSE_DEBUG
bool "Verbose debug messages"
default DEBUG
diff --git a/xen/common/softirq.c b/xen/common/softirq.c
index 67c84ba..152788a 100644
--- a/xen/common/softirq.c
+++ b/xen/common/softirq.c
@@ -13,6 +13,7 @@
#include <xen/mm.h>
#include <xen/preempt.h>
#include <xen/sched.h>
+#include <xen/trace.h>
#include <xen/rcupdate.h>
#include <xen/softirq.h>
@@ -25,6 +26,57 @@ static softirq_handler softirq_handlers[NR_SOFTIRQS];
static DEFINE_PER_CPU(cpumask_t, batch_mask);
static DEFINE_PER_CPU(unsigned int, batching);
+#ifdef CONFIG_TRACE_SOFTIRQS
+#define trace_raise(n) TRACE_1D(TRC_XEN_SIRQ_RAISE, n)
+static inline void trace_raise_cpu(unsigned int nr, unsigned int cpu)
+{
+ struct {
+ uint16_t cpu, nr;
+ } d;
+
+ if ( likely(!tb_init_done) )
+ return;
+
+ d.nr = nr;
+ d.cpu = cpu;
+ __trace_var(TRC_XEN_SIRQ_RAISE_CPU, 1, sizeof(d), &d);
+}
+static inline
+void trace_raise_mask(unsigned int nr, const cpumask_t *m)
+{
+ struct {
+ uint32_t nr;
+ uint32_t mask[6];
+ } d;
+
+ if ( likely(!tb_init_done) )
+ return;
+
+ d.nr = nr;
+ memset(d.mask, 0, sizeof(d.mask));
+ memcpy(d.mask, m, min(sizeof(d.mask), sizeof(cpumask_t)));
+ __trace_var(TRC_XEN_SIRQ_RAISE_MASK, 1, sizeof(d), &d);
+}
+static inline void trace_handler(unsigned int nr, unsigned int cpu)
+{
+ struct {
+ uint16_t pending, nr;
+ } d;
+
+ if ( likely(!tb_init_done) )
+ return;
+
+ d.nr = nr;
+ d.pending = (uint16_t)softirq_pending(cpu);
+ __trace_var(TRC_XEN_SIRQ_HANDLER, 1, sizeof(d), &d);
+}
+#else /* !TRACE_SOFTIRQS */
+#define trace_raise(n) do {} while ( 0 )
+#define trace_raise_cpu(n, c) do {} while ( 0 )
+#define trace_raise_mask(n, m) do {} while ( 0 )
+#define trace_handler(n, c) do {} while ( 0 )
+#endif /* TRACE_SOFTIRQS */
+
static void __do_softirq(unsigned long ignore_mask)
{
unsigned int i, cpu = smp_processor_id();
@@ -43,6 +95,7 @@ static void __do_softirq(unsigned long ignore_mask)
i = find_first_set_bit(pending);
clear_bit(i, &softirq_pending(cpu));
+ trace_handler(i, cpu);
(*softirq_handlers[i])();
}
}
@@ -71,6 +124,8 @@ void cpumask_raise_softirq(const cpumask_t *mask, unsigned int nr)
unsigned int cpu, this_cpu = smp_processor_id();
cpumask_t send_mask, *raise_mask;
+ trace_raise_mask(nr, mask);
+
if ( !per_cpu(batching, this_cpu) || in_irq() )
{
cpumask_clear(&send_mask);
@@ -93,6 +148,8 @@ void cpu_raise_softirq(unsigned int cpu, unsigned int nr)
{
unsigned int this_cpu = smp_processor_id();
+ trace_raise_cpu(nr, cpu);
+
if ( test_and_set_bit(nr, &softirq_pending(cpu))
|| (cpu == this_cpu)
|| arch_skip_send_event_check(cpu) )
@@ -125,6 +182,7 @@ void cpu_raise_softirq_batch_finish(void)
void raise_softirq(unsigned int nr)
{
+ trace_raise(nr);
set_bit(nr, &softirq_pending(smp_processor_id()));
}
diff --git a/xen/include/public/trace.h b/xen/include/public/trace.h
index 50b4fcc..9d065aa 100644
--- a/xen/include/public/trace.h
+++ b/xen/include/public/trace.h
@@ -95,6 +95,7 @@
/* Trace subclasses for Xen internals */
#define TRC_XEN_RCU 0x01001000 /* RCU traces */
+#define TRC_XEN_SIRQ 0x01002000 /* Traces relating to softirqs */
/* Trace events per class */
#define TRC_LOST_RECORDS (TRC_GEN + 1)
@@ -292,6 +293,12 @@
#define TRC_XEN_RCU_DO_CALLBKS (TRC_XEN_RCU + 0x7)
#define TRC_XEN_RCU_PENDING (TRC_XEN_RCU + 0x8)
+/* Trace events for softirqs */
+#define TRC_XEN_SIRQ_HANDLER (TRC_XEN_SIRQ + 0x1)
+#define TRC_XEN_SIRQ_RAISE_MASK (TRC_XEN_SIRQ + 0x2)
+#define TRC_XEN_SIRQ_RAISE_CPU (TRC_XEN_SIRQ + 0x3)
+#define TRC_XEN_SIRQ_RAISE (TRC_XEN_SIRQ + 0x4)
+
/*
* Event Flags
*
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 11/15] tools: tracing: handle RCU events in xentrace and xenalyze
2017-06-01 17:33 [PATCH 00/15] xen/tools: add tracing to various Xen subsystems Dario Faggioli
` (9 preceding siblings ...)
2017-06-01 17:34 ` [PATCH 10/15] xen: trace softirqs Dario Faggioli
@ 2017-06-01 17:34 ` Dario Faggioli
2017-06-01 17:35 ` [PATCH 12/15] xen: trace tasklets Dario Faggioli
` (5 subsequent siblings)
16 siblings, 0 replies; 68+ messages in thread
From: Dario Faggioli @ 2017-06-01 17:34 UTC (permalink / raw)
To: xen-devel; +Cc: George Dunlap, Wei Liu, Ian Jackson
Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
---
Cc: George Dunlap <george.dunlap@eu.citrix.com>
Cc: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Wei Liu <wei.liu2@citrix.com>
---
tools/xentrace/formats | 5 +++
tools/xentrace/xenalyze.c | 82 +++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 87 insertions(+)
diff --git a/tools/xentrace/formats b/tools/xentrace/formats
index 77dbd93..615ba7d 100644
--- a/tools/xentrace/formats
+++ b/tools/xentrace/formats
@@ -214,6 +214,11 @@
0x01001007 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) rcu_do_callbacks [ ]
0x01001008 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) rcu_pending [ %(1)d ]
+0x01002001 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) softirq_handler [ nr:pending_mask = 0x%(1)08x ]
+0x01002002 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) softirq_raise_cpumask [ nr = %(1)d, CPU mask: 0x%(7)08x%(6)08x%(5)08x%(4)08x%(3)08x%(2)08x ]
+0x01002003 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) softirq_raise_cpu [ nr:cpu = 0x%(1)08x ]
+0x01002004 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) softirq_raise [ nr = %(1)d ]
+
0x00084001 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) hpet create [ tn = %(1)d, irq = %(2)d, delta = 0x%(4)08x%(3)08x, period = 0x%(6)08x%(5)08x ]
0x00084002 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) pit create [ delta = 0x%(1)016x, period = 0x%(2)016x ]
0x00084003 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) rtc create [ delta = 0x%(1)016x , period = 0x%(2)016x ]
diff --git a/tools/xentrace/xenalyze.c b/tools/xentrace/xenalyze.c
index e98596a..5cb0f3b 100644
--- a/tools/xentrace/xenalyze.c
+++ b/tools/xentrace/xenalyze.c
@@ -8684,6 +8684,84 @@ void rcu_process(struct pcpu_info *p) {
}
}
+void softirq_process(struct pcpu_info *p) {
+ struct record_info *ri = &p->ri;
+
+ switch ( ri->event )
+ {
+ case TRC_XEN_SIRQ_RAISE:
+ {
+ struct {
+ uint32_t nr;
+ } *r = (typeof(r))ri->d;
+
+ if ( opt.dump_all )
+ {
+ printf(" %s raise_softirq nr %u\n",
+ ri->dump_header,
+ r->nr);
+ }
+ break;
+ }
+ case TRC_XEN_SIRQ_RAISE_CPU:
+ {
+ struct {
+ uint16_t cpu, nr;
+ } *r = (typeof(r))ri->d;
+
+ if ( opt.dump_all )
+ {
+ printf(" %s raise_softirq nr %u on cpu %u\n",
+ ri->dump_header,
+ r->nr,
+ r->cpu);
+ }
+ break;
+ }
+ case TRC_XEN_SIRQ_RAISE_MASK:
+ {
+ struct {
+ uint32_t nr;
+ uint32_t mask[6];
+ } *r = (typeof(r))ri->d;
+
+ if ( opt.dump_all )
+ {
+ int i;
+
+ i = 5;
+ while ( i >= 0 && !r->mask[i] ) i--;
+ printf(" %s raise_softirq nr %u on cpumask 0x",
+ ri->dump_header,
+ r->nr);
+ for ( ; i >= 0 ; i-- )
+ printf("%08x", r->mask[i]);
+ printf("\n");
+ }
+ break;
+ }
+ case TRC_XEN_SIRQ_HANDLER:
+ {
+ struct {
+ uint16_t pending, nr;
+ } *r = (typeof(r))ri->d;
+
+ if ( opt.dump_all )
+ {
+ printf(" %s softirq_handler nr %u, pending = 0x%08x\n",
+ ri->dump_header,
+ r->nr,
+ r->pending);
+ }
+ break;
+ }
+ default:
+ if( opt.dump_all )
+ dump_generic(stdout, ri);
+ break;
+ }
+}
+
#define TRC_HW_SUB_PM 1
#define TRC_HW_SUB_IRQ 2
void hw_process(struct pcpu_info *p)
@@ -8703,6 +8781,7 @@ void hw_process(struct pcpu_info *p)
}
#define TRC_XEN_SUB_RCU 1
+#define TRC_XEN_SUB_SIRQ 2
void xen_process(struct pcpu_info *p)
{
struct record_info *ri = &p->ri;
@@ -8712,6 +8791,9 @@ void xen_process(struct pcpu_info *p)
case TRC_XEN_SUB_RCU:
rcu_process(p);
break;
+ case TRC_XEN_SUB_SIRQ:
+ softirq_process(p);
+ break;
}
}
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 12/15] xen: trace tasklets
2017-06-01 17:33 [PATCH 00/15] xen/tools: add tracing to various Xen subsystems Dario Faggioli
` (10 preceding siblings ...)
2017-06-01 17:34 ` [PATCH 11/15] tools: tracing: handle RCU events in xentrace and xenalyze Dario Faggioli
@ 2017-06-01 17:35 ` Dario Faggioli
2017-06-09 10:59 ` Jan Beulich
2017-06-01 17:35 ` [PATCH 13/15] tools: tracing: handle tasklets events in xentrace and xenalyze Dario Faggioli
` (4 subsequent siblings)
16 siblings, 1 reply; 68+ messages in thread
From: Dario Faggioli @ 2017-06-01 17:35 UTC (permalink / raw)
To: xen-devel
Cc: Stefano Stabellini, George Dunlap, Andrew Cooper, Tim Deegan,
Jan Beulich
Making it possible generate events showing the
activity and the behavior of tasklets.
Gate this with its specific Kconfig option (under
CONFIG_TRACING), and keep it in disabled state by
default.
---
Cc: George Dunlap <George.Dunlap@eu.citrix.com>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>
Cc: Jan Beulich <jbeulich@suse.com>
Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: Stefano Stabellini <sstabellini@kernel.org>
Cc: Tim Deegan <tim@xen.org>
---
xen/Kconfig.debug | 8 ++++
xen/common/tasklet.c | 87 ++++++++++++++++++++++++++++++++++++++++++++
xen/include/public/trace.h | 9 +++++
3 files changed, 104 insertions(+)
diff --git a/xen/Kconfig.debug b/xen/Kconfig.debug
index 4979e83..f7b2e06 100644
--- a/xen/Kconfig.debug
+++ b/xen/Kconfig.debug
@@ -132,6 +132,14 @@ config TRACE_SOFTIRQS
Makes it possible to generate events related to raising and
handling of softirqs within Xen.
+config TRACE_TASKLETS
+ bool "Trace when tasklets are queued and scheduled" if EXPERT = "y"
+ default n
+ depends on TRACING
+ ---help---
+ Make it possible to generate events related to scheduling,
+ queueing and running of tasklets within Xen.
+
config VERBOSE_DEBUG
bool "Verbose debug messages"
default DEBUG
diff --git a/xen/common/tasklet.c b/xen/common/tasklet.c
index 365a777..20b9d68 100644
--- a/xen/common/tasklet.c
+++ b/xen/common/tasklet.c
@@ -30,10 +30,87 @@ static DEFINE_PER_CPU(struct list_head, softirq_tasklet_list);
/* Protects all lists and tasklet structures. */
static DEFINE_SPINLOCK(tasklet_lock);
+#ifdef CONFIG_TRACE_TASKLETS
+static inline void trace_enqueue(const struct tasklet *t)
+{
+ uint64_t addr;
+
+ if ( likely(!tb_init_done) )
+ return;
+
+ addr = (uint64_t)t->func;
+ __trace_var(TRC_XEN_TASKLET_ENQUEUE, 0, sizeof(addr), &addr);
+}
+static inline void trace_schedule(const struct tasklet *t)
+{
+ struct {
+ uint64_t addr;
+ int16_t sched_on, is_sirq;
+ } d;
+
+ if ( likely(!tb_init_done) )
+ return;
+
+ d.addr = (uint64_t)t->func;
+ d.sched_on = t->scheduled_on;
+ d.is_sirq = t->is_softirq;
+ __trace_var(TRC_XEN_TASKLET_SCHEDULE, 1, sizeof(d), &d);
+}
+static inline void trace_work(const struct tasklet *t)
+{
+ uint64_t addr;
+
+ if ( likely(!tb_init_done) )
+ return;
+
+ addr = (uint64_t)t->func;
+ __trace_var(TRC_XEN_TASKLET_WORK, 1, sizeof(addr), &addr);
+}
+static inline void trace_kill(const struct tasklet *t)
+{
+ struct {
+ uint64_t addr;
+ int16_t sched_on, is_run;
+ } d;
+
+ if ( likely(!tb_init_done) )
+ return;
+
+ d.addr = (uint64_t)t->func;
+ d.sched_on = t->scheduled_on;
+ d.is_run = t->is_running;
+ __trace_var(TRC_XEN_TASKLET_KILL, 0, sizeof(d), &d);
+}
+static inline void trace_init(const struct tasklet *t)
+{
+ struct {
+ uint64_t addr;
+ uint32_t is_sirq;
+ } d;
+
+ if ( likely(!tb_init_done) )
+ return;
+
+ d.addr = (uint64_t)t->func;
+ d.is_sirq = t->is_softirq;
+ __trace_var(TRC_XEN_TASKLET_INIT, 0, sizeof(d), &d);
+}
+#define trace_migrate() TRACE_0D(TRC_XEN_TASKLET_MIGR);
+#else
+#define trace_enqueue(t) do {} while ( 0 )
+#define trace_schedule(t) do {} while ( 0 )
+#define trace_work(t) do {} while ( 0 )
+#define trace_kill(t) do {} while ( 0 )
+#define trace_migrate() do {} while ( 0 )
+#define trace_init(t) do {} while ( 0 )
+#endif /* TRACE_TASKLETS */
+
static void tasklet_enqueue(struct tasklet *t)
{
unsigned int cpu = t->scheduled_on;
+ trace_enqueue(t);
+
if ( t->is_softirq )
{
struct list_head *list = &per_cpu(softirq_tasklet_list, cpu);
@@ -60,6 +137,7 @@ void tasklet_schedule_on_cpu(struct tasklet *t, unsigned int cpu)
if ( tasklets_initialised && !t->is_dead )
{
t->scheduled_on = cpu;
+ trace_schedule(t);
if ( !t->is_running )
{
list_del(&t->list);
@@ -91,6 +169,7 @@ static void do_tasklet_work(unsigned int cpu, struct list_head *list)
spin_unlock_irq(&tasklet_lock);
sync_local_execstate();
+ trace_work(t);
t->func(t->data);
spin_lock_irq(&tasklet_lock);
@@ -158,6 +237,7 @@ void tasklet_kill(struct tasklet *t)
list_del_init(&t->list);
}
+ trace_kill(t);
t->scheduled_on = -1;
t->is_dead = 1;
@@ -178,6 +258,11 @@ static void migrate_tasklets_from_cpu(unsigned int cpu, struct list_head *list)
spin_lock_irqsave(&tasklet_lock, flags);
+ if ( list_empty(list) )
+ goto out;
+
+ trace_migrate();
+
while ( !list_empty(list) )
{
t = list_entry(list->next, struct tasklet, list);
@@ -187,6 +272,7 @@ static void migrate_tasklets_from_cpu(unsigned int cpu, struct list_head *list)
tasklet_enqueue(t);
}
+ out:
spin_unlock_irqrestore(&tasklet_lock, flags);
}
@@ -198,6 +284,7 @@ void tasklet_init(
t->scheduled_on = -1;
t->func = func;
t->data = data;
+ trace_init(t);
}
void softirq_tasklet_init(
diff --git a/xen/include/public/trace.h b/xen/include/public/trace.h
index 9d065aa..acee7d5 100644
--- a/xen/include/public/trace.h
+++ b/xen/include/public/trace.h
@@ -96,6 +96,7 @@
/* Trace subclasses for Xen internals */
#define TRC_XEN_RCU 0x01001000 /* RCU traces */
#define TRC_XEN_SIRQ 0x01002000 /* Traces relating to softirqs */
+#define TRC_XEN_TSKLT 0x01004000 /* Traces relating to tasklets */
/* Trace events per class */
#define TRC_LOST_RECORDS (TRC_GEN + 1)
@@ -299,6 +300,14 @@
#define TRC_XEN_SIRQ_RAISE_CPU (TRC_XEN_SIRQ + 0x3)
#define TRC_XEN_SIRQ_RAISE (TRC_XEN_SIRQ + 0x4)
+/* Trace events for tasklets */
+#define TRC_XEN_TASKLET_ENQUEUE (TRC_XEN_TSKLT + 0x1)
+#define TRC_XEN_TASKLET_SCHEDULE (TRC_XEN_TSKLT + 0x2)
+#define TRC_XEN_TASKLET_WORK (TRC_XEN_TSKLT + 0x3)
+#define TRC_XEN_TASKLET_KILL (TRC_XEN_TSKLT + 0x4)
+#define TRC_XEN_TASKLET_INIT (TRC_XEN_TSKLT + 0x5)
+#define TRC_XEN_TASKLET_MIGR (TRC_XEN_TSKLT + 0x6)
+
/*
* Event Flags
*
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 13/15] tools: tracing: handle tasklets events in xentrace and xenalyze
2017-06-01 17:33 [PATCH 00/15] xen/tools: add tracing to various Xen subsystems Dario Faggioli
` (11 preceding siblings ...)
2017-06-01 17:35 ` [PATCH 12/15] xen: trace tasklets Dario Faggioli
@ 2017-06-01 17:35 ` Dario Faggioli
2017-06-01 17:35 ` [PATCH 14/15] xen: trace timers Dario Faggioli
` (3 subsequent siblings)
16 siblings, 0 replies; 68+ messages in thread
From: Dario Faggioli @ 2017-06-01 17:35 UTC (permalink / raw)
To: xen-devel; +Cc: George Dunlap, Wei Liu, Ian Jackson
Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
---
Cc: George Dunlap <george.dunlap@eu.citrix.com>
Cc: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Wei Liu <wei.liu2@citrix.com>
---
tools/xentrace/formats | 7 +++
tools/xentrace/xenalyze.c | 92 +++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 99 insertions(+)
diff --git a/tools/xentrace/formats b/tools/xentrace/formats
index 615ba7d..7d59db7 100644
--- a/tools/xentrace/formats
+++ b/tools/xentrace/formats
@@ -219,6 +219,13 @@
0x01002003 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) softirq_raise_cpu [ nr:cpu = 0x%(1)08x ]
0x01002004 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) softirq_raise [ nr = %(1)d ]
+0x01004001 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) tasklet_enqueue [ fn = 0x%(2)08x%(1)08x ]
+0x01004002 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) tasklet_schedule [ fn = 0x%(2)08x%(1)08x, is_sirq:sched_on = 0x%(3)08x ]
+0x01004003 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) tasklet_work [ fn = 0x%(2)08x%(1)08x ]
+0x01004004 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) tasklet_kill [ fn = 0x%(2)08x%(1)08x, is_run:sched_on = 0x%(3)08x ]
+0x01004005 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) tasklet_init [ fn = 0x%(2)08x%(1)08x, is_sirq = %(3)d ]
+0x01004006 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) tasklet_migr [ ]
+
0x00084001 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) hpet create [ tn = %(1)d, irq = %(2)d, delta = 0x%(4)08x%(3)08x, period = 0x%(6)08x%(5)08x ]
0x00084002 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) pit create [ delta = 0x%(1)016x, period = 0x%(2)016x ]
0x00084003 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) rtc create [ delta = 0x%(1)016x , period = 0x%(2)016x ]
diff --git a/tools/xentrace/xenalyze.c b/tools/xentrace/xenalyze.c
index 5cb0f3b..eecdd61 100644
--- a/tools/xentrace/xenalyze.c
+++ b/tools/xentrace/xenalyze.c
@@ -8762,6 +8762,94 @@ void softirq_process(struct pcpu_info *p) {
}
}
+void tasklet_process(struct pcpu_info *p) {
+ struct record_info *ri = &p->ri;
+
+ switch ( ri->event )
+ {
+ case TRC_XEN_TASKLET_ENQUEUE:
+ {
+ struct {
+ uint64_t addr;
+ } *r = (typeof(r))ri->d;
+
+ if ( opt.dump_all )
+ {
+ printf(" %s tasklet_enqueue fn=%p\n",
+ ri->dump_header, (void*)r->addr);
+ }
+ break;
+ }
+ case TRC_XEN_TASKLET_SCHEDULE:
+ {
+ struct {
+ uint64_t addr;
+ int16_t sched_on, is_sirq;
+ } *r = (typeof(r))ri->d;
+
+ if ( opt.dump_all )
+ {
+ printf(" %s tasklet_schedule fn=%p, sched_on=%d%s\n",
+ ri->dump_header, (void*)r->addr, r->sched_on,
+ r->is_sirq ? " (softirq)" : "");
+ }
+ break;
+ }
+ case TRC_XEN_TASKLET_WORK:
+ {
+ struct {
+ uint64_t addr;
+ } *r = (typeof(r))ri->d;
+
+ if ( opt.dump_all )
+ {
+ printf(" %s tasklet_do_work fn=%p\n",
+ ri->dump_header, (void*)r->addr);
+ }
+ break;
+ }
+ case TRC_XEN_TASKLET_KILL:
+ {
+ struct {
+ uint64_t addr;
+ int16_t sched_on, is_run;
+ } *r = (typeof(r))ri->d;
+
+ if ( opt.dump_all )
+ {
+ printf(" %s tasklet_kill fn=%p\n, sched_on=%d, is_running=%d\n",
+ ri->dump_header, (void*)r->addr, r->sched_on, r->is_run);
+ }
+ break;
+ }
+ case TRC_XEN_TASKLET_INIT:
+ {
+ struct {
+ uint64_t addr;
+ uint32_t is_sirq;
+ } *r = (typeof(r))ri->d;
+
+ if ( opt.dump_all )
+ {
+ printf(" %s tasklet_init fn=%p%s\n",
+ ri->dump_header, (void*)r->addr,
+ r->is_sirq ? ", (softirq)" : "");
+ }
+ break;
+ }
+ case TRC_XEN_TASKLET_MIGR:
+ {
+ if ( opt.dump_all )
+ printf(" %s tasklet_migrate\n", ri->dump_header);
+ break;
+ }
+ default:
+ if( opt.dump_all )
+ dump_generic(stdout, ri);
+ break;
+ }
+}
+
#define TRC_HW_SUB_PM 1
#define TRC_HW_SUB_IRQ 2
void hw_process(struct pcpu_info *p)
@@ -8782,6 +8870,7 @@ void hw_process(struct pcpu_info *p)
#define TRC_XEN_SUB_RCU 1
#define TRC_XEN_SUB_SIRQ 2
+#define TRC_XEN_SUB_TSKLT 4
void xen_process(struct pcpu_info *p)
{
struct record_info *ri = &p->ri;
@@ -8794,6 +8883,9 @@ void xen_process(struct pcpu_info *p)
case TRC_XEN_SUB_SIRQ:
softirq_process(p);
break;
+ case TRC_XEN_SUB_TSKLT:
+ tasklet_process(p);
+ break;
}
}
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 14/15] xen: trace timers
2017-06-01 17:33 [PATCH 00/15] xen/tools: add tracing to various Xen subsystems Dario Faggioli
` (12 preceding siblings ...)
2017-06-01 17:35 ` [PATCH 13/15] tools: tracing: handle tasklets events in xentrace and xenalyze Dario Faggioli
@ 2017-06-01 17:35 ` Dario Faggioli
2017-06-01 17:35 ` [PATCH 15/15] tools: tracing: handle timers events in xentrace and xenalyze Dario Faggioli
` (2 subsequent siblings)
16 siblings, 0 replies; 68+ messages in thread
From: Dario Faggioli @ 2017-06-01 17:35 UTC (permalink / raw)
To: xen-devel
Cc: Stefano Stabellini, George Dunlap, Andrew Cooper, Tim Deegan,
Jan Beulich
Making it possible generate events showing the
activity and the behavior of timers.
Gate this with its specific Kconfig option (under
CONFIG_TRACING), and keep it in disabled state by
default.
---
Cc: George Dunlap <George.Dunlap@eu.citrix.com>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>
Cc: Jan Beulich <jbeulich@suse.com>
Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: Stefano Stabellini <sstabellini@kernel.org>
Cc: Tim Deegan <tim@xen.org>
---
xen/Kconfig.debug | 9 +++
xen/common/timer.c | 158 ++++++++++++++++++++++++++++++++++++++++++--
xen/include/public/trace.h | 12 +++
3 files changed, 173 insertions(+), 6 deletions(-)
diff --git a/xen/Kconfig.debug b/xen/Kconfig.debug
index f7b2e06..e363435 100644
--- a/xen/Kconfig.debug
+++ b/xen/Kconfig.debug
@@ -140,6 +140,15 @@ config TRACE_TASKLETS
Make it possible to generate events related to scheduling,
queueing and running of tasklets within Xen.
++config TRACE_TIMERS
+ bool "Trace when timers are armed and fires" if EXPERT = "y"
+ default n
+ depends on TRACING
+ ---help---
+ Makes it possible to generate events related to the timers
+ subsystems, such as, creation, initialization, queueing,
+ firing and removal of timers.
+
config VERBOSE_DEBUG
bool "Verbose debug messages"
default DEBUG
diff --git a/xen/common/timer.c b/xen/common/timer.c
index d9ff669..8daeeb3 100644
--- a/xen/common/timer.c
+++ b/xen/common/timer.c
@@ -43,6 +43,136 @@ static DEFINE_RCU_READ_LOCK(timer_cpu_read_lock);
DEFINE_PER_CPU(s_time_t, timer_deadline);
+#ifdef CONFIG_TRACE_TIMERS
+static inline void trace_rm_entry(const struct timer *t)
+{
+ struct {
+ uint64_t addr;
+ uint16_t status, cpu;
+ } d;
+
+ if ( likely(!tb_init_done) )
+ return;
+
+ d.addr = (uint64_t)t;
+ d.cpu = t->cpu;
+ d.status = t->status;
+ __trace_var(TRC_XEN_TIMER_RMENTRY, 0, sizeof(d), &d);
+}
+static inline void trace_add_entry(const struct timer *t)
+{
+ struct {
+ uint64_t addr;
+ uint16_t status, cpu;
+ } d;
+
+ if ( likely(!tb_init_done) )
+ return;
+
+ d.addr = (uint64_t)t;
+ d.cpu = t->cpu;
+ d.status = t->status;
+ __trace_var(TRC_XEN_TIMER_ADDENTRY, 0, sizeof(d), &d);
+}
+static inline void trace_set(const struct timer *t)
+{
+ struct {
+ uint64_t addr, addr_fn;
+ int64_t expires_in;
+ } d;
+
+ if ( likely(!tb_init_done) )
+ return;
+
+ d.addr = (uint64_t)t;
+ d.addr_fn = (uint64_t)t->function;
+ d.expires_in = t->expires - NOW();
+ __trace_var(TRC_XEN_TIMER_SET, 1, sizeof(d), &d);
+}
+static inline void trace_stop(const struct timer *t)
+{
+ uint64_t addr;
+
+ if ( likely(!tb_init_done) )
+ return;
+
+ addr = (uint64_t)t;
+ __trace_var(TRC_XEN_TIMER_STOP, 0, sizeof(addr), &addr);
+}
+static inline void trace_migrate(const struct timer *t, unsigned int cpu)
+{
+ struct {
+ uint64_t addr;
+ uint16_t new_cpu, old_cpu;
+ } d;
+
+ if ( likely(!tb_init_done) )
+ return;
+
+ d.addr = (uint64_t)t;
+ d.old_cpu = t->cpu;
+ d.new_cpu = cpu;
+ __trace_var(TRC_XEN_TIMER_MIGRATE, 0, sizeof(d), &d);
+}
+static inline void trace_kill(const struct timer *t)
+{
+ uint64_t addr;
+
+ if ( likely(!tb_init_done) )
+ return;
+
+ addr = (uint64_t)t;
+ __trace_var(TRC_XEN_TIMER_KILL, 0, sizeof(addr), &addr);
+}
+static inline void trace_exec(const struct timer *t)
+{
+ struct {
+ uint64_t addr;
+ int64_t tardiness;
+ } d;
+
+ if ( likely(!tb_init_done) )
+ return;
+
+ d.addr = (uint64_t)t;
+ d.tardiness = NOW() - t->expires;
+ __trace_var(TRC_XEN_TIMER_EXEC, 1, sizeof(d), &d);
+}
+static inline void trace_reprogr(s_time_t n, s_time_t d)
+{
+ uint64_t deadline = d - n;
+
+ if ( likely(!tb_init_done) )
+ return;
+
+ __trace_var(TRC_XEN_TIMER_REPRGR, 1, sizeof(deadline), &deadline);
+}
+static inline void trace_hoverfl(unsigned int old_lim, unsigned int new_lim)
+{
+ struct {
+ uint16_t new_limit, old_limit;
+ } d;
+
+ if ( likely(!tb_init_done) )
+ return;
+
+ d.old_limit = old_lim;
+ d.new_limit = new_lim;
+ __trace_var(TRC_XEN_TIMER_EXEC, 0, sizeof(d), &d);
+}
+#define trace_hoverfl(o, n) TRACE_2D(TRC_XEN_TIMER_HOVERFL, o, n)
+#else /* !TRACE_TIMERS */
+#define trace_rm_entry(t) do {} while ( 0 )
+#define trace_add_entry(t) do {} while ( 0 )
+#define trace_set(t) do {} while ( 0 )
+#define trace_stop(t) do {} while ( 0 )
+#define trace_migrate(t, c) do {} while ( 0 )
+#define trace_kill(t) do {} while ( 0 )
+#define trace_exec(t) do {} while ( 0 )
+#define trace_reprogr(n, d) do {} while ( 0 )
+#define trace_hoverfl(o, n) do {} while ( 0 )
+#endif /* TRACE_TIMERS */
+
/****************************************************************************
* HEAP OPERATIONS.
*/
@@ -175,6 +305,8 @@ static int remove_entry(struct timer *t)
struct timers *timers = &per_cpu(timers, t->cpu);
int rc;
+ trace_rm_entry(t);
+
switch ( t->status )
{
case TIMER_STATUS_in_heap:
@@ -204,11 +336,14 @@ static int add_entry(struct timer *t)
t->status = TIMER_STATUS_in_heap;
rc = add_to_heap(timers->heap, t);
if ( t->heap_offset != 0 )
- return rc;
+ goto out;
/* Fall back to adding to the slower linked list. */
t->status = TIMER_STATUS_in_list;
- return add_to_list(&timers->list, t);
+ rc = add_to_list(&timers->list, t);
+ out:
+ trace_add_entry(t);
+ return rc;
}
static inline void activate_timer(struct timer *timer)
@@ -307,6 +442,8 @@ void set_timer(struct timer *timer, s_time_t expires)
if ( !timer_lock_irqsave(timer, flags) )
return;
+ trace_set(timer);
+
if ( active_timer(timer) )
deactivate_timer(timer);
@@ -325,6 +462,8 @@ void stop_timer(struct timer *timer)
if ( !timer_lock_irqsave(timer, flags) )
return;
+ trace_stop(timer);
+
if ( active_timer(timer) )
deactivate_timer(timer);
@@ -335,7 +474,6 @@ void stop_timer(struct timer *timer)
void migrate_timer(struct timer *timer, unsigned int new_cpu)
{
unsigned int old_cpu;
- bool_t active;
unsigned long flags;
rcu_read_lock(&timer_cpu_read_lock);
@@ -369,15 +507,16 @@ void migrate_timer(struct timer *timer, unsigned int new_cpu)
rcu_read_unlock(&timer_cpu_read_lock);
- active = active_timer(timer);
- if ( active )
+ trace_migrate(timer, new_cpu);
+
+ if ( active_timer(timer) )
deactivate_timer(timer);
list_del(&timer->inactive);
write_atomic(&timer->cpu, new_cpu);
list_add(&timer->inactive, &per_cpu(timers, new_cpu).inactive);
- if ( active )
+ if ( active_timer(timer) )
activate_timer(timer);
spin_unlock(&per_cpu(timers, old_cpu).lock);
@@ -395,6 +534,8 @@ void kill_timer(struct timer *timer)
if ( !timer_lock_irqsave(timer, flags) )
return;
+ trace_kill(timer);
+
if ( active_timer(timer) )
deactivate_timer(timer);
@@ -421,6 +562,7 @@ static void execute_timer(struct timers *ts, struct timer *t)
ts->running = t;
spin_unlock_irq(&ts->lock);
+ trace_exec(t);
(*fn)(data);
spin_lock_irq(&ts->lock);
ts->running = NULL;
@@ -443,6 +585,7 @@ static void timer_softirq_action(void)
int old_limit = GET_HEAP_LIMIT(heap);
int new_limit = ((old_limit + 1) << 4) - 1;
struct timer **newheap = xmalloc_array(struct timer *, new_limit + 1);
+
if ( newheap != NULL )
{
spin_lock_irq(&ts->lock);
@@ -454,6 +597,7 @@ static void timer_softirq_action(void)
xfree(heap);
heap = newheap;
}
+ trace_hoverfl(old_limit, new_limit);
}
spin_lock_irq(&ts->lock);
@@ -495,6 +639,8 @@ static void timer_softirq_action(void)
this_cpu(timer_deadline) =
(deadline == STIME_MAX) ? 0 : MAX(deadline, now + timer_slop);
+ trace_reprogr(now, this_cpu(timer_deadline));
+
if ( !reprogram_timer(this_cpu(timer_deadline)) )
raise_softirq(TIMER_SOFTIRQ);
diff --git a/xen/include/public/trace.h b/xen/include/public/trace.h
index acee7d5..9983ce8 100644
--- a/xen/include/public/trace.h
+++ b/xen/include/public/trace.h
@@ -97,6 +97,7 @@
#define TRC_XEN_RCU 0x01001000 /* RCU traces */
#define TRC_XEN_SIRQ 0x01002000 /* Traces relating to softirqs */
#define TRC_XEN_TSKLT 0x01004000 /* Traces relating to tasklets */
+#define TRC_XEN_TIMER 0x01008000 /* Trace relating to timer events */
/* Trace events per class */
#define TRC_LOST_RECORDS (TRC_GEN + 1)
@@ -308,6 +309,17 @@
#define TRC_XEN_TASKLET_INIT (TRC_XEN_TSKLT + 0x5)
#define TRC_XEN_TASKLET_MIGR (TRC_XEN_TSKLT + 0x6)
+/* Trace events for timers */
+#define TRC_XEN_TIMER_RMENTRY (TRC_XEN_TIMER + 0x1)
+#define TRC_XEN_TIMER_ADDENTRY (TRC_XEN_TIMER + 0x2)
+#define TRC_XEN_TIMER_SET (TRC_XEN_TIMER + 0x3)
+#define TRC_XEN_TIMER_STOP (TRC_XEN_TIMER + 0x4)
+#define TRC_XEN_TIMER_MIGRATE (TRC_XEN_TIMER + 0x5)
+#define TRC_XEN_TIMER_KILL (TRC_XEN_TIMER + 0x6)
+#define TRC_XEN_TIMER_EXEC (TRC_XEN_TIMER + 0x7)
+#define TRC_XEN_TIMER_REPRGR (TRC_XEN_TIMER + 0x8)
+#define TRC_XEN_TIMER_HOVERFL (TRC_XEN_TIMER + 0x9)
+
/*
* Event Flags
*
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 15/15] tools: tracing: handle timers events in xentrace and xenalyze
2017-06-01 17:33 [PATCH 00/15] xen/tools: add tracing to various Xen subsystems Dario Faggioli
` (13 preceding siblings ...)
2017-06-01 17:35 ` [PATCH 14/15] xen: trace timers Dario Faggioli
@ 2017-06-01 17:35 ` Dario Faggioli
2017-06-07 14:13 ` [PATCH 00/15] xen/tools: add tracing to various Xen subsystems Konrad Rzeszutek Wilk
2017-06-13 16:34 ` George Dunlap
16 siblings, 0 replies; 68+ messages in thread
From: Dario Faggioli @ 2017-06-01 17:35 UTC (permalink / raw)
To: xen-devel; +Cc: George Dunlap, Wei Liu, Ian Jackson
Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
---
Cc: George Dunlap <george.dunlap@eu.citrix.com>
Cc: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Wei Liu <wei.liu2@citrix.com>
---
tools/xentrace/formats | 10 +++
tools/xentrace/xenalyze.c | 141 +++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 151 insertions(+)
diff --git a/tools/xentrace/formats b/tools/xentrace/formats
index 7d59db7..c2fdead 100644
--- a/tools/xentrace/formats
+++ b/tools/xentrace/formats
@@ -226,6 +226,16 @@
0x01004005 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) tasklet_init [ fn = 0x%(2)08x%(1)08x, is_sirq = %(3)d ]
0x01004006 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) tasklet_migr [ ]
+0x01008001 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) timer_rmentry [ t = 0x%(2)08x%(1)08x, cpu:status = %(3)08x ]
+0x01008002 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) timer_addentry [ t = 0x%(2)08x%(1)08x, cpu:status = %(3)08x ]
+0x01008003 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) timer_set [ t = 0x%(2)08x%(1)08x, fn = 0x%(4)08x%(3)08x exp=0x%(6)08x%(5)08x ]
+0x01008004 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) timer_stop [ t = 0x%(2)08x%(1)08x ]
+0x01008005 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) timer_migr [ t = 0x%(2)08x%(1)08x ]
+0x01008006 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) timer_kill [ t = 0x%(2)08x%(1)08x ]
+0x01008007 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) timer_exec [ t = 0x%(2)08x%(1)08x, tard=0x%(4)08x%(3)08x ]
+0x01008008 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) timer_reprogr [ deadline=0x%(2)08x%(1)08x ]
+0x01008009 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) timer_hoverfl [ old_lim:new_lim=0x%(1)08x ]
+
0x00084001 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) hpet create [ tn = %(1)d, irq = %(2)d, delta = 0x%(4)08x%(3)08x, period = 0x%(6)08x%(5)08x ]
0x00084002 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) pit create [ delta = 0x%(1)016x, period = 0x%(2)016x ]
0x00084003 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) rtc create [ delta = 0x%(1)016x , period = 0x%(2)016x ]
diff --git a/tools/xentrace/xenalyze.c b/tools/xentrace/xenalyze.c
index eecdd61..c34479c 100644
--- a/tools/xentrace/xenalyze.c
+++ b/tools/xentrace/xenalyze.c
@@ -8850,6 +8850,143 @@ void tasklet_process(struct pcpu_info *p) {
}
}
+void timer_process(struct pcpu_info *p) {
+ struct record_info *ri = &p->ri;
+
+ switch ( ri->event )
+ {
+ case TRC_XEN_TIMER_RMENTRY:
+ {
+ struct {
+ uint64_t addr;
+ uint16_t status, cpu;
+ } *r = (typeof(r))ri->d;
+
+ if ( opt.dump_all )
+ {
+ printf(" %s timer_rm_entry t=%p, cpu=%u, status=0x%x\n",
+ ri->dump_header, (void*)r->addr, r->cpu, r->status);
+ }
+ break;
+ }
+ case TRC_XEN_TIMER_ADDENTRY:
+ {
+ struct {
+ uint64_t addr;
+ uint16_t status, cpu;
+ } *r = (typeof(r))ri->d;
+
+ if ( opt.dump_all )
+ {
+ printf(" %s timer_add_entry t=%p, cpu=%u, status=0x%x\n",
+ ri->dump_header, (void*)r->addr, r->cpu, r->status);
+ }
+ break;
+ }
+ case TRC_XEN_TIMER_SET:
+ {
+ struct {
+ uint64_t addr, addr_fn;
+ int64_t expires;
+ } *r = (typeof(r))ri->d;
+
+ if ( opt.dump_all )
+ {
+ printf(" %s timer_set t=%p, fn=%p, expires_in=%ld.%luus\n",
+ ri->dump_header, (void*)r->addr, (void*)r->addr_fn,
+ (long int)(r->expires / 1000),
+ (unsigned long int)labs(r->expires % 1000));
+ }
+ break;
+ }
+ case TRC_XEN_TIMER_STOP:
+ {
+ struct {
+ uint64_t addr;
+ } *r = (typeof(r))ri->d;
+
+ if ( opt.dump_all )
+ {
+ printf(" %s timer_stop t=%p\n", ri->dump_header, (void*)r->addr);
+ }
+ break;
+ }
+ case TRC_XEN_TIMER_MIGRATE:
+ {
+ struct {
+ uint64_t addr;
+ uint16_t new_cpu, old_cpu;
+ } *r = (typeof(r))ri->d;
+
+ if ( opt.dump_all )
+ {
+ printf(" %s timer_migrate t=%p, cpu=%u, new_cpu=%u\n",
+ ri->dump_header, (void*)r->addr, r->old_cpu, r->new_cpu);
+ }
+ break;
+ }
+ case TRC_XEN_TIMER_KILL:
+ {
+ struct {
+ uint64_t addr;
+ } *r = (typeof(r))ri->d;
+
+ if ( opt.dump_all )
+ {
+ printf(" %s timer_kill t=%p\n", ri->dump_header, (void*)r->addr);
+ }
+ break;
+ }
+ case TRC_XEN_TIMER_EXEC:
+ {
+ struct {
+ uint64_t addr;
+ int64_t tardiness;
+ } *r = (typeof(r))ri->d;
+
+ if ( opt.dump_all )
+ {
+ printf(" %s timer_exec t=%p, tardiness=%ld.%luus\n",
+ ri->dump_header, (void*)r->addr,
+ (long int)(r->tardiness / 1000),
+ (unsigned long int)labs(r->tardiness % 1000));
+ }
+ break;
+ }
+ case TRC_XEN_TIMER_REPRGR:
+ {
+ struct {
+ uint64_t deadline;
+ } *r = (typeof(r))ri->d;
+
+ if ( opt.dump_all )
+ {
+ printf(" %s timer_reprogr deadline=%lu.%luus\n",
+ ri->dump_header, (unsigned long int)(r->deadline / 1000),
+ (unsigned long int)(r->deadline % 1000));
+ }
+ break;
+ }
+ case TRC_XEN_TIMER_HOVERFL:
+ {
+ struct {
+ uint16_t new_limit, old_limit;
+ } *r = (typeof(r))ri->d;
+
+ if ( opt.dump_all )
+ {
+ printf(" %s timer_heap_overflow, limit=%u, new_limit=%u\n",
+ ri->dump_header, r->old_limit, r->new_limit);
+ }
+ break;
+ }
+ default:
+ if( opt.dump_all )
+ dump_generic(stdout, ri);
+ break;
+ }
+}
+
#define TRC_HW_SUB_PM 1
#define TRC_HW_SUB_IRQ 2
void hw_process(struct pcpu_info *p)
@@ -8871,6 +9008,7 @@ void hw_process(struct pcpu_info *p)
#define TRC_XEN_SUB_RCU 1
#define TRC_XEN_SUB_SIRQ 2
#define TRC_XEN_SUB_TSKLT 4
+#define TRC_XEN_SUB_TIMER 8
void xen_process(struct pcpu_info *p)
{
struct record_info *ri = &p->ri;
@@ -8886,6 +9024,9 @@ void xen_process(struct pcpu_info *p)
case TRC_XEN_SUB_TSKLT:
tasklet_process(p);
break;
+ case TRC_XEN_SUB_TIMER:
+ timer_process(p);
+ break;
}
}
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 68+ messages in thread
* Re: [PATCH 02/15] xen: tracing: avoid checking tb_init_done multiple times.
2017-06-01 17:33 ` [PATCH 02/15] xen: tracing: avoid checking tb_init_done multiple times Dario Faggioli
@ 2017-06-01 17:53 ` Andrew Cooper
2017-06-01 23:08 ` Dario Faggioli
2017-06-07 14:46 ` Jan Beulich
2017-06-08 14:37 ` George Dunlap
2 siblings, 1 reply; 68+ messages in thread
From: Andrew Cooper @ 2017-06-01 17:53 UTC (permalink / raw)
To: Dario Faggioli, xen-devel; +Cc: George Dunlap
On 01/06/17 18:33, Dario Faggioli wrote:
> In fact, when calling __trace_var() directly, we can
> assume that tb_init_done has been checked to be true,
> and the if is hence redundant.
>
> While there, also:
> - still in __trace_var(), move the check that the event
> is actually being traced up a little bit (to bail as
> soon as possible, if it is not);
> - make it explicit that tb_init_done is likely 0 in
> trace_will_trace_event(), as it is almost everywhere
> in the code.
>
> Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
> ---
> Cc: George Dunlap <george.dunlap@eu.citrix.com>
> ---
> xen/common/trace.c | 8 +++-----
> 1 file changed, 3 insertions(+), 5 deletions(-)
>
> diff --git a/xen/common/trace.c b/xen/common/trace.c
> index 4fedc26..f29cd4c 100644
> --- a/xen/common/trace.c
> +++ b/xen/common/trace.c
> @@ -311,7 +311,7 @@ static int tb_set_size(unsigned int pages)
>
> int trace_will_trace_event(u32 event)
> {
> - if ( !tb_init_done )
> + if ( likely(!tb_init_done) )
> return 0;
>
> /*
> @@ -691,7 +691,8 @@ void __trace_var(u32 event, bool_t cycles, unsigned int extra,
> unsigned int extra_word;
> bool_t started_below_highwater;
You probably want an ASSERT(tb_init_done) here to keep callers in check.
Otherwise, Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>
>
> - if( !tb_init_done )
> + /* If the event is not interesting, bail, as early as possible */
> + if ( (tb_event_mask & event) == 0 )
> return;
>
> /* Convert byte count into word count, rounding up */
> @@ -705,9 +706,6 @@ void __trace_var(u32 event, bool_t cycles, unsigned int extra,
> /* Round size up to nearest word */
> extra = extra_word * sizeof(u32);
>
> - if ( (tb_event_mask & event) == 0 )
> - return;
> -
> /* match class */
> if ( ((tb_event_mask >> TRC_CLS_SHIFT) & (event >> TRC_CLS_SHIFT)) == 0 )
> return;
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 03/15] xen/tools: tracing: several improvements on IRQs tracing
2017-06-01 17:33 ` [PATCH 03/15] xen/tools: tracing: several improvements on IRQs tracing Dario Faggioli
@ 2017-06-01 18:02 ` Andrew Cooper
2017-06-01 23:12 ` Dario Faggioli
2017-06-07 15:05 ` Jan Beulich
2017-06-08 14:59 ` George Dunlap
2 siblings, 1 reply; 68+ messages in thread
From: Andrew Cooper @ 2017-06-01 18:02 UTC (permalink / raw)
To: Dario Faggioli, xen-devel
Cc: George Dunlap, Wei Liu, Ian Jackson, Jan Beulich
On 01/06/17 18:33, Dario Faggioli wrote:
> More specifically:
> - the handling of the TRC_HW_IRQ_HANDLED is fixed, both in
> xentrace_format and in xenalyze;
> - simple events for recording when we enter and exit the
> do_IRQ function, as well as when we deal with a guest
> IRQ, are added;
> - tracing of IRQs handled with direct vectors is also
> added.
>
> With all the above, a trace will now look like this (using
> xenalyze):
>
> 0.001299072 .x- d32768v5 irq_enter, irq 80000000
> 0.001299072 .x- d32768v5 irq_direct, vec fa, handler = 0xffff82d08026d7e4
Please consistently use 0x$hex. vec in particular has 10/16ths of its
values which are completely ambiguous between hex and decimal.
~Andrew
> 0.001300014 .x- d32768v5 raise_softirq nr 0
> 0.001300487 .x- d32768v5 irq_exit irq 80000000, in_irq = 0
>
> Or like this:
>
> 0.049437892 -|- d32767v12 irq_enter, irq 4
> 0.049437892 -|- d32767v12 irq_handled irq 4, 85428 cycles
> 0.049474336 -|- d32767v12 irq_exit irq 4, status = 0x0, in_irq = 0
>
> Making it much easier to figure out when interrupt
> processing start, what it does and when it ends.
>
> Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
> ---
> Cc: George Dunlap <george.dunlap@eu.citrix.com>
> Cc: Ian Jackson <ian.jackson@eu.citrix.com>
> Cc: Wei Liu <wei.liu2@citrix.com>
> Cc: Jan Beulich <jbeulich@suse.com>
> Cc: Andrew Cooper <andrew.cooper3@citrix.com>
> ---
> tools/xentrace/formats | 6 ++++
> tools/xentrace/xenalyze.c | 56 +++++++++++++++++++++++++++++++++++-----
> xen/arch/x86/irq.c | 61 +++++++++++++++++++++++++++++++++++++-------
> xen/include/public/trace.h | 4 +++
> 4 files changed, 109 insertions(+), 18 deletions(-)
>
> diff --git a/tools/xentrace/formats b/tools/xentrace/formats
> index 8b31780..00c29b8 100644
> --- a/tools/xentrace/formats
> +++ b/tools/xentrace/formats
> @@ -197,7 +197,11 @@
> 0x00802005 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) move_vector [ irq = %(1)d had vector 0x%(2)x on CPU%(3)d ]
> 0x00802006 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) assign_vector [ irq = %(1)d = vector 0x%(2)x, CPU mask: 0x%(3)08x ]
> 0x00802007 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) bogus_vector [ 0x%(1)x ]
> -0x00802008 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) do_irq [ irq = %(1)d, began = %(2)dus, ended = %(3)dus ]
> +0x00802008 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) irq_handled [ irq = %(1)d, dur = 0x%(3)08x%(2)08x ]
> +0x00802009 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) direct_vector [ vector 0x%(1)x, handler = 0x%(3)08x%(2)08x ]
> +0x0080200a CPU%(cpu)d %(tsc)d (+%(reltsc)8d) do_irq [ irq = %(1)d ]
> +0x0080200b CPU%(cpu)d %(tsc)d (+%(reltsc)8d) do_guest_irq [ irq = %(1)d ]
> +0x0080200c CPU%(cpu)d %(tsc)d (+%(reltsc)8d) irq_exit [ irq = %(1)d, status = 0x%(2)x, in_irq = %(3)d ]
>
> 0x00084001 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) hpet create [ tn = %(1)d, irq = %(2)d, delta = 0x%(4)08x%(3)08x, period = 0x%(6)08x%(5)08x ]
> 0x00084002 CPU%(cpu)d %(tsc)d (+%(reltsc)8d) pit create [ delta = 0x%(1)016x, period = 0x%(2)016x ]
> diff --git a/tools/xentrace/xenalyze.c b/tools/xentrace/xenalyze.c
> index fa608ad..063eee7 100644
> --- a/tools/xentrace/xenalyze.c
> +++ b/tools/xentrace/xenalyze.c
> @@ -8380,19 +8380,45 @@ void irq_process(struct pcpu_info *p) {
> }
> break;
> }
> - case TRC_HW_IRQ_HANDLED:
> + case TRC_HW_IRQ_ENTER:
> + case TRC_HW_IRQ_GUEST:
> {
> struct {
> - int irq, start_tsc, end_tsc;
> + int32_t irq;
> } *r = (typeof(r))ri->d;
> - int arctime;
>
> - arctime = r->end_tsc - r->start_tsc;
> if ( opt.dump_all )
> {
> - printf(" %s irq_handled irq %x %d (%d,%d)\n",
> - ri->dump_header,
> - r->irq, arctime, r->start_tsc, r->end_tsc);
> + printf(" %s irq_%s, irq %x\n", ri->dump_header,
> + ri->event == TRC_HW_IRQ_ENTER ? "enter" : "guest", r->irq);
irq %#x
> + }
> + break;
> + }
> + case TRC_HW_IRQ_DIRECT_VECTOR:
> + {
> + struct {
> + uint32_t vec;
> + uint64_t handler;
> + } __attribute__((packed)) *r = (typeof(r))ri->d;
> +
> + if ( opt.dump_all )
> + {
> + printf(" %s irq_direct, vec %x, handler = %p\n",
> + ri->dump_header, r->vec, (void*)r->handler);
vec %02x
> + }
> + break;
> + }
> + case TRC_HW_IRQ_HANDLED:
> + {
> + struct {
> + int32_t irq;
> + uint64_t arctime;
> + } __attribute__((packed)) *r = (typeof(r))ri->d;
> +
> + if ( opt.dump_all )
> + {
> + printf(" %s irq_handled irq %x, %"PRIu64" cycles\n",
irq %#x
> + ri->dump_header, r->irq, r->arctime);
> }
> if ( opt.scatterplot_irq )
> {
> @@ -8407,6 +8433,22 @@ void irq_process(struct pcpu_info *p) {
> }
> break;
> }
> + case TRC_HW_IRQ_EXIT:
> + {
> + struct {
> + int32_t irq, status;
> + uint32_t in_irq;
> + } *r = (typeof(r))ri->d;
> +
> + if ( opt.dump_all )
> + {
> + printf(" %s irq_exit irq %x", ri->dump_header, r->irq);
> + if ( r->status != -1 )
> + printf(", status = 0x%x", r->status);
> + printf(", in_irq = %d\n", r->in_irq);
> + }
> + break;
> + }
> case TRC_HW_IRQ_ASSIGN_VECTOR:
> {
> struct {
> diff --git a/xen/arch/x86/irq.c b/xen/arch/x86/irq.c
> index 676ba52..f5d9302 100644
> --- a/xen/arch/x86/irq.c
> +++ b/xen/arch/x86/irq.c
> @@ -100,11 +100,15 @@ static void trace_irq_mask(u32 event, int irq, int vector, cpumask_t *mask)
> unsigned int irq:16, vec:16;
> unsigned int mask[6];
> } d;
> +
> + if ( likely(!tb_init_done) )
> + return;
> +
> d.irq = irq;
> d.vec = vector;
> memset(d.mask, 0, sizeof(d.mask));
> memcpy(d.mask, mask, min(sizeof(d.mask), sizeof(cpumask_t)));
> - trace_var(event, 1, sizeof(d), &d);
> + __trace_var(event, 1, sizeof(d), &d);
> }
>
> static int __init __bind_irq_vector(int irq, int vector, const cpumask_t *cpu_mask)
> @@ -804,23 +808,54 @@ void alloc_direct_apic_vector(
> spin_unlock(&lock);
> }
>
> +static inline void trace_irq_handled(int irq, uint64_t time)
> +{
> + struct __packed {
> + int32_t irq;
> + uint64_t time;
> + } d;
> +
> + if ( likely(!tb_init_done) )
> + return;
> +
> + d.irq = irq;
> + d.time = time;
> + __trace_var(TRC_HW_IRQ_HANDLED, 0, sizeof(d), &d);
> +}
> +
> void do_IRQ(struct cpu_user_regs *regs)
> {
> struct irqaction *action;
> - uint32_t tsc_in;
> - struct irq_desc *desc;
> + uint64_t tsc_in = 0;
> + struct irq_desc *desc = NULL;
> unsigned int vector = (u8)regs->entry_vector;
> int irq = __get_cpu_var(vector_irq[vector]);
> struct cpu_user_regs *old_regs = set_irq_regs(regs);
>
> perfc_incr(irqs);
> this_cpu(irq_count)++;
> + TRACE_1D(TRC_HW_IRQ_ENTER, irq);
> irq_enter();
>
> - if (irq < 0) {
> - if (direct_apic_vector[vector] != NULL) {
> + if (irq < 0)
Spaces.
> + {
> + if (direct_apic_vector[vector] != NULL)
Spaces.
> + {
> + if ( unlikely(tb_init_done) )
> + {
> + struct __packed {
> + uint32_t vec;
> + uint64_t handler;
> + } d;
> +
> + d.vec = vector;
> + d.handler = (uint64_t)direct_apic_vector[vector];
As there is no code inbetween, you can do this as a straight
initialisation of d.
i.e.
} d = { vector, (uint64_t)direct_apic_vector[vector] };
Otherwise, Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>
> + __trace_var(TRC_HW_IRQ_DIRECT_VECTOR, 0, sizeof(d), &d);
> + }
> (*direct_apic_vector[vector])(regs);
> - } else {
> + }
> + else
> + {
> const char *kind = ", LAPIC";
>
> if ( apic_isr_read(vector) )
> @@ -884,9 +919,13 @@ void do_IRQ(struct cpu_user_regs *regs)
> desc->rl_quantum_start = now;
> }
>
> - tsc_in = tb_init_done ? get_cycles() : 0;
> + if ( unlikely(tb_init_done) )
> + {
> + __trace_var(TRC_HW_IRQ_GUEST, 0, sizeof(irq), &irq);
> + tsc_in = get_cycles();
> + }
> __do_IRQ_guest(irq);
> - TRACE_3D(TRC_HW_IRQ_HANDLED, irq, tsc_in, get_cycles());
> + trace_irq_handled(irq, get_cycles() - tsc_in);
> goto out_no_end;
> }
>
> @@ -907,9 +946,10 @@ void do_IRQ(struct cpu_user_regs *regs)
> {
> desc->status &= ~IRQ_PENDING;
> spin_unlock_irq(&desc->lock);
> - tsc_in = tb_init_done ? get_cycles() : 0;
> + if ( unlikely(tb_init_done) )
> + tsc_in = get_cycles();
> action->handler(irq, action->dev_id, regs);
> - TRACE_3D(TRC_HW_IRQ_HANDLED, irq, tsc_in, get_cycles());
> + trace_irq_handled(irq, get_cycles() - tsc_in);
> spin_lock_irq(&desc->lock);
> }
>
> @@ -922,6 +962,7 @@ void do_IRQ(struct cpu_user_regs *regs)
> spin_unlock(&desc->lock);
> out_no_unlock:
> irq_exit();
> + TRACE_3D(TRC_HW_IRQ_EXIT, irq, desc == NULL ? -1 : desc->status, in_irq());
> set_irq_regs(old_regs);
> }
>
> diff --git a/xen/include/public/trace.h b/xen/include/public/trace.h
> index 7f2e891..f66a7af 100644
> --- a/xen/include/public/trace.h
> +++ b/xen/include/public/trace.h
> @@ -271,6 +271,10 @@
> #define TRC_HW_IRQ_ASSIGN_VECTOR (TRC_HW_IRQ + 0x6)
> #define TRC_HW_IRQ_UNMAPPED_VECTOR (TRC_HW_IRQ + 0x7)
> #define TRC_HW_IRQ_HANDLED (TRC_HW_IRQ + 0x8)
> +#define TRC_HW_IRQ_DIRECT_VECTOR (TRC_HW_IRQ + 0x9)
> +#define TRC_HW_IRQ_ENTER (TRC_HW_IRQ + 0xA)
> +#define TRC_HW_IRQ_GUEST (TRC_HW_IRQ + 0xB)
> +#define TRC_HW_IRQ_EXIT (TRC_HW_IRQ + 0xC)
>
> /*
> * Event Flags
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 05/15] xen: make it possible to disable tracing in Kconfig.
2017-06-01 17:34 ` [PATCH 05/15] xen: make it possible to disable tracing in Kconfig Dario Faggioli
@ 2017-06-01 18:43 ` Andrew Cooper
2017-06-07 11:01 ` Julien Grall
2017-06-07 15:14 ` Jan Beulich
2017-06-08 15:17 ` George Dunlap
2 siblings, 1 reply; 68+ messages in thread
From: Andrew Cooper @ 2017-06-01 18:43 UTC (permalink / raw)
To: Dario Faggioli, xen-devel
Cc: Stefano Stabellini, Wei Liu, George Dunlap, Doug Goldstein,
Ian Jackson, Julien Grall, Jan Beulich
On 01/06/17 18:34, Dario Faggioli wrote:
> And compile it out of the hypervisor entirely.
>
> Code and other sections' sizes change as follows.
>
> Output of `size`:
> vanilla patched-Y patched-N
> text 1929007 1929007 1902783
> data 337784 337784 337688
> bss 1310464 1310464 1310336
>
> Output of `size -A`:
> vanilla patched-Y patched-N
> .text 1372602 1372602 1348026
> .rodata 312152 312152 310680
> .init.text 244209 244209 244033
> .init.data 224576 224576 224576
> .data 57472 57472 57376
> .bss 1310464 1310464 1310336
> Total 23026516 23027008 22858069
>
> No functional change intended.
>
> Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
I'm tempted to ack it on that diffstat alone ;p
However, I've got some suggestions to improve it.
> ---
> Cc: George Dunlap <George.Dunlap@eu.citrix.com>
> Cc: Andrew Cooper <andrew.cooper3@citrix.com>
> Cc: Jan Beulich <jbeulich@suse.com>
> Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> Cc: Stefano Stabellini <sstabellini@kernel.org>
> Cc: Julien Grall <julien.grall@arm.com>
> Cc: Ian Jackson <ian.jackson@eu.citrix.com>
> Cc: Wei Liu <wei.liu2@citrix.com>
> Cc: Doug Goldstein <cardoe@cardoe.com>
> ---
> xen/Kconfig.debug | 8 ++++++++
> xen/arch/x86/hvm/svm/entry.S | 2 ++
> xen/arch/x86/trace.c | 23 +++++++++++++++++++++++
> xen/common/trace.c | 39 +++++++++++++++++++++++++++++++++++----
> xen/drivers/cpufreq/utility.c | 7 +++++--
> xen/include/xen/trace.h | 30 +++++++++++++++++++++++++++++-
> 6 files changed, 102 insertions(+), 7 deletions(-)
>
> diff --git a/xen/Kconfig.debug b/xen/Kconfig.debug
> index 689f297..374c1c0 100644
> --- a/xen/Kconfig.debug
> +++ b/xen/Kconfig.debug
> @@ -98,6 +98,14 @@ config PERF_ARRAYS
> ---help---
> Enables software performance counter array histograms.
>
> +config TRACING
> + bool "Tracing"
> + default y
> + ---help---
> + Enables collecting traces of events occurring in the hypervisor
> + in per-CPU ring buffers. The 'xentrace' tool can be used to read
> + the buffers and dump the content on the disk.
The grammar here is a little unusual. How about extending it a little
to this?
The tracing infrastructure allows Xen to record a log of events in
per-CPU ring buffers, in binary form. These rings can be accessed in
userspace with the 'xentrace' tool, and interpreted using 'xenalyze' or
'xentrace_format'.
> +
>
> config VERBOSE_DEBUG
> bool "Verbose debug messages"
> diff --git a/xen/arch/x86/hvm/svm/entry.S b/xen/arch/x86/hvm/svm/entry.S
> index a4ab40a..ea4a106 100644
> --- a/xen/arch/x86/hvm/svm/entry.S
> +++ b/xen/arch/x86/hvm/svm/entry.S
> @@ -61,10 +61,12 @@ __UNLIKELY_END(nsvm_hap)
>
> call svm_asid_handle_vmrun
>
> +#ifdef CONFIG_TRACING
> cmpb $0,tb_init_done(%rip)
> UNLIKELY_START(nz, svm_trace)
> call svm_trace_vmentry
> UNLIKELY_END(svm_trace)
> +#endif
You can also enclose svm_trace_vmentry() in svm.c with #ifdefary
>
> mov VCPU_svm_vmcb(%rbx),%rcx
> mov UREGS_rax(%rsp),%rax
> diff --git a/xen/arch/x86/trace.c b/xen/arch/x86/trace.c
> index 4a953c5..411f798 100644
> --- a/xen/arch/x86/trace.c
> +++ b/xen/arch/x86/trace.c
> @@ -1,3 +1,4 @@
> +#ifdef CONFIG_TRACING
> #include <xen/init.h>
> #include <xen/kernel.h>
> #include <xen/lib.h>
> @@ -157,3 +158,25 @@ void __trace_ptwr_emulation(unsigned long addr, l1_pgentry_t npte)
> __trace_var(event, 1/*tsc*/, sizeof(d), &d);
> }
> }
> +#else /* !CONFIG_TRACING */
> +#include <xen/kernel.h>
> +#include <xen/trace.h>
> +
> +void __trace_pv_trap(int trapnr, unsigned long eip,
> + int use_error_code, unsigned error_code)
> +{
> +}
> +void __trace_pv_page_fault(unsigned long addr, unsigned error_code)
> +{
> +}
> +void __trace_trap_one_addr(unsigned event, unsigned long va)
> +{
> +}
> +void __trace_trap_two_addr(unsigned event, unsigned long va1,
> + unsigned long va2)
> +{
> +}
> +void __trace_ptwr_emulation(unsigned long addr, l1_pgentry_t npte)
> +{
> +}
These should all be static inlines in a header file. This will avoid
forcing the compiler to insert a bunch of empty functions and have to
call them, and it allows you to modify the x86 Makefile to be
andrewcoop@andrewcoop:/local/xen.git/xen$ git diff
diff --git a/xen/arch/x86/Makefile b/xen/arch/x86/Makefile
index 93ead6e..affeb22 100644
--- a/xen/arch/x86/Makefile
+++ b/xen/arch/x86/Makefile
@@ -59,7 +59,7 @@ obj-y += srat.o
obj-y += string.o
obj-y += sysctl.o
obj-y += time.o
-obj-y += trace.o
+obj-$(CONFIG_TRACE) += trace.o
obj-y += traps.o
obj-y += usercopy.o
obj-y += x86_emulate.o
If you are feeling super keen, it would be great to move trace.c into
the pv/ directory (as it is entirely PV specific), in line with cleanup
that myself and Wei are slowly doing.
If you are going to do the move, it would be better to split that out
into a separate patch ahead of introducing CONFIG_TRACE, and take the
opportunity to fix the trailing whitespace and style issues in the file.
> +#endif /* CONFIG_TRACING */
> diff --git a/xen/common/trace.c b/xen/common/trace.c
> index f29cd4c..2c18462 100644
> --- a/xen/common/trace.c
> +++ b/xen/common/trace.c
> @@ -48,6 +48,11 @@ static unsigned int opt_tevt_mask;
> integer_param("tbuf_size", opt_tbuf_size);
> integer_param("tevt_mask", opt_tevt_mask);
>
> +#ifdef CONFIG_TRACING
> +/* a flag recording whether initialization has been done */
> +/* or more properly, if the tbuf subsystem is enabled right now */
> +int tb_init_done __read_mostly;
> +
> /* Pointers to the meta-data objects for all system trace buffers */
> static struct t_info *t_info;
> static unsigned int t_info_pages;
> @@ -64,10 +69,6 @@ static u32 t_buf_highwater;
> static DEFINE_PER_CPU(unsigned long, lost_records);
> static DEFINE_PER_CPU(unsigned long, lost_records_first_tsc);
>
> -/* a flag recording whether initialization has been done */
> -/* or more properly, if the tbuf subsystem is enabled right now */
> -int tb_init_done __read_mostly;
> -
> /* which CPUs tracing is enabled on */
> static cpumask_t tb_cpu_mask;
>
> @@ -868,6 +869,36 @@ void __trace_hypercall(uint32_t event, unsigned long op,
>
> __trace_var(event, 1, sizeof(uint32_t) * (1 + (a - d.args)), &d);
> }
> +#else /* !CONFIG_TRACING */
Newlines here please.
> +void __init init_trace_bufs(void)
> +{
> + opt_tbuf_size = 0;
Why do you need this? I can't see it referenced anywhere outside of
CONFIG_TRACING.
> +}
> +
> +int tb_control(xen_sysctl_tbuf_op_t *tbc)
> +{
This also should be a static inline which return -EOPNOTSUPP.
Doing so would also allow common/trace.c to become obj-$(CONFIG_TRACE)
as well.
> + static DEFINE_SPINLOCK(lock);
> + int rc = 0;
> +
> + spin_lock(&lock);
> +
> + switch ( tbc->cmd )
> + {
> + case XEN_SYSCTL_TBUFOP_get_info:
> + tbc->evt_mask = 0;
> + tbc->buffer_mfn = 0;
> + tbc->size = 0;
> + break;
> + default:
> + rc = -ENOSYS;
> + break;
> + }
> +
> + spin_unlock(&lock);
> +
> + return rc;
> +}
> +#endif /* CONFIG_TRACING */
>
> /*
> * Local variables:
> diff --git a/xen/drivers/cpufreq/utility.c b/xen/drivers/cpufreq/utility.c
> index 53879fe..b686a9d 100644
> --- a/xen/drivers/cpufreq/utility.c
> +++ b/xen/drivers/cpufreq/utility.c
> @@ -362,11 +362,14 @@ int __cpufreq_driver_target(struct cpufreq_policy *policy,
>
> if (cpu_online(policy->cpu) && cpufreq_driver->target)
> {
> - unsigned int prev_freq = policy->cur;
> + uint32_t d[2] = { policy->cur, 0 };
>
> retval = cpufreq_driver->target(policy, target_freq, relation);
> if ( retval == 0 )
> - TRACE_2D(TRC_PM_FREQ_CHANGE, prev_freq/1000, policy->cur/1000);
> + {
> + d[1] = policy->cur/1000;
> + trace_var(TRC_PM_FREQ_CHANGE, 1, sizeof(d), d);
> + }
> }
Is this hunk supposed to live in this patch?
>
> return retval;
> diff --git a/xen/include/xen/trace.h b/xen/include/xen/trace.h
> index 12966ea..d1b6c70 100644
> --- a/xen/include/xen/trace.h
> +++ b/xen/include/xen/trace.h
> @@ -21,7 +21,11 @@
> #ifndef __XEN_TRACE_H__
> #define __XEN_TRACE_H__
>
> +#ifdef CONFIG_TRACING
> extern int tb_init_done;
> +#else
> +#define tb_init_done 0
> +#endif
>
> #include <public/sysctl.h>
> #include <public/trace.h>
> @@ -33,6 +37,7 @@ void init_trace_bufs(void);
> /* used to retrieve the physical address of the trace buffers */
> int tb_control(struct xen_sysctl_tbuf_op *tbc);
>
> +#ifdef CONFIG_TRACING
> int trace_will_trace_event(u32 event);
>
> void __trace_var(u32 event, bool_t cycles, unsigned int extra, const void *);
> @@ -113,7 +118,7 @@ void __trace_hypercall(uint32_t event, unsigned long op,
> } \
> } while ( 0 )
>
> -#define TRACE_6D(_e,d1,d2,d3,d4,d5,d6) \
> +#define TRACE_6D(_e,d1,d2,d3,d4,d5,d6) \
> do { \
> if ( unlikely(tb_init_done) ) \
> { \
> @@ -127,5 +132,28 @@ void __trace_hypercall(uint32_t event, unsigned long op,
> __trace_var(_e, 1, sizeof(_d), _d); \
> } \
> } while ( 0 )
> +#else /* !CONFIG_TRACING */
> +#define trace_will_trace_event(u) (0)
This needs to be a static inline, otherwise you get different behaviour
WRT evaluating its arguments.
trace_will_trace_event() also looks like it is in need of turning into a
bool function.
~Andrew
> +static inline void __trace_var(u32 event, bool_t cycles, unsigned int extra,
> + const void *extra_data)
> +{
> +}
> +static inline void trace_var(u32 event, int cycles, int extra,
> + const void *extra_data)
> +{
> +}
> +static inline void __trace_hypercall(uint32_t event, unsigned long op,
> + const xen_ulong_t *args)
> +{
> +}
> +
> +#define TRACE_0D(e) do {} while ( 0 )
> +#define TRACE_1D(e,d1) do {} while ( 0 )
> +#define TRACE_2D(e,d1,d2) do {} while ( 0 )
> +#define TRACE_3D(e,d1,d2,d3) do {} while ( 0 )
> +#define TRACE_4D(e,d1,d2,d3,d4) do {} while ( 0 )
> +#define TRACE_5D(e,d1,d2,d3,d4,d5) do {} while ( 0 )
> +#define TRACE_6D(e,d1,d2,d3,d4,d5,d6) do {} while ( 0 )
> +#endif /* CONFIG_TRACING */
>
> #endif /* __XEN_TRACE_H__ */
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 68+ messages in thread
* Re: [PATCH 06/15] xen: trace IRQ enabling/disabling
2017-06-01 17:34 ` [PATCH 06/15] xen: trace IRQ enabling/disabling Dario Faggioli
@ 2017-06-01 19:08 ` Andrew Cooper
2017-06-01 23:42 ` Dario Faggioli
2017-06-07 11:16 ` Julien Grall
` (2 subsequent siblings)
3 siblings, 1 reply; 68+ messages in thread
From: Andrew Cooper @ 2017-06-01 19:08 UTC (permalink / raw)
To: Dario Faggioli, xen-devel
Cc: George Dunlap, Julien Grall, Stefano Stabellini,
Jennifer Herbert, Jan Beulich
On 01/06/17 18:34, Dario Faggioli wrote:
> Trace when interrupts are disabled and (re)enabled.
> Basically, we replace the IRQ disabling and enabling
> functions with helpers that does the same, but also
> output the proper trace record.
>
> For putting in the record something that will let
> us identify _where_ in the code (i.e., in what function)
> the IRQ manipulation is happening, use either:
> - current_text_addr(),
> - or __builtin_return_address(0).
>
> In fact, depending on whether the disabling/enabling
> happens in macros (like for local_irq_disable() and
> local_irq_enable()) or in actual functions (like in
> spin_lock_irq*()), it is either:
> - the actual content of the instruction pointer when
> IRQ are disabled/enabled,
> - or the return address of the utility function where
> IRQ are disabled/enabled,
> that will tell us what it is the actual piece of code
> that is asking for the IRQ manipulation operation.
>
> Gate this with its specific Kconfig option, and keep
> it in disabled state by default (i.e., don't build it,
> if not explicitly specified), as the impact on
> performance may be non negligible.
>
> Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
This reminds me that I need to dust off my series which fixes
local_irq_save() to not capture its variables by name.
I.e. it would be used as "flags = local_irq_save();" in the normal C way.
I'm fairly sure I can also improve the generated assembly.
> ---
> Cc: George Dunlap <george.dunlap@eu.citrix.com>
> Cc: Stefano Stabellini <sstabellini@kernel.org>
> Cc: Julien Grall <julien.grall@arm.com>
> Cc: Jan Beulich <jbeulich@suse.com>
> Cc: Andrew Cooper <andrew.cooper3@citrix.com>
> Cc: Jennifer Herbert <jennifer.herbert@citrix.com>
> ---
> xen/Kconfig.debug | 11 ++++-
> xen/common/spinlock.c | 16 +++++--
> xen/common/trace.c | 10 +++-
> xen/include/asm-arm/arm32/system.h | 12 +++++
> xen/include/asm-arm/arm64/system.h | 12 +++++
> xen/include/asm-x86/system.h | 85 ++++++++++++++++++++++++++++++++++--
> xen/include/public/trace.h | 2 +
> xen/include/xen/rwlock.h | 33 +++++++++++---
> 8 files changed, 161 insertions(+), 20 deletions(-)
>
> diff --git a/xen/Kconfig.debug b/xen/Kconfig.debug
> index 374c1c0..81910c9 100644
> --- a/xen/Kconfig.debug
> +++ b/xen/Kconfig.debug
> @@ -98,7 +98,7 @@ config PERF_ARRAYS
> ---help---
> Enables software performance counter array histograms.
>
> -config TRACING
> +menuconfig TRACING
> bool "Tracing"
> default y
> ---help---
> @@ -106,6 +106,15 @@ config TRACING
> in per-CPU ring buffers. The 'xentrace' tool can be used to read
> the buffers and dump the content on the disk.
>
> +config TRACE_IRQSOFF
> + bool "Trace when IRQs are disabled and (re)enabled" if EXPERT = "y"
> + default n
> + depends on TRACING
> + ---help---
> + Makes it possible to generate events _every_ time IRQs are disabled
> + and (re)enabled, with also an indication of where that happened.
> + Note that this comes with high overead and produces huge amount of
> + tracing data.
You've got mixed tabs/spaces here.
>
> config VERBOSE_DEBUG
> bool "Verbose debug messages"
> diff --git a/xen/common/spinlock.c b/xen/common/spinlock.c
> index 2a06406..33b903e 100644
> --- a/xen/common/spinlock.c
> +++ b/xen/common/spinlock.c
> @@ -150,7 +150,9 @@ void _spin_lock(spinlock_t *lock)
> void _spin_lock_irq(spinlock_t *lock)
> {
> ASSERT(local_irq_is_enabled());
> - local_irq_disable();
> + _local_irq_disable();
> + if ( unlikely(tb_init_done) )
> + trace_irq_disable_ret();
> _spin_lock(lock);
> }
Is it sensible to do this way around?
By writing the trace record while interrupts are disabled, you do
prevent nesting in the general case (but not in NMIs/MCEs or the
irqsave() variants), but you increase the critical region while
interrupts are disabled.
Alternatively, writing the trace record outside of the critical region
would reduce the size of the region. Interpretation logic already needs
to cope with nesting, so is one extra level of nesting in this corner
case a problem?
Does the logic cope with the fact that interrupt gates automatically
disable interrupts?
~Andrew
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 02/15] xen: tracing: avoid checking tb_init_done multiple times.
2017-06-01 17:53 ` Andrew Cooper
@ 2017-06-01 23:08 ` Dario Faggioli
0 siblings, 0 replies; 68+ messages in thread
From: Dario Faggioli @ 2017-06-01 23:08 UTC (permalink / raw)
To: Andrew Cooper, xen-devel; +Cc: George Dunlap
[-- Attachment #1.1: Type: text/plain, Size: 2151 bytes --]
On Thu, 2017-06-01 at 18:53 +0100, Andrew Cooper wrote:
> On 01/06/17 18:33, Dario Faggioli wrote:
> > diff --git a/xen/common/trace.c b/xen/common/trace.c
> > index 4fedc26..f29cd4c 100644
> > --- a/xen/common/trace.c
> > +++ b/xen/common/trace.c
> > @@ -691,7 +691,8 @@ void __trace_var(u32 event, bool_t cycles,
> > unsigned int extra,
> > unsigned int extra_word;
> > bool_t started_below_highwater;
>
> You probably want an ASSERT(tb_init_done) here to keep callers in
> check.
>
Indeed! And in fact, I did have one. But only to discover that it
triggers! :-O
After looking at this better, I figured out that the existing check
(the one that this patch kills) is not just redundant, but only a best
effort measure.
In fact, we still haven't acquired any locks here. If I put the
ASSERT() and, between when I call __trace_var() and when the ASSERT()
is reached, someone manages to set tb_init_done=0, the ASSERT(), as
said, fires.
However, that is no better in current code. In fact, it is very well
possible that someone manages to set tb_init_done=0 *right after*, here
in __trace_var(), we check it with the if.
Actually, I think having the check there is misleading, because it
makes one think that tb_init_done is guaranteed to be and stay 0 below
it, while that isn't true at all. And this is therefore just another
reason to get rid of it, IMO. But, sadly, I can't replace it with an
ASSERT(). :-(
> Otherwise, Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>
>
Thanks, let me know if this still stands.
> >
> > - if( !tb_init_done )
> > + /* If the event is not interesting, bail, as early as possible
> > */
> > + if ( (tb_event_mask & event) == 0 )
> > return;
> >
> > /* Convert byte count into word count, rounding up */
>
Regards,
Dario
--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
[-- Attachment #2: Type: text/plain, Size: 127 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 03/15] xen/tools: tracing: several improvements on IRQs tracing
2017-06-01 18:02 ` Andrew Cooper
@ 2017-06-01 23:12 ` Dario Faggioli
0 siblings, 0 replies; 68+ messages in thread
From: Dario Faggioli @ 2017-06-01 23:12 UTC (permalink / raw)
To: Andrew Cooper, xen-devel; +Cc: George Dunlap, Wei Liu, Ian Jackson, Jan Beulich
[-- Attachment #1.1: Type: text/plain, Size: 2240 bytes --]
On Thu, 2017-06-01 at 19:02 +0100, Andrew Cooper wrote:
> On 01/06/17 18:33, Dario Faggioli wrote:
> > More specifically:
> > - the handling of the TRC_HW_IRQ_HANDLED is fixed, both in
> > xentrace_format and in xenalyze;
> > - simple events for recording when we enter and exit the
> > do_IRQ function, as well as when we deal with a guest
> > IRQ, are added;
> > - tracing of IRQs handled with direct vectors is also
> > added.
> >
> > With all the above, a trace will now look like this (using
> > xenalyze):
> >
> > 0.001299072 .x- d32768v5 irq_enter, irq 80000000
> > 0.001299072 .x- d32768v5 irq_direct, vec fa, handler =
> > 0xffff82d08026d7e4
>
> Please consistently use 0x$hex. vec in particular has 10/16ths of
> its
> values which are completely ambiguous between hex and decimal.
>
I fully agree. The file is not super consistent about this already, and
I was basically following the suit of the majority of existing cases.
But I really do agree, so I'll probably make a pre-patch, that converts
everyone to 0x$hex, and then use it in this patch as well.
> > + {
> > + if ( unlikely(tb_init_done) )
> > + {
> > + struct __packed {
> > + uint32_t vec;
> > + uint64_t handler;
> > + } d;
> > +
> > + d.vec = vector;
> > + d.handler = (uint64_t)direct_apic_vector[vector];
>
> As there is no code inbetween, you can do this as a straight
> initialisation of d.
>
> i.e.
>
> } d = { vector, (uint64_t)direct_apic_vector[vector] };
>
> Otherwise, Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>
>
Ok, thanks. I'll take care of this, and of all the others print format
and style comments.
Regards,
Dario
--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
[-- Attachment #2: Type: text/plain, Size: 127 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 06/15] xen: trace IRQ enabling/disabling
2017-06-01 19:08 ` Andrew Cooper
@ 2017-06-01 23:42 ` Dario Faggioli
2017-06-08 15:51 ` George Dunlap
2017-06-08 16:05 ` Jan Beulich
0 siblings, 2 replies; 68+ messages in thread
From: Dario Faggioli @ 2017-06-01 23:42 UTC (permalink / raw)
To: Andrew Cooper, xen-devel
Cc: George Dunlap, Julien Grall, Stefano Stabellini,
Jennifer Herbert, Jan Beulich
[-- Attachment #1.1: Type: text/plain, Size: 3327 bytes --]
On Thu, 2017-06-01 at 20:08 +0100, Andrew Cooper wrote:
> On 01/06/17 18:34, Dario Faggioli wrote:
> > diff --git a/xen/common/spinlock.c b/xen/common/spinlock.c
> > index 2a06406..33b903e 100644
> > --- a/xen/common/spinlock.c
> > +++ b/xen/common/spinlock.c
> > @@ -150,7 +150,9 @@ void _spin_lock(spinlock_t *lock)
> > void _spin_lock_irq(spinlock_t *lock)
> > {
> > ASSERT(local_irq_is_enabled());
> > - local_irq_disable();
> > + _local_irq_disable();
> > + if ( unlikely(tb_init_done) )
> > + trace_irq_disable_ret();
> > _spin_lock(lock);
> > }
>
> Is it sensible to do this way around?
>
Well, I guess either variant has up and down sides, and it looks to me
that there is no way to measure this, without interfering with the
behavior of the thing being measured ("Once upon a time, there was a
cat, who lived in a box. His name was Schrödinger..." :-D :-D :-D)
> By writing the trace record while interrupts are disabled, you do
> prevent nesting in the general case (but not in NMIs/MCEs or the
> irqsave() variants),
>
Forgive the ignorance, what's special about NMIs/MCAs that is relevant
for this? About _irqsave(), I'm also not sure what you mean, as
_irqsave() is already done differently than this.
> but you increase the critical region while
> interrupts are disabled.
>
I know.
> Alternatively, writing the trace record outside of the critical
> region
> would reduce the size of the region. Interpretation logic already
> needs
> to cope with nesting, so is one extra level of nesting in this corner
> case a problem?
>
TBH, I was indeed trying to offer to the component that will be looking
at and interpreting the data, the as clear as possible view on when
IRQs are _actually_ disabled and enabled. As in, if nesting occurs,
only the event corresponding to:
- the first local_irq_disable()
- the last local_irq_enable()
I.e., to not require that it (the interpretation logic) does understand
nesting.
But more than this, what I was concerned about was the accuracy of the
reporting itself. In fact, if I do as you suggest, I can be interrupted
(as interrupts are still on) after having called trace_irq_disable().
That means the report will show higher IRQ disabled time period, for
this instance, than what the reality is.
And the same is true on the enabling side, when I call
trace_irq_enable() _before_ re-enabling interrupts, which has the same
bad effect on the length of IRQ disabled section.
Maybe, considering that anything that will interrupt me in these cases,
are other interrupts, that will most likely disable interrupts
themselves, I should not worry too much about this... What do you
think?
> Does the logic cope with the fact that interrupt gates automatically
> disable interrupts?
>
Ah, right. No, it does not. I probably should mention this in the
changelog. Any ideas of how to deal with that? If yes, I'm more than
happy to fix this...
Thanks and Regards,
Dario
--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
[-- Attachment #2: Type: text/plain, Size: 127 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 05/15] xen: make it possible to disable tracing in Kconfig.
2017-06-01 18:43 ` Andrew Cooper
@ 2017-06-07 11:01 ` Julien Grall
0 siblings, 0 replies; 68+ messages in thread
From: Julien Grall @ 2017-06-07 11:01 UTC (permalink / raw)
To: Andrew Cooper, Dario Faggioli, xen-devel
Cc: Stefano Stabellini, Wei Liu, George Dunlap, Doug Goldstein,
Ian Jackson, Jan Beulich
Hi,
On 01/06/17 19:43, Andrew Cooper wrote:
> On 01/06/17 18:34, Dario Faggioli wrote:
> These should all be static inlines in a header file. This will avoid
> forcing the compiler to insert a bunch of empty functions and have to
> call them, and it allows you to modify the x86 Makefile to be
>
> andrewcoop@andrewcoop:/local/xen.git/xen$ git diff
> diff --git a/xen/arch/x86/Makefile b/xen/arch/x86/Makefile
> index 93ead6e..affeb22 100644
> --- a/xen/arch/x86/Makefile
> +++ b/xen/arch/x86/Makefile
> @@ -59,7 +59,7 @@ obj-y += srat.o
> obj-y += string.o
> obj-y += sysctl.o
> obj-y += time.o
> -obj-y += trace.o
> +obj-$(CONFIG_TRACE) += trace.o
> obj-y += traps.o
> obj-y += usercopy.o
> obj-y += x86_emulate.o
I would use similar things for common/trace.c too. This would let the
user remove all tracing code at build time.
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 06/15] xen: trace IRQ enabling/disabling
2017-06-01 17:34 ` [PATCH 06/15] xen: trace IRQ enabling/disabling Dario Faggioli
2017-06-01 19:08 ` Andrew Cooper
@ 2017-06-07 11:16 ` Julien Grall
2017-06-07 15:22 ` Dario Faggioli
2017-06-08 16:01 ` George Dunlap
2017-06-09 10:41 ` Jan Beulich
3 siblings, 1 reply; 68+ messages in thread
From: Julien Grall @ 2017-06-07 11:16 UTC (permalink / raw)
To: Dario Faggioli, xen-devel
Cc: George Dunlap, Andrew Cooper, Stefano Stabellini,
Jennifer Herbert, Jan Beulich
Hi Dario,
On 01/06/17 18:34, Dario Faggioli wrote:
> Trace when interrupts are disabled and (re)enabled.
> Basically, we replace the IRQ disabling and enabling
> functions with helpers that does the same, but also
> output the proper trace record.
>
> For putting in the record something that will let
> us identify _where_ in the code (i.e., in what function)
> the IRQ manipulation is happening, use either:
> - current_text_addr(),
> - or __builtin_return_address(0).
>
> In fact, depending on whether the disabling/enabling
> happens in macros (like for local_irq_disable() and
> local_irq_enable()) or in actual functions (like in
> spin_lock_irq*()), it is either:
> - the actual content of the instruction pointer when
> IRQ are disabled/enabled,
> - or the return address of the utility function where
> IRQ are disabled/enabled,
> that will tell us what it is the actual piece of code
> that is asking for the IRQ manipulation operation.
>
> Gate this with its specific Kconfig option, and keep
> it in disabled state by default (i.e., don't build it,
> if not explicitly specified), as the impact on
> performance may be non negligible.
>
> Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
> ---
> Cc: George Dunlap <george.dunlap@eu.citrix.com>
> Cc: Stefano Stabellini <sstabellini@kernel.org>
> Cc: Julien Grall <julien.grall@arm.com>
> Cc: Jan Beulich <jbeulich@suse.com>
> Cc: Andrew Cooper <andrew.cooper3@citrix.com>
> Cc: Jennifer Herbert <jennifer.herbert@citrix.com>
> ---
> xen/Kconfig.debug | 11 ++++-
> xen/common/spinlock.c | 16 +++++--
> xen/common/trace.c | 10 +++-
> xen/include/asm-arm/arm32/system.h | 12 +++++
> xen/include/asm-arm/arm64/system.h | 12 +++++
> xen/include/asm-x86/system.h | 85 ++++++++++++++++++++++++++++++++++--
> xen/include/public/trace.h | 2 +
> xen/include/xen/rwlock.h | 33 +++++++++++---
> 8 files changed, 161 insertions(+), 20 deletions(-)
>
> diff --git a/xen/Kconfig.debug b/xen/Kconfig.debug
> index 374c1c0..81910c9 100644
> --- a/xen/Kconfig.debug
> +++ b/xen/Kconfig.debug
> @@ -98,7 +98,7 @@ config PERF_ARRAYS
> ---help---
> Enables software performance counter array histograms.
>
> -config TRACING
> +menuconfig TRACING
> bool "Tracing"
> default y
> ---help---
> @@ -106,6 +106,15 @@ config TRACING
> in per-CPU ring buffers. The 'xentrace' tool can be used to read
> the buffers and dump the content on the disk.
>
> +config TRACE_IRQSOFF
> + bool "Trace when IRQs are disabled and (re)enabled" if EXPERT = "y"
> + default n
> + depends on TRACING
> + ---help---
> + Makes it possible to generate events _every_ time IRQs are disabled
> + and (re)enabled, with also an indication of where that happened.
> + Note that this comes with high overead and produces huge amount of
> + tracing data.
>
> config VERBOSE_DEBUG
> bool "Verbose debug messages"
> diff --git a/xen/common/spinlock.c b/xen/common/spinlock.c
> index 2a06406..33b903e 100644
> --- a/xen/common/spinlock.c
> +++ b/xen/common/spinlock.c
> @@ -150,7 +150,9 @@ void _spin_lock(spinlock_t *lock)
> void _spin_lock_irq(spinlock_t *lock)
> {
> ASSERT(local_irq_is_enabled());
> - local_irq_disable();
> + _local_irq_disable();
> + if ( unlikely(tb_init_done) )
__trace_var already contain a "if ( !tb_init_done )". It sounds
pointless to do it twice, and also I think it is not obvious to
understand the meaning of the check (i.e what is tb_init_done).
Would it be possible to hide this check and avoid check tb_init_done twice?
> diff --git a/xen/include/asm-arm/arm32/system.h b/xen/include/asm-arm/arm32/system.h
> index c617b40..20871ad 100644
> --- a/xen/include/asm-arm/arm32/system.h
> +++ b/xen/include/asm-arm/arm32/system.h
> @@ -4,6 +4,8 @@
>
> #include <asm/arm32/cmpxchg.h>
>
> +#include <xen/trace.h>
> +
> #define local_irq_disable() asm volatile ( "cpsid i @ local_irq_disable\n" : : : "cc" )
> #define local_irq_enable() asm volatile ( "cpsie i @ local_irq_enable\n" : : : "cc" )
>
> @@ -41,6 +43,16 @@ static inline int local_irq_is_enabled(void)
> #define local_abort_enable() __asm__("cpsie a @ __sta\n" : : : "memory", "cc")
> #define local_abort_disable() __asm__("cpsid a @ __sta\n" : : : "memory", "cc")
>
> +/* We do not support tracing (at all) yet */
I know that some bits are missing for ARM, but the patch #5 contradicts
this comment as you have CONFIG_TRACE=y by default.
> +#define trace_irq_disable_ret() do { } while ( 0 )
> +#define trace_irq_enable_ret() do { } while ( 0 )
> +#define trace_irq_save_ret(_x) do { } while ( 0 )
> +#define trace_irq_restore_ret(_x) do { } while ( 0 )
> +#define _local_irq_disable() local_irq_disable()
> +#define _local_irq_enable() local_irq_enable()
> +#define _local_irq_save(_x) local_irq_save(_x)
> +#define _local_irq_restore(_x) local_irq_restore(_x)
> +
This does not need to be duplicated in both asm-arm/arm{32,64}/system.h.
You could directly implement them in asm-arm/system.h.
Also, I would prefer if you stay consistent with x86. I.e non-underscore
versions are calling the underscore versions and not the invert.
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 00/15] xen/tools: add tracing to various Xen subsystems
2017-06-01 17:33 [PATCH 00/15] xen/tools: add tracing to various Xen subsystems Dario Faggioli
` (14 preceding siblings ...)
2017-06-01 17:35 ` [PATCH 15/15] tools: tracing: handle timers events in xentrace and xenalyze Dario Faggioli
@ 2017-06-07 14:13 ` Konrad Rzeszutek Wilk
2017-06-08 16:45 ` Dario Faggioli
2017-06-13 16:34 ` George Dunlap
16 siblings, 1 reply; 68+ messages in thread
From: Konrad Rzeszutek Wilk @ 2017-06-07 14:13 UTC (permalink / raw)
To: Dario Faggioli
Cc: Stefano Stabellini, Wei Liu, George Dunlap, Andrew Cooper,
Ian Jackson, Tim Deegan, Julien Grall, Jan Beulich,
Jennifer Herbert, xen-devel, Doug Goldstein
On Thu, Jun 01, 2017 at 07:33:33PM +0200, Dario Faggioli wrote:
> Hello,
>
> While chasing and dealing with bugs, over this last period, I've found myself
> augmenting Xen with quite a few new tracing capabilities, especially focusing
> on:
> - IRQ being disabled and (re)enabled (in addition to the already existing
> tracing of IRQ related activity that we have);
> - RCU;
> - softirqs (I think I sent a preliminary version of this, long ago, but can't
> be sure);
> - tasklets;
> - timers;
>
> And, apart from the first 4 patches (which are random, but still tracing
> related, of course, improvements), this is what this patch series does: it adds
> tracing to the Xen susystems listed above.
>
> That happens, one subsystem after another, in patches 6 to 15.
>
> Patch 5 deserves special mention. In fact, now that we have Kconfig, I thought
> it could be a nice thing to make it possible to select, at build config time,
> whether we want tracing or not, in the hypervisor (like, for instance, we do
> for performance counters).
Did you have thoughts on perhaps using asm goto as an
alterantive to unlikely?
In Linux it is called jump labels or such - the idea is that the
code has (by default and on x86) five NOP instructions. But you
can patch it over and add an call to the unlikely code.
But perhaps that is more of an future idea as looking at the Linux code
it looks quite large and not that simple.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 01/15] xen: in do_softirq() sample smp_processor_id() once and for all.
2017-06-01 17:33 ` [PATCH 01/15] xen: in do_softirq() sample smp_processor_id() once and for all Dario Faggioli
@ 2017-06-07 14:38 ` Jan Beulich
2017-06-08 14:12 ` George Dunlap
2017-06-08 14:20 ` George Dunlap
1 sibling, 1 reply; 68+ messages in thread
From: Jan Beulich @ 2017-06-07 14:38 UTC (permalink / raw)
To: Dario Faggioli
Cc: Stefano Stabellini, George Dunlap, Andrew Cooper, Tim Deegan,
Julien Grall, xen-devel
>>> On 01.06.17 at 19:33, <dario.faggioli@citrix.com> wrote:
> In fact, right now, we read it at every iteration of the loop.
> The reason it's done like this is how context switch was handled
> on IA64 (see commit ae9bfcdc, "[XEN] Various softirq cleanups" [1]).
>
> However:
> 1) we don't have IA64 any longer, and all the achitectures that
> we do support, are ok with sampling once and for all;
> 2) sampling at every iteration (slightly) affect performance;
> 3) sampling at every iteration is misleading, as it makes people
> believe that it is currently possible that SCHEDULE_SOFTIRQ
> moves the execution flow on another CPU (and the comment,
> by reinforcing this belief, makes things even worse!).
>
> Therefore, let's:
> - do the sampling once and for all, and remove the comment;
> - leave an ASSERT() around, so that, if context switching
> logic changes (in current or new arches), we will notice.
I'm not convinced. The comment clearly says "may", and I don't
think the performance overhead of smp_processor_id() is that
high. Keeping the code as is allows some future port to do
context switches like ia64 did, if that's more efficient there.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 02/15] xen: tracing: avoid checking tb_init_done multiple times.
2017-06-01 17:33 ` [PATCH 02/15] xen: tracing: avoid checking tb_init_done multiple times Dario Faggioli
2017-06-01 17:53 ` Andrew Cooper
@ 2017-06-07 14:46 ` Jan Beulich
2017-06-07 15:55 ` Dario Faggioli
2017-06-08 14:37 ` George Dunlap
2 siblings, 1 reply; 68+ messages in thread
From: Jan Beulich @ 2017-06-07 14:46 UTC (permalink / raw)
To: Dario Faggioli; +Cc: George Dunlap, xen-devel
>>> On 01.06.17 at 19:33, <dario.faggioli@citrix.com> wrote:
> In fact, when calling __trace_var() directly, we can
> assume that tb_init_done has been checked to be true,
> and the if is hence redundant.
The "assume" here worries me: What if there's a single place
somewhere that a grep can't easily find where no check is
present? Is it certain that ...
> @@ -691,7 +691,8 @@ void __trace_var(u32 event, bool_t cycles, unsigned int extra,
> unsigned int extra_word;
> bool_t started_below_highwater;
>
> - if( !tb_init_done )
> + /* If the event is not interesting, bail, as early as possible */
> + if ( (tb_event_mask & event) == 0 )
> return;
... this check would always be false then (i.e. tb_event_mask is
always zero) in that case?
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 03/15] xen/tools: tracing: several improvements on IRQs tracing
2017-06-01 17:33 ` [PATCH 03/15] xen/tools: tracing: several improvements on IRQs tracing Dario Faggioli
2017-06-01 18:02 ` Andrew Cooper
@ 2017-06-07 15:05 ` Jan Beulich
2017-06-07 15:45 ` Dario Faggioli
2017-06-08 14:59 ` George Dunlap
2 siblings, 1 reply; 68+ messages in thread
From: Jan Beulich @ 2017-06-07 15:05 UTC (permalink / raw)
To: Dario Faggioli
Cc: George Dunlap, Andrew Cooper, Wei Liu, Ian Jackson, xen-devel
>>> On 01.06.17 at 19:33, <dario.faggioli@citrix.com> wrote:
> More specifically:
> - the handling of the TRC_HW_IRQ_HANDLED is fixed, both in
> xentrace_format and in xenalyze;
> - simple events for recording when we enter and exit the
> do_IRQ function, as well as when we deal with a guest
> IRQ, are added;
On x86. What about ARM?
> - tracing of IRQs handled with direct vectors is also
> added.
>
> With all the above, a trace will now look like this (using
> xenalyze):
>
> 0.001299072 .x- d32768v5 irq_enter, irq 80000000
> 0.001299072 .x- d32768v5 irq_direct, vec fa, handler = 0xffff82d08026d7e4
> 0.001300014 .x- d32768v5 raise_softirq nr 0
> 0.001300487 .x- d32768v5 irq_exit irq 80000000, in_irq = 0
The IRQ number looks bogus here, and I'm not convinced giving
a bogus example in a commit message is a good idea. I realize
this is presumably a result of vector_irq[] being initialized to
INT_MIN, but I would say the trace points would then better be
moved so that no bogus data is being recorded.
> @@ -884,9 +919,13 @@ void do_IRQ(struct cpu_user_regs *regs)
> desc->rl_quantum_start = now;
> }
>
> - tsc_in = tb_init_done ? get_cycles() : 0;
> + if ( unlikely(tb_init_done) )
> + {
> + __trace_var(TRC_HW_IRQ_GUEST, 0, sizeof(irq), &irq);
> + tsc_in = get_cycles();
> + }
> __do_IRQ_guest(irq);
> - TRACE_3D(TRC_HW_IRQ_HANDLED, irq, tsc_in, get_cycles());
> + trace_irq_handled(irq, get_cycles() - tsc_in);
> goto out_no_end;
> }
Before this change, the get_cycles() invocation was after
the tb_init_done check. Now you move it ahead of it (as
the function arguments are evaluated before executing the
function body) - are you sure all compiler versions will suitably
re-order this?
Additionally I'm afraid this will trigger compiler warnings on at
least some compiler versions, as tsc_in is now possibly
uninitialized (and there's no clear way to disprove this for the
compiler, again because function arguments are being
evaluated before the function body is reached).
As to get_cycles() itself - is the relatively imprecise time
stamp it produces really good enough for tracing? I notice
that there are only very few other users of that function.
> @@ -922,6 +962,7 @@ void do_IRQ(struct cpu_user_regs *regs)
> spin_unlock(&desc->lock);
> out_no_unlock:
> irq_exit();
> + TRACE_3D(TRC_HW_IRQ_EXIT, irq, desc == NULL ? -1 : desc->status, in_irq());
The ordering of irq_{enter,exit}() and TRACE_{1,3}D() may be
preferable from a trace quality pov, but as far as the system is
concerned you're adding code which runs in interrupt context
without being aware of that. This may be a latent issue.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 05/15] xen: make it possible to disable tracing in Kconfig.
2017-06-01 17:34 ` [PATCH 05/15] xen: make it possible to disable tracing in Kconfig Dario Faggioli
2017-06-01 18:43 ` Andrew Cooper
@ 2017-06-07 15:14 ` Jan Beulich
2017-06-08 15:16 ` George Dunlap
2017-06-08 15:17 ` George Dunlap
2 siblings, 1 reply; 68+ messages in thread
From: Jan Beulich @ 2017-06-07 15:14 UTC (permalink / raw)
To: Dario Faggioli
Cc: Stefano Stabellini, Wei Liu, George Dunlap, Andrew Cooper,
Ian Jackson, Julien Grall, xen-devel, Doug Goldstein
>>> On 01.06.17 at 19:34, <dario.faggioli@citrix.com> wrote:
> --- a/xen/Kconfig.debug
> +++ b/xen/Kconfig.debug
> @@ -98,6 +98,14 @@ config PERF_ARRAYS
> ---help---
> Enables software performance counter array histograms.
>
> +config TRACING
> + bool "Tracing"
> + default y
default DEBUG (you don't want to suggest turning it on for a
release build)
> --- a/xen/common/trace.c
> +++ b/xen/common/trace.c
> @@ -48,6 +48,11 @@ static unsigned int opt_tevt_mask;
> integer_param("tbuf_size", opt_tbuf_size);
> integer_param("tevt_mask", opt_tevt_mask);
>
> +#ifdef CONFIG_TRACING
> +/* a flag recording whether initialization has been done */
> +/* or more properly, if the tbuf subsystem is enabled right now */
> +int tb_init_done __read_mostly;
Switch to bool at once? And in any event correct the comment
style please.
> --- a/xen/drivers/cpufreq/utility.c
> +++ b/xen/drivers/cpufreq/utility.c
> @@ -362,11 +362,14 @@ int __cpufreq_driver_target(struct cpufreq_policy *policy,
>
> if (cpu_online(policy->cpu) && cpufreq_driver->target)
> {
> - unsigned int prev_freq = policy->cur;
> + uint32_t d[2] = { policy->cur, 0 };
Notwithstanding Andrew's question about the placement of this
hunk the ", 0" seems pointless to me.
> @@ -33,6 +37,7 @@ void init_trace_bufs(void);
> /* used to retrieve the physical address of the trace buffers */
> int tb_control(struct xen_sysctl_tbuf_op *tbc);
>
> +#ifdef CONFIG_TRACING
> int trace_will_trace_event(u32 event);
Please have a blank line between these, two of them around the
matching #else, and one ahead of the #endif.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 06/15] xen: trace IRQ enabling/disabling
2017-06-07 11:16 ` Julien Grall
@ 2017-06-07 15:22 ` Dario Faggioli
2017-06-09 10:51 ` Julien Grall
0 siblings, 1 reply; 68+ messages in thread
From: Dario Faggioli @ 2017-06-07 15:22 UTC (permalink / raw)
To: Julien Grall, xen-devel
Cc: George Dunlap, Andrew Cooper, Stefano Stabellini,
Jennifer Herbert, Jan Beulich
[-- Attachment #1.1: Type: text/plain, Size: 4018 bytes --]
On Wed, 2017-06-07 at 12:16 +0100, Julien Grall wrote:
> Hi Dario,
>
Hi,
> On 01/06/17 18:34, Dario Faggioli wrote:
> > diff --git a/xen/common/spinlock.c b/xen/common/spinlock.c
> > index 2a06406..33b903e 100644
> > --- a/xen/common/spinlock.c
> > +++ b/xen/common/spinlock.c
> > @@ -150,7 +150,9 @@ void _spin_lock(spinlock_t *lock)
> > void _spin_lock_irq(spinlock_t *lock)
> > {
> > ASSERT(local_irq_is_enabled());
> > - local_irq_disable();
> > + _local_irq_disable();
> > + if ( unlikely(tb_init_done) )
>
> __trace_var already contain a "if ( !tb_init_done )". It sounds
> pointless to do it twice, and also I think it is not obvious to
> understand the meaning of the check (i.e what is tb_init_done).
>
> Would it be possible to hide this check and avoid check tb_init_done
> twice?
>
I totally agree. And in fact, in another patch, I remove the
tb_init_done check in __trace_var(). Reason behind this is that all the
callers of __trace_var() (the main one being trace_var()), already
checks for tb_init_done themselves.
In fact, the explicit check in the caller, is the (only) basis on which
one decides to call either trace_var() or __trace_var().
This here is one of the call sites where I think the check is better
done in the caller.
> > diff --git a/xen/include/asm-arm/arm32/system.h b/xen/include/asm-
> > arm/arm32/system.h
> > index c617b40..20871ad 100644
> > --- a/xen/include/asm-arm/arm32/system.h
> > +++ b/xen/include/asm-arm/arm32/system.h
> > @@ -4,6 +4,8 @@
> >
> > #include <asm/arm32/cmpxchg.h>
> >
> > +#include <xen/trace.h>
> > +
> > #define local_irq_disable() asm volatile ( "cpsid i @
> > local_irq_disable\n" : : : "cc" )
> > #define local_irq_enable() asm volatile ( "cpsie i @
> > local_irq_enable\n" : : : "cc" )
> >
> > @@ -41,6 +43,16 @@ static inline int local_irq_is_enabled(void)
> > #define local_abort_enable() __asm__("cpsie a @ __sta\n" : : :
> > "memory", "cc")
> > #define local_abort_disable() __asm__("cpsid a @ __sta\n" : : :
> > "memory", "cc")
> >
> > +/* We do not support tracing (at all) yet */
>
> I know that some bits are missing for ARM, but the patch #5
> contradicts
> this comment as you have CONFIG_TRACE=y by default.
>
No sure what you mean. Tracing is de facto on by default right now,
despite it not being implemented for ARM. So what I'm doing is
basically keeping things as they are.
If you think it should be off by default, well, let's talk about it...
but I'm not sure this is really what you are saying/asking.
> > +#define trace_irq_disable_ret() do { } while ( 0 )
> > +#define trace_irq_enable_ret() do { } while ( 0 )
> > +#define trace_irq_save_ret(_x) do { } while ( 0 )
> > +#define trace_irq_restore_ret(_x) do { } while ( 0 )
> > +#define _local_irq_disable() local_irq_disable()
> > +#define _local_irq_enable() local_irq_enable()
> > +#define _local_irq_save(_x) local_irq_save(_x)
> > +#define _local_irq_restore(_x) local_irq_restore(_x)
> > +
>
> This does not need to be duplicated in both asm-
> arm/arm{32,64}/system.h.
> You could directly implement them in asm-arm/system.h.
>
Ok, thanks.
> Also, I would prefer if you stay consistent with x86. I.e non-
> underscore
> versions are calling the underscore versions and not the invert.
>
Well, I know it is counterintuitive, but it's the easiest way of
getting over this, i.e., without the need of a lot of stubs, and
touching as few existing code as possible.
But I certainly can give it a try doing it as you say, and see how it
ends up looking like.
Regards,
Dario
--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
[-- Attachment #2: Type: text/plain, Size: 127 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 03/15] xen/tools: tracing: several improvements on IRQs tracing
2017-06-07 15:05 ` Jan Beulich
@ 2017-06-07 15:45 ` Dario Faggioli
2017-06-07 15:58 ` Jan Beulich
0 siblings, 1 reply; 68+ messages in thread
From: Dario Faggioli @ 2017-06-07 15:45 UTC (permalink / raw)
To: Jan Beulich; +Cc: George Dunlap, Andrew Cooper, Wei Liu, Ian Jackson, xen-devel
[-- Attachment #1.1: Type: text/plain, Size: 5014 bytes --]
On Wed, 2017-06-07 at 09:05 -0600, Jan Beulich wrote:
> > > > On 01.06.17 at 19:33, <dario.faggioli@citrix.com> wrote:
> >
> > More specifically:
> > - the handling of the TRC_HW_IRQ_HANDLED is fixed, both in
> > xentrace_format and in xenalyze;
> > - simple events for recording when we enter and exit the
> > do_IRQ function, as well as when we deal with a guest
> > IRQ, are added;
>
> On x86. What about ARM?
>
We don't have tracing at all on ARM. I guess I can mention this in the
commit message (and, perhaps, even add an 'x86' tag to the subject).
> > - tracing of IRQs handled with direct vectors is also
> > added.
> >
> > With all the above, a trace will now look like this (using
> > xenalyze):
> >
> > 0.001299072 .x- d32768v5 irq_enter, irq 80000000
> > 0.001299072 .x- d32768v5 irq_direct, vec fa, handler =
> > 0xffff82d08026d7e4
> > 0.001300014 .x- d32768v5 raise_softirq nr 0
> > 0.001300487 .x- d32768v5 irq_exit irq 80000000, in_irq = 0
>
> The IRQ number looks bogus here, and I'm not convinced giving
> a bogus example in a commit message is a good idea. I realize
> this is presumably a result of vector_irq[] being initialized to
> INT_MIN, but I would say the trace points would then better be
> moved so that no bogus data is being recorded.
>
Ok, I'll have a look at how to achieve that.
> > @@ -884,9 +919,13 @@ void do_IRQ(struct cpu_user_regs *regs)
> > desc->rl_quantum_start = now;
> > }
> >
> > - tsc_in = tb_init_done ? get_cycles() : 0;
> > + if ( unlikely(tb_init_done) )
> > + {
> > + __trace_var(TRC_HW_IRQ_GUEST, 0, sizeof(irq), &irq);
> > + tsc_in = get_cycles();
> > + }
> > __do_IRQ_guest(irq);
> > - TRACE_3D(TRC_HW_IRQ_HANDLED, irq, tsc_in, get_cycles());
> > + trace_irq_handled(irq, get_cycles() - tsc_in);
> > goto out_no_end;
> > }
>
> Before this change, the get_cycles() invocation was after
> the tb_init_done check. Now you move it ahead of it (as
> the function arguments are evaluated before executing the
> function body) - are you sure all compiler versions will suitably
> re-order this?
>
> Additionally I'm afraid this will trigger compiler warnings on at
> least some compiler versions, as tsc_in is now possibly
> uninitialized (and there's no clear way to disprove this for the
> compiler, again because function arguments are being
> evaluated before the function body is reached).
>
I understand and (now that I see it) agree with your remark on when
get_cycles() is called. I'll reorganize things so that the patched
behavior matches the existing one.
OTOH, I'm not sure I see the "potentially uninitialized" issue for
tsc_in, but since I'm reworking the code, I'll keep that in mind too.
> As to get_cycles() itself - is the relatively imprecise time
> stamp it produces really good enough for tracing? I notice
> that there are only very few other users of that function.
>
Yes, that's also something I was wondering. It's what is being used
currently, so I thought it was like that for a reason, and that it
wasn't this patch job to change that.
But if it's judged to be fine to turn this into NOW() (done either
here, or in a dedicated patch)., I'm all for it.
> > @@ -922,6 +962,7 @@ void do_IRQ(struct cpu_user_regs *regs)
> > spin_unlock(&desc->lock);
> > out_no_unlock:
> > irq_exit();
> > + TRACE_3D(TRC_HW_IRQ_EXIT, irq, desc == NULL ? -1 : desc-
> > >status, in_irq());
>
> The ordering of irq_{enter,exit}() and TRACE_{1,3}D() may be
> preferable from a trace quality pov, but as far as the system is
> concerned you're adding code which runs in interrupt context
> without being aware of that. This may be a latent issue.
>
Sorry, I don't understand your concern(s). What is it that I am not
aware of, and what is it that could be a latent issue?
About the position of those tracepoints: the nice thing about IRQ_EXIT
following irq_exit() is that I can record the result of in_irq(), which
will tell whether or not we're dealing with a nested interrupt. I don't
do the same for IRQ_ENTER, but, not that I think about it, I guess I
could.
However, if what you're saying is that they need to stay within the
irq_enter()-irq_exit() section, I can certainly make that happen. The
trace needs to be interpreted knowing where the tracepoints are anyway
(and that's true for each and every event already), so no big deal,
really.
Thanks and Regards,
Dario
--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
[-- Attachment #2: Type: text/plain, Size: 127 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 02/15] xen: tracing: avoid checking tb_init_done multiple times.
2017-06-07 14:46 ` Jan Beulich
@ 2017-06-07 15:55 ` Dario Faggioli
2017-06-07 16:06 ` Jan Beulich
0 siblings, 1 reply; 68+ messages in thread
From: Dario Faggioli @ 2017-06-07 15:55 UTC (permalink / raw)
To: Jan Beulich; +Cc: George Dunlap, xen-devel
[-- Attachment #1.1: Type: text/plain, Size: 1864 bytes --]
On Wed, 2017-06-07 at 08:46 -0600, Jan Beulich wrote:
> > > > On 01.06.17 at 19:33, <dario.faggioli@citrix.com> wrote:
> >
> > In fact, when calling __trace_var() directly, we can
> > assume that tb_init_done has been checked to be true,
> > and the if is hence redundant.
>
> The "assume" here worries me: What if there's a single place
> somewhere that a grep can't easily find where no check is
> present? Is it certain that ...
>
> > @@ -691,7 +691,8 @@ void __trace_var(u32 event, bool_t cycles,
> > unsigned int extra,
> > unsigned int extra_word;
> > bool_t started_below_highwater;
> >
> > - if( !tb_init_done )
> > + /* If the event is not interesting, bail, as early as possible
> > */
> > + if ( (tb_event_mask & event) == 0 )
> > return;
>
> ... this check would always be false then (i.e. tb_event_mask is
> always zero) in that case?
>
As said when replying to Andrew, I originally put an ASSERT() there.
That made me realize, though, that the existing if(!tb_init_done) is
ineffective and potentially racy (or, if you want to be kind "it's a
best effort kind of measure") already.
In fact, even right now, without my patches, we don't hold the tracing
lock when we execute that if. Nothing prevents tb_init_done to become 0
_right_after_ we saw it being 1 and decide to go ahead.
This, to me, looks like an even more compelling reason to remove it.
But I think I can improve the commit message so that it explains this
thing that I just said above too.
Thanks and Regards,
Dario
--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
[-- Attachment #2: Type: text/plain, Size: 127 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 03/15] xen/tools: tracing: several improvements on IRQs tracing
2017-06-07 15:45 ` Dario Faggioli
@ 2017-06-07 15:58 ` Jan Beulich
2017-06-08 14:53 ` George Dunlap
0 siblings, 1 reply; 68+ messages in thread
From: Jan Beulich @ 2017-06-07 15:58 UTC (permalink / raw)
To: Dario Faggioli
Cc: George Dunlap, Andrew Cooper, Wei Liu, Ian Jackson, xen-devel
>>> On 07.06.17 at 17:45, <dario.faggioli@citrix.com> wrote:
> On Wed, 2017-06-07 at 09:05 -0600, Jan Beulich wrote:
>> > > > On 01.06.17 at 19:33, <dario.faggioli@citrix.com> wrote:
>> > @@ -884,9 +919,13 @@ void do_IRQ(struct cpu_user_regs *regs)
>> > desc->rl_quantum_start = now;
>> > }
>> >
>> > - tsc_in = tb_init_done ? get_cycles() : 0;
>> > + if ( unlikely(tb_init_done) )
>> > + {
>> > + __trace_var(TRC_HW_IRQ_GUEST, 0, sizeof(irq), &irq);
>> > + tsc_in = get_cycles();
>> > + }
>> > __do_IRQ_guest(irq);
>> > - TRACE_3D(TRC_HW_IRQ_HANDLED, irq, tsc_in, get_cycles());
>> > + trace_irq_handled(irq, get_cycles() - tsc_in);
>> > goto out_no_end;
>> > }
>>
>> Before this change, the get_cycles() invocation was after
>> the tb_init_done check. Now you move it ahead of it (as
>> the function arguments are evaluated before executing the
>> function body) - are you sure all compiler versions will suitably
>> re-order this?
>>
>> Additionally I'm afraid this will trigger compiler warnings on at
>> least some compiler versions, as tsc_in is now possibly
>> uninitialized (and there's no clear way to disprove this for the
>> compiler, again because function arguments are being
>> evaluated before the function body is reached).
>>
> I understand and (now that I see it) agree with your remark on when
> get_cycles() is called. I'll reorganize things so that the patched
> behavior matches the existing one.
>
> OTOH, I'm not sure I see the "potentially uninitialized" issue for
> tsc_in, but since I'm reworking the code, I'll keep that in mind too.
You initialize tsc_in only when tb_init_done is set, but you use
it without that conditional. And even if you used it under that
same conditional, older compiler versions aren't able to track
that fact (same conditional) and raise a warning anyway.
>> As to get_cycles() itself - is the relatively imprecise time
>> stamp it produces really good enough for tracing? I notice
>> that there are only very few other users of that function.
>>
> Yes, that's also something I was wondering. It's what is being used
> currently, so I thought it was like that for a reason, and that it
> wasn't this patch job to change that.
>
> But if it's judged to be fine to turn this into NOW() (done either
> here, or in a dedicated patch)., I'm all for it.
Right, I didn't necessarily mean for this to be done here, I merely
wanted to raise the fact (remembering that for the other time
handling code using "synced" TSC reads was quite a bit of an
accuracy improvement).
>> > @@ -922,6 +962,7 @@ void do_IRQ(struct cpu_user_regs *regs)
>> > spin_unlock(&desc->lock);
>> > out_no_unlock:
>> > irq_exit();
>> > + TRACE_3D(TRC_HW_IRQ_EXIT, irq, desc == NULL ? -1 : desc-
>> > >status, in_irq());
>>
>> The ordering of irq_{enter,exit}() and TRACE_{1,3}D() may be
>> preferable from a trace quality pov, but as far as the system is
>> concerned you're adding code which runs in interrupt context
>> without being aware of that. This may be a latent issue.
>>
> Sorry, I don't understand your concern(s). What is it that I am not
> aware of, and what is it that could be a latent issue?
>
> About the position of those tracepoints: the nice thing about IRQ_EXIT
> following irq_exit() is that I can record the result of in_irq(), which
> will tell whether or not we're dealing with a nested interrupt. I don't
> do the same for IRQ_ENTER, but, not that I think about it, I guess I
> could.
>
> However, if what you're saying is that they need to stay within the
> irq_enter()-irq_exit() section, I can certainly make that happen. The
> trace needs to be interpreted knowing where the tracepoints are anyway
> (and that's true for each and every event already), so no big deal,
> really.
Yes, it's really just this latter fact - it would feel more safe from
an overall system pov if you had
irq_enter();
TRACE_1D();
...
TRACE_3D();
irq_exit();
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 02/15] xen: tracing: avoid checking tb_init_done multiple times.
2017-06-07 15:55 ` Dario Faggioli
@ 2017-06-07 16:06 ` Jan Beulich
2017-06-08 14:34 ` George Dunlap
0 siblings, 1 reply; 68+ messages in thread
From: Jan Beulich @ 2017-06-07 16:06 UTC (permalink / raw)
To: Dario Faggioli; +Cc: George Dunlap, xen-devel
>>> On 07.06.17 at 17:55, <dario.faggioli@citrix.com> wrote:
> On Wed, 2017-06-07 at 08:46 -0600, Jan Beulich wrote:
>> > > > On 01.06.17 at 19:33, <dario.faggioli@citrix.com> wrote:
>> >
>> > In fact, when calling __trace_var() directly, we can
>> > assume that tb_init_done has been checked to be true,
>> > and the if is hence redundant.
>>
>> The "assume" here worries me: What if there's a single place
>> somewhere that a grep can't easily find where no check is
>> present? Is it certain that ...
>>
>> > @@ -691,7 +691,8 @@ void __trace_var(u32 event, bool_t cycles,
>> > unsigned int extra,
>> > unsigned int extra_word;
>> > bool_t started_below_highwater;
>> >
>> > - if( !tb_init_done )
>> > + /* If the event is not interesting, bail, as early as possible
>> > */
>> > + if ( (tb_event_mask & event) == 0 )
>> > return;
>>
>> ... this check would always be false then (i.e. tb_event_mask is
>> always zero) in that case?
>>
> As said when replying to Andrew, I originally put an ASSERT() there.
>
> That made me realize, though, that the existing if(!tb_init_done) is
> ineffective and potentially racy (or, if you want to be kind "it's a
> best effort kind of measure") already.
>
> In fact, even right now, without my patches, we don't hold the tracing
> lock when we execute that if. Nothing prevents tb_init_done to become 0
> _right_after_ we saw it being 1 and decide to go ahead.
>
> This, to me, looks like an even more compelling reason to remove it.
> But I think I can improve the commit message so that it explains this
> thing that I just said above too.
Well, my question wasn't about a possible race (as the code would
need to be able to deal with that no matter what change you do
here), but about the case where tb_init_done has never been set.
Would tb_event_mask be reliably zero in that case, no matter
what other operations may have been performed?
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 01/15] xen: in do_softirq() sample smp_processor_id() once and for all.
2017-06-07 14:38 ` Jan Beulich
@ 2017-06-08 14:12 ` George Dunlap
0 siblings, 0 replies; 68+ messages in thread
From: George Dunlap @ 2017-06-08 14:12 UTC (permalink / raw)
To: Jan Beulich, Dario Faggioli
Cc: Stefano Stabellini, George Dunlap, Andrew Cooper, Tim Deegan,
Julien Grall, xen-devel
On 07/06/17 15:38, Jan Beulich wrote:
>>>> On 01.06.17 at 19:33, <dario.faggioli@citrix.com> wrote:
>> In fact, right now, we read it at every iteration of the loop.
>> The reason it's done like this is how context switch was handled
>> on IA64 (see commit ae9bfcdc, "[XEN] Various softirq cleanups" [1]).
>>
>> However:
>> 1) we don't have IA64 any longer, and all the achitectures that
>> we do support, are ok with sampling once and for all;
>> 2) sampling at every iteration (slightly) affect performance;
>> 3) sampling at every iteration is misleading, as it makes people
>> believe that it is currently possible that SCHEDULE_SOFTIRQ
>> moves the execution flow on another CPU (and the comment,
>> by reinforcing this belief, makes things even worse!).
>>
>> Therefore, let's:
>> - do the sampling once and for all, and remove the comment;
>> - leave an ASSERT() around, so that, if context switching
>> logic changes (in current or new arches), we will notice.
>
> I'm not convinced. The comment clearly says "may", and I don't
> think the performance overhead of smp_processor_id() is that
> high. Keeping the code as is allows some future port to do
> context switches like ia64 did, if that's more efficient there.
In English, "may" not only means "has a possibility of not happening",
but also means "has a possibility of happening". That's false in the
hypervisor code the way it's currently written. If I said, "Bring a
water pistol to the XenSummit because Jan may burst into flames" I think
you'd say I was implying something false. :-)
I think it would be better to remove it. There are doubtless *lots* of
places in the code now that implicitly rely on context_switch() never
returning. The ASSERT() will catch this issue quickly enough if we ever
re-introduce that behavior.
On the other hand, unless there's a noticeable performance overhead,
this isn't a big issue.
-George
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 01/15] xen: in do_softirq() sample smp_processor_id() once and for all.
2017-06-01 17:33 ` [PATCH 01/15] xen: in do_softirq() sample smp_processor_id() once and for all Dario Faggioli
2017-06-07 14:38 ` Jan Beulich
@ 2017-06-08 14:20 ` George Dunlap
2017-06-08 14:42 ` Jan Beulich
1 sibling, 1 reply; 68+ messages in thread
From: George Dunlap @ 2017-06-08 14:20 UTC (permalink / raw)
To: Dario Faggioli, xen-devel
Cc: Stefano Stabellini, George Dunlap, Andrew Cooper, Tim Deegan,
Julien Grall, Jan Beulich
On 01/06/17 18:33, Dario Faggioli wrote:
> In fact, right now, we read it at every iteration of the loop.
> The reason it's done like this is how context switch was handled
> on IA64 (see commit ae9bfcdc, "[XEN] Various softirq cleanups" [1]).
>
> However:
> 1) we don't have IA64 any longer, and all the achitectures that
> we do support, are ok with sampling once and for all;
> 2) sampling at every iteration (slightly) affect performance;
> 3) sampling at every iteration is misleading, as it makes people
> believe that it is currently possible that SCHEDULE_SOFTIRQ
> moves the execution flow on another CPU (and the comment,
> by reinforcing this belief, makes things even worse!).
>
> Therefore, let's:
> - do the sampling once and for all, and remove the comment;
"Once and for all" has a much stronger sense of finality than you're
using here: reading smp_processor_id() in that function "once and for
all" would mean you would never have to read it on that function, on
that particular core, ever again, until the end of the universe. :-)
I think I'd say, "only once". The scope of the "only" in that case is
"this function call", not "all calls forever".
With that changed:
Reviewed-by: George Dunlap <george.dunlap@citrix.com>
(Although we till need Jan's acquiescence.)
> - leave an ASSERT() around, so that, if context switching
> logic changes (in current or new arches), we will notice.
>
> [1] Some more (historical) information here:
> http://old-list-archives.xenproject.org/archives/html/xen-devel/2006-06/msg01262.html
>
> Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
> ---
> Cc: Andrew Cooper <andrew.cooper3@citrix.com>
> Cc: George Dunlap <George.Dunlap@eu.citrix.com>
> Cc: Jan Beulich <jbeulich@suse.com>
> Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> Cc: Stefano Stabellini <sstabellini@kernel.org>
> Cc: Julien Grall <julien.grall@arm.com>
> Cc: Tim Deegan <tim@xen.org>
> ---
> xen/common/softirq.c | 8 ++------
> 1 file changed, 2 insertions(+), 6 deletions(-)
>
> diff --git a/xen/common/softirq.c b/xen/common/softirq.c
> index ac12cf8..67c84ba 100644
> --- a/xen/common/softirq.c
> +++ b/xen/common/softirq.c
> @@ -27,16 +27,12 @@ static DEFINE_PER_CPU(unsigned int, batching);
>
> static void __do_softirq(unsigned long ignore_mask)
> {
> - unsigned int i, cpu;
> + unsigned int i, cpu = smp_processor_id();
> unsigned long pending;
>
> for ( ; ; )
> {
> - /*
> - * Initialise @cpu on every iteration: SCHEDULE_SOFTIRQ may move
> - * us to another processor.
> - */
> - cpu = smp_processor_id();
> + ASSERT(cpu == smp_processor_id());
>
> if ( rcu_pending(cpu) )
> rcu_check_callbacks(cpu);
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 02/15] xen: tracing: avoid checking tb_init_done multiple times.
2017-06-07 16:06 ` Jan Beulich
@ 2017-06-08 14:34 ` George Dunlap
0 siblings, 0 replies; 68+ messages in thread
From: George Dunlap @ 2017-06-08 14:34 UTC (permalink / raw)
To: Jan Beulich, Dario Faggioli; +Cc: George Dunlap, xen-devel
On 07/06/17 17:06, Jan Beulich wrote:
>>>> On 07.06.17 at 17:55, <dario.faggioli@citrix.com> wrote:
>> On Wed, 2017-06-07 at 08:46 -0600, Jan Beulich wrote:
>>>>>> On 01.06.17 at 19:33, <dario.faggioli@citrix.com> wrote:
>>>>
>>>> In fact, when calling __trace_var() directly, we can
>>>> assume that tb_init_done has been checked to be true,
>>>> and the if is hence redundant.
>>>
>>> The "assume" here worries me: What if there's a single place
>>> somewhere that a grep can't easily find where no check is
>>> present? Is it certain that ...
>>>
>>>> @@ -691,7 +691,8 @@ void __trace_var(u32 event, bool_t cycles,
>>>> unsigned int extra,
>>>> unsigned int extra_word;
>>>> bool_t started_below_highwater;
>>>>
>>>> - if( !tb_init_done )
>>>> + /* If the event is not interesting, bail, as early as possible
>>>> */
>>>> + if ( (tb_event_mask & event) == 0 )
>>>> return;
>>>
>>> ... this check would always be false then (i.e. tb_event_mask is
>>> always zero) in that case?
>>>
>> As said when replying to Andrew, I originally put an ASSERT() there.
>>
>> That made me realize, though, that the existing if(!tb_init_done) is
>> ineffective and potentially racy (or, if you want to be kind "it's a
>> best effort kind of measure") already.
>>
>> In fact, even right now, without my patches, we don't hold the tracing
>> lock when we execute that if. Nothing prevents tb_init_done to become 0
>> _right_after_ we saw it being 1 and decide to go ahead.
>>
>> This, to me, looks like an even more compelling reason to remove it.
>> But I think I can improve the commit message so that it explains this
>> thing that I just said above too.
>
> Well, my question wasn't about a possible race (as the code would
> need to be able to deal with that no matter what change you do
> here), but about the case where tb_init_done has never been set.
> Would tb_event_mask be reliably zero in that case, no matter
> what other operations may have been performed?
Looks like tb_event_mask can be set even if opt_tbuf_size == 0; so yes,
if someone enabled the event mask before allocating any buffers, *and*
there was a bug where __trace_var() was called without first checking
tb_init_done, then we might get past this point.
But it looks like that's still OK -- we'll then bail when our bit in
tb_cpu_mask isn't set, or finally bail when we find t_bufs to be zero.
-George
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 02/15] xen: tracing: avoid checking tb_init_done multiple times.
2017-06-01 17:33 ` [PATCH 02/15] xen: tracing: avoid checking tb_init_done multiple times Dario Faggioli
2017-06-01 17:53 ` Andrew Cooper
2017-06-07 14:46 ` Jan Beulich
@ 2017-06-08 14:37 ` George Dunlap
2 siblings, 0 replies; 68+ messages in thread
From: George Dunlap @ 2017-06-08 14:37 UTC (permalink / raw)
To: Dario Faggioli, xen-devel; +Cc: George Dunlap
On 01/06/17 18:33, Dario Faggioli wrote:
> In fact, when calling __trace_var() directly, we can
> assume that tb_init_done has been checked to be true,
> and the if is hence redundant.
You probably want to adjust the comment before the smp_rmb() that
mentions tb_init_done as well.
Other than that, looks good.
-George
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 01/15] xen: in do_softirq() sample smp_processor_id() once and for all.
2017-06-08 14:20 ` George Dunlap
@ 2017-06-08 14:42 ` Jan Beulich
0 siblings, 0 replies; 68+ messages in thread
From: Jan Beulich @ 2017-06-08 14:42 UTC (permalink / raw)
To: Dario Faggioli, George Dunlap
Cc: Stefano Stabellini, George Dunlap, AndrewCooper, Tim Deegan,
Julien Grall, xen-devel
>>> On 08.06.17 at 16:20, <george.dunlap@citrix.com> wrote:
> On 01/06/17 18:33, Dario Faggioli wrote:
>> In fact, right now, we read it at every iteration of the loop.
>> The reason it's done like this is how context switch was handled
>> on IA64 (see commit ae9bfcdc, "[XEN] Various softirq cleanups" [1]).
>>
>> However:
>> 1) we don't have IA64 any longer, and all the achitectures that
>> we do support, are ok with sampling once and for all;
>> 2) sampling at every iteration (slightly) affect performance;
>> 3) sampling at every iteration is misleading, as it makes people
>> believe that it is currently possible that SCHEDULE_SOFTIRQ
>> moves the execution flow on another CPU (and the comment,
>> by reinforcing this belief, makes things even worse!).
>>
>> Therefore, let's:
>> - do the sampling once and for all, and remove the comment;
>
> "Once and for all" has a much stronger sense of finality than you're
> using here: reading smp_processor_id() in that function "once and for
> all" would mean you would never have to read it on that function, on
> that particular core, ever again, until the end of the universe. :-)
>
> I think I'd say, "only once". The scope of the "only" in that case is
> "this function call", not "all calls forever".
>
> With that changed:
>
> Reviewed-by: George Dunlap <george.dunlap@citrix.com>
>
> (Although we till need Jan's acquiescence.)
I've voiced my opinion, but I don't mean to block the patch. After
all there's no active issue the change introduces.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 03/15] xen/tools: tracing: several improvements on IRQs tracing
2017-06-07 15:58 ` Jan Beulich
@ 2017-06-08 14:53 ` George Dunlap
2017-06-08 15:34 ` Jan Beulich
0 siblings, 1 reply; 68+ messages in thread
From: George Dunlap @ 2017-06-08 14:53 UTC (permalink / raw)
To: Jan Beulich, Dario Faggioli
Cc: George Dunlap, Andrew Cooper, Wei Liu, Ian Jackson, xen-devel
On 07/06/17 16:58, Jan Beulich wrote:
>>>> On 07.06.17 at 17:45, <dario.faggioli@citrix.com> wrote:
>> On Wed, 2017-06-07 at 09:05 -0600, Jan Beulich wrote:
>>>>>> On 01.06.17 at 19:33, <dario.faggioli@citrix.com> wrote:
>>>> @@ -884,9 +919,13 @@ void do_IRQ(struct cpu_user_regs *regs)
>>>> desc->rl_quantum_start = now;
>>>> }
>>>>
>>>> - tsc_in = tb_init_done ? get_cycles() : 0;
>>>> + if ( unlikely(tb_init_done) )
>>>> + {
>>>> + __trace_var(TRC_HW_IRQ_GUEST, 0, sizeof(irq), &irq);
>>>> + tsc_in = get_cycles();
>>>> + }
>>>> __do_IRQ_guest(irq);
>>>> - TRACE_3D(TRC_HW_IRQ_HANDLED, irq, tsc_in, get_cycles());
>>>> + trace_irq_handled(irq, get_cycles() - tsc_in);
>>>> goto out_no_end;
>>>> }
>>>
>>> Before this change, the get_cycles() invocation was after
>>> the tb_init_done check. Now you move it ahead of it (as
>>> the function arguments are evaluated before executing the
>>> function body) - are you sure all compiler versions will suitably
>>> re-order this?
>>>
>>> Additionally I'm afraid this will trigger compiler warnings on at
>>> least some compiler versions, as tsc_in is now possibly
>>> uninitialized (and there's no clear way to disprove this for the
>>> compiler, again because function arguments are being
>>> evaluated before the function body is reached).
>>>
>> I understand and (now that I see it) agree with your remark on when
>> get_cycles() is called. I'll reorganize things so that the patched
>> behavior matches the existing one.
>>
>> OTOH, I'm not sure I see the "potentially uninitialized" issue for
>> tsc_in, but since I'm reworking the code, I'll keep that in mind too.
>
> You initialize tsc_in only when tb_init_done is set, but you use
> it without that conditional. And even if you used it under that
> same conditional, older compiler versions aren't able to track
> that fact (same conditional) and raise a warning anyway.
This patch changes thing to initialize tsc_in on declaration (at the top
of the function).
If we want to keep the compiler "uninitialized variable" analysis
around, that would have to go away and we'd want to do something like
was there before. (I'm ambivalent about it, but I know Jan likes it.)
-George
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 03/15] xen/tools: tracing: several improvements on IRQs tracing
2017-06-01 17:33 ` [PATCH 03/15] xen/tools: tracing: several improvements on IRQs tracing Dario Faggioli
2017-06-01 18:02 ` Andrew Cooper
2017-06-07 15:05 ` Jan Beulich
@ 2017-06-08 14:59 ` George Dunlap
2 siblings, 0 replies; 68+ messages in thread
From: George Dunlap @ 2017-06-08 14:59 UTC (permalink / raw)
To: Dario Faggioli, xen-devel
Cc: George Dunlap, Wei Liu, Ian Jackson, Jan Beulich, Andrew Cooper
On 01/06/17 18:33, Dario Faggioli wrote:
> More specifically:
> - the handling of the TRC_HW_IRQ_HANDLED is fixed, both in
> xentrace_format and in xenalyze;
> - simple events for recording when we enter and exit the
> do_IRQ function, as well as when we deal with a guest
> IRQ, are added;
> - tracing of IRQs handled with direct vectors is also
> added.
>
> With all the above, a trace will now look like this (using
> xenalyze):
>
> 0.001299072 .x- d32768v5 irq_enter, irq 80000000
> 0.001299072 .x- d32768v5 irq_direct, vec fa, handler = 0xffff82d08026d7e4
> 0.001300014 .x- d32768v5 raise_softirq nr 0
> 0.001300487 .x- d32768v5 irq_exit irq 80000000, in_irq = 0
>
> Or like this:
>
> 0.049437892 -|- d32767v12 irq_enter, irq 4
> 0.049437892 -|- d32767v12 irq_handled irq 4, 85428 cycles
> 0.049474336 -|- d32767v12 irq_exit irq 4, status = 0x0, in_irq = 0
>
> Making it much easier to figure out when interrupt
> processing start, what it does and when it ends.
>
> Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
Generally looks good to me. I mostly agree with the commentary as well.
-George
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 04/15] tools: xenalyze: fix dumping of PM_IDLE events.
2017-06-01 17:34 ` [PATCH 04/15] tools: xenalyze: fix dumping of PM_IDLE events Dario Faggioli
@ 2017-06-08 15:06 ` George Dunlap
0 siblings, 0 replies; 68+ messages in thread
From: George Dunlap @ 2017-06-08 15:06 UTC (permalink / raw)
To: Dario Faggioli; +Cc: xen-devel
On Thu, Jun 1, 2017 at 6:34 PM, Dario Faggioli
<dario.faggioli@citrix.com> wrote:
> In fact, not all the information present in the trace
> record were used and printed.
>
> Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
Reviewed-by: George Dunlap <george.dunlap@citrix.com>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 05/15] xen: make it possible to disable tracing in Kconfig.
2017-06-07 15:14 ` Jan Beulich
@ 2017-06-08 15:16 ` George Dunlap
2017-06-08 15:35 ` Jan Beulich
0 siblings, 1 reply; 68+ messages in thread
From: George Dunlap @ 2017-06-08 15:16 UTC (permalink / raw)
To: Jan Beulich, Dario Faggioli
Cc: Stefano Stabellini, Wei Liu, George Dunlap, Andrew Cooper,
Ian Jackson, Julien Grall, xen-devel, Doug Goldstein
On 07/06/17 16:14, Jan Beulich wrote:
>>>> On 01.06.17 at 19:34, <dario.faggioli@citrix.com> wrote:
>> --- a/xen/Kconfig.debug
>> +++ b/xen/Kconfig.debug
>> @@ -98,6 +98,14 @@ config PERF_ARRAYS
>> ---help---
>> Enables software performance counter array histograms.
>>
>> +config TRACING
>> + bool "Tracing"
>> + default y
>
> default DEBUG (you don't want to suggest turning it on for a
> release build)
In the past I've found it pretty useful to have on in release builds of
XenServer, to analyze performance problems a customer was having, and
sometimes to see where the guest was crashing.
-George
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 05/15] xen: make it possible to disable tracing in Kconfig.
2017-06-01 17:34 ` [PATCH 05/15] xen: make it possible to disable tracing in Kconfig Dario Faggioli
2017-06-01 18:43 ` Andrew Cooper
2017-06-07 15:14 ` Jan Beulich
@ 2017-06-08 15:17 ` George Dunlap
2 siblings, 0 replies; 68+ messages in thread
From: George Dunlap @ 2017-06-08 15:17 UTC (permalink / raw)
To: Dario Faggioli, xen-devel
Cc: Stefano Stabellini, Wei Liu, George Dunlap, Andrew Cooper,
Ian Jackson, Julien Grall, Jan Beulich, Doug Goldstein
On 01/06/17 18:34, Dario Faggioli wrote:
> And compile it out of the hypervisor entirely.
>
> Code and other sections' sizes change as follows.
>
> Output of `size`:
> vanilla patched-Y patched-N
> text 1929007 1929007 1902783
> data 337784 337784 337688
> bss 1310464 1310464 1310336
>
> Output of `size -A`:
> vanilla patched-Y patched-N
> .text 1372602 1372602 1348026
> .rodata 312152 312152 310680
> .init.text 244209 244209 244033
> .init.data 224576 224576 224576
> .data 57472 57472 57376
> .bss 1310464 1310464 1310336
> Total 23026516 23027008 22858069
>
> No functional change intended.
>
> Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
Thanks Dario -- patch looks good overall; I agree with all of Jan and
Andy's comments that I haven't responded to. :-)
-George
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 03/15] xen/tools: tracing: several improvements on IRQs tracing
2017-06-08 14:53 ` George Dunlap
@ 2017-06-08 15:34 ` Jan Beulich
0 siblings, 0 replies; 68+ messages in thread
From: Jan Beulich @ 2017-06-08 15:34 UTC (permalink / raw)
To: Dario Faggioli, George Dunlap
Cc: George Dunlap, Andrew Cooper, Wei Liu, Ian Jackson, xen-devel
>>> On 08.06.17 at 16:53, <george.dunlap@citrix.com> wrote:
> On 07/06/17 16:58, Jan Beulich wrote:
>>>>> On 07.06.17 at 17:45, <dario.faggioli@citrix.com> wrote:
>>> On Wed, 2017-06-07 at 09:05 -0600, Jan Beulich wrote:
>>>>>>> On 01.06.17 at 19:33, <dario.faggioli@citrix.com> wrote:
>>>>> @@ -884,9 +919,13 @@ void do_IRQ(struct cpu_user_regs *regs)
>>>>> desc->rl_quantum_start = now;
>>>>> }
>>>>>
>>>>> - tsc_in = tb_init_done ? get_cycles() : 0;
>>>>> + if ( unlikely(tb_init_done) )
>>>>> + {
>>>>> + __trace_var(TRC_HW_IRQ_GUEST, 0, sizeof(irq), &irq);
>>>>> + tsc_in = get_cycles();
>>>>> + }
>>>>> __do_IRQ_guest(irq);
>>>>> - TRACE_3D(TRC_HW_IRQ_HANDLED, irq, tsc_in, get_cycles());
>>>>> + trace_irq_handled(irq, get_cycles() - tsc_in);
>>>>> goto out_no_end;
>>>>> }
>>>>
>>>> Before this change, the get_cycles() invocation was after
>>>> the tb_init_done check. Now you move it ahead of it (as
>>>> the function arguments are evaluated before executing the
>>>> function body) - are you sure all compiler versions will suitably
>>>> re-order this?
>>>>
>>>> Additionally I'm afraid this will trigger compiler warnings on at
>>>> least some compiler versions, as tsc_in is now possibly
>>>> uninitialized (and there's no clear way to disprove this for the
>>>> compiler, again because function arguments are being
>>>> evaluated before the function body is reached).
>>>>
>>> I understand and (now that I see it) agree with your remark on when
>>> get_cycles() is called. I'll reorganize things so that the patched
>>> behavior matches the existing one.
>>>
>>> OTOH, I'm not sure I see the "potentially uninitialized" issue for
>>> tsc_in, but since I'm reworking the code, I'll keep that in mind too.
>>
>> You initialize tsc_in only when tb_init_done is set, but you use
>> it without that conditional. And even if you used it under that
>> same conditional, older compiler versions aren't able to track
>> that fact (same conditional) and raise a warning anyway.
>
> This patch changes thing to initialize tsc_in on declaration (at the top
> of the function).
Oh, it's only now that I actually see an initializer is being added.
I'm sorry for the noise then.
> If we want to keep the compiler "uninitialized variable" analysis
> around, that would have to go away and we'd want to do something like
> was there before. (I'm ambivalent about it, but I know Jan likes it.)
No, I'm fine with the change. I was just being blind.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 05/15] xen: make it possible to disable tracing in Kconfig.
2017-06-08 15:16 ` George Dunlap
@ 2017-06-08 15:35 ` Jan Beulich
2017-06-08 15:37 ` George Dunlap
0 siblings, 1 reply; 68+ messages in thread
From: Jan Beulich @ 2017-06-08 15:35 UTC (permalink / raw)
To: Dario Faggioli, George Dunlap
Cc: Stefano Stabellini, Wei Liu, George Dunlap, Andrew Cooper,
Ian Jackson, Julien Grall, xen-devel, Doug Goldstein
>>> On 08.06.17 at 17:16, <george.dunlap@citrix.com> wrote:
> On 07/06/17 16:14, Jan Beulich wrote:
>>>>> On 01.06.17 at 19:34, <dario.faggioli@citrix.com> wrote:
>>> --- a/xen/Kconfig.debug
>>> +++ b/xen/Kconfig.debug
>>> @@ -98,6 +98,14 @@ config PERF_ARRAYS
>>> ---help---
>>> Enables software performance counter array histograms.
>>>
>>> +config TRACING
>>> + bool "Tracing"
>>> + default y
>>
>> default DEBUG (you don't want to suggest turning it on for a
>> release build)
>
> In the past I've found it pretty useful to have on in release builds of
> XenServer, to analyze performance problems a customer was having, and
> sometimes to see where the guest was crashing.
Well, that's your choice. I'm only asking to default to off in release
builds, not to prevent enabling it.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 05/15] xen: make it possible to disable tracing in Kconfig.
2017-06-08 15:35 ` Jan Beulich
@ 2017-06-08 15:37 ` George Dunlap
2017-06-08 15:44 ` Jan Beulich
0 siblings, 1 reply; 68+ messages in thread
From: George Dunlap @ 2017-06-08 15:37 UTC (permalink / raw)
To: Jan Beulich, Dario Faggioli
Cc: Stefano Stabellini, Wei Liu, George Dunlap, Andrew Cooper,
Ian Jackson, Julien Grall, xen-devel, Doug Goldstein
On 08/06/17 16:35, Jan Beulich wrote:
>>>> On 08.06.17 at 17:16, <george.dunlap@citrix.com> wrote:
>> On 07/06/17 16:14, Jan Beulich wrote:
>>>>>> On 01.06.17 at 19:34, <dario.faggioli@citrix.com> wrote:
>>>> --- a/xen/Kconfig.debug
>>>> +++ b/xen/Kconfig.debug
>>>> @@ -98,6 +98,14 @@ config PERF_ARRAYS
>>>> ---help---
>>>> Enables software performance counter array histograms.
>>>>
>>>> +config TRACING
>>>> + bool "Tracing"
>>>> + default y
>>>
>>> default DEBUG (you don't want to suggest turning it on for a
>>> release build)
>>
>> In the past I've found it pretty useful to have on in release builds of
>> XenServer, to analyze performance problems a customer was having, and
>> sometimes to see where the guest was crashing.
>
> Well, that's your choice. I'm only asking to default to off in release
> builds, not to prevent enabling it.
Of course; the question is, as you say, what we're "suggesting"; and
what people will experience if they just go with the default.
If it were up to me I'd leave it on by default, but I don't have a
strong opinion.
-George
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 05/15] xen: make it possible to disable tracing in Kconfig.
2017-06-08 15:37 ` George Dunlap
@ 2017-06-08 15:44 ` Jan Beulich
0 siblings, 0 replies; 68+ messages in thread
From: Jan Beulich @ 2017-06-08 15:44 UTC (permalink / raw)
To: George Dunlap
Cc: Stefano Stabellini, Wei Liu, George Dunlap, Andrew Cooper,
Dario Faggioli, Doug Goldstein, Julien Grall, xen-devel,
Ian Jackson
>>> On 08.06.17 at 17:37, <george.dunlap@citrix.com> wrote:
> On 08/06/17 16:35, Jan Beulich wrote:
>>>>> On 08.06.17 at 17:16, <george.dunlap@citrix.com> wrote:
>>> On 07/06/17 16:14, Jan Beulich wrote:
>>>>>>> On 01.06.17 at 19:34, <dario.faggioli@citrix.com> wrote:
>>>>> --- a/xen/Kconfig.debug
>>>>> +++ b/xen/Kconfig.debug
>>>>> @@ -98,6 +98,14 @@ config PERF_ARRAYS
>>>>> ---help---
>>>>> Enables software performance counter array histograms.
>>>>>
>>>>> +config TRACING
>>>>> + bool "Tracing"
>>>>> + default y
>>>>
>>>> default DEBUG (you don't want to suggest turning it on for a
>>>> release build)
>>>
>>> In the past I've found it pretty useful to have on in release builds of
>>> XenServer, to analyze performance problems a customer was having, and
>>> sometimes to see where the guest was crashing.
>>
>> Well, that's your choice. I'm only asking to default to off in release
>> builds, not to prevent enabling it.
>
> Of course; the question is, as you say, what we're "suggesting"; and
> what people will experience if they just go with the default.
>
> If it were up to me I'd leave it on by default, but I don't have a
> strong opinion.
If you look at that section in Kconfig.debug, nothing defaults to
y, and I think this is appropriate for debug options in general. I
additionally think that defaults suggested in EXPERT mode should
match the values automatically chosen in non-EXPERT mode.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 06/15] xen: trace IRQ enabling/disabling
2017-06-01 23:42 ` Dario Faggioli
@ 2017-06-08 15:51 ` George Dunlap
2017-06-08 16:05 ` Jan Beulich
1 sibling, 0 replies; 68+ messages in thread
From: George Dunlap @ 2017-06-08 15:51 UTC (permalink / raw)
To: Dario Faggioli, Andrew Cooper, xen-devel
Cc: George Dunlap, Julien Grall, Stefano Stabellini,
Jennifer Herbert, Jan Beulich
On 02/06/17 00:42, Dario Faggioli wrote:
> On Thu, 2017-06-01 at 20:08 +0100, Andrew Cooper wrote:
>> On 01/06/17 18:34, Dario Faggioli wrote:
>>> diff --git a/xen/common/spinlock.c b/xen/common/spinlock.c
>>> index 2a06406..33b903e 100644
>>> --- a/xen/common/spinlock.c
>>> +++ b/xen/common/spinlock.c
>>> @@ -150,7 +150,9 @@ void _spin_lock(spinlock_t *lock)
>>> void _spin_lock_irq(spinlock_t *lock)
>>> {
>>> ASSERT(local_irq_is_enabled());
>>> - local_irq_disable();
>>> + _local_irq_disable();
>>> + if ( unlikely(tb_init_done) )
>>> + trace_irq_disable_ret();
>>> _spin_lock(lock);
>>> }
>>
>> Is it sensible to do this way around?
>>
> Well, I guess either variant has up and down sides, and it looks to me
> that there is no way to measure this, without interfering with the
> behavior of the thing being measured ("Once upon a time, there was a
> cat, who lived in a box. His name was Schrödinger..." :-D :-D :-D)
>
>> By writing the trace record while interrupts are disabled, you do
>> prevent nesting in the general case (but not in NMIs/MCEs or the
>> irqsave() variants),
>>
> Forgive the ignorance, what's special about NMIs/MCAs that is relevant
> for this? About _irqsave(), I'm also not sure what you mean, as
> _irqsave() is already done differently than this.
>
>> but you increase the critical region while
>> interrupts are disabled.
>>
> I know.
>
>> Alternatively, writing the trace record outside of the critical
>> region
>> would reduce the size of the region. Interpretation logic already
>> needs
>> to cope with nesting, so is one extra level of nesting in this corner
>> case a problem?
>>
> TBH, I was indeed trying to offer to the component that will be looking
> at and interpreting the data, the as clear as possible view on when
> IRQs are _actually_ disabled and enabled. As in, if nesting occurs,
> only the event corresponding to:
> - the first local_irq_disable()
> - the last local_irq_enable()
>
> I.e., to not require that it (the interpretation logic) does understand
> nesting.
>
> But more than this, what I was concerned about was the accuracy of the
> reporting itself. In fact, if I do as you suggest, I can be interrupted
> (as interrupts are still on) after having called trace_irq_disable().
> That means the report will show higher IRQ disabled time period, for
> this instance, than what the reality is.
>
> And the same is true on the enabling side, when I call
> trace_irq_enable() _before_ re-enabling interrupts, which has the same
> bad effect on the length of IRQ disabled section.
>
> Maybe, considering that anything that will interrupt me in these cases,
> are other interrupts, that will most likely disable interrupts
> themselves, I should not worry too much about this... What do you
> think?
I think it would make the analysis easier to do it the way you've done
it. It certainly does increase the length of the actual critical
section, but we're already assuming that all of this code is going to be
disabled unless people specifically want to do IRQ analysis, so I don't
think that should be an issue.
-George
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 06/15] xen: trace IRQ enabling/disabling
2017-06-01 17:34 ` [PATCH 06/15] xen: trace IRQ enabling/disabling Dario Faggioli
2017-06-01 19:08 ` Andrew Cooper
2017-06-07 11:16 ` Julien Grall
@ 2017-06-08 16:01 ` George Dunlap
2017-06-08 16:11 ` Dario Faggioli
2017-06-09 10:41 ` Jan Beulich
3 siblings, 1 reply; 68+ messages in thread
From: George Dunlap @ 2017-06-08 16:01 UTC (permalink / raw)
To: Dario Faggioli, xen-devel
Cc: Stefano Stabellini, George Dunlap, Andrew Cooper,
Jennifer Herbert, Julien Grall, Jan Beulich
On 01/06/17 18:34, Dario Faggioli wrote:
> Trace when interrupts are disabled and (re)enabled.
> Basically, we replace the IRQ disabling and enabling
> functions with helpers that does the same, but also
> output the proper trace record.
>
> For putting in the record something that will let
> us identify _where_ in the code (i.e., in what function)
> the IRQ manipulation is happening, use either:
> - current_text_addr(),
> - or __builtin_return_address(0).
>
> In fact, depending on whether the disabling/enabling
> happens in macros (like for local_irq_disable() and
> local_irq_enable()) or in actual functions (like in
> spin_lock_irq*()), it is either:
> - the actual content of the instruction pointer when
> IRQ are disabled/enabled,
> - or the return address of the utility function where
> IRQ are disabled/enabled,
> that will tell us what it is the actual piece of code
> that is asking for the IRQ manipulation operation.
>
> Gate this with its specific Kconfig option, and keep
> it in disabled state by default (i.e., don't build it,
> if not explicitly specified), as the impact on
> performance may be non negligible.
>
> Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
> ---
> Cc: George Dunlap <george.dunlap@eu.citrix.com>
> Cc: Stefano Stabellini <sstabellini@kernel.org>
> Cc: Julien Grall <julien.grall@arm.com>
> Cc: Jan Beulich <jbeulich@suse.com>
> Cc: Andrew Cooper <andrew.cooper3@citrix.com>
> Cc: Jennifer Herbert <jennifer.herbert@citrix.com>
> ---
> xen/Kconfig.debug | 11 ++++-
> xen/common/spinlock.c | 16 +++++--
> xen/common/trace.c | 10 +++-
> xen/include/asm-arm/arm32/system.h | 12 +++++
> xen/include/asm-arm/arm64/system.h | 12 +++++
> xen/include/asm-x86/system.h | 85 ++++++++++++++++++++++++++++++++++--
> xen/include/public/trace.h | 2 +
> xen/include/xen/rwlock.h | 33 +++++++++++---
> 8 files changed, 161 insertions(+), 20 deletions(-)
>
> diff --git a/xen/Kconfig.debug b/xen/Kconfig.debug
> index 374c1c0..81910c9 100644
> --- a/xen/Kconfig.debug
> +++ b/xen/Kconfig.debug
> @@ -98,7 +98,7 @@ config PERF_ARRAYS
> ---help---
> Enables software performance counter array histograms.
>
> -config TRACING
> +menuconfig TRACING
> bool "Tracing"
> default y
> ---help---
> @@ -106,6 +106,15 @@ config TRACING
> in per-CPU ring buffers. The 'xentrace' tool can be used to read
> the buffers and dump the content on the disk.
>
> +config TRACE_IRQSOFF
> + bool "Trace when IRQs are disabled and (re)enabled" if EXPERT = "y"
> + default n
> + depends on TRACING
> + ---help---
> + Makes it possible to generate events _every_ time IRQs are disabled
> + and (re)enabled, with also an indication of where that happened.
> + Note that this comes with high overead and produces huge amount of
> + tracing data.
*overhead (yours is missing an 'h')
I think I might emphasize that the overhead well be present even when
tracing is off. What about something like this?
"Note that this comes with an overhead even when tracing is disabled;
and has a high overhead and produces a large amount of tracing data when
enabled."
With that change:
Acked-by: George Dunlap <george.dunlap@citrix.com>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 06/15] xen: trace IRQ enabling/disabling
2017-06-01 23:42 ` Dario Faggioli
2017-06-08 15:51 ` George Dunlap
@ 2017-06-08 16:05 ` Jan Beulich
1 sibling, 0 replies; 68+ messages in thread
From: Jan Beulich @ 2017-06-08 16:05 UTC (permalink / raw)
To: Dario Faggioli
Cc: Stefano Stabellini, George Dunlap, Andrew Cooper,
Jennifer Herbert, Julien Grall, xen-devel
>>> On 02.06.17 at 01:42, <dario.faggioli@citrix.com> wrote:
> On Thu, 2017-06-01 at 20:08 +0100, Andrew Cooper wrote:
>> By writing the trace record while interrupts are disabled, you do
>> prevent nesting in the general case (but not in NMIs/MCEs or the
>> irqsave() variants),
>>
> Forgive the ignorance, what's special about NMIs/MCAs that is relevant
> for this?
NMI/#MC can nest nevertheless, so preventing nesting in the
common case doesn't mean you won't see nesting at all.
>> Does the logic cope with the fact that interrupt gates automatically
>> disable interrupts?
>>
> Ah, right. No, it does not. I probably should mention this in the
> changelog. Any ideas of how to deal with that? If yes, I'm more than
> happy to fix this...
Well, respective entry points will need to update tracking state.
See how Linux has placed certain macros to that effect on
various entry paths.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 06/15] xen: trace IRQ enabling/disabling
2017-06-08 16:01 ` George Dunlap
@ 2017-06-08 16:11 ` Dario Faggioli
0 siblings, 0 replies; 68+ messages in thread
From: Dario Faggioli @ 2017-06-08 16:11 UTC (permalink / raw)
To: George Dunlap, xen-devel
Cc: Stefano Stabellini, George Dunlap, Andrew Cooper,
Jennifer Herbert, Julien Grall, Jan Beulich
[-- Attachment #1.1: Type: text/plain, Size: 1890 bytes --]
On Thu, 2017-06-08 at 17:01 +0100, George Dunlap wrote:
> On 01/06/17 18:34, Dario Faggioli wrote:
> > diff --git a/xen/Kconfig.debug b/xen/Kconfig.debug
> > index 374c1c0..81910c9 100644
> > --- a/xen/Kconfig.debug
> > +++ b/xen/Kconfig.debug
> > @@ -98,7 +98,7 @@ config PERF_ARRAYS
> > ---help---
> > Enables software performance counter array histograms.
> >
> > -config TRACING
> > +menuconfig TRACING
> > bool "Tracing"
> > default y
> > ---help---
> > @@ -106,6 +106,15 @@ config TRACING
> > in per-CPU ring buffers. The 'xentrace' tool can be used
> > to read
> > the buffers and dump the content on the disk.
> >
> > +config TRACE_IRQSOFF
> > + bool "Trace when IRQs are disabled and (re)enabled" if
> > EXPERT = "y"
> > + default n
> > + depends on TRACING
> > + ---help---
> > + Makes it possible to generate events _every_ time IRQs
> > are disabled
> > + and (re)enabled, with also an indication of where that
> > happened.
> > + Note that this comes with high overead and produces huge
> > amount of
> > + tracing data.
>
> I think I might emphasize that the overhead well be present even when
> tracing is off. What about something like this?
>
Yes, good point.
> "Note that this comes with an overhead even when tracing is disabled;
> and has a high overhead and produces a large amount of tracing data
> when
> enabled."
>
I like it, I'll go for it.
> With that change:
>
> Acked-by: George Dunlap <george.dunlap@citrix.com>
>
Thanks and Regards,
Dario
--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
[-- Attachment #2: Type: text/plain, Size: 127 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 00/15] xen/tools: add tracing to various Xen subsystems
2017-06-07 14:13 ` [PATCH 00/15] xen/tools: add tracing to various Xen subsystems Konrad Rzeszutek Wilk
@ 2017-06-08 16:45 ` Dario Faggioli
0 siblings, 0 replies; 68+ messages in thread
From: Dario Faggioli @ 2017-06-08 16:45 UTC (permalink / raw)
To: Konrad Rzeszutek Wilk
Cc: Stefano Stabellini, Wei Liu, George Dunlap, Andrew Cooper,
Ian Jackson, Tim Deegan, Julien Grall, Jan Beulich,
Jennifer Herbert, xen-devel, Doug Goldstein
[-- Attachment #1.1: Type: text/plain, Size: 1501 bytes --]
On Wed, 2017-06-07 at 10:13 -0400, Konrad Rzeszutek Wilk wrote:
> On Thu, Jun 01, 2017 at 07:33:33PM +0200, Dario Faggioli wrote:
> >
> > Patch 5 deserves special mention. In fact, now that we have
> > Kconfig, I thought
> > it could be a nice thing to make it possible to select, at build
> > config time,
> > whether we want tracing or not, in the hypervisor (like, for
> > instance, we do
> > for performance counters).
>
> Did you have thoughts on perhaps using asm goto as an
> alterantive to unlikely?
>
> In Linux it is called jump labels or such - the idea is that the
> code has (by default and on x86) five NOP instructions. But you
> can patch it over and add an call to the unlikely code.
>
Yes, I know. I've never actually looked at the code, but I know they do
that, and I think it's cool.
> But perhaps that is more of an future idea as looking at the Linux
> code
> it looks quite large and not that simple.
>
I would love for us to do something similar in Xen. I've _thought_
about that many times, but that's it. :-/
Let's see... right now, I can't look into this, as I agree with you
that it would be a major piece of work.
But yes, it's been in my thoughts! :-)
Regards,
Dario
--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
[-- Attachment #2: Type: text/plain, Size: 127 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 06/15] xen: trace IRQ enabling/disabling
2017-06-01 17:34 ` [PATCH 06/15] xen: trace IRQ enabling/disabling Dario Faggioli
` (2 preceding siblings ...)
2017-06-08 16:01 ` George Dunlap
@ 2017-06-09 10:41 ` Jan Beulich
3 siblings, 0 replies; 68+ messages in thread
From: Jan Beulich @ 2017-06-09 10:41 UTC (permalink / raw)
To: Dario Faggioli
Cc: Stefano Stabellini, George Dunlap, Andrew Cooper,
Jennifer Herbert, Julien Grall, xen-devel
>>> On 01.06.17 at 19:34, <dario.faggioli@citrix.com> wrote:
> @@ -106,6 +106,15 @@ config TRACING
> in per-CPU ring buffers. The 'xentrace' tool can be used to read
> the buffers and dump the content on the disk.
>
> +config TRACE_IRQSOFF
> + bool "Trace when IRQs are disabled and (re)enabled" if EXPERT = "y"
> + default n
I'm not aware of any behavioral difference between this an not
specifying a default at all, so I'd like to ask to follow suit (see
surrounding code) and omit this line.
> --- a/xen/common/spinlock.c
> +++ b/xen/common/spinlock.c
> @@ -150,7 +150,9 @@ void _spin_lock(spinlock_t *lock)
> void _spin_lock_irq(spinlock_t *lock)
> {
> ASSERT(local_irq_is_enabled());
> - local_irq_disable();
> + _local_irq_disable();
Hmm, looks you're introducing new non-static, underscore-prefixed
symbols. Please don't, use e.g. a "arch_" prefix or a "_notrace" suffix
instead.
> --- a/xen/include/asm-x86/system.h
> +++ b/xen/include/asm-x86/system.h
> @@ -5,6 +5,8 @@
> #include <xen/bitops.h>
> #include <asm/processor.h>
>
> +#include <xen/trace.h>
Why?
> @@ -214,6 +216,79 @@ static always_inline unsigned long __xadd(
> "ri" ( (x) & X86_EFLAGS_IF ) ); \
> })
>
> +#ifdef CONFIG_TRACE_IRQSOFF
> +
> +#define TRACE_LOCAL_ADDR ((uint64_t) current_text_addr())
> +#define TRACE_RET_ADDR ((uint64_t) __builtin_return_address(0))
> +
> +#define trace_irq_disable(_a) \
> +({ \
> + uint64_t addr = _a; \
> + __trace_var(TRC_HW_IRQ_DISABLE, 1, sizeof(addr), &addr); \
> +})
> +#define trace_irq_enable(_a) \
> +({ \
> + uint64_t addr = _a; \
> + __trace_var(TRC_HW_IRQ_ENABLE, 1, sizeof(addr), &addr); \
> +})
> +#define trace_irq_save(_x, _a) \
> +({ \
> + uint64_t addr = _a; \
> + if ( _x & X86_EFLAGS_IF ) \
> + __trace_var(TRC_HW_IRQ_DISABLE, 1, sizeof(addr), &addr); \
> +})
> +#define trace_irq_restore(_x, _a) \
> +({ \
> + uint64_t addr = _a; \
> + if ( _x & X86_EFLAGS_IF ) \
> + __trace_var(TRC_HW_IRQ_ENABLE, 1, sizeof(addr), &addr); \
> +})
> +
> +#define trace_irq_disable_ret() trace_irq_disable(TRACE_RET_ADDR)
> +#define trace_irq_enable_ret() trace_irq_enable(TRACE_RET_ADDR)
> +#define trace_irq_save_ret(_x) trace_irq_save(_x, TRACE_RET_ADDR)
> +#define trace_irq_restore_ret(_x) trace_irq_restore(_x, TRACE_RET_ADDR)
> +
> +#define local_irq_disable() \
> +({ \
> + bool_t irqon = local_irq_is_enabled(); \
> + _local_irq_disable(); \
> + if ( unlikely(tb_init_done && irqon) ) \
> + trace_irq_disable(TRACE_LOCAL_ADDR); \
> +})
> +
> +#define local_irq_enable() \
> +({ \
> + if ( unlikely(tb_init_done) ) \
> + trace_irq_enable(TRACE_LOCAL_ADDR); \
> + _local_irq_enable(); \
> +})
> +
> +#define local_irq_save(_x) \
> +({ \
> + local_save_flags(_x); \
> + _local_irq_disable(); \
> + if ( unlikely(tb_init_done) ) \
> + trace_irq_save(_x, TRACE_LOCAL_ADDR); \
> +})
> +
> +#define local_irq_restore(_x) \
> +({ \
> + if ( unlikely(tb_init_done) ) \
> + trace_irq_restore(_x, TRACE_LOCAL_ADDR); \
> + _local_irq_restore(_x); \
> +})
> +#else /* !TRACE_IRQSOFF */
> +#define trace_irq_disable_ret() do { } while ( 0 )
> +#define trace_irq_enable_ret() do { } while ( 0 )
> +#define trace_irq_save_ret(_x) do { } while ( 0 )
> +#define trace_irq_restore_ret(_x) do { } while ( 0 )
> +#define local_irq_disable() _local_irq_disable()
> +#define local_irq_enable() _local_irq_enable()
> +#define local_irq_save(_x) _local_irq_save(_x)
> +#define local_irq_restore(_x) _local_irq_restore(_x)
> +#endif /* TRACE_IRQSOFF */
None of this looks to be arch-specific.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 08/15] xen: trace RCU behavior
2017-06-01 17:34 ` [PATCH 08/15] xen: trace RCU behavior Dario Faggioli
@ 2017-06-09 10:48 ` Jan Beulich
2017-06-13 16:05 ` George Dunlap
1 sibling, 0 replies; 68+ messages in thread
From: Jan Beulich @ 2017-06-09 10:48 UTC (permalink / raw)
To: Dario Faggioli
Cc: Stefano Stabellini, George Dunlap, Andrew Cooper, Tim Deegan, xen-devel
>>> On 01.06.17 at 19:34, <dario.faggioli@citrix.com> wrote:
> --- a/xen/common/rcupdate.c
> +++ b/xen/common/rcupdate.c
> @@ -92,6 +92,57 @@ static int qhimark = 10000;
> static int qlowmark = 100;
> static int rsinterval = 1000;
>
> +#ifdef CONFIG_TRACE_RCU
> +static inline void trace_call_rcu(void *func)
> +{
> + uint64_t addr = (uint64_t)func;
I've probably overlooked something similar in patch 6 - why uint64_t
rather than unsigned long? Does this even build without warning for
ARM32?
> + if ( likely(!tb_init_done) )
> + return;
> +
> + __trace_var(TRC_XEN_RCU_CALL_RCU, 0, sizeof(addr), &addr);
> +}
> +static inline void trace_start_batch(const cpumask_t *m)
Blank lines between functions please.
> +{
> + uint32_t mask[6];
> +
> + if ( likely(!tb_init_done) )
> + return;
> +
> + memset(mask, 0, sizeof(mask));
Perhaps better use {} as initializer?
> + memcpy(mask, m, min(sizeof(mask), sizeof(cpumask_t)));
sizeof(*m)
> +#define trace_force_qstate() TRACE_0D(TRC_XEN_RCU_FORCE_QSTATE)
> +#define trace_cpu_quiet() TRACE_0D(TRC_XEN_RCU_CPU_QUIET)
> +#define trace_check_qstate(p) TRACE_1D(TRC_XEN_RCU_CHECK_QSTATE, p)
> +#define trace_do_callbacks() TRACE_0D(TRC_XEN_RCU_DO_CALLBKS)
> +#define trace_pending(p) TRACE_1D(TRC_XEN_RCU_PENDING, p)
> +#else /* !TRACE_RCU */
> +#define trace_call_rcu(f) do {} while ( 0 )
> +#define trace_start_batch(m) do {} while ( 0 )
> +#define trace_do_batch(f, q) do {} while ( 0 )
> +#define trace_force_qstate() do {} while ( 0 )
> +#define trace_cpu_quiet() do {} while ( 0 )
> +#define trace_check_qstate(p) do {} while ( 0 )
> +#define trace_do_callbacks() do {} while ( 0 )
> +#define trace_pending(p) do {} while ( 0 )
Here and elsewhere please make sure you evaluate macro
arguments consistently (i.e. exactly once regardless of tracing
being enabled or disabled).
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 06/15] xen: trace IRQ enabling/disabling
2017-06-07 15:22 ` Dario Faggioli
@ 2017-06-09 10:51 ` Julien Grall
2017-06-09 10:53 ` Julien Grall
2017-06-09 10:55 ` George Dunlap
0 siblings, 2 replies; 68+ messages in thread
From: Julien Grall @ 2017-06-09 10:51 UTC (permalink / raw)
To: Dario Faggioli, xen-devel
Cc: George Dunlap, Andrew Cooper, Stefano Stabellini,
Jennifer Herbert, Jan Beulich
On 07/06/17 16:22, Dario Faggioli wrote:
> On Wed, 2017-06-07 at 12:16 +0100, Julien Grall wrote:
>> Hi Dario,
>>
> Hi,
>
>> On 01/06/17 18:34, Dario Faggioli wrote:
>>> diff --git a/xen/common/spinlock.c b/xen/common/spinlock.c
>>> index 2a06406..33b903e 100644
>>> --- a/xen/common/spinlock.c
>>> +++ b/xen/common/spinlock.c
>>> @@ -150,7 +150,9 @@ void _spin_lock(spinlock_t *lock)
>>> void _spin_lock_irq(spinlock_t *lock)
>>> {
>>> ASSERT(local_irq_is_enabled());
>>> - local_irq_disable();
>>> + _local_irq_disable();
>>> + if ( unlikely(tb_init_done) )
>>
>> __trace_var already contain a "if ( !tb_init_done )". It sounds
>> pointless to do it twice, and also I think it is not obvious to
>> understand the meaning of the check (i.e what is tb_init_done).
>>
>> Would it be possible to hide this check and avoid check tb_init_done
>> twice?
>>
> I totally agree. And in fact, in another patch, I remove the
> tb_init_done check in __trace_var(). Reason behind this is that all the
> callers of __trace_var() (the main one being trace_var()), already
> checks for tb_init_done themselves.
>
> In fact, the explicit check in the caller, is the (only) basis on which
> one decides to call either trace_var() or __trace_var().
>
> This here is one of the call sites where I think the check is better
> done in the caller.
I admit, it is a bit confusing to require the caller to check
tb_init_done. It adds more code and IHMO counterintuitive.
>
>>> diff --git a/xen/include/asm-arm/arm32/system.h b/xen/include/asm-
>>> arm/arm32/system.h
>>> index c617b40..20871ad 100644
>>> --- a/xen/include/asm-arm/arm32/system.h
>>> +++ b/xen/include/asm-arm/arm32/system.h
>>> @@ -4,6 +4,8 @@
>>>
>>> #include <asm/arm32/cmpxchg.h>
>>>
>>> +#include <xen/trace.h>
>>> +
>>> #define local_irq_disable() asm volatile ( "cpsid i @
>>> local_irq_disable\n" : : : "cc" )
>>> #define local_irq_enable() asm volatile ( "cpsie i @
>>> local_irq_enable\n" : : : "cc" )
>>>
>>> @@ -41,6 +43,16 @@ static inline int local_irq_is_enabled(void)
>>> #define local_abort_enable() __asm__("cpsie a @ __sta\n" : : :
>>> "memory", "cc")
>>> #define local_abort_disable() __asm__("cpsid a @ __sta\n" : : :
>>> "memory", "cc")
>>>
>>> +/* We do not support tracing (at all) yet */
>>
>> I know that some bits are missing for ARM, but the patch #5
>> contradicts
>> this comment as you have CONFIG_TRACE=y by default.
>>
> No sure what you mean. Tracing is de facto on by default right now,
> despite it not being implemented for ARM. So what I'm doing is
> basically keeping things as they are.
>
> If you think it should be off by default, well, let's talk about it...
> but I'm not sure this is really what you are saying/asking.
I mean that CONFIG_TRACE=y by default gives the impression that tracing
is supported on ARM. This is not the case, hence your comment. So I
think it should be off my default until we get the support. BTW, it is
only a couple of simple patches but they never made upstreamed :/
>>> +#define trace_irq_disable_ret() do { } while ( 0 )
>>> +#define trace_irq_enable_ret() do { } while ( 0 )
>>> +#define trace_irq_save_ret(_x) do { } while ( 0 )
>>> +#define trace_irq_restore_ret(_x) do { } while ( 0 )
>>> +#define _local_irq_disable() local_irq_disable()
>>> +#define _local_irq_enable() local_irq_enable()
>>> +#define _local_irq_save(_x) local_irq_save(_x)
>>> +#define _local_irq_restore(_x) local_irq_restore(_x)
>>> +
>>
>> This does not need to be duplicated in both asm-
>> arm/arm{32,64}/system.h.
>> You could directly implement them in asm-arm/system.h.
>>
> Ok, thanks.
>
>> Also, I would prefer if you stay consistent with x86. I.e non-
>> underscore
>> versions are calling the underscore versions and not the invert.
>>
> Well, I know it is counterintuitive, but it's the easiest way of
> getting over this, i.e., without the need of a lot of stubs, and
> touching as few existing code as possible.
Can you detail here? The only things I was asking is to do:
#define _local_irq_disable() ......
#define local_irq_disable() _local_irq_disable.
The callers would still use local_irq_disable() and the code would be
consistent with x86.
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 10/15] xen: trace softirqs
2017-06-01 17:34 ` [PATCH 10/15] xen: trace softirqs Dario Faggioli
@ 2017-06-09 10:51 ` Jan Beulich
0 siblings, 0 replies; 68+ messages in thread
From: Jan Beulich @ 2017-06-09 10:51 UTC (permalink / raw)
To: Dario Faggioli
Cc: Stefano Stabellini, George Dunlap, Andrew Cooper, Tim Deegan, xen-devel
>>> On 01.06.17 at 19:34, <dario.faggioli@citrix.com> wrote:
> --- a/xen/Kconfig.debug
> +++ b/xen/Kconfig.debug
> @@ -124,6 +124,14 @@ config TRACE_RCU
> Makes it possible generate events showing the activity and the
> behavior of the RCU subsystem.
>
> +config TRACE_SOFTIRQS
> + bool "Trace when softirqs are raised and handled" if EXPERT = "y"
> + default n
> + depends on TRACING
> + ---help---
> + Makes it possible to generate events related to raising and
> + handling of softirqs within Xen.
Indentation (and again no default please). Other comments given
earlier apply here (and generally to other patches in this series).
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 06/15] xen: trace IRQ enabling/disabling
2017-06-09 10:51 ` Julien Grall
@ 2017-06-09 10:53 ` Julien Grall
2017-06-09 10:55 ` George Dunlap
1 sibling, 0 replies; 68+ messages in thread
From: Julien Grall @ 2017-06-09 10:53 UTC (permalink / raw)
To: Dario Faggioli, xen-devel
Cc: George Dunlap, Andrew Cooper, Stefano Stabellini,
Jennifer Herbert, Jan Beulich
On 09/06/17 11:51, Julien Grall wrote:
>
>
> On 07/06/17 16:22, Dario Faggioli wrote:
>> On Wed, 2017-06-07 at 12:16 +0100, Julien Grall wrote:
>>> Hi Dario,
>>>
>> Hi,
>>
>>> On 01/06/17 18:34, Dario Faggioli wrote:
>>>> diff --git a/xen/common/spinlock.c b/xen/common/spinlock.c
>>>> index 2a06406..33b903e 100644
>>>> --- a/xen/common/spinlock.c
>>>> +++ b/xen/common/spinlock.c
>>>> @@ -150,7 +150,9 @@ void _spin_lock(spinlock_t *lock)
>>>> void _spin_lock_irq(spinlock_t *lock)
>>>> {
>>>> ASSERT(local_irq_is_enabled());
>>>> - local_irq_disable();
>>>> + _local_irq_disable();
>>>> + if ( unlikely(tb_init_done) )
>>>
>>> __trace_var already contain a "if ( !tb_init_done )". It sounds
>>> pointless to do it twice, and also I think it is not obvious to
>>> understand the meaning of the check (i.e what is tb_init_done).
>>>
>>> Would it be possible to hide this check and avoid check tb_init_done
>>> twice?
>>>
>> I totally agree. And in fact, in another patch, I remove the
>> tb_init_done check in __trace_var(). Reason behind this is that all the
>> callers of __trace_var() (the main one being trace_var()), already
>> checks for tb_init_done themselves.
>>
>> In fact, the explicit check in the caller, is the (only) basis on which
>> one decides to call either trace_var() or __trace_var().
>>
>> This here is one of the call sites where I think the check is better
>> done in the caller.
>
> I admit, it is a bit confusing to require the caller to check
> tb_init_done. It adds more code and IHMO counterintuitive.
BTW, I am not responsible of this code, so if the REST maintainers are
happy with that then fine.
>
>>
>>>> diff --git a/xen/include/asm-arm/arm32/system.h b/xen/include/asm-
>>>> arm/arm32/system.h
>>>> index c617b40..20871ad 100644
>>>> --- a/xen/include/asm-arm/arm32/system.h
>>>> +++ b/xen/include/asm-arm/arm32/system.h
>>>> @@ -4,6 +4,8 @@
>>>>
>>>> #include <asm/arm32/cmpxchg.h>
>>>>
>>>> +#include <xen/trace.h>
>>>> +
>>>> #define local_irq_disable() asm volatile ( "cpsid i @
>>>> local_irq_disable\n" : : : "cc" )
>>>> #define local_irq_enable() asm volatile ( "cpsie i @
>>>> local_irq_enable\n" : : : "cc" )
>>>>
>>>> @@ -41,6 +43,16 @@ static inline int local_irq_is_enabled(void)
>>>> #define local_abort_enable() __asm__("cpsie a @ __sta\n" : : :
>>>> "memory", "cc")
>>>> #define local_abort_disable() __asm__("cpsid a @ __sta\n" : : :
>>>> "memory", "cc")
>>>>
>>>> +/* We do not support tracing (at all) yet */
>>>
>>> I know that some bits are missing for ARM, but the patch #5
>>> contradicts
>>> this comment as you have CONFIG_TRACE=y by default.
>>>
>> No sure what you mean. Tracing is de facto on by default right now,
>> despite it not being implemented for ARM. So what I'm doing is
>> basically keeping things as they are.
>>
>> If you think it should be off by default, well, let's talk about it...
>> but I'm not sure this is really what you are saying/asking.
>
> I mean that CONFIG_TRACE=y by default gives the impression that tracing
> is supported on ARM. This is not the case, hence your comment. So I
> think it should be off my default until we get the support. BTW, it is
> only a couple of simple patches but they never made upstreamed :/
>
>>>> +#define trace_irq_disable_ret() do { } while ( 0 )
>>>> +#define trace_irq_enable_ret() do { } while ( 0 )
>>>> +#define trace_irq_save_ret(_x) do { } while ( 0 )
>>>> +#define trace_irq_restore_ret(_x) do { } while ( 0 )
>>>> +#define _local_irq_disable() local_irq_disable()
>>>> +#define _local_irq_enable() local_irq_enable()
>>>> +#define _local_irq_save(_x) local_irq_save(_x)
>>>> +#define _local_irq_restore(_x) local_irq_restore(_x)
>>>> +
>>>
>>> This does not need to be duplicated in both asm-
>>> arm/arm{32,64}/system.h.
>>> You could directly implement them in asm-arm/system.h.
>>>
>> Ok, thanks.
>>
>>> Also, I would prefer if you stay consistent with x86. I.e non-
>>> underscore
>>> versions are calling the underscore versions and not the invert.
>>>
>> Well, I know it is counterintuitive, but it's the easiest way of
>> getting over this, i.e., without the need of a lot of stubs, and
>> touching as few existing code as possible.
>
> Can you detail here? The only things I was asking is to do:
>
> #define _local_irq_disable() ......
>
> #define local_irq_disable() _local_irq_disable.
>
> The callers would still use local_irq_disable() and the code would be
> consistent with x86.
>
> Cheers,
>
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 06/15] xen: trace IRQ enabling/disabling
2017-06-09 10:51 ` Julien Grall
2017-06-09 10:53 ` Julien Grall
@ 2017-06-09 10:55 ` George Dunlap
2017-06-09 11:00 ` Julien Grall
1 sibling, 1 reply; 68+ messages in thread
From: George Dunlap @ 2017-06-09 10:55 UTC (permalink / raw)
To: Julien Grall, Dario Faggioli, xen-devel
Cc: George Dunlap, Andrew Cooper, Stefano Stabellini,
Jennifer Herbert, Jan Beulich
On 09/06/17 11:51, Julien Grall wrote:
>
>
> On 07/06/17 16:22, Dario Faggioli wrote:
>> On Wed, 2017-06-07 at 12:16 +0100, Julien Grall wrote:
>>> Hi Dario,
>>>
>> Hi,
>>
>>> On 01/06/17 18:34, Dario Faggioli wrote:
>>>> diff --git a/xen/common/spinlock.c b/xen/common/spinlock.c
>>>> index 2a06406..33b903e 100644
>>>> --- a/xen/common/spinlock.c
>>>> +++ b/xen/common/spinlock.c
>>>> @@ -150,7 +150,9 @@ void _spin_lock(spinlock_t *lock)
>>>> void _spin_lock_irq(spinlock_t *lock)
>>>> {
>>>> ASSERT(local_irq_is_enabled());
>>>> - local_irq_disable();
>>>> + _local_irq_disable();
>>>> + if ( unlikely(tb_init_done) )
>>>
>>> __trace_var already contain a "if ( !tb_init_done )". It sounds
>>> pointless to do it twice, and also I think it is not obvious to
>>> understand the meaning of the check (i.e what is tb_init_done).
>>>
>>> Would it be possible to hide this check and avoid check tb_init_done
>>> twice?
>>>
>> I totally agree. And in fact, in another patch, I remove the
>> tb_init_done check in __trace_var(). Reason behind this is that all the
>> callers of __trace_var() (the main one being trace_var()), already
>> checks for tb_init_done themselves.
>>
>> In fact, the explicit check in the caller, is the (only) basis on which
>> one decides to call either trace_var() or __trace_var().
>>
>> This here is one of the call sites where I think the check is better
>> done in the caller.
>
> I admit, it is a bit confusing to require the caller to check
> tb_init_done. It adds more code and IHMO counterintuitive.
Because a number of the traces include a certain amount if "marshalling"
work ahead of time; there's no point in making space for a struct on the
stack, carefully reading and sorting all kinds if things to put into it,
only to have the data ignored because tracing isn't enabled (which is by
far the most common case).
-George
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 12/15] xen: trace tasklets
2017-06-01 17:35 ` [PATCH 12/15] xen: trace tasklets Dario Faggioli
@ 2017-06-09 10:59 ` Jan Beulich
2017-06-09 11:17 ` Dario Faggioli
0 siblings, 1 reply; 68+ messages in thread
From: Jan Beulich @ 2017-06-09 10:59 UTC (permalink / raw)
To: Dario Faggioli
Cc: Stefano Stabellini, George Dunlap, Andrew Cooper, Tim Deegan, xen-devel
>>> On 01.06.17 at 19:35, <dario.faggioli@citrix.com> wrote:
> --- a/xen/common/tasklet.c
> +++ b/xen/common/tasklet.c
> @@ -30,10 +30,87 @@ static DEFINE_PER_CPU(struct list_head, softirq_tasklet_list);
> /* Protects all lists and tasklet structures. */
> static DEFINE_SPINLOCK(tasklet_lock);
>
> +#ifdef CONFIG_TRACE_TASKLETS
> +static inline void trace_enqueue(const struct tasklet *t)
> +{
> + uint64_t addr;
> +
> + if ( likely(!tb_init_done) )
> + return;
> +
> + addr = (uint64_t)t->func;
> + __trace_var(TRC_XEN_TASKLET_ENQUEUE, 0, sizeof(addr), &addr);
> +}
> +static inline void trace_schedule(const struct tasklet *t)
> +{
> + struct {
> + uint64_t addr;
> + int16_t sched_on, is_sirq;
> + } d;
> +
> + if ( likely(!tb_init_done) )
> + return;
> +
> + d.addr = (uint64_t)t->func;
> + d.sched_on = t->scheduled_on;
> + d.is_sirq = t->is_softirq;
> + __trace_var(TRC_XEN_TASKLET_SCHEDULE, 1, sizeof(d), &d);
> +}
> +static inline void trace_work(const struct tasklet *t)
> +{
> + uint64_t addr;
> +
> + if ( likely(!tb_init_done) )
> + return;
> +
> + addr = (uint64_t)t->func;
> + __trace_var(TRC_XEN_TASKLET_WORK, 1, sizeof(addr), &addr);
> +}
> +static inline void trace_kill(const struct tasklet *t)
> +{
> + struct {
> + uint64_t addr;
> + int16_t sched_on, is_run;
> + } d;
> +
> + if ( likely(!tb_init_done) )
> + return;
> +
> + d.addr = (uint64_t)t->func;
> + d.sched_on = t->scheduled_on;
> + d.is_run = t->is_running;
> + __trace_var(TRC_XEN_TASKLET_KILL, 0, sizeof(d), &d);
> +}
> +static inline void trace_init(const struct tasklet *t)
> +{
> + struct {
> + uint64_t addr;
> + uint32_t is_sirq;
> + } d;
> +
> + if ( likely(!tb_init_done) )
> + return;
> +
> + d.addr = (uint64_t)t->func;
> + d.is_sirq = t->is_softirq;
> + __trace_var(TRC_XEN_TASKLET_INIT, 0, sizeof(d), &d);
> +}
> +#define trace_migrate() TRACE_0D(TRC_XEN_TASKLET_MIGR);
> +#else
> +#define trace_enqueue(t) do {} while ( 0 )
> +#define trace_schedule(t) do {} while ( 0 )
> +#define trace_work(t) do {} while ( 0 )
> +#define trace_kill(t) do {} while ( 0 )
> +#define trace_migrate() do {} while ( 0 )
> +#define trace_init(t) do {} while ( 0 )
> +#endif /* TRACE_TASKLETS */
Seeing how such additions add up, I think I'd prefer if you put them
into header files instead of cluttering source files this way. You
could have one such header per traceable component.
> @@ -178,6 +258,11 @@ static void migrate_tasklets_from_cpu(unsigned int cpu, struct list_head *list)
>
> spin_lock_irqsave(&tasklet_lock, flags);
>
> + if ( list_empty(list) )
> + goto out;
> +
> + trace_migrate();
> +
> while ( !list_empty(list) )
Two alternatives:
if ( !list_empty(list) )
trace_migrate();
(avoiding the goto) or
if ( list_empty(list) )
goto out;
trace_migrate();
do {
...
} while ( !list_empty(list) );
(avoiding the redundant check).
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 06/15] xen: trace IRQ enabling/disabling
2017-06-09 10:55 ` George Dunlap
@ 2017-06-09 11:00 ` Julien Grall
0 siblings, 0 replies; 68+ messages in thread
From: Julien Grall @ 2017-06-09 11:00 UTC (permalink / raw)
To: George Dunlap, Dario Faggioli, xen-devel
Cc: George Dunlap, Andrew Cooper, Stefano Stabellini,
Jennifer Herbert, Jan Beulich
On 09/06/17 11:55, George Dunlap wrote:
> On 09/06/17 11:51, Julien Grall wrote:
>>
>>
>> On 07/06/17 16:22, Dario Faggioli wrote:
>>> On Wed, 2017-06-07 at 12:16 +0100, Julien Grall wrote:
>>>> Hi Dario,
>>>>
>>> Hi,
>>>
>>>> On 01/06/17 18:34, Dario Faggioli wrote:
>>>>> diff --git a/xen/common/spinlock.c b/xen/common/spinlock.c
>>>>> index 2a06406..33b903e 100644
>>>>> --- a/xen/common/spinlock.c
>>>>> +++ b/xen/common/spinlock.c
>>>>> @@ -150,7 +150,9 @@ void _spin_lock(spinlock_t *lock)
>>>>> void _spin_lock_irq(spinlock_t *lock)
>>>>> {
>>>>> ASSERT(local_irq_is_enabled());
>>>>> - local_irq_disable();
>>>>> + _local_irq_disable();
>>>>> + if ( unlikely(tb_init_done) )
>>>>
>>>> __trace_var already contain a "if ( !tb_init_done )". It sounds
>>>> pointless to do it twice, and also I think it is not obvious to
>>>> understand the meaning of the check (i.e what is tb_init_done).
>>>>
>>>> Would it be possible to hide this check and avoid check tb_init_done
>>>> twice?
>>>>
>>> I totally agree. And in fact, in another patch, I remove the
>>> tb_init_done check in __trace_var(). Reason behind this is that all the
>>> callers of __trace_var() (the main one being trace_var()), already
>>> checks for tb_init_done themselves.
>>>
>>> In fact, the explicit check in the caller, is the (only) basis on which
>>> one decides to call either trace_var() or __trace_var().
>>>
>>> This here is one of the call sites where I think the check is better
>>> done in the caller.
>>
>> I admit, it is a bit confusing to require the caller to check
>> tb_init_done. It adds more code and IHMO counterintuitive.
>
> Because a number of the traces include a certain amount if "marshalling"
> work ahead of time; there's no point in making space for a struct on the
> stack, carefully reading and sorting all kinds if things to put into it,
> only to have the data ignored because tracing isn't enabled (which is by
> far the most common case).
At the moment all the callers have the semantic:
if ( !unlikely(tb_init_done) )
trace_irq_*
So why not hiding that in the macro trace_irq_disable/trace_irq_enable...?
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 12/15] xen: trace tasklets
2017-06-09 10:59 ` Jan Beulich
@ 2017-06-09 11:17 ` Dario Faggioli
2017-06-09 11:29 ` Jan Beulich
0 siblings, 1 reply; 68+ messages in thread
From: Dario Faggioli @ 2017-06-09 11:17 UTC (permalink / raw)
To: Jan Beulich
Cc: Stefano Stabellini, George Dunlap, Andrew Cooper, Tim Deegan, xen-devel
[-- Attachment #1.1: Type: text/plain, Size: 2149 bytes --]
On Fri, 2017-06-09 at 04:59 -0600, Jan Beulich wrote:
> > > > On 01.06.17 at 19:35, <dario.faggioli@citrix.com> wrote:
> > --- a/xen/common/tasklet.c
> > +++ b/xen/common/tasklet.c
> > @@ -30,10 +30,87 @@ static DEFINE_PER_CPU(struct list_head,
> > [...]
> > +#else
> >
> > +#define trace_enqueue(t) do {} while ( 0 )
> > +#define trace_schedule(t) do {} while ( 0 )
> > +#define trace_work(t) do {} while ( 0 )
> > +#define trace_kill(t) do {} while ( 0 )
> > +#define trace_migrate() do {} while ( 0 )
> > +#define trace_init(t) do {} while ( 0 )
> > +#endif /* TRACE_TASKLETS */
>
> Seeing how such additions add up, I think I'd prefer if you put them
> into header files instead of cluttering source files this way. You
> could have one such header per traceable component.
>
Right, good point.
As a matter of fact, the components dealt with in this series, have
their own header already, such as:
xen/include/xen/rcupdate.h
xen/include/xen/timer.h
xen/include/xen/softirq.h
xen/include/xen/tasklet.h
I guess I can put these there?
> > @@ -178,6 +258,11 @@ static void migrate_tasklets_from_cpu(unsigned
> > int cpu, struct list_head *list)
> >
> > spin_lock_irqsave(&tasklet_lock, flags);
> >
> > + if ( list_empty(list) )
> > + goto out;
> > +
> > + trace_migrate();
> > +
> > while ( !list_empty(list) )
>
> Two alternatives:
>
> if ( !list_empty(list) )
> trace_migrate();
>
> (avoiding the goto) or
>
> if ( list_empty(list) )
> goto out;
>
> trace_migrate();
>
> do {
> ...
> } while ( !list_empty(list) );
>
> (avoiding the redundant check).
>
Sure.
Thanks and Regards,
Dario
--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
[-- Attachment #2: Type: text/plain, Size: 127 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 12/15] xen: trace tasklets
2017-06-09 11:17 ` Dario Faggioli
@ 2017-06-09 11:29 ` Jan Beulich
0 siblings, 0 replies; 68+ messages in thread
From: Jan Beulich @ 2017-06-09 11:29 UTC (permalink / raw)
To: Dario Faggioli
Cc: Stefano Stabellini, George Dunlap, Andrew Cooper, Tim Deegan, xen-devel
>>> On 09.06.17 at 13:17, <dario.faggioli@citrix.com> wrote:
> On Fri, 2017-06-09 at 04:59 -0600, Jan Beulich wrote:
>> > > > On 01.06.17 at 19:35, <dario.faggioli@citrix.com> wrote:
>> > --- a/xen/common/tasklet.c
>> > +++ b/xen/common/tasklet.c
>> > @@ -30,10 +30,87 @@ static DEFINE_PER_CPU(struct list_head,
>> > [...]
>
>> > +#else
>> >
>> > +#define trace_enqueue(t) do {} while ( 0 )
>> > +#define trace_schedule(t) do {} while ( 0 )
>> > +#define trace_work(t) do {} while ( 0 )
>> > +#define trace_kill(t) do {} while ( 0 )
>> > +#define trace_migrate() do {} while ( 0 )
>> > +#define trace_init(t) do {} while ( 0 )
>> > +#endif /* TRACE_TASKLETS */
>>
>> Seeing how such additions add up, I think I'd prefer if you put them
>> into header files instead of cluttering source files this way. You
>> could have one such header per traceable component.
>>
> Right, good point.
>
> As a matter of fact, the components dealt with in this series, have
> their own header already, such as:
>
> xen/include/xen/rcupdate.h
> xen/include/xen/timer.h
> xen/include/xen/softirq.h
> xen/include/xen/tasklet.h
>
> I guess I can put these there?
No, please don't - these definitions don't need to be seen by
everyone including these headers. Instead I'd like to suggest
having a separate trace/ directory where they could all go (at
once eliminating the need to prefix their names with e.g.
trace_).
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 07/15] tools: tracing: handle IRQs on/off events in xentrace and xenalyze
2017-06-01 17:34 ` [PATCH 07/15] tools: tracing: handle IRQs on/off events in xentrace and xenalyze Dario Faggioli
@ 2017-06-13 15:58 ` George Dunlap
0 siblings, 0 replies; 68+ messages in thread
From: George Dunlap @ 2017-06-13 15:58 UTC (permalink / raw)
To: Dario Faggioli, xen-devel
Cc: George Dunlap, Andrew Cooper, Ian Jackson, Wei Liu, Jennifer Herbert
On 01/06/17 18:34, Dario Faggioli wrote:
> so the trace will show properly decoded info,
> rather than just a bunch of hex codes.
>
> Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
Acked-by: George Dunlap <george.dunlap@citrix.com>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 08/15] xen: trace RCU behavior
2017-06-01 17:34 ` [PATCH 08/15] xen: trace RCU behavior Dario Faggioli
2017-06-09 10:48 ` Jan Beulich
@ 2017-06-13 16:05 ` George Dunlap
1 sibling, 0 replies; 68+ messages in thread
From: George Dunlap @ 2017-06-13 16:05 UTC (permalink / raw)
To: Dario Faggioli, xen-devel
Cc: Stefano Stabellini, George Dunlap, Andrew Cooper, Tim Deegan,
Jan Beulich
On 01/06/17 18:34, Dario Faggioli wrote:
> Making it possible generate events showing the
> activity and the behavior of the RCU subsystem.
>
> Gate this with its specific Kconfig option (under
> CONFIG_TRACING), and keep it in disabled state by
> default.
> ---
> Cc: George Dunlap <George.Dunlap@eu.citrix.com>
> Cc: Andrew Cooper <andrew.cooper3@citrix.com>
> Cc: Jan Beulich <jbeulich@suse.com>
> Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> Cc: Stefano Stabellini <sstabellini@kernel.org>
> Cc: Tim Deegan <tim@xen.org>
> ---
> xen/Kconfig.debug | 8 ++++
> xen/common/rcupdate.c | 82 ++++++++++++++++++++++++++++++++++++++++----
> xen/include/public/trace.h | 14 ++++++++
> 3 files changed, 97 insertions(+), 7 deletions(-)
>
> diff --git a/xen/Kconfig.debug b/xen/Kconfig.debug
> index 81910c9..fd5cccc 100644
> --- a/xen/Kconfig.debug
> +++ b/xen/Kconfig.debug
> @@ -116,6 +116,14 @@ config TRACE_IRQSOFF
> Note that this comes with high overead and produces huge amount of
> tracing data.
>
> +config TRACE_RCU
> + bool "Trace RCU behavior" if EXPERT = "y"
> + default n
> + depends on TRACING
> + ---help---
> + Makes it possible generate events showing the activity and the
> + behavior of the RCU subsystem.
> +
> config VERBOSE_DEBUG
> bool "Verbose debug messages"
> default DEBUG
> diff --git a/xen/common/rcupdate.c b/xen/common/rcupdate.c
> index 8cc5a82..f160582 100644
> --- a/xen/common/rcupdate.c
> +++ b/xen/common/rcupdate.c
> @@ -92,6 +92,57 @@ static int qhimark = 10000;
> static int qlowmark = 100;
> static int rsinterval = 1000;
>
> +#ifdef CONFIG_TRACE_RCU
> +static inline void trace_call_rcu(void *func)
> +{
> + uint64_t addr = (uint64_t)func;
> +
> + if ( likely(!tb_init_done) )
> + return;
> +
> + __trace_var(TRC_XEN_RCU_CALL_RCU, 0, sizeof(addr), &addr);
> +}
> +static inline void trace_start_batch(const cpumask_t *m)
> +{
> + uint32_t mask[6];
> +
> + if ( likely(!tb_init_done) )
> + return;
> +
> + memset(mask, 0, sizeof(mask));
> + memcpy(mask, m, min(sizeof(mask), sizeof(cpumask_t)));
> + __trace_var(TRC_XEN_RCU_START_BATCH, 0, sizeof(mask), &mask);
Another possibility here would be to say something like:
size = min(sizeof(mask), sizeof(cpumask_t))
memcpy(mask, m, size)
__trace_var(..., 0, size, &mask)
That would result in a more efficient log on systems which, say, had
defied NR_CPUS to something relatively small for efficiency reasons; and
you wouldn't need to clear the unused bits.
Everything else looks good (with Jan's comments).
-George
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 09/15] tools: tracing: handle RCU events in xentrace and xenalyze
2017-06-01 17:34 ` [PATCH 09/15] tools: tracing: handle RCU events in xentrace and xenalyze Dario Faggioli
@ 2017-06-13 16:12 ` George Dunlap
0 siblings, 0 replies; 68+ messages in thread
From: George Dunlap @ 2017-06-13 16:12 UTC (permalink / raw)
To: Dario Faggioli, xen-devel; +Cc: George Dunlap, Wei Liu, Ian Jackson
On 01/06/17 18:34, Dario Faggioli wrote:
> Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
Looks good; you'll just have to adjust the START_BATCH code if you make
that a variable-sized record.
If that doesn't end up happening:
Reviewed-by: George Dunlap <george.dunlap@citrix.com>
-George
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 00/15] xen/tools: add tracing to various Xen subsystems
2017-06-01 17:33 [PATCH 00/15] xen/tools: add tracing to various Xen subsystems Dario Faggioli
` (15 preceding siblings ...)
2017-06-07 14:13 ` [PATCH 00/15] xen/tools: add tracing to various Xen subsystems Konrad Rzeszutek Wilk
@ 2017-06-13 16:34 ` George Dunlap
16 siblings, 0 replies; 68+ messages in thread
From: George Dunlap @ 2017-06-13 16:34 UTC (permalink / raw)
To: Dario Faggioli, xen-devel
Cc: Stefano Stabellini, Wei Liu, George Dunlap, Andrew Cooper,
Ian Jackson, Tim Deegan, Julien Grall, Jan Beulich,
Jennifer Herbert, Doug Goldstein
On 01/06/17 18:33, Dario Faggioli wrote:
> Hello,
>
> While chasing and dealing with bugs, over this last period, I've found myself
> augmenting Xen with quite a few new tracing capabilities, especially focusing
> on:
> - IRQ being disabled and (re)enabled (in addition to the already existing
> tracing of IRQ related activity that we have);
> - RCU;
> - softirqs (I think I sent a preliminary version of this, long ago, but can't
> be sure);
> - tasklets;
> - timers;
>
> And, apart from the first 4 patches (which are random, but still tracing
> related, of course, improvements), this is what this patch series does: it adds
> tracing to the Xen susystems listed above.
>
> That happens, one subsystem after another, in patches 6 to 15.
>
> Patch 5 deserves special mention. In fact, now that we have Kconfig, I thought
> it could be a nice thing to make it possible to select, at build config time,
> whether we want tracing or not, in the hypervisor (like, for instance, we do
> for performance counters).
>
> To be honest, my goal was to be able to compile tracing off, and run
> benchmarks, to assess how much of a overhead tracing introduces, but then I
> decided it was worth doing this properly, and now sending it. I am still
> running those benchmarks. Preliminary results seems to be showing that having
> tracing support in the hypervisor does not (when it's disabled, of course)
> introduce too much overhead. Still, I think it could be useful, to people that
> wants a very specificly tailored version of Xen (embedded, small footprint,
> etc.), to be able to rip it off nice and easily (e.g., like OpenXT guys did for
> schedulers).
>
> Of course, I will report here what I find, when benchmarks will finish running.
> (In the meantime, patch 5 has some data about .text section shrinking in its
> changelog.)
>
> I also thought, for similar reasons, that it would have been good to be able to
> also individually enable or disable the new tracing I'm introducing with this
> series. This potentially applies even to the tracing we already have in tree
> (e.g., one may want to have tracing compiled in, for everything except that for
> scheduling), and I'd be up for working on this. However, in this series, I am
> touching really hot paths (with the exception, maybe, of RCU), and so I think
> it is even more important to be able to disable tracing for them, for minimum
> overhead.
>
> For instance, the IRQ enabling and disabling tracing, I find it really really
> usable for understanding certain class of behavior, and, with some scripting
> and some more gnuplot "magic" (which I'll also share shortly) we can also use
> it to automatically measure and graph for how long interrupt are kept disabled,
> in various places within the hypervisor (Jennifer has done a similar analysis
> for XenServer, a while back). But it is very invasive, so you want to be able
> to turn it on and off.
> As said, I don't have all the result I need to present conclusions, but what I
> see in preliminary data is that, although the tracing we currently have in Xen
> is not too bad, performance wise, this new IRQs on/off tracing does have an
> impact on performance, just for being there in the code (i.e., even when it is
> there in the code but is *disabled*).
>
> Of course, all this fine grain control of tracing options is hidden under
> XEN_CONFIG_EXPERT, and all the new tracing is disabled by default.
>
> There is a branch with this series here:
> git://xenbits.xen.org/people/dariof/xen.git rel/tracing/xen-internals
> http://xenbits.xen.org/gitweb/?p=people/dariof/xen.git;a=shortlog;h=refs/heads/rel/tracing/xen-internals
>
> And Travis is happy about it:
> https://travis-ci.org/fdario/xen/builds/238421024
>
> Let me know what you think.
>
> Thanks and Regards,
> Dario
> ---
> Dario Faggioli (15):
> xen: in do_softirq() sample smp_processor_id() once and for all.
> xen: tracing: avoid checking tb_init_done multiple times.
> xen/tools: tracing: several improvements on IRQs tracing
> tools: xenalyze: fix dumping of PM_IDLE events.
> xen: make it possible to disable tracing in Kconfig.
> xen: trace IRQ enabling/disabling
> tools: tracing: handle IRQs on/off events in xentrace and xenalyze
> xen: trace RCU behavior
> tools: tracing: handle RCU events in xentrace and xenalyze
> xen: trace softirqs
> tools: tracing: handle RCU events in xentrace and xenalyze
> xen: trace tasklets
> tools: tracing: handle tasklets events in xentrace and xenalyze
> xen: trace timers
> tools: tracing: handle timers events in xentrace and xenalyze
Patches 10-15 look good to me too:
Acked-by: George Dunlap <george.dunlap@citrix.com>
-George
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 68+ messages in thread
end of thread, other threads:[~2017-06-13 16:34 UTC | newest]
Thread overview: 68+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-06-01 17:33 [PATCH 00/15] xen/tools: add tracing to various Xen subsystems Dario Faggioli
2017-06-01 17:33 ` [PATCH 01/15] xen: in do_softirq() sample smp_processor_id() once and for all Dario Faggioli
2017-06-07 14:38 ` Jan Beulich
2017-06-08 14:12 ` George Dunlap
2017-06-08 14:20 ` George Dunlap
2017-06-08 14:42 ` Jan Beulich
2017-06-01 17:33 ` [PATCH 02/15] xen: tracing: avoid checking tb_init_done multiple times Dario Faggioli
2017-06-01 17:53 ` Andrew Cooper
2017-06-01 23:08 ` Dario Faggioli
2017-06-07 14:46 ` Jan Beulich
2017-06-07 15:55 ` Dario Faggioli
2017-06-07 16:06 ` Jan Beulich
2017-06-08 14:34 ` George Dunlap
2017-06-08 14:37 ` George Dunlap
2017-06-01 17:33 ` [PATCH 03/15] xen/tools: tracing: several improvements on IRQs tracing Dario Faggioli
2017-06-01 18:02 ` Andrew Cooper
2017-06-01 23:12 ` Dario Faggioli
2017-06-07 15:05 ` Jan Beulich
2017-06-07 15:45 ` Dario Faggioli
2017-06-07 15:58 ` Jan Beulich
2017-06-08 14:53 ` George Dunlap
2017-06-08 15:34 ` Jan Beulich
2017-06-08 14:59 ` George Dunlap
2017-06-01 17:34 ` [PATCH 04/15] tools: xenalyze: fix dumping of PM_IDLE events Dario Faggioli
2017-06-08 15:06 ` George Dunlap
2017-06-01 17:34 ` [PATCH 05/15] xen: make it possible to disable tracing in Kconfig Dario Faggioli
2017-06-01 18:43 ` Andrew Cooper
2017-06-07 11:01 ` Julien Grall
2017-06-07 15:14 ` Jan Beulich
2017-06-08 15:16 ` George Dunlap
2017-06-08 15:35 ` Jan Beulich
2017-06-08 15:37 ` George Dunlap
2017-06-08 15:44 ` Jan Beulich
2017-06-08 15:17 ` George Dunlap
2017-06-01 17:34 ` [PATCH 06/15] xen: trace IRQ enabling/disabling Dario Faggioli
2017-06-01 19:08 ` Andrew Cooper
2017-06-01 23:42 ` Dario Faggioli
2017-06-08 15:51 ` George Dunlap
2017-06-08 16:05 ` Jan Beulich
2017-06-07 11:16 ` Julien Grall
2017-06-07 15:22 ` Dario Faggioli
2017-06-09 10:51 ` Julien Grall
2017-06-09 10:53 ` Julien Grall
2017-06-09 10:55 ` George Dunlap
2017-06-09 11:00 ` Julien Grall
2017-06-08 16:01 ` George Dunlap
2017-06-08 16:11 ` Dario Faggioli
2017-06-09 10:41 ` Jan Beulich
2017-06-01 17:34 ` [PATCH 07/15] tools: tracing: handle IRQs on/off events in xentrace and xenalyze Dario Faggioli
2017-06-13 15:58 ` George Dunlap
2017-06-01 17:34 ` [PATCH 08/15] xen: trace RCU behavior Dario Faggioli
2017-06-09 10:48 ` Jan Beulich
2017-06-13 16:05 ` George Dunlap
2017-06-01 17:34 ` [PATCH 09/15] tools: tracing: handle RCU events in xentrace and xenalyze Dario Faggioli
2017-06-13 16:12 ` George Dunlap
2017-06-01 17:34 ` [PATCH 10/15] xen: trace softirqs Dario Faggioli
2017-06-09 10:51 ` Jan Beulich
2017-06-01 17:34 ` [PATCH 11/15] tools: tracing: handle RCU events in xentrace and xenalyze Dario Faggioli
2017-06-01 17:35 ` [PATCH 12/15] xen: trace tasklets Dario Faggioli
2017-06-09 10:59 ` Jan Beulich
2017-06-09 11:17 ` Dario Faggioli
2017-06-09 11:29 ` Jan Beulich
2017-06-01 17:35 ` [PATCH 13/15] tools: tracing: handle tasklets events in xentrace and xenalyze Dario Faggioli
2017-06-01 17:35 ` [PATCH 14/15] xen: trace timers Dario Faggioli
2017-06-01 17:35 ` [PATCH 15/15] tools: tracing: handle timers events in xentrace and xenalyze Dario Faggioli
2017-06-07 14:13 ` [PATCH 00/15] xen/tools: add tracing to various Xen subsystems Konrad Rzeszutek Wilk
2017-06-08 16:45 ` Dario Faggioli
2017-06-13 16:34 ` George Dunlap
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).