Perf ABI versioning

Perf ABI versioning

[Thomas Renninger]

Hi,

as I recently cleaned up the power events with
some format/abi adjustings...

I realized that it was rather easy to adjust
the stuff in drivers/tools/perf, because there
are some nice facilities to query which events
are available.

I looked at pytimechart, there it's not that nice
and convenient would be some kind of version number.

pytimechart can parse traces which are simply copied
away from /sys/kernel/debug/trace and to check for
available events and their format, one would have to
do a separate, ugly to implement, iteration of parsing.

Would it make sense to introduce a trace
ABI version number?
It would be enough to increase it by one every time
any event (don't know about other perf facilities) is
added, removed or modified.
Ideally the modifications are tracked with a short
description in a file like Documentation/trace/format.changes

Untested, but the patch at the end should simply add
a version number on top of
/sys/kernel/debug/trace file

While this should be enough for the kernel, perf record
would need to include it into it's binary perf.data format
as well?

Then userspace apps could easily test whether supported
events would be included at all, whether only a subset of
supported events were made available and which format they have.

Does this make sense?

Thanks,

    Thomas

Re: Perf ABI versioning

[Thomas Renninger]

and the tiny (untested) patch which should
add a version on top of /sys/kernel/debug/trace:

---
 Documentation/trace/events.changes |    6 ++++++
 kernel/trace/trace.c               |    1 +
 kernel/trace/trace.h               |    2 ++
 3 files changed, 9 insertions(+), 0 deletions(-)

diff --git a/Documentation/trace/events.changes b/Documentation/trace/events.changes
new file mode 100644
index 0000000..bd29abc
--- /dev/null
+++ b/Documentation/trace/events.changes
@@ -0,0 +1,6 @@
+Version 2:
+<Modifications are tracked here>
+
+Version 1:
+Initial version.
+Available events and their formats:
diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
index dc53ecb..602885a 100644
--- a/kernel/trace/trace.c
+++ b/kernel/trace/trace.c
@@ -2043,6 +2043,7 @@ void trace_default_header(struct seq_file *m)
 {
 	struct trace_iterator *iter = m->private;
 
+	seq_puts(m, "# Perf ABI Version %d\n", PERF_ABI_VERSION);
 	if (iter->iter_flags & TRACE_FILE_LAT_FMT) {
 		/* print nothing if the buffers are empty */
 		if (trace_empty(iter))
diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h
index 9021f8c..c23a7a2 100644
--- a/kernel/trace/trace.h
+++ b/kernel/trace/trace.h
@@ -13,6 +13,8 @@
 #include <linux/trace_seq.h>
 #include <linux/ftrace_event.h>
 
+#define PERF_ABI_VERSION 1
+
 enum trace_type {
 	__TRACE_FIRST_TYPE = 0,
 

Re: Perf ABI versioning

[Pierre Tardy]

On Mon, Jan 24, 2011 at 5:04 PM, Thomas Renninger <trenn@suse.de> wrote:
> Hi,
>
> as I recently cleaned up the power events with
> some format/abi adjustings...
>
> I realized that it was rather easy to adjust
> the stuff in drivers/tools/perf, because there
> are some nice facilities to query which events
> are available.
>
> I looked at pytimechart, there it's not that nice
> and convenient would be some kind of version number.

pytimechart does pattern matching to be able to work with different
version of the ftrace formats.
For a same trace type, it will try different trace format, until it
finds a match.
e.g.:
    ('irq_handler_entry', 'irq=%d handler=%s','irq','name'),
    ('irq_handler_entry', 'irq=%d name=%s','irq','name'),

The problem of this approach is obviously that you have to modify
pytimechart each time there is a new format, it cannot just look at
the trace format and just ignore new fields that it dont care of.

Please also note, that pytimechart also have a perf-python backend
that may not have such issues in dealing with new events fields, but
is actually sensible to event fields renaming.

> Would it make sense to introduce a trace
> ABI version number?
So as soon as pytimechart is the only user of this versionning, I'm
not sure it is really needed.

Regards,
Pierre

Re: Perf ABI versioning

[Jean Pihet]

Hi Thomas,

On Mon, Jan 24, 2011 at 5:04 PM, Thomas Renninger <trenn@suse.de> wrote:
> Hi,
>
> as I recently cleaned up the power events with
> some format/abi adjustings...
>
> I realized that it was rather easy to adjust
> the stuff in drivers/tools/perf, because there
> are some nice facilities to query which events
> are available.
>
> I looked at pytimechart, there it's not that nice
> and convenient would be some kind of version number.
>
> pytimechart can parse traces which are simply copied
> away from /sys/kernel/debug/trace and to check for
> available events and their format, one would have to
> do a separate, ugly to implement, iteration of parsing.
>
> Would it make sense to introduce a trace
> ABI version number?
> It would be enough to increase it by one every time
> any event (don't know about other perf facilities) is
> added, removed or modified.
> Ideally the modifications are tracked with a short
> description in a file like Documentation/trace/format.changes
>
> Untested, but the patch at the end should simply add
> a version number on top of
> /sys/kernel/debug/trace file

If needed the ABI versioning should at least contain a major and minor
number, i.e. 'x.y'.

The big question IMO is if it is needed or not. Is this a real problem
in the userland? Are the apps always fitting the kernel ABI in a
packaged distro?

>
> While this should be enough for the kernel, perf record
> would need to include it into it's binary perf.data format
> as well?
>
> Then userspace apps could easily test whether supported
> events would be included at all, whether only a subset of
> supported events were made available and which format they have.
>
> Does this make sense?
>
> Thanks,
>
>    Thomas
>

Regards,
Jean
--
To unsubscribe from this list: send the line "unsubscribe linux-trace-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Perf ABI versioning

[Frederic Weisbecker]

On Mon, Jan 24, 2011 at 06:28:54PM +0100, Jean Pihet wrote:
> Hi Thomas,
> 
> On Mon, Jan 24, 2011 at 5:04 PM, Thomas Renninger <trenn@suse.de> wrote:
> > Hi,
> >
> > as I recently cleaned up the power events with
> > some format/abi adjustings...
> >
> > I realized that it was rather easy to adjust
> > the stuff in drivers/tools/perf, because there
> > are some nice facilities to query which events
> > are available.
> >
> > I looked at pytimechart, there it's not that nice
> > and convenient would be some kind of version number.
> >
> > pytimechart can parse traces which are simply copied
> > away from /sys/kernel/debug/trace and to check for
> > available events and their format, one would have to
> > do a separate, ugly to implement, iteration of parsing.
> >
> > Would it make sense to introduce a trace
> > ABI version number?
> > It would be enough to increase it by one every time
> > any event (don't know about other perf facilities) is
> > added, removed or modified.
> > Ideally the modifications are tracked with a short
> > description in a file like Documentation/trace/format.changes
> >
> > Untested, but the patch at the end should simply add
> > a version number on top of
> > /sys/kernel/debug/trace file
> 
> If needed the ABI versioning should at least contain a major and minor
> number, i.e. 'x.y'.
> 
> The big question IMO is if it is needed or not. Is this a real problem
> in the userland? Are the apps always fitting the kernel ABI in a
> packaged distro?

This may help solving some tracepoint ABI issues that we have.
But let me add more tracing people and LKML in Cc in an answer to the explanation
of the idea, to discuss that more widely.

Re: Perf ABI versioning

[Mathieu Desnoyers]

[ added Ingo, Steven, Thomas and Peter to CC list ]

* Thomas Renninger (trenn@suse.de) wrote:
> Hi,
> 
> as I recently cleaned up the power events with
> some format/abi adjustings...
> 
> I realized that it was rather easy to adjust
> the stuff in drivers/tools/perf, because there
> are some nice facilities to query which events
> are available.
> 
> I looked at pytimechart, there it's not that nice
> and convenient would be some kind of version number.
> 
> pytimechart can parse traces which are simply copied
> away from /sys/kernel/debug/trace and to check for
> available events and their format, one would have to
> do a separate, ugly to implement, iteration of parsing.
> 
> Would it make sense to introduce a trace
> ABI version number?
> It would be enough to increase it by one every time
> any event (don't know about other perf facilities) is
> added, removed or modified.
> Ideally the modifications are tracked with a short
> description in a file like Documentation/trace/format.changes
> 
> Untested, but the patch at the end should simply add
> a version number on top of
> /sys/kernel/debug/trace file
> 
> While this should be enough for the kernel, perf record
> would need to include it into it's binary perf.data format
> as well?
> 
> Then userspace apps could easily test whether supported
> events would be included at all, whether only a subset of
> supported events were made available and which format they have.
> 
> Does this make sense?

Hi Thomas,

I think it might make more sense to make pytimechart read the
/sys/kernel/debug/tracing/events/*/*/format files to find out if the
print-format for the events are as expected. I think this is much more adapted
to a development scheme where subsystem maintainers are responsible for changing
the content of their own events than a single global version number that needs
to be bumped synchronously for everything.

I would agree, though, about the need for a version number on the way
/sys/kernel/debug/tracing/events exports the binary event type and text-output
formatting information: in order to cope with changes to the way this metadata
is exported, we'd need such a versioning number.

But again, trying to version every tiny change to the fields per se won't scale
with the development process given that we pull instrumentation changes from
many different repository concurrently. The field layout information is there to
deal with this problem, pytimechart should perhaps start using it.

Thanks,

Mathieu

> 
> Thanks,
> 
>     Thomas

-- 
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com

Re: Perf ABI versioning

[Frederic Weisbecker]

(Adding LKML and more tracing people in Cc)

On Mon, Jan 24, 2011 at 05:04:00PM +0100, Thomas Renninger wrote:
> Hi,
> 
> as I recently cleaned up the power events with
> some format/abi adjustings...
> 
> I realized that it was rather easy to adjust
> the stuff in drivers/tools/perf, because there
> are some nice facilities to query which events
> are available.
> 
> I looked at pytimechart, there it's not that nice
> and convenient would be some kind of version number.
> 
> pytimechart can parse traces which are simply copied
> away from /sys/kernel/debug/trace and to check for
> available events and their format, one would have to
> do a separate, ugly to implement, iteration of parsing.
> 
> Would it make sense to introduce a trace
> ABI version number?
> It would be enough to increase it by one every time
> any event (don't know about other perf facilities) is
> added, removed or modified.
> Ideally the modifications are tracked with a short
> description in a file like Documentation/trace/format.changes
> 
> Untested, but the patch at the end should simply add
> a version number on top of
> /sys/kernel/debug/trace file
> 
> While this should be enough for the kernel, perf record
> would need to include it into it's binary perf.data format
> as well?
> 
> Then userspace apps could easily test whether supported
> events would be included at all, whether only a subset of
> supported events were made available and which format they have.
> 
> Does this make sense?

This may be generally useful to help dealing with tracepoint ABI
changes.

But instead of a global tracing ABI number, I would rather suggest
one number per tracepoint subsystem (sched, power, etc...).

Ideally it would be per event, but sometimes those events tend to be
renamed or a whole tracepoint subsystem refactored (see workqueue
lately). Hence it might be better per subsystem.

Now these numbers can be increased in the case of new fields added,
fields removed, fields renamed, new tracepoints, removed tracepoints,
renamed tracepoints.

But I think we shouldn't do that for pure ascii format changes, because
we created the format to avoid fixing that on the rock. I know that
would help Pytimechart, and it's a precious tool that I wish we can help
the development. But we shouldn't encourage the pattern match way of
parsing events, instead of using the format file.

Also, tracking the changes in the Documentation is going to be a
nightmare. I suggest developers who want to dig into details of
an ABI change to check the code or the format themselves.

And BTW we can probably provide more granularity in the versioning
with a major number for ABI changes that may break tools (field or
tracepoint removal, rename, etc...) and a minor number for changes
that probably won't break tools but could bring them more informations:
new tracepoint fields, new tracepoints, etc... so we can tell the tools
about that.

Hm?

Re: Perf ABI versioning

[Peter Zijlstra]

On Mon, 2011-01-24 at 21:34 +0100, Frederic Weisbecker wrote:
> 
> But instead of a global tracing ABI number, I would rather suggest
> one number per tracepoint subsystem (sched, power, etc...).
> 
> Ideally it would be per event, but sometimes those events tend to be
> renamed or a whole tracepoint subsystem refactored (see workqueue
> lately). Hence it might be better per subsystem. 

What's wrong with what we have? the /format file is pretty unique to
function as a version number of you use a hash over it.

Re: Perf ABI versioning

[Frederic Weisbecker]

On Mon, Jan 24, 2011 at 09:39:03PM +0100, Peter Zijlstra wrote:
> On Mon, 2011-01-24 at 21:34 +0100, Frederic Weisbecker wrote:
> > 
> > But instead of a global tracing ABI number, I would rather suggest
> > one number per tracepoint subsystem (sched, power, etc...).
> > 
> > Ideally it would be per event, but sometimes those events tend to be
> > renamed or a whole tracepoint subsystem refactored (see workqueue
> > lately). Hence it might be better per subsystem. 
> 
> What's wrong with what we have? the /format file is pretty unique to
> function as a version number of you use a hash over it.

Yeah we could in fact use it to find if fields have been added,
removed. Right, all in one that looks enough to me.

Re: Perf ABI versioning

[Thomas Renninger]

Hi,

Pierre may correct me if I am wrong, even pytimechart is not big,
I realized that I am not that good in python code reading...

On Monday 24 January 2011 21:12:25 Mathieu Desnoyers wrote:
> [ added Ingo, Steven, Thomas and Peter to CC list ]
> 
> I would agree, though, about the need for a version number on the way
> /sys/kernel/debug/tracing/events exports the binary event type and text-output
> formatting information: in order to cope with changes to the way this metadata
> is exported, we'd need such a versioning number.
pytimechart simply works on reading /sys/kernel/debug/trace
The big advantage is that you can copy things away to another machine.
I expect the same concept is with perf record and perf.data.
Theoretically you can attach perf.data to a bug report and everybody (considering
the userspace tool is new and maintained enough) can process the data.

I do not know the binary format of this file, but a version number would be
an easy way for userspace tools to figure out which events are supported
and in which format.

> But again, trying to version every tiny change to the fields per se won't scale
> with the development process given that we pull instrumentation changes from
> many different repository concurrently. The field layout information is there to
> deal with this problem, pytimechart should perhaps start using it.
I see that there are quite some changes, but why wouldn't it scale?
Even if every git commit automatically would increases the version, a userspace
tool would only pick out very specific versions it cares about:

if (version > 2000) {
    /* event x got introduced, try to catch it as well */
}
if (version < 3000) {
   /* event y did not have field z */
}
...

I would not introduce a major.minor just for the sake of it.

About Frederic's comment:
> Also, tracking the changes in the Documentation is going to be a
> nightmare. I suggest developers who want to dig into details of
> an ABI change to check the code or the format themselves.
If you want an ABI and userspace tools being able to cope with
it across kernel versions , there should be some documentation as well.
Let it only be one sentence containing the event to grep for. You can then
catch the details from the git commit id.

       Thomas

Re: Perf ABI versioning

[Ingo Molnar]

* Frederic Weisbecker <fweisbec@gmail.com> wrote:

> This may be generally useful to help dealing with tracepoint ABI changes.
> 
> But instead of a global tracing ABI number, I would rather suggest one number per 
> tracepoint subsystem (sched, power, etc...).

Nooooooooooo ... !!! :-)

Please lets stop this madness before it gets too serious: we dont do ABI version 
numbering in Linux, full stop.

We use 'natural' ABIs where the lack of an ABI component triggers some sort of 
clean, finegrained error. Like a -EINVAL on a not-yet-implemented ABI component, a 
non-existent file entry, or -ENOSYS on a non-existent syscall.

Such a design is arbitrarily backportable or forward portable, it's extensible and 
it is actually maintainable.

In the ABI version numbering direction lies Windows madness ...

Thanks,

	Ingo

Re: Perf ABI versioning

[Thomas Renninger]

On Monday 24 January 2011 22:28:38 Ingo Molnar wrote:
> 
> * Frederic Weisbecker <fweisbec@gmail.com> wrote:
> 
> > This may be generally useful to help dealing with tracepoint ABI changes.
> > 
> > But instead of a global tracing ABI number, I would rather suggest one number per 
> > tracepoint subsystem (sched, power, etc...).
> 
> Nooooooooooo ... !!! :-)
> 
> Please lets stop this madness before it gets too serious: we dont do ABI version 
> numbering in Linux, full stop.
Ok... :)
 
> We use 'natural' ABIs where the lack of an ABI component triggers some sort of 
> clean, finegrained error. Like a -EINVAL on a not-yet-implemented ABI component, a 
> non-existent file entry, or -ENOSYS on a non-existent syscall.
If the whole stuff matures, someone might think about a nice concept to be able
to shift around perf.data binaries which can be processed by userspace tools,
independent of the kernel version running below on which the perf.data
was produced.

Was just an idea, I agree that it's far away from being practical,
please ignore the thread...

Sorry for the noise,

   Thomas

Re: Perf ABI versioning

[Arnaldo Carvalho de Melo]

Em Mon, Jan 24, 2011 at 10:49:13PM +0100, Thomas Renninger escreveu:
> On Monday 24 January 2011 22:28:38 Ingo Molnar wrote:
> > * Frederic Weisbecker <fweisbec@gmail.com> wrote:
> > 
> > > This may be generally useful to help dealing with tracepoint ABI changes.
> > > 
> > > But instead of a global tracing ABI number, I would rather suggest one number per 
> > > tracepoint subsystem (sched, power, etc...).
> > 
> > Nooooooooooo ... !!! :-)
> > 
> > Please lets stop this madness before it gets too serious: we dont do ABI version 
> > numbering in Linux, full stop.
> Ok... :)
>  
> > We use 'natural' ABIs where the lack of an ABI component triggers some sort of 
> > clean, finegrained error. Like a -EINVAL on a not-yet-implemented ABI component, a 
> > non-existent file entry, or -ENOSYS on a non-existent syscall.
> If the whole stuff matures, someone might think about a nice concept to be able
> to shift around perf.data binaries which can be processed by userspace tools,
> independent of the kernel version running below on which the perf.data
> was produced.

We try to do that already, things like sample_id_all that is a feature
present on a new kernel are capability queried by trying to ask the
kernel for it and if it fails, cope with that and do as before it, in
some cases warning the user if the fallback can lead to inacuracies.

Shifting around perf.data files accross arquitectures and OS/vendor
releases is an everyday goal.

I.e. the new perf binary works with older kernels.

We also try hard to keep the latest tools building on older kernels.

Using the /format file hashed as peterz said is elegant, fine grained
content based ABI number :-)

- Arnaldo

Re: Perf ABI versioning

[Frederic Weisbecker]

On Mon, Jan 24, 2011 at 10:28:38PM +0100, Ingo Molnar wrote:
> 
> * Frederic Weisbecker <fweisbec@gmail.com> wrote:
> 
> > This may be generally useful to help dealing with tracepoint ABI changes.
> > 
> > But instead of a global tracing ABI number, I would rather suggest one number per 
> > tracepoint subsystem (sched, power, etc...).
> 
> Nooooooooooo ... !!! :-)
> 
> Please lets stop this madness before it gets too serious: we dont do ABI version 
> numbering in Linux, full stop.
> 
> We use 'natural' ABIs where the lack of an ABI component triggers some sort of 
> clean, finegrained error. Like a -EINVAL on a not-yet-implemented ABI component, a 
> non-existent file entry, or -ENOSYS on a non-existent syscall.
> 
> Such a design is arbitrarily backportable or forward portable, it's extensible and 
> it is actually maintainable.
> 
> In the ABI version numbering direction lies Windows madness ...
> 
> Thanks,
> 
> 	Ingo

Yeah. But the tracepoint case looked to me quite special as sometimes
the ABI doesn't evolved smoothly like in the scheme you describe (see
the power events case).

OTOH, checking the format file if we have lost or gained fields must be enough
already to, indeed, avoid a versioning madness :)

Re: Perf ABI versioning

[Thomas Renninger]

On Monday 24 January 2011 22:57:55 Arnaldo Carvalho de Melo wrote:
> Em Mon, Jan 24, 2011 at 10:49:13PM +0100, Thomas Renninger escreveu:
... 
> Using the /format file hashed as peterz said is elegant, fine grained
> content based ABI number :-)
Ok thanks, I should have had a closer look at the binary format.

It may make sense at some time to provide a library so that other tools
can make use of the most convenient functions in tools/perf/utils/*
as well?

I'll have a look at the powertop perf implementations I wanted to
look at anyway and try to understand the stuff a bit better.

Thanks,

       Thomas

Re: Perf ABI versioning

[Frederic Weisbecker]

On Mon, Jan 24, 2011 at 11:30:01PM +0100, Thomas Renninger wrote:
> On Monday 24 January 2011 22:57:55 Arnaldo Carvalho de Melo wrote:
> > Em Mon, Jan 24, 2011 at 10:49:13PM +0100, Thomas Renninger escreveu:
> ... 
> > Using the /format file hashed as peterz said is elegant, fine grained
> > content based ABI number :-)
> Ok thanks, I should have had a closer look at the binary format.
> 
> It may make sense at some time to provide a library so that other tools
> can make use of the most convenient functions in tools/perf/utils/*
> as well?

I personally wish the other way around: having Pytimechart merged in perf :)
If I remember correctly, the problem was that we don't have support for
trace_printk with perf?

But sure if more out of tree tools show up and want to use tracepoints,
we can think about librarizing it.

(Even though my secret hope is that
we build a nice one tool that can answer most tracing needs together rather
than many scattered ad hoc pieces.)

Re: Perf ABI versioning

[Arnaldo Carvalho de Melo]

Em Mon, Jan 24, 2011 at 11:46:12PM +0100, Frederic Weisbecker escreveu:
> On Mon, Jan 24, 2011 at 11:30:01PM +0100, Thomas Renninger wrote:
> > On Monday 24 January 2011 22:57:55 Arnaldo Carvalho de Melo wrote:
> > > Em Mon, Jan 24, 2011 at 10:49:13PM +0100, Thomas Renninger escreveu:
> > ... 
> > > Using the /format file hashed as peterz said is elegant, fine grained
> > > content based ABI number :-)
> > Ok thanks, I should have had a closer look at the binary format.
> > 
> > It may make sense at some time to provide a library so that other tools
> > can make use of the most convenient functions in tools/perf/utils/*
> > as well?
> 
> I personally wish the other way around: having Pytimechart merged in perf :)
> If I remember correctly, the problem was that we don't have support for
> trace_printk with perf?
> 
> But sure if more out of tree tools show up and want to use tracepoints,
> we can think about librarizing it.
> 
> (Even though my secret hope is that
> we build a nice one tool that can answer most tracing needs together rather
> than many scattered ad hoc pieces.)

Yes, there is this conflict of tools/perf/ people coming from the
kernel, where we can change APIs as we see fit as long as we pay the
price of going over the users to fix them up.

And the desire of having something that can be used by projects living
outside the kernel source repo.

There are at least two efforts underway that are paving the way to
having something usable by third parties, Borislav Petkov's patchseries
to have some liblk/libtrace and my work for .39 to have a python perf.so
binding.

I think we should try as hard as possible to get things in tools/ so
that we have as long an experience in trying to get good abstractions as
possible before we decide on a v1 "outsiders API" :-)

- Arnaldo

P.S. Most of my work is in my perf/core branch in my git.k.o tree,
both mine and Borislavs are a wersearch away :)