[PATCH] tracing: Export key trace event symbols

[PATCH] tracing: Export key trace event symbols

[Ron Rechenmacher]

If symbols are not exported, modules can no longer register additional
(module specified) tracepoints like they use to be able to (i.e linux-3.15.x).
Somewhere on or about commit de7b2973903c6cc50b31ee5682a69b2219b9919d
(Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Date:   Tue Apr 8 17:26:21 2014 -0400
tracepoint: Use struct pointer instead of name hash for reg/unreg tracepoints)
modules which attempted to register additional tracing functions would
get "Unknown symbol" errors. For example: "... Unknown symbol
__tracepoint_sched_switch (err 0)"
Symbols can be exported using the kernel's EXPORT_TRACEPOINT_SYMBOL_GPL macro
to allow modules to once again register their own tracing functions (for at
least some key points in the kernel as provided by this patch).

Signed-off-by: Ron Rechenmacher <ron@fnal.gov>
Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=96051
---
  kernel/irq/handle.c               |    6 ++++++
  kernel/softirq.c                  |    6 ++++++
  kernel/trace/trace_sched_switch.c |    5 +++++
  kernel/trace/trace_syscalls.c     |    6 ++++++
  4 files changed, 23 insertions(+), 0 deletions(-)

diff --git a/kernel/irq/handle.c b/kernel/irq/handle.c
index 470d08c..a98d763 100644
--- a/kernel/irq/handle.c
+++ b/kernel/irq/handle.c
@@ -20,6 +20,12 @@

  #include "internals.h"

+/*
+ * Allow modules to register additional trace routines
+ */
+EXPORT_TRACEPOINT_SYMBOL_GPL(irq_handler_entry);
+EXPORT_TRACEPOINT_SYMBOL_GPL(irq_handler_exit);
+
  /**
   * handle_bad_irq - handle spurious and unhandled irqs
   * @irq:       the interrupt number
diff --git a/kernel/softirq.c b/kernel/softirq.c
index fca82c3..af6fa2e 100644
--- a/kernel/softirq.c
+++ b/kernel/softirq.c
@@ -52,6 +52,12 @@ irq_cpustat_t irq_stat[NR_CPUS] ____cacheline_aligned;
  EXPORT_SYMBOL(irq_stat);
  #endif

+/*
+ * Allow modules to register additional trace routines
+ */
+EXPORT_TRACEPOINT_SYMBOL_GPL(softirq_entry);
+EXPORT_TRACEPOINT_SYMBOL_GPL(softirq_exit);
+
  static struct softirq_action softirq_vec[NR_SOFTIRQS] __cacheline_aligned_in_smp;

  DEFINE_PER_CPU(struct task_struct *, ksoftirqd);
diff --git a/kernel/trace/trace_sched_switch.c b/kernel/trace/trace_sched_switch.c
index 7e62c0a..2ea1a6a 100644
--- a/kernel/trace/trace_sched_switch.c
+++ b/kernel/trace/trace_sched_switch.c
@@ -14,6 +14,11 @@

  #include "trace.h"

+/*
+ * Allow modules to register additional trace routines
+ */
+EXPORT_TRACEPOINT_SYMBOL_GPL(sched_switch);
+
  static struct trace_array      *ctx_trace;
  static int __read_mostly       tracer_enabled;
  static int                     sched_ref;
diff --git a/kernel/trace/trace_syscalls.c b/kernel/trace/trace_syscalls.c
index ee7b5a0..6c3bbd0 100644
--- a/kernel/trace/trace_syscalls.c
+++ b/kernel/trace/trace_syscalls.c
@@ -9,6 +9,12 @@
  #include "trace_output.h"
  #include "trace.h"

+/*
+ * Allow modules to register additional trace routines
+ */
+EXPORT_TRACEPOINT_SYMBOL_GPL(sys_enter);
+EXPORT_TRACEPOINT_SYMBOL_GPL(sys_exit);
+
  static DEFINE_MUTEX(syscall_trace_lock);
  static int sys_refcount_enter;
  static int sys_refcount_exit;

Re: [PATCH] tracing: Export key trace event symbols

[Christoph Hellwig]

On Mon, Apr 20, 2015 at 04:38:11PM -0500, Ron Rechenmacher wrote:
> If symbols are not exported, modules can no longer register additional
> (module specified) tracepoints like they use to be able to (i.e linux-3.15.x).
> Somewhere on or about commit de7b2973903c6cc50b31ee5682a69b2219b9919d
> (Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> Date:   Tue Apr 8 17:26:21 2014 -0400
> tracepoint: Use struct pointer instead of name hash for reg/unreg tracepoints)
> modules which attempted to register additional tracing functions would
> get "Unknown symbol" errors. For example: "... Unknown symbol
> __tracepoint_sched_switch (err 0)"
> Symbols can be exported using the kernel's EXPORT_TRACEPOINT_SYMBOL_GPL macro
> to allow modules to once again register their own tracing functions (for at
> least some key points in the kernel as provided by this patch).

Which (in-tree) module fails with this?  I don't think anyone should
actually register a symbol.

Re: [PATCH] tracing: Export key trace event symbols

[Ron Rechenmacher]

Christoph Hellwig wrote on 04/21/15 01:10:
>
> Which (in-tree) module fails with this?  I don't think anyone should
> actually register a symbol.
>

I see you (Christoph Hellwig) have asked this question in a similar context
(see https://patches.linaro.org/28821/).
This question does not seem to make sense because:
1) the external module is not registering a _symbol_ but more
precisely a tracepoint _function_ as the whole tracepoint system allows for
_multiple_ functions to be called for each tracepoint declared in the kernel.
2) It's not the point that an in-tree module would fail.  Again, the tracepoint
system allows for _multiple_functions_ to be defined/registered for each tracepoint
and _in_the_earlier_kernels_(i.e. 3.10.x and many others),_external_modules_could_
_register_ one or more _additional_functions_ to be called.

IF you're specifically saying that external modules should not register additional
tracepoint functions, my question would simply be: why do you think this?

To give you an example of the usefulness of continuing to allow this (continuation
from earlier kernels): the kernel scheduling has a tracepoint defined; of course a
critical operation for any kernel. I use to be able to insert a module which would
collect my own statistics on when and what switching was going on on what CPU cores.
I can think of many other potential reasons that this would be useful for external
modules. To think that tracepoints would only be useful for in-tree development is,
perhaps, (not meaning to offend) short sighted.

-- 
Ron Rechenmacher
Engineer
Fermi National Accelerator Laboratory
Batavia, IL  60510

Re: [PATCH] tracing: Export key trace event symbols

[Ron Rechenmacher]

I see in the reference I mentioned below (https://patches.linaro.org/28821/),
and in the current mm source, that some tracepoint symbols are already EXPORTed,
but not _GPL. I do not know the fine points between "GPL-ed" and "non-GPL-ed" symbol
exporting.  Would it make a difference if my patch proposed non-GPL exporting?

Ron Rechenmacher wrote on 04/21/15 07:04:
>
>
> Christoph Hellwig wrote on 04/21/15 01:10:
>>
>> Which (in-tree) module fails with this?  I don't think anyone should
>> actually register a symbol.
>>
>
> I see you (Christoph Hellwig) have asked this question in a similar context
> (see https://patches.linaro.org/28821/).
> This question does not seem to make sense because:
> 1) the external module is not registering a _symbol_ but more
> precisely a tracepoint _function_ as the whole tracepoint system allows for
> _multiple_ functions to be called for each tracepoint declared in the kernel.
> 2) It's not the point that an in-tree module would fail.  Again, the tracepoint
> system allows for _multiple_functions_ to be defined/registered for each tracepoint
> and _in_the_earlier_kernels_(i.e. 3.10.x and many others),_external_modules_could_
> _register_ one or more _additional_functions_ to be called.
>
> IF you're specifically saying that external modules should not register additional
> tracepoint functions, my question would simply be: why do you think this?
>
> To give you an example of the usefulness of continuing to allow this (continuation
> from earlier kernels): the kernel scheduling has a tracepoint defined; of course a
> critical operation for any kernel. I use to be able to insert a module which would
> collect my own statistics on when and what switching was going on on what CPU cores.
> I can think of many other potential reasons that this would be useful for external
> modules. To think that tracepoints would only be useful for in-tree development is,
> perhaps, (not meaning to offend) short sighted.
>

-- 
Ron Rechenmacher
Engineer
Fermi National Accelerator Laboratory
Batavia, IL  60510

Re: [PATCH] tracing: Export key trace event symbols

[Steven Rostedt]

On Tue, 21 Apr 2015 07:19:30 -0500
Ron Rechenmacher <ron@fnal.gov> wrote:

> I see in the reference I mentioned below (https://patches.linaro.org/28821/),
> and in the current mm source, that some tracepoint symbols are already EXPORTed,
> but not _GPL. I do not know the fine points between "GPL-ed" and "non-GPL-ed" symbol
> exporting.  Would it make a difference if my patch proposed non-GPL exporting?

The only reason there's non GPL exports for tracepoints is because
tracepoints were used in header files, and called by static inlines
like kfree() and such that were already exported non-gpl.

I believe we removed all tracepoints from static inlines and headers
because they were causing other problems (you don't want tracepoint
code inlined all over the place, it bloats the kernel).

I may look into cleaning up the kernel and removing the
EXPORT_TRACEPOINT_SYMBOL().

-- Steve

Re: [PATCH] tracing: Export key trace event symbols

[Christoph Hellwig]

Hi Ron,

the Linux kernel is not a shared library, but an integrated project
that happens to be split into loadable modules.

Please send your module that uses the tracepoints and we can start
discussing if it makes sense to exports the symbols for it, or if
we can implement it in a better way.

Re: [PATCH] tracing: Export key trace event symbols

[Ron Rechenmacher]

Hi Christoph,

My module, and information about it, can be found at:
https://cdcvs.fnal.gov/redmine/projects/trace
https://cdcvs.fnal.gov/redmine/projects/trace/repository/show/src_module
with the header at
https://cdcvs.fnal.gov/redmine/projects/trace/repository/raw/include/trace.h

It seems that you are the person that I have to convince in order to get
my patch accepted. Is this true?

Can you tell me how:
./kernel/trace/power-traces.c:17:EXPORT_TRACEPOINT_SYMBOL_GPL(power_start);
./kernel/trace/power-traces.c:19:EXPORT_TRACEPOINT_SYMBOL_GPL(cpu_idle);
should be allowed (how it fits with your definition of the Linux kernel below) and
my proposed EXPORTS not?
I hope that you will not suggest removing the above 2 exports.

Thanks,
Ron

Christoph Hellwig wrote on 04/21/15 07:22:
> Hi Ron,
>
> the Linux kernel is not a shared library, but an integrated project
> that happens to be split into loadable modules.
>
> Please send your module that uses the tracepoints and we can start
> discussing if it makes sense to exports the symbols for it, or if
> we can implement it in a better way.
>

-- 
Ron Rechenmacher
Engineer
Fermi National Accelerator Laboratory
Batavia, IL  60510

Re: [PATCH] tracing: Export key trace event symbols

[Christoph Hellwig]

On Tue, Apr 21, 2015 at 08:13:24AM -0500, Ron Rechenmacher wrote:
> Hi Christoph,
> 
> My module, and information about it, can be found at:
> https://cdcvs.fnal.gov/redmine/projects/trace
> https://cdcvs.fnal.gov/redmine/projects/trace/repository/show/src_module
> with the header at
> https://cdcvs.fnal.gov/redmine/projects/trace/repository/raw/include/trace.h

And now send a patch including a rationale on why we need it.  I haven't
looked deep enough, but so far I can't see any real value add in it.

> 
> It seems that you are the person that I have to convince in order to get
> my patch accepted. Is this true?

In the end you'll need to convince a maintainer to apply it.

> Can you tell me how:
> ./kernel/trace/power-traces.c:17:EXPORT_TRACEPOINT_SYMBOL_GPL(power_start);

I can't find a tracepoint with that name in the current tree.

> ./kernel/trace/power-traces.c:19:EXPORT_TRACEPOINT_SYMBOL_GPL(cpu_idle);

This one is used by the potentially modular intel_idle (now cpuids)
driver, so it's exported for a reason.

> should be allowed (how it fits with your definition of the Linux kernel below) and
> my proposed EXPORTS not?
> I hope that you will not suggest removing the above 2 exports.

It's perfectly fine to remove unused exports, and we do it regularly

Re: [PATCH] tracing: Export key trace event symbols

[Ron Rechenmacher]

But why export anything _GPL in the first place?


Christoph Hellwig wrote on 04/21/15 08:23:
> On Tue, Apr 21, 2015 at 08:13:24AM -0500, Ron Rechenmacher wrote:
>> Hi Christoph,
>>
>> My module, and information about it, can be found at:
>> https://cdcvs.fnal.gov/redmine/projects/trace
>> https://cdcvs.fnal.gov/redmine/projects/trace/repository/show/src_module
>> with the header at
>> https://cdcvs.fnal.gov/redmine/projects/trace/repository/raw/include/trace.h
>
> And now send a patch including a rationale on why we need it.  I haven't
> looked deep enough, but so far I can't see any real value add in it.
>
>>
>> It seems that you are the person that I have to convince in order to get
>> my patch accepted. Is this true?
>
> In the end you'll need to convince a maintainer to apply it.
>
>> Can you tell me how:
>> ./kernel/trace/power-traces.c:17:EXPORT_TRACEPOINT_SYMBOL_GPL(power_start);
>
> I can't find a tracepoint with that name in the current tree.
>
>> ./kernel/trace/power-traces.c:19:EXPORT_TRACEPOINT_SYMBOL_GPL(cpu_idle);
>
> This one is used by the potentially modular intel_idle (now cpuids)
> driver, so it's exported for a reason.
>
>> should be allowed (how it fits with your definition of the Linux kernel below) and
>> my proposed EXPORTS not?
>> I hope that you will not suggest removing the above 2 exports.
>
> It's perfectly fine to remove unused exports, and we do it regularly
>

-- 
Ron Rechenmacher
Engineer
Fermi National Accelerator Laboratory
Batavia, IL  60510

Re: [PATCH] tracing: Export key trace event symbols

[Steven Rostedt]

On Tue, 21 Apr 2015 08:26:26 -0500
Ron Rechenmacher <ron@fnal.gov> wrote:

> But why export anything _GPL in the first place?
> 

To make it clear that they are derivatives of the kernel. An export is
just to allow modules access to the functions, they were not added for
out of tree modules. Out of tree modules just happen to be lucky to
have them available.

Note, if you can present a good case to why they should be exported,
then we should have no problem exporting them. But I just looked at
your code, and I have some questions about it.

EXPORT_SYMBOL( traceControl_p );
EXPORT_SYMBOL( traceEntries_p );
EXPORT_SYMBOL( traceNamLvls_p );
EXPORT_SYMBOL( trace_allow_printk );

You have non GPL exported functions. Why? Is this used by non GPL code?

MODULE_LICENSE("GPL"); /* Get rid of taint message by declaring code as GPL */

That comment is very telling.

Maybe I'm reading into things, but until we understand exactly why you
need these symbols exported, we wont exported them. There's no kernel
ABI that we must stick to. Only the user space ABI is what we keep
stable.

-- Steve

Re: [PATCH] tracing: Export key trace event symbols

[Ron Rechenmacher]

Thanks for looking at the code and providing the feedback.

Using EXPORT_SYMBOL was a mistake. I just changed them to EXPORT_SYMBOL_GPL.
I've been doing this sort of "tracing" (which I consider the "minimum level of
complexity" tracing) for about 30 years. I've never made any money on this, nor do
I care to. For the last 25 years, I've worked at a US Dept. of Energy research facility
and consider all the stuff I do to be in the public domain. I, in no way, want to suggest
or even care if other want to use it or not. I won't even suggest that you might need it.
It has allowed me to do great work/research and I will patch the kernel manually it need be.
But if there is anything else I should do to the code to make is more public, please let me
know.

Exactly why I _need_ these symbols exported?:

When data acquistion systems I develop or am involved with at Fermilab, have near-realtime
performance issues, _my_ tracing system is what _I_ use. There is a user-space component to
it that can be used with or without the kernel module. Using the module allows me to
see interactions with the Linux OS.

Sure, there are other tools -- none which provide me "the minimum level of complexity"
( About once a year I take a week or 2 and learn the latest of what's available. )

There has been times in the past, when I've had a major experiment collaboration
load a patched kernel, but there is a certain level of inconvenience associated with that.


Currently, the major experiments are using a derivative of Redhat EL6 and there is not
a problem with creating and loading my module. I'm optimistic that there will not
be a problem with EL7 which has a base kernel derived form Linux-3.10.

I am, however, doing research on high speed networking and we are using the Linux-3.16.1
kernel, which I am currently patching.  In the future, it would be nice not to have to
patch the kernel.

The kernel currently exports:
      EXPORT_TRACEPOINT_SYMBOL_GPL(power_start)
and  EXPORT_TRACEPOINT_SYMBOL_GPL(cpu_idle)

I understand that "Only the user space ABI is what we keep stable" and that before
commit de7b2973903c (tracepoint: Use struct pointer instead of name hash for reg/unreg
tracepoints) it was just "lucky" that I could have my module work.
But, I can image how these (the above EXPORTs) would be very useful for external
modules to help with custom power management. And, I suspect that if they went away,
someone would complain.

I would like to have my proposed patch included to provide similar
usefulness for my custom tracing (for the sake of science).

Please accept my patch.

Thanks for you consideration and if there is specific help/suggestion you can offer to
get my patch accepted, please let me know.

Thanks again for the discussion.

--Ron


Steven Rostedt wrote on 04/21/15 08:53:
> On Tue, 21 Apr 2015 08:26:26 -0500
> Ron Rechenmacher <ron@fnal.gov> wrote:
>
>> But why export anything _GPL in the first place?
>>
>
> To make it clear that they are derivatives of the kernel. An export is
> just to allow modules access to the functions, they were not added for
> out of tree modules. Out of tree modules just happen to be lucky to
> have them available.
>
> Note, if you can present a good case to why they should be exported,
> then we should have no problem exporting them. But I just looked at
> your code, and I have some questions about it.
>
> EXPORT_SYMBOL( traceControl_p );
> EXPORT_SYMBOL( traceEntries_p );
> EXPORT_SYMBOL( traceNamLvls_p );
> EXPORT_SYMBOL( trace_allow_printk );
>
> You have non GPL exported functions. Why? Is this used by non GPL code?
>
> MODULE_LICENSE("GPL"); /* Get rid of taint message by declaring code as GPL */
>
> That comment is very telling.
>
> Maybe I'm reading into things, but until we understand exactly why you
> need these symbols exported, we wont exported them. There's no kernel
> ABI that we must stick to. Only the user space ABI is what we keep
> stable.
>
> -- Steve
>

-- 
Ron Rechenmacher
Engineer
Fermi National Accelerator Laboratory
Batavia, IL  60510

Re: [PATCH] tracing: Export key trace event symbols

[Steven Rostedt]

On Tue, 21 Apr 2015 10:00:25 -0500
Ron Rechenmacher <ron@fnal.gov> wrote:

> Thanks for looking at the code and providing the feedback.
> 
> Using EXPORT_SYMBOL was a mistake. I just changed them to EXPORT_SYMBOL_GPL.
> I've been doing this sort of "tracing" (which I consider the "minimum level of
> complexity" tracing) for about 30 years. I've never made any money on this, nor do
> I care to. For the last 25 years, I've worked at a US Dept. of Energy research facility
> and consider all the stuff I do to be in the public domain. I, in no way, want to suggest
> or even care if other want to use it or not. I won't even suggest that you might need it.
> It has allowed me to do great work/research and I will patch the kernel manually it need be.
> But if there is anything else I should do to the code to make is more public, please let me
> know.

Heh, OK, I believe you :-)  it's just that we need to be careful about
this, as Christoph is even in the middle of litigation about GPL abuses.

> 
> Exactly why I _need_ these symbols exported?:
> 
> When data acquistion systems I develop or am involved with at Fermilab, have near-realtime
> performance issues, _my_ tracing system is what _I_ use. There is a user-space component to
> it that can be used with or without the kernel module. Using the module allows me to
> see interactions with the Linux OS.

I understand the _my_ tracing system. Before ftrace, I used logdev. Not
quite 30 years, but it started sometime in '98.

> 
> Sure, there are other tools -- none which provide me "the minimum level of complexity"
> ( About once a year I take a week or 2 and learn the latest of what's available. )

If you have any suggestions to make the in kernel tracer better, please
feel free to make comments.

> 
> There has been times in the past, when I've had a major experiment collaboration
> load a patched kernel, but there is a certain level of inconvenience associated with that.
> 
> 
> Currently, the major experiments are using a derivative of Redhat EL6 and there is not
> a problem with creating and loading my module. I'm optimistic that there will not
> be a problem with EL7 which has a base kernel derived form Linux-3.10.
> 
> I am, however, doing research on high speed networking and we are using the Linux-3.16.1
> kernel, which I am currently patching.  In the future, it would be nice not to have to
> patch the kernel.
> 
> The kernel currently exports:
>       EXPORT_TRACEPOINT_SYMBOL_GPL(power_start)
> and  EXPORT_TRACEPOINT_SYMBOL_GPL(cpu_idle)
> 
> I understand that "Only the user space ABI is what we keep stable" and that before
> commit de7b2973903c (tracepoint: Use struct pointer instead of name hash for reg/unreg
> tracepoints) it was just "lucky" that I could have my module work.
> But, I can image how these (the above EXPORTs) would be very useful for external
> modules to help with custom power management. And, I suspect that if they went away,
> someone would complain.
> 
> I would like to have my proposed patch included to provide similar
> usefulness for my custom tracing (for the sake of science).
> 
> Please accept my patch.

OK, lets see what you are doing in your patch:

+/*
+ * Allow modules to register additional trace routines
+ */
+EXPORT_TRACEPOINT_SYMBOL_GPL(irq_handler_entry);
+EXPORT_TRACEPOINT_SYMBOL_GPL(irq_handler_exit);
+

Instead of just saying "Allow modules to register additional trace
routines", please explain what its used for. What are you measuring
externally? Just the timings of the interrupts?

I'm also curious to know why ftrace isn't good enough.

Maybe the better solution is to add something to ftrace that does what
your tracer currently does?

-- Steve

> 
> Thanks for you consideration and if there is specific help/suggestion you can offer to
> get my patch accepted, please let me know.
> 
> Thanks again for the discussion.

Re: [PATCH] tracing: Export key trace event symbols

[Ron Rechenmacher]

Steven Rostedt wrote on 04/21/15 10:49:
[snip]
>
> OK, lets see what you are doing in your patch:

Thanks for helping/discussing!

>
> +/*
> + * Allow modules to register additional trace routines
> + */
> +EXPORT_TRACEPOINT_SYMBOL_GPL(irq_handler_entry);
> +EXPORT_TRACEPOINT_SYMBOL_GPL(irq_handler_exit);
> +
>
> Instead of just saying "Allow modules to register additional trace
> routines", please explain what its used for. What are you measuring
> externally? Just the timings of the interrupts?

How about "Allow for module (internal or external) to register trace function which
shows (or measures) interactions with user-space executables more directly.  Knowing
when interrupts happen and how they can subtly impact user-space application timing
is important in some application."

>
> I'm also curious to know why ftrace isn't good enough.

ftrace (if one considers ltt-ng) is probably capable of doing what my trace does
except that the timestamp is not TOD (Time Of Day) -- but probably/maybe could
be made to do so???

It's a matter of complexity or "start up cost" (or entry barrier).

I'm not the best at articulating all there is to TRACE succinctly, but lets
start with the fact that the user-space component of TRACE is basically just a header
file that defines the TRACE printf-like macro and handles the initialization which is
basically mmap-ing a file (as a trace buffer (with control structure)) and is,
for the most part, Unix OS independent.

The TRACE module, in addition to registering TRACE compatible tracing functions,
also creates a virtual file that the user-space TRACE-enable program(s) can mmap just
like another regular file (it doesn't know it's a kernel virtual file).
[As a side point, it's just for convenience that one or more user-space programs are
configured (via env. var.) to use the kernel virtual file as the buffer; they could use
any other regular file or each program could use a separate file.]

The module becomes critical when tracing OS key interactions (scheduling and interrupts)
with the userspace application(s) is necessary.


>
> Maybe the better solution is to add something to ftrace that does what
> your tracer currently does?

Although there is significant overlap between my TRACE and ftrace, the key
design criteria is such that trying to make ftrace do what is desired of TRACE
would be impractical.  Ftrace is more of a developer tool while TRACE is more of
an end-user utility, where simplicity of use is paramount.

Thanks for your help,
Ron

PS. Below, I've included a portion of a print out from the user-space TRACE control
program printing the buffer and showing the exact impact that an interrupt
has on a program looping.  Just a contrived example.

>
> -- Steve
>
>>
>> Thanks for you consideration and if there is specific help/suggestion you can offer to
>> get my patch accepted, please let me know.
>>
>> Thanks again for the discussion.
/home/ron/work/tracePrj/trace
cluck :^) TRACE_SHOW=HxNTiICLR
--04/21_16:42:24--
/home/ron/work/tracePrj/trace
cluck :^) tshow 10000 | tdelta -stats | { for xx in 1 2;do read h; echo "$h";done;tail;}
idx           us_tod     delta   tid TID cpu lv r msg
---- ---------------- --------- ----- --- --- -- - -----------------------------
9996 1429652195997224         1 40744  18  17  0 . Hello 9996. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
5000000000 15
9997 1429652195997223         1 40744  18  17  0 . Hello 9997. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
5000000000 15
9998 1429652195997222         1 40744  18  17  0 . Hello 9998. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
5000000000 15
9999 1429652195997221         1 40744  18  17  0 . Hello 9999. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
5000000000 15
cpu="0"
                   min       -13
                   max        17
                   tot      7872
                   ave 0.7872787
                   cnt      9999
--04/21_16:42:27--
/home/ron/work/tracePrj/trace
cluck :^) tshow 10000 | tdelta -stats | { for xx in 1 2;do read h; echo "$h";done;grep -C15 '2        17';}
idx           us_tod     delta   tid TID cpu lv r msg
---- ---------------- --------- ----- --- --- -- - -----------------------------
9438 1429652195997676         1 40744  18  17  0 . Hello 9470. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
5000000000 15
9439 1429652195997676         0 40744  18  17  0 . Hello 9471. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
5000000000 15
9440 1429652195997675         1 40744  18  17  0 . Hello 9472. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
5000000000 15
9441 1429652195997674         1 40744  18  17  0 . Hello 9473. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
5000000000 15
9442 1429652195997673         1 40744  18  17  0 . Hello 9474. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
5000000000 15
9443 1429652195997672         1 40744  18  17  0 . Hello 9475. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
5000000000 15
9444 1429652195997672         0 40744  18  17  0 . Hello 9476. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
5000000000 15
9445 1429652195997671         1 40744  18  17  0 . Hello 9477. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
5000000000 15
9446 1429652195997670         1 40744  18  17  0 . Hello 9478. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
5000000000 15
9447 1429652195997667         3 40744   0  17 28 . sirqexit: cpu=17 vec_nr=9
9448 1429652195997664         3 40744   0  17 27 . sirqenter: cpu=17 vec_nr=9
9449 1429652195997662         2 40744   0  17 28 . sirqexit: cpu=17 vec_nr=1
9450 1429652195997660         2 40744   0  17 27 . sirqenter: cpu=17 vec_nr=1
9451 1429652195997656         4     0   0  13 28 . sirqexit: cpu=13 vec_nr=9
9452 1429652195997669       -13 40744  18  17  0 . Hello 9479. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
5000000000 15
9453 1429652195997652        17     0   0  13 27 1 sirqenter: cpu=13 vec_nr=9
9454 1429652195997652         0 40744  18  17  0 . Hello 9480. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
5000000000 15
9455 1429652195997651         1 40744  18  17  0 . Hello 9481. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
5000000000 15
9456 1429652195997649         2     0   0  13 28 1 sirqexit: cpu=13 vec_nr=1
9457 1429652195997649         0 40744  18  17  0 . Hello 9482. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
5000000000 15
9458 1429652195997648         1     0   0  12 28 . sirqexit: cpu=12 vec_nr=9
9459 1429652195997648         0 40744  18  17  0 . Hello 9483. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
5000000000 15
9460 1429652195997647         1 40744  18  17  0 . Hello 9484. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
5000000000 15
9461 1429652195997646         1 40744  18  17  0 . Hello 9485. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
5000000000 15
9462 1429652195997645         1     0   0  13 27 1 sirqenter: cpu=13 vec_nr=1
9463 1429652195997645         0 40744  18  17  0 1 Hello 9486. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
5000000000 15
9464 1429652195997644         1     0   0  12 27 . sirqenter: cpu=12 vec_nr=9
9465 1429652195997643         1 40744  18  17  0 . Hello 9487. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
5000000000 15
9466 1429652195997642         1 40744  18  17  0 . Hello 9488. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
5000000000 15
9467 1429652195997642         0     0   0  12 28 . sirqexit: cpu=12 vec_nr=1
9468 1429652195997641         1 40744  18  17  0 . Hello 9489. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
5000000000 15
--04/21_17:15:52--



-- 
Ron Rechenmacher
Engineer
Fermi National Accelerator Laboratory
Batavia, IL  60510

Re: [PATCH] tracing: Export key trace event symbols

[Steven Rostedt]

[ Added a bunch of people that use perf ;-) ]

On Tue, 21 Apr 2015 17:23:45 -0500
Ron Rechenmacher <ron@fnal.gov> wrote:

> 
> Steven Rostedt wrote on 04/21/15 10:49:
> [snip]
> >
> > OK, lets see what you are doing in your patch:
> 
> Thanks for helping/discussing!
> 
> >
> > +/*
> > + * Allow modules to register additional trace routines
> > + */
> > +EXPORT_TRACEPOINT_SYMBOL_GPL(irq_handler_entry);
> > +EXPORT_TRACEPOINT_SYMBOL_GPL(irq_handler_exit);
> > +
> >
> > Instead of just saying "Allow modules to register additional trace
> > routines", please explain what its used for. What are you measuring
> > externally? Just the timings of the interrupts?
> 
> How about "Allow for module (internal or external) to register trace function which
> shows (or measures) interactions with user-space executables more directly.  Knowing
> when interrupts happen and how they can subtly impact user-space application timing
> is important in some application."
> 
> >
> > I'm also curious to know why ftrace isn't good enough.
> 
> ftrace (if one considers ltt-ng) is probably capable of doing what my trace does
> except that the timestamp is not TOD (Time Of Day) -- but probably/maybe could
> be made to do so???
> 
> It's a matter of complexity or "start up cost" (or entry barrier).
> 
> I'm not the best at articulating all there is to TRACE succinctly, but lets
> start with the fact that the user-space component of TRACE is basically just a header
> file that defines the TRACE printf-like macro and handles the initialization which is
> basically mmap-ing a file (as a trace buffer (with control structure)) and is,
> for the most part, Unix OS independent.

Heh, that sounds like the perf interface. Have you thought about using
that instead?

Look at what PowerTop does: https://01.org/powertop

You can use the perf interface to mmap a memory buffer and read the
data in its raw format. To parse the data, you need to read the format
files in /sys/kernel/debug/tracing/events, but there's a library that
does that. Unfortunately, that library isn't in distros yet (we really
need to get that done). But you can just copy the code from the kernel
source tree from tools/lib/traceevents/ and use that as perf does (as
well as powertop does too).

The format files tells you how to get the data from the raw binary
format.

If you use the perf interface, you can have userspace tools do custom
things with the events as they happen. This may make your utility more
robust in the long run.

I'd be happy to help you understand how to use the traceevent library. I
don't use the perf interface, but you can get examples from the
PowerTop code.

-- Steve


> 
> The TRACE module, in addition to registering TRACE compatible tracing functions,
> also creates a virtual file that the user-space TRACE-enable program(s) can mmap just
> like another regular file (it doesn't know it's a kernel virtual file).
> [As a side point, it's just for convenience that one or more user-space programs are
> configured (via env. var.) to use the kernel virtual file as the buffer; they could use
> any other regular file or each program could use a separate file.]
> 
> The module becomes critical when tracing OS key interactions (scheduling and interrupts)
> with the userspace application(s) is necessary.
> 
> 
> >
> > Maybe the better solution is to add something to ftrace that does what
> > your tracer currently does?
> 
> Although there is significant overlap between my TRACE and ftrace, the key
> design criteria is such that trying to make ftrace do what is desired of TRACE
> would be impractical.  Ftrace is more of a developer tool while TRACE is more of
> an end-user utility, where simplicity of use is paramount.
> 
> Thanks for your help,
> Ron
> 
> PS. Below, I've included a portion of a print out from the user-space TRACE control
> program printing the buffer and showing the exact impact that an interrupt
> has on a program looping.  Just a contrived example.
> 
> >
> > -- Steve
> >
> >>
> >> Thanks for you consideration and if there is specific help/suggestion you can offer to
> >> get my patch accepted, please let me know.
> >>
> >> Thanks again for the discussion.
> /home/ron/work/tracePrj/trace
> cluck :^) TRACE_SHOW=HxNTiICLR
> --04/21_16:42:24--
> /home/ron/work/tracePrj/trace
> cluck :^) tshow 10000 | tdelta -stats | { for xx in 1 2;do read h; echo "$h";done;tail;}
> idx           us_tod     delta   tid TID cpu lv r msg
> ---- ---------------- --------- ----- --- --- -- - -----------------------------
> 9996 1429652195997224         1 40744  18  17  0 . Hello 9996. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
> 5000000000 15
> 9997 1429652195997223         1 40744  18  17  0 . Hello 9997. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
> 5000000000 15
> 9998 1429652195997222         1 40744  18  17  0 . Hello 9998. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
> 5000000000 15
> 9999 1429652195997221         1 40744  18  17  0 . Hello 9999. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
> 5000000000 15
> cpu="0"
>                    min       -13
>                    max        17
>                    tot      7872
>                    ave 0.7872787
>                    cnt      9999
> --04/21_16:42:27--
> /home/ron/work/tracePrj/trace
> cluck :^) tshow 10000 | tdelta -stats | { for xx in 1 2;do read h; echo "$h";done;grep -C15 '2        17';}
> idx           us_tod     delta   tid TID cpu lv r msg
> ---- ---------------- --------- ----- --- --- -- - -----------------------------
> 9438 1429652195997676         1 40744  18  17  0 . Hello 9470. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
> 5000000000 15
> 9439 1429652195997676         0 40744  18  17  0 . Hello 9471. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
> 5000000000 15
> 9440 1429652195997675         1 40744  18  17  0 . Hello 9472. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
> 5000000000 15
> 9441 1429652195997674         1 40744  18  17  0 . Hello 9473. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
> 5000000000 15
> 9442 1429652195997673         1 40744  18  17  0 . Hello 9474. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
> 5000000000 15
> 9443 1429652195997672         1 40744  18  17  0 . Hello 9475. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
> 5000000000 15
> 9444 1429652195997672         0 40744  18  17  0 . Hello 9476. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
> 5000000000 15
> 9445 1429652195997671         1 40744  18  17  0 . Hello 9477. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
> 5000000000 15
> 9446 1429652195997670         1 40744  18  17  0 . Hello 9478. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
> 5000000000 15
> 9447 1429652195997667         3 40744   0  17 28 . sirqexit: cpu=17 vec_nr=9
> 9448 1429652195997664         3 40744   0  17 27 . sirqenter: cpu=17 vec_nr=9
> 9449 1429652195997662         2 40744   0  17 28 . sirqexit: cpu=17 vec_nr=1
> 9450 1429652195997660         2 40744   0  17 27 . sirqenter: cpu=17 vec_nr=1
> 9451 1429652195997656         4     0   0  13 28 . sirqexit: cpu=13 vec_nr=9
> 9452 1429652195997669       -13 40744  18  17  0 . Hello 9479. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
> 5000000000 15
> 9453 1429652195997652        17     0   0  13 27 1 sirqenter: cpu=13 vec_nr=9
> 9454 1429652195997652         0 40744  18  17  0 . Hello 9480. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
> 5000000000 15
> 9455 1429652195997651         1 40744  18  17  0 . Hello 9481. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
> 5000000000 15
> 9456 1429652195997649         2     0   0  13 28 1 sirqexit: cpu=13 vec_nr=1
> 9457 1429652195997649         0 40744  18  17  0 . Hello 9482. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
> 5000000000 15
> 9458 1429652195997648         1     0   0  12 28 . sirqexit: cpu=12 vec_nr=9
> 9459 1429652195997648         0 40744  18  17  0 . Hello 9483. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
> 5000000000 15
> 9460 1429652195997647         1 40744  18  17  0 . Hello 9484. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
> 5000000000 15
> 9461 1429652195997646         1 40744  18  17  0 . Hello 9485. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
> 5000000000 15
> 9462 1429652195997645         1     0   0  13 27 1 sirqenter: cpu=13 vec_nr=1
> 9463 1429652195997645         0 40744  18  17  0 1 Hello 9486. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
> 5000000000 15
> 9464 1429652195997644         1     0   0  12 27 . sirqenter: cpu=12 vec_nr=9
> 9465 1429652195997643         1 40744  18  17  0 . Hello 9487. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
> 5000000000 15
> 9466 1429652195997642         1 40744  18  17  0 . Hello 9488. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
> 5000000000 15
> 9467 1429652195997642         0     0   0  12 28 . sirqexit: cpu=12 vec_nr=1
> 9468 1429652195997641         1 40744  18  17  0 . Hello 9489. "c 2.5 5 5000000000 15" should be repeated here: c 2.5 5 
> 5000000000 15
> --04/21_17:15:52--
> 
> 
> 

Re: [PATCH] tracing: Export key trace event symbols

[Ron Rechenmacher]

Steven Rostedt wrote on 04/21/15 17:44:
>
> [ Added a bunch of people that use perf ;-) ]
>

Thanks. I hope they can help me get the patch accepted.

[snip]
>>>
>>> +/*
>>> + * Allow modules to register additional trace routines
>>> + */
>>> +EXPORT_TRACEPOINT_SYMBOL_GPL(irq_handler_entry);
>>> +EXPORT_TRACEPOINT_SYMBOL_GPL(irq_handler_exit);
>>> +
>>>
>>> Instead of just saying "Allow modules to register additional trace
>>> routines", please explain what its used for. What are you measuring
>>> externally? Just the timings of the interrupts?
>>
>> How about "Allow for module (internal or external) to register trace function which
>> shows (or measures) interactions with user-space executables more directly.  Knowing
>> when interrupts happen and how they can subtly impact user-space application timing
>> is important in some application."
>>
>>>
>>> I'm also curious to know why ftrace isn't good enough.
>>
>> ftrace (if one considers ltt-ng) is probably capable of doing what my trace does
>> except that the timestamp is not TOD (Time Of Day) -- but probably/maybe could
>> be made to do so???
>>
>> It's a matter of complexity or "start up cost" (or entry barrier).
>>
>> I'm not the best at articulating all there is to TRACE succinctly, but lets
>> start with the fact that the user-space component of TRACE is basically just a header
>> file that defines the TRACE printf-like macro and handles the initialization which is
>> basically mmap-ing a file (as a trace buffer (with control structure)) and is,
>> for the most part, Unix OS independent.
>
> Heh, that sounds like the perf interface. Have you thought about using
> that instead?
>
> Look at what PowerTop does: https://01.org/powertop
>
> You can use the perf interface to mmap a memory buffer and read the
> data in its raw format. To parse the data, you need to read the format

I've looked at the above reference briefly and it appears that user-space
would be mmapping the buffer read-only. Is that correct?
I've also looked at lttng and I've said before, the "start-up cost" is too high.
My TRACE module creates the virtual file and allows the user-space program(s) to
mmap a portion of it read-only (to protect the kernel) and then the buffer portion
read-write - so, yes, any user-space program _could_ simply scribble over the
buffer-portion, but what they usually end up doing it _writing_ their traces into
the appropriate "slot" memory along side of slots that the kernel could be writing the
key event TRACEs to.
I'm not trying to promote the use of my TRACE, I'm just trying to show
that there is a set of valid reasons (current and potential future) for accepting my patch.

> files in /sys/kernel/debug/tracing/events, but there's a library that
> does that. Unfortunately, that library isn't in distros yet (we really
> need to get that done). But you can just copy the code from the kernel
> source tree from tools/lib/traceevents/ and use that as perf does (as
> well as powertop does too).
>
> The format files tells you how to get the data from the raw binary
> format.
>
> If you use the perf interface, you can have userspace tools do custom
> things with the events as they happen. This may make your utility more
> robust in the long run.
>
> I'd be happy to help you understand how to use the traceevent library. I
> don't use the perf interface, but you can get examples from the
> PowerTop code.

I'd rather not get into the "you should stop doing your thing and just/only do
something else" mode, if you don't mind. As I've said before, I believe my TRACEing
is just the right level of complexity for the total system analysis that I do and
I have been periodically checking/searching what's out there.

A college of mine and I actually used perf, just today, to gain an understanding
of what is going on in our "Nova data acquistion" system at fermilab. While it was
useful, it is not a replacement for TRACE.

Thanks,
Ron

>
> -- Steve
>
[snip]

-- 
Ron Rechenmacher
Engineer
Fermi National Accelerator Laboratory
Batavia, IL  60510

Re: [PATCH] tracing: Export key trace event symbols

[Steven Rostedt]

On Tue, 21 Apr 2015 21:24:51 -0500
Ron Rechenmacher <ron@fnal.gov> wrote:


> I've looked at the above reference briefly and it appears that user-space
> would be mmapping the buffer read-only. Is that correct?

Correct, but I'm sure we could still add something (if it doesn't
already exist) to have userspace write into the buffer. Ftrace has that
with the trace_marker file.

> I've also looked at lttng and I've said before, the "start-up cost" is too high.

I'm not sure what you mean by "start-up cost". Note, I'm not actually
talking about using userspace perf. I'm talking about using the perf
interface directly with your program like powertop does.

> My TRACE module creates the virtual file and allows the user-space program(s) to
> mmap a portion of it read-only (to protect the kernel) and then the buffer portion
> read-write - so, yes, any user-space program _could_ simply scribble over the
> buffer-portion, but what they usually end up doing it _writing_ their traces into
> the appropriate "slot" memory along side of slots that the kernel could be writing the
> key event TRACEs to.
> I'm not trying to promote the use of my TRACE, I'm just trying to show
> that there is a set of valid reasons (current and potential future) for accepting my patch.
> 


> 
> I'd rather not get into the "you should stop doing your thing and just/only do
> something else" mode, if you don't mind. As I've said before, I believe my TRACEing
> is just the right level of complexity for the total system analysis that I do and
> I have been periodically checking/searching what's out there.

Understand. This was my feeling with logdev. I ported it year after
year (from '98 all the way to 2007 - when ftrace made it obsolete, I
even ported it to the xen hypervisor!).

What I learned was that something that was so useful to me could also
be very useful for others. You got my attention. I want to understand
why this works for you so well, and perhaps we can modify the existing
tools to make them handle all the requirements that your TRACE tool
does or at the minimum, have something your TRACE tool can hook into.

ftrace is the result of logdev being merged with the latency-tracer
that was in the -rt patch. I'm not saying stop doing your thing, I'm
asking what is different about your thing that we can make our tools
better to do it too. Your thing is obviously beneficial to you. I'm
guessing it can be beneficial to others outside the fermilab.


> 
> A college of mine and I actually used perf, just today, to gain an understanding
> of what is going on in our "Nova data acquistion" system at fermilab. While it was
> useful, it is not a replacement for TRACE.

Again, I did not mean to use the perf user space tool. I was wondering
if it would be possible to have TRACE use the perf kernel interface.

-- Steve

Re: [PATCH] tracing: Export key trace event symbols

[Steven Rostedt]

On Wed, 22 Apr 2015 08:53:14 -0400
Steven Rostedt <rostedt@goodmis.org> wrote:
 
> > 
> > I'd rather not get into the "you should stop doing your thing and just/only do
> > something else" mode, if you don't mind. As I've said before, I believe my TRACEing
> > is just the right level of complexity for the total system analysis that I do and
> > I have been periodically checking/searching what's out there.
> 
> Understand. This was my feeling with logdev. I ported it year after

That was suppose to be "I understand.", that is, I understand what you
meant. It was not suppose to sound like I'm telling you to "understand".

-- Steve

> year (from '98 all the way to 2007 - when ftrace made it obsolete, I
> even ported it to the xen hypervisor!).

Re: [PATCH] tracing: Export key trace event symbols

[Arnaldo Carvalho de Melo]

Em Wed, Apr 22, 2015 at 08:53:14AM -0400, Steven Rostedt escreveu:
> On Tue, 21 Apr 2015 21:24:51 -0500
> Ron Rechenmacher <ron@fnal.gov> wrote:
> > I've looked at the above reference briefly and it appears that user-space
> > would be mmapping the buffer read-only. Is that correct?
> 
> Correct, but I'm sure we could still add something (if it doesn't
> already exist) to have userspace write into the buffer. Ftrace has that
> with the trace_marker file.

There is something in the works, I guess Pawell Moll (sp) was working on it, and
David Ahern (CCed) should know, David?
 
> > I've also looked at lttng and I've said before, the "start-up cost" is too high.
> 
> I'm not sure what you mean by "start-up cost". Note, I'm not actually
> talking about using userspace perf. I'm talking about using the perf
> interface directly with your program like powertop does.
> 
> > My TRACE module creates the virtual file and allows the user-space program(s) to
> > mmap a portion of it read-only (to protect the kernel) and then the buffer portion
> > read-write - so, yes, any user-space program _could_ simply scribble over the
> > buffer-portion, but what they usually end up doing it _writing_ their traces into
> > the appropriate "slot" memory along side of slots that the kernel could be writing the
> > key event TRACEs to.
> > I'm not trying to promote the use of my TRACE, I'm just trying to show
> > that there is a set of valid reasons (current and potential future) for accepting my patch.

> > I'd rather not get into the "you should stop doing your thing and just/only do
> > something else" mode, if you don't mind. As I've said before, I believe my TRACEing
> > is just the right level of complexity for the total system analysis that I do and
> > I have been periodically checking/searching what's out there.
 
> Understand. This was my feeling with logdev. I ported it year after
> year (from '98 all the way to 2007 - when ftrace made it obsolete, I
> even ported it to the xen hypervisor!).
 
> What I learned was that something that was so useful to me could also
> be very useful for others. You got my attention. I want to understand
> why this works for you so well, and perhaps we can modify the existing
> tools to make them handle all the requirements that your TRACE tool
> does or at the minimum, have something your TRACE tool can hook into.
> 
> ftrace is the result of logdev being merged with the latency-tracer
> that was in the -rt patch. I'm not saying stop doing your thing, I'm
> asking what is different about your thing that we can make our tools
> better to do it too. Your thing is obviously beneficial to you. I'm
> guessing it can be beneficial to others outside the fermilab.
 
> > A college of mine and I actually used perf, just today, to gain an understanding
> > of what is going on in our "Nova data acquistion" system at fermilab. While it was
> > useful, it is not a replacement for TRACE.
 
> Again, I did not mean to use the perf user space tool. I was wondering
> if it would be possible to have TRACE use the perf kernel interface.

Well, but then it can influence how perf works, be it in the kernel (this
userspace event insertion in the ring buffer, in the works, may be suitable
here, may be not), be it in the tooling side, where there is a strace like tool
that could do parts of what you want.

- Arnaldo

Re: [PATCH] tracing: Export key trace event symbols

[David Ahern]

On 4/22/15 8:47 AM, Arnaldo Carvalho de Melo wrote:
> Em Wed, Apr 22, 2015 at 08:53:14AM -0400, Steven Rostedt escreveu:
>> >On Tue, 21 Apr 2015 21:24:51 -0500
>> >Ron Rechenmacher<ron@fnal.gov>  wrote:
>>> > >I've looked at the above reference briefly and it appears that user-space
>>> > >would be mmapping the buffer read-only. Is that correct?
>> >
>> >Correct, but I'm sure we could still add something (if it doesn't
>> >already exist) to have userspace write into the buffer. Ftrace has that
>> >with the trace_marker file.
> There is something in the works, I guess Pawell Moll (sp) was working on it, and
> David Ahern (CCed) should know, David?
>

I played around with generating perf events in userspace with the 
intention of having the userspace events get merged with kernel events 
during the processing stage, but I did not take it to the point of 
integrating into perf. This was around October 2013. I got distracted 
with other topics and have not come back to it.

Pawel has a patch that allows userspace to inject events into the stream 
via ioctl calls.

Stephane also injects events for JIT.

One of the key requirements is a common time basis (e.g., 
CLOCK_MONOTONIC or PERF_CLOCK) to be able to merge the events properly. 
I have a kernel module that exports perf_clock to userspace via 
clock_gettime; the 4.1 kernel should have the code that allows the clock 
id to be specified providing a solution to this problem.

David

Re: [PATCH] tracing: Export key trace event symbols

[Steven Rostedt]

On Wed, 22 Apr 2015 09:36:20 -0600
David Ahern <dsahern@gmail.com> wrote:

> One of the key requirements is a common time basis (e.g., 
> CLOCK_MONOTONIC or PERF_CLOCK) to be able to merge the events properly. 
> I have a kernel module that exports perf_clock to userspace via 
> clock_gettime; the 4.1 kernel should have the code that allows the clock 
> id to be specified providing a solution to this problem.

Ron,

I should have warned you. This is normal Linux kernel development.
Someone posts a simple patch to make something of there's work better,
and it ends up becoming a massive undertaking to come up with a solution
that makes Linux in general a better operating system ;-)

-- Steve

Re: [PATCH] tracing: Export key trace event symbols

[Ron Rechenmacher]

Thanks, Steve :)

I was working on a reply to your previous email but I'll reply here.
I'm happy with how this is going and I'm really appreciative of how
you have/are handling (managing) this. Thanks!

Assuming I'm only getting the tip of the ice berg of "normal Linux kernel
development," I don't know if it's reasonable to achieve a balance between
discussing "accepting the patch" and "what I'm doing with TRACE."

If I may, I can separate the two as follows (and focusing more on "accepting the patch"):
In my 30 year evolution of TRACE, a significant point came (somewhere in the
1990's) when, with Vxworks, I noticed that they had a "task switch hook add" routine
and I was able to incorporate kernel task switching events in with the application
traces. My first patches to the linux-2.2 kernel included a patch to the
scheduler and interrupts came quickly there after.  I was always wondering
if Linux would one day provide "task switch hook" capability.
In early 2013, when I started the v3 of TRACE, I was extremely happy to find
the symbol "register_trace_sched_switch" --- this was all I needed and I thought
"they did it."

With the understanding, as has been mentioned before, that "only the user space ABI is
what we keep stable" and that before commit de7b2973903c (tracepoint: Use struct pointer
instead of name hash for reg/unreg tracepoints) it was just "lucky" that I could use
the register_tracepoint_* routines as somewhat generic "add hook" routines (from
external modules), is it now reasonable to accept my patch to allow external modules
to "add hooks" to key events in Linux?  Why or why not?

Thanks,
Ron


Steven Rostedt wrote on 04/22/15 10:44:
> On Wed, 22 Apr 2015 09:36:20 -0600
> David Ahern <dsahern@gmail.com> wrote:
>
>> One of the key requirements is a common time basis (e.g.,
>> CLOCK_MONOTONIC or PERF_CLOCK) to be able to merge the events properly.
>> I have a kernel module that exports perf_clock to userspace via
>> clock_gettime; the 4.1 kernel should have the code that allows the clock
>> id to be specified providing a solution to this problem.
>
> Ron,
>
> I should have warned you. This is normal Linux kernel development.
> Someone posts a simple patch to make something of there's work better,
> and it ends up becoming a massive undertaking to come up with a solution
> that makes Linux in general a better operating system ;-)
>
> -- Steve
>

-- 
Ron Rechenmacher
Engineer
Fermi National Accelerator Laboratory
Batavia, IL  60510

Re: [PATCH] tracing: Export key trace event symbols

[Steven Rostedt]

On Wed, 22 Apr 2015 11:35:41 -0500
Ron Rechenmacher <ron@fnal.gov> wrote:

 
> With the understanding, as has been mentioned before, that "only the user space ABI is
> what we keep stable" and that before commit de7b2973903c (tracepoint: Use struct pointer
> instead of name hash for reg/unreg tracepoints) it was just "lucky" that I could use
> the register_tracepoint_* routines as somewhat generic "add hook" routines (from
> external modules), is it now reasonable to accept my patch to allow external modules
> to "add hooks" to key events in Linux?  Why or why not?
> 

As has been mentioned before, the EXPORT_SYMBOL*() calls is for in-tree
modules that need them. If there's no in-tree users of the exported
symbol, that can be grounds for removing them.

But that is more of a guideline than a rule. Preferably, we want people
using the in-tree mechanisms, because as I have said before, if
something is useful for you, it is most likely useful for others. Even
if it is your own home grown utility. You've been keeping it around for
30 years and that says something. It means that it is probably useful
for other people as well.

We resist making small changes for out-of-tree modules because we want
that functionality in tree. Now, if for some reason, we agree that the
functionality does not belong in tree, and all maintainers involved are
OK, we can add a hook for you. But that wont happen until we are
convinced that the functionality you want belongs out of tree and the
in tree mechanisms are not good enough for you.

I'll admit it adds more burden on you. But the net goal of Linux kernel
development is not it improve usability for each individual, but to
improve the infrastructure as a whole, such that the community benefits
in the end, and not just for a few individuals. Yes, it takes more
time, thought and effort, but in the long run, it's a net gain for
everyone (including you :-).

-- Steve

Re: [PATCH] tracing: Export key trace event symbols

[Pawel Moll]

On Wed, 2015-04-22 at 16:36 +0100, David Ahern wrote:
> On 4/22/15 8:47 AM, Arnaldo Carvalho de Melo wrote:
> > Em Wed, Apr 22, 2015 at 08:53:14AM -0400, Steven Rostedt escreveu:
> >> >On Tue, 21 Apr 2015 21:24:51 -0500
> >> >Ron Rechenmacher<ron@fnal.gov>  wrote:
> >>> > >I've looked at the above reference briefly and it appears that user-space
> >>> > >would be mmapping the buffer read-only. Is that correct?
> >> >
> >> >Correct, but I'm sure we could still add something (if it doesn't
> >> >already exist) to have userspace write into the buffer. Ftrace has that
> >> >with the trace_marker file.
> > There is something in the works, I guess Pawell Moll (sp) was working on it, and
> > David Ahern (CCed) should know, David?
> >
> 
> I played around with generating perf events in userspace with the 
> intention of having the userspace events get merged with kernel events 
> during the processing stage, but I did not take it to the point of 
> integrating into perf. This was around October 2013. I got distracted 
> with other topics and have not come back to it.
> 
> Pawel has a patch that allows userspace to inject events into the stream 
> via ioctl calls.

In the last version it was even a prctl - no need for a perf file
descriptor any more :-)

But the patch requires more care if it's to go in, so I'm open to people
screaming "yes, we need it!" ;-)

> Stephane also injects events for JIT.

But that, as far as I understand, happens in userspace (as in: nothing
goes down to kernel)?

> One of the key requirements is a common time basis (e.g., 
> CLOCK_MONOTONIC or PERF_CLOCK) to be able to merge the events properly. 
> I have a kernel module that exports perf_clock to userspace via 
> clock_gettime; the 4.1 kernel should have the code that allows the clock 
> id to be specified providing a solution to this problem.

It's in! :-) After all these years...

http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=34f439278cef7b1177f8ce24f9fc81dfc6221d3b

Pawel

Re: [PATCH] tracing: Export key trace event symbols

[Pawel Moll]

On Thu, 2015-04-23 at 16:28 +0100, Pawel Moll wrote:
> On Wed, 2015-04-22 at 16:36 +0100, David Ahern wrote:
> > On 4/22/15 8:47 AM, Arnaldo Carvalho de Melo wrote:
> > > Em Wed, Apr 22, 2015 at 08:53:14AM -0400, Steven Rostedt escreveu:
> > >> >On Tue, 21 Apr 2015 21:24:51 -0500
> > >> >Ron Rechenmacher<ron@fnal.gov>  wrote:
> > >>> > >I've looked at the above reference briefly and it appears that user-space
> > >>> > >would be mmapping the buffer read-only. Is that correct?
> > >> >
> > >> >Correct, but I'm sure we could still add something (if it doesn't
> > >> >already exist) to have userspace write into the buffer. Ftrace has that
> > >> >with the trace_marker file.
> > > There is something in the works, I guess Pawell Moll (sp) was working on it, and
> > > David Ahern (CCed) should know, David?
> > >
> > 
> > I played around with generating perf events in userspace with the 
> > intention of having the userspace events get merged with kernel events 
> > during the processing stage, but I did not take it to the point of 
> > integrating into perf. This was around October 2013. I got distracted 
> > with other topics and have not come back to it.
> > 
> > Pawel has a patch that allows userspace to inject events into the stream 
> > via ioctl calls.
> 
> In the last version it was even a prctl - no need for a perf file
> descriptor any more :-)
> 
> But the patch requires more care if it's to go in, so I'm open to people
> screaming "yes, we need it!" ;-)

Forgot to quote the link:

http://article.gmane.org/gmane.linux.kernel.api/5851

Pawel

Re: [PATCH] tracing: Export key trace event symbols

[Steven Rostedt]

On Tue, 21 Apr 2015 06:23:55 -0700
Christoph Hellwig <hch@infradead.org> wrote:

> > 
> > It seems that you are the person that I have to convince in order to get
> > my patch accepted. Is this true?
> 
> In the end you'll need to convince a maintainer to apply it.

But a Nack from you will keep us from applying it ;-)

-- Steve

Re: [PATCH] tracing: Export key trace event symbols

[Steven Rostedt]

On Mon, 20 Apr 2015 16:38:11 -0500
Ron Rechenmacher <ron@fnal.gov> wrote:

> If symbols are not exported, modules can no longer register additional
> (module specified) tracepoints like they use to be able to (i.e linux-3.15.x).
> Somewhere on or about commit de7b2973903c6cc50b31ee5682a69b2219b9919d
> (Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> Date:   Tue Apr 8 17:26:21 2014 -0400
> tracepoint: Use struct pointer instead of name hash for reg/unreg tracepoints)
> modules which attempted to register additional tracing functions would
> get "Unknown symbol" errors. For example: "... Unknown symbol
> __tracepoint_sched_switch (err 0)"
> Symbols can be exported using the kernel's EXPORT_TRACEPOINT_SYMBOL_GPL macro
> to allow modules to once again register their own tracing functions (for at
> least some key points in the kernel as provided by this patch).
> 
> Signed-off-by: Ron Rechenmacher <ron@fnal.gov>
> Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=96051

Hi Ron,

I was talking with Mathieu on IRC and asked him how LTTng gets its
kernel tracepoints, and he told me he uses
for_each_kernel_tracepoint(). That will iterate over all tracepoints
that have been added in the kernel (and is exported GPL).

You can still use that to get the handle onto any tracepoint you need.
It's pretty straight forward (I just wrote a simple module to test it
out), and just compare against the tp->name, to find what you want.

I still would like to get more usage out of the internal code, but this
is your work around you wanted. No need to export new symbols. Just a
little more setup time on module load.

-- Steve

Here's my mod....


#include <linux/module.h>
#include <linux/ftrace.h>
#include <linux/tracepoint.h>

static func(struct tracepoint *tp, void *ignore)
{
	printk("tracepoint: %s\n", tp->name);
}

static int __init my_tp_init(void)
{
	for_each_kernel_tracepoint(func, NULL);
	return 0;
}

static void __exit my_tp_exit(void)
{
}

module_init(my_tp_init);
module_exit(my_tp_exit);

MODULE_AUTHOR("My name here");
MODULE_DESCRIPTION("Me!");
MODULE_LICENSE("GPL");

Re: [PATCH] tracing: Export key trace event symbols

[Mathieu Desnoyers]

----- Original Message -----
> On Mon, 20 Apr 2015 16:38:11 -0500
> Ron Rechenmacher <ron@fnal.gov> wrote:
> 
> > If symbols are not exported, modules can no longer register additional
> > (module specified) tracepoints like they use to be able to (i.e
> > linux-3.15.x).
> > Somewhere on or about commit de7b2973903c6cc50b31ee5682a69b2219b9919d
> > (Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> > Date:   Tue Apr 8 17:26:21 2014 -0400
> > tracepoint: Use struct pointer instead of name hash for reg/unreg
> > tracepoints)
> > modules which attempted to register additional tracing functions would
> > get "Unknown symbol" errors. For example: "... Unknown symbol
> > __tracepoint_sched_switch (err 0)"
> > Symbols can be exported using the kernel's EXPORT_TRACEPOINT_SYMBOL_GPL
> > macro
> > to allow modules to once again register their own tracing functions (for at
> > least some key points in the kernel as provided by this patch).
> > 
> > Signed-off-by: Ron Rechenmacher <ron@fnal.gov>
> > Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=96051
> 
> Hi Ron,
> 
> I was talking with Mathieu on IRC and asked him how LTTng gets its
> kernel tracepoints, and he told me he uses
> for_each_kernel_tracepoint(). That will iterate over all tracepoints
> that have been added in the kernel (and is exported GPL).
> 
> You can still use that to get the handle onto any tracepoint you need.
> It's pretty straight forward (I just wrote a simple module to test it
> out), and just compare against the tp->name, to find what you want.
> 
> I still would like to get more usage out of the internal code, but this
> is your work around you wanted. No need to export new symbols. Just a
> little more setup time on module load.

Hi Ron,

Quoting a snippet of your earlier emails:

"ftrace (if one considers ltt-ng) is probably capable of doing what my trace does
except that the timestamp is not TOD (Time Of Day) -- but probably/maybe could
be made to do so???"

FYI, LTTng and Ftrace are two different projects. LTTng features
am out-of-tree kernel and a user-space tracer, whereas Ftrace features
a kernel tracer, readily available from the Linux kernel sources.

The timestamps can now be the monotonic clock for each of perf, ftrace,
and lttng, thanks to the work on nmi-safe clock source done by Thomas
Gleixner (merged in 3.17).

With the nmi-safe clocks, you'll be able to correlate samples
and traces taken by perf, ftrace, and lttng-modules with LTTng
userspace traces (LTTng-UST). This has been one of the goals of the
Common Trace Format (CTF) since its creation. You might want to look
at the perf-to-ctf conversion feature merged in Linux 4.0.

Best regards,

Mathieu



> 
> -- Steve
> 
> Here's my mod....
> 
> 
> #include <linux/module.h>
> #include <linux/ftrace.h>
> #include <linux/tracepoint.h>
> 
> static func(struct tracepoint *tp, void *ignore)
> {
> 	printk("tracepoint: %s\n", tp->name);
> }
> 
> static int __init my_tp_init(void)
> {
> 	for_each_kernel_tracepoint(func, NULL);
> 	return 0;
> }
> 
> static void __exit my_tp_exit(void)
> {
> }
> 
> module_init(my_tp_init);
> module_exit(my_tp_exit);
> 
> MODULE_AUTHOR("My name here");
> MODULE_DESCRIPTION("Me!");
> MODULE_LICENSE("GPL");
> 

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

Re: [PATCH] tracing: Export key trace event symbols

[Ron Rechenmacher]

Steven Rostedt wrote on 04/24/15 16:24:
> On Mon, 20 Apr 2015 16:38:11 -0500
> Ron Rechenmacher <ron@fnal.gov> wrote:
>
>> If symbols are not exported, modules can no longer register additional
>> (module specified) tracepoints like they use to be able to (i.e linux-3.15.x).
>> Somewhere on or about commit de7b2973903c6cc50b31ee5682a69b2219b9919d
>> (Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
>> Date:   Tue Apr 8 17:26:21 2014 -0400
>> tracepoint: Use struct pointer instead of name hash for reg/unreg tracepoints)
>> modules which attempted to register additional tracing functions would
>> get "Unknown symbol" errors. For example: "... Unknown symbol
>> __tracepoint_sched_switch (err 0)"
>> Symbols can be exported using the kernel's EXPORT_TRACEPOINT_SYMBOL_GPL macro
>> to allow modules to once again register their own tracing functions (for at
>> least some key points in the kernel as provided by this patch).
>>
>> Signed-off-by: Ron Rechenmacher <ron@fnal.gov>
>> Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=96051
>
> Hi Ron,
>
> I was talking with Mathieu on IRC and asked him how LTTng gets its
> kernel tracepoints, and he told me he uses
> for_each_kernel_tracepoint(). That will iterate over all tracepoints
> that have been added in the kernel (and is exported GPL).
>
> You can still use that to get the handle onto any tracepoint you need.
> It's pretty straight forward (I just wrote a simple module to test it
> out), and just compare against the tp->name, to find what you want.

Excellent. Thanks Steve. I was able to get this to work 100% for me.
What should be done about the bug report (96051)?
Will you handle it or should I change it to "resolved as invalid"?

>
> I still would like to get more usage out of the internal code, but this
> is your work around you wanted. No need to export new symbols. Just a
> little more setup time on module load.

Do you think for_each_kernel_tracepoint() and, additionally, the
tracepoint_probe_register() and tracepoint_probe_unregister() functions
will be around for quite some time? I would imagine as long as
LTTng uses them.

Thanks,
Ron

>
> -- Steve
>
> Here's my mod....
>
>
> #include <linux/module.h>
> #include <linux/ftrace.h>
> #include <linux/tracepoint.h>
>
> static func(struct tracepoint *tp, void *ignore)
> {
> 	printk("tracepoint: %s\n", tp->name);
> }
>
> static int __init my_tp_init(void)
> {
> 	for_each_kernel_tracepoint(func, NULL);
> 	return 0;
> }
>
> static void __exit my_tp_exit(void)
> {
> }
>
> module_init(my_tp_init);
> module_exit(my_tp_exit);
>
> MODULE_AUTHOR("My name here");
> MODULE_DESCRIPTION("Me!");
> MODULE_LICENSE("GPL");
>

-- 
Ron Rechenmacher
Engineer
Fermi National Accelerator Laboratory
Batavia, IL  60510