Linux-Trace-Users Archive on lore.kernel.org
 help / color / Atom feed
* increase size of number of possible tracing events
@ 2019-06-21 22:14 Anthony Eden
  2019-06-21 23:13 ` Steven Rostedt
  2019-06-26 15:46 ` Steven Rostedt
  0 siblings, 2 replies; 6+ messages in thread
From: Anthony Eden @ 2019-06-21 22:14 UTC (permalink / raw)
  To: linux-trace-users

I am trying to develop a patch which increases the maximum number of
possible tracing events (i.e. TRACE_EVENT_TYPE_MAX). I naively tried
changing unsigned short trace_entry::type to unsigned int (that simply
broke tracing).

Short background: I am placing a UProbe tracepoint at the start of
every basic block for every DSO in a dynamically-linked program. I
recently hit the ceiling for the maximum number of tracepoints with
dnsmasq;

/usr/bin/dnsmasq:            12826 basic blocks
/usr/lib/libc.so.6:          60511 basic blocks
/lib64/ld-linux-x86-64.so.2: 6270 basic blocks
linux-vdso.so.1:             36 basic blocks

Writing to /sys/kernel/debug/tracing/uprobe_events returns -ENODEV. If
anyone could provide some guidance towards raising the limit, it would
be much appreciated.

Thanks,
-Anthony

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: increase size of number of possible tracing events
  2019-06-21 22:14 increase size of number of possible tracing events Anthony Eden
@ 2019-06-21 23:13 ` Steven Rostedt
  2019-06-26 15:46 ` Steven Rostedt
  1 sibling, 0 replies; 6+ messages in thread
From: Steven Rostedt @ 2019-06-21 23:13 UTC (permalink / raw)
  To: Anthony Eden, linux-trace-users

I'm currently traveling (replying on my phone so excuse the top post), but what you want will require a kernel change.

-- Steve


On June 21, 2019 6:14:54 PM EDT, Anthony Eden <aeden@csail.mit.edu> wrote:
>I am trying to develop a patch which increases the maximum number of
>possible tracing events (i.e. TRACE_EVENT_TYPE_MAX). I naively tried
>changing unsigned short trace_entry::type to unsigned int (that simply
>broke tracing).
>
>Short background: I am placing a UProbe tracepoint at the start of
>every basic block for every DSO in a dynamically-linked program. I
>recently hit the ceiling for the maximum number of tracepoints with
>dnsmasq;
>
>/usr/bin/dnsmasq:            12826 basic blocks
>/usr/lib/libc.so.6:          60511 basic blocks
>/lib64/ld-linux-x86-64.so.2: 6270 basic blocks
>linux-vdso.so.1:             36 basic blocks
>
>Writing to /sys/kernel/debug/tracing/uprobe_events returns -ENODEV. If
>anyone could provide some guidance towards raising the limit, it would
>be much appreciated.
>
>Thanks,
>-Anthony

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity and top posting.

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: increase size of number of possible tracing events
  2019-06-21 22:14 increase size of number of possible tracing events Anthony Eden
  2019-06-21 23:13 ` Steven Rostedt
@ 2019-06-26 15:46 ` Steven Rostedt
  2019-06-27 16:44   ` Anthony Eden
  1 sibling, 1 reply; 6+ messages in thread
From: Steven Rostedt @ 2019-06-26 15:46 UTC (permalink / raw)
  To: Anthony Eden; +Cc: linux-trace-users, Slavomir Kaslev

On Fri, 21 Jun 2019 18:14:54 -0400
Anthony Eden <aeden@csail.mit.edu> wrote:

> I am trying to develop a patch which increases the maximum number of
> possible tracing events (i.e. TRACE_EVENT_TYPE_MAX). I naively tried
> changing unsigned short trace_entry::type to unsigned int (that simply
> broke tracing).
> 
> Short background: I am placing a UProbe tracepoint at the start of
> every basic block for every DSO in a dynamically-linked program. I
> recently hit the ceiling for the maximum number of tracepoints with
> dnsmasq;
> 
> /usr/bin/dnsmasq:            12826 basic blocks
> /usr/lib/libc.so.6:          60511 basic blocks
> /lib64/ld-linux-x86-64.so.2: 6270 basic blocks
> linux-vdso.so.1:             36 basic blocks
> 
> Writing to /sys/kernel/debug/tracing/uprobe_events returns -ENODEV. If
> anyone could provide some guidance towards raising the limit, it would
> be much appreciated.
>

I'm not sure having over 65535 different events is a good idea. Perhaps
we should create a different type of uprobe that can be reused, as I'm
sure you are not making each of these events record different data.
Basically make it similar to the function tracing event, which is a
single event that records the function that it traces, and not
thousands of different individual events.

Would something like that be helpful for you? It still requires a
kernel change, but that is still possible.

-- Steve

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: increase size of number of possible tracing events
  2019-06-26 15:46 ` Steven Rostedt
@ 2019-06-27 16:44   ` Anthony Eden
  2019-06-30 14:31     ` Anthony Eden
  0 siblings, 1 reply; 6+ messages in thread
From: Anthony Eden @ 2019-06-27 16:44 UTC (permalink / raw)
  To: Steven Rostedt; +Cc: linux-trace-users, kaslevs

> I'm not sure having over 65535 different events is a good idea. Perhaps
> we should create a different type of uprobe that can be reused, as I'm
> sure you are not making each of these events record different data.
> Basically make it similar to the function tracing event, which is a
> single event that records the function that it traces, and not
> thousands of different individual events.
>
> Would something like that be helpful for you? It still requires a
> kernel change, but that is still possible.

You're right; the events record no data. Optimizing for this use-case
sounds reasonable.

It would be very helpful. The basic-block traces produced by this
mechanism allow us to identify bugs introduced by transformations we
make on the binaries. Kernel changes are no problem.

-Anthony

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: increase size of number of possible tracing events
  2019-06-27 16:44   ` Anthony Eden
@ 2019-06-30 14:31     ` Anthony Eden
  2019-07-01 12:58       ` Frank Ch. Eigler
  0 siblings, 1 reply; 6+ messages in thread
From: Anthony Eden @ 2019-06-30 14:31 UTC (permalink / raw)
  To: Steven Rostedt; +Cc: linux-trace-users, kaslevs

Using ptrace to do this turns out to be absurdly slow, as one might
expect (I wrote a python script for lldb). If anyone would like to see
the actual numbers let me know.

-Anthony

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: increase size of number of possible tracing events
  2019-06-30 14:31     ` Anthony Eden
@ 2019-07-01 12:58       ` Frank Ch. Eigler
  0 siblings, 0 replies; 6+ messages in thread
From: Frank Ch. Eigler @ 2019-07-01 12:58 UTC (permalink / raw)
  To: Anthony Eden; +Cc: Steven Rostedt, linux-trace-users, kaslevs

Hi -

> Using ptrace to do this turns out to be absurdly slow, as one might
> expect (I wrote a python script for lldb). If anyone would like to see
> the actual numbers let me know.

In the mean time, you can also try systemtap userspace probes.  These
normally use kernel uprobes, but not the ftrace/perf-event mechanism,
so are not limited in number.  A user can also opt to configure pure
userspace dyninst to instrument.

- FChE

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, back to index

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-06-21 22:14 increase size of number of possible tracing events Anthony Eden
2019-06-21 23:13 ` Steven Rostedt
2019-06-26 15:46 ` Steven Rostedt
2019-06-27 16:44   ` Anthony Eden
2019-06-30 14:31     ` Anthony Eden
2019-07-01 12:58       ` Frank Ch. Eigler

Linux-Trace-Users Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-trace-users/0 linux-trace-users/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-trace-users linux-trace-users/ https://lore.kernel.org/linux-trace-users \
		linux-trace-users@vger.kernel.org
	public-inbox-index linux-trace-users

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-trace-users


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git