increase size of number of possible tracing events

increase size of number of possible tracing events

[Anthony Eden]

I am trying to develop a patch which increases the maximum number of
possible tracing events (i.e. TRACE_EVENT_TYPE_MAX). I naively tried
changing unsigned short trace_entry::type to unsigned int (that simply
broke tracing).

Short background: I am placing a UProbe tracepoint at the start of
every basic block for every DSO in a dynamically-linked program. I
recently hit the ceiling for the maximum number of tracepoints with
dnsmasq;

/usr/bin/dnsmasq:            12826 basic blocks
/usr/lib/libc.so.6:          60511 basic blocks
/lib64/ld-linux-x86-64.so.2: 6270 basic blocks
linux-vdso.so.1:             36 basic blocks

Writing to /sys/kernel/debug/tracing/uprobe_events returns -ENODEV. If
anyone could provide some guidance towards raising the limit, it would
be much appreciated.

Thanks,
-Anthony

Re: increase size of number of possible tracing events

[Steven Rostedt]

I'm currently traveling (replying on my phone so excuse the top post), but what you want will require a kernel change.

-- Steve


On June 21, 2019 6:14:54 PM EDT, Anthony Eden <aeden@csail.mit.edu> wrote:
>I am trying to develop a patch which increases the maximum number of
>possible tracing events (i.e. TRACE_EVENT_TYPE_MAX). I naively tried
>changing unsigned short trace_entry::type to unsigned int (that simply
>broke tracing).
>
>Short background: I am placing a UProbe tracepoint at the start of
>every basic block for every DSO in a dynamically-linked program. I
>recently hit the ceiling for the maximum number of tracepoints with
>dnsmasq;
>
>/usr/bin/dnsmasq:            12826 basic blocks
>/usr/lib/libc.so.6:          60511 basic blocks
>/lib64/ld-linux-x86-64.so.2: 6270 basic blocks
>linux-vdso.so.1:             36 basic blocks
>
>Writing to /sys/kernel/debug/tracing/uprobe_events returns -ENODEV. If
>anyone could provide some guidance towards raising the limit, it would
>be much appreciated.
>
>Thanks,
>-Anthony

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity and top posting.

Re: increase size of number of possible tracing events

[Steven Rostedt]

On Fri, 21 Jun 2019 18:14:54 -0400
Anthony Eden <aeden@csail.mit.edu> wrote:

> I am trying to develop a patch which increases the maximum number of
> possible tracing events (i.e. TRACE_EVENT_TYPE_MAX). I naively tried
> changing unsigned short trace_entry::type to unsigned int (that simply
> broke tracing).
> 
> Short background: I am placing a UProbe tracepoint at the start of
> every basic block for every DSO in a dynamically-linked program. I
> recently hit the ceiling for the maximum number of tracepoints with
> dnsmasq;
> 
> /usr/bin/dnsmasq:            12826 basic blocks
> /usr/lib/libc.so.6:          60511 basic blocks
> /lib64/ld-linux-x86-64.so.2: 6270 basic blocks
> linux-vdso.so.1:             36 basic blocks
> 
> Writing to /sys/kernel/debug/tracing/uprobe_events returns -ENODEV. If
> anyone could provide some guidance towards raising the limit, it would
> be much appreciated.
>

I'm not sure having over 65535 different events is a good idea. Perhaps
we should create a different type of uprobe that can be reused, as I'm
sure you are not making each of these events record different data.
Basically make it similar to the function tracing event, which is a
single event that records the function that it traces, and not
thousands of different individual events.

Would something like that be helpful for you? It still requires a
kernel change, but that is still possible.

-- Steve

Re: increase size of number of possible tracing events

[Anthony Eden]

> I'm not sure having over 65535 different events is a good idea. Perhaps
> we should create a different type of uprobe that can be reused, as I'm
> sure you are not making each of these events record different data.
> Basically make it similar to the function tracing event, which is a
> single event that records the function that it traces, and not
> thousands of different individual events.
>
> Would something like that be helpful for you? It still requires a
> kernel change, but that is still possible.

You're right; the events record no data. Optimizing for this use-case
sounds reasonable.

It would be very helpful. The basic-block traces produced by this
mechanism allow us to identify bugs introduced by transformations we
make on the binaries. Kernel changes are no problem.

-Anthony

Re: increase size of number of possible tracing events

[Anthony Eden]

Using ptrace to do this turns out to be absurdly slow, as one might
expect (I wrote a python script for lldb). If anyone would like to see
the actual numbers let me know.

-Anthony

Re: increase size of number of possible tracing events

[Frank Ch. Eigler]

Hi -

> Using ptrace to do this turns out to be absurdly slow, as one might
> expect (I wrote a python script for lldb). If anyone would like to see
> the actual numbers let me know.

In the mean time, you can also try systemtap userspace probes.  These
normally use kernel uprobes, but not the ftrace/perf-event mechanism,
so are not limited in number.  A user can also opt to configure pure
userspace dyninst to instrument.

- FChE