[lttng-dev] Trying to understand use of lttng enable-event --kernel --userspace-probe=

[lttng-dev] Trying to understand use of lttng enable-event --kernel --userspace-probe=

[Brian Hutchinson via lttng-dev]

Hi,

I'm trying to figure out how to use uprobes with lttng.

I can't use a normal uprobe for a line number just using the address I
want to probe obtained from objdump?  As in:

echo 'p /usr/local/bin/my_app:0x2c3a8' >>
/sys/kernel/debug/tracing/uprobe_events

... which isn't a function entry, it's just a line of code I want to probe on.

This link says it has to be elf or sdt:
https://lttng.org/man/1/lttng-enable-event/v2.11/#doc-opt--userspace-probe

So can I not probe on just a line of code by specifying an address???

It doesn't look like these methods above will do what I'm wanting to
do.  I've tried to find examples of using enable-event --kernel
--userspace-probe= but there doesn't appear to be many.

Thanks,

Brian
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Trying to understand use of lttng enable-event --kernel --userspace-probe=

[Mathieu Desnoyers via lttng-dev]

On 2023-05-16 22:11, Brian Hutchinson via lttng-dev wrote:
> Hi,
> 
> I'm trying to figure out how to use uprobes with lttng.
> 
> I can't use a normal uprobe for a line number just using the address I
> want to probe obtained from objdump?  As in:
> 
> echo 'p /usr/local/bin/my_app:0x2c3a8' >>
> /sys/kernel/debug/tracing/uprobe_events
> 
> ... which isn't a function entry, it's just a line of code I want to probe on.
> 
> This link says it has to be elf or sdt:
> https://lttng.org/man/1/lttng-enable-event/v2.11/#doc-opt--userspace-probe
> 
> So can I not probe on just a line of code by specifying an address???
> 
> It doesn't look like these methods above will do what I'm wanting to
> do.  I've tried to find examples of using enable-event --kernel
> --userspace-probe= but there doesn't appear to be many.
> 

There are examples here:

https://lttng.org/docs/v2.13/#doc-enabling-disabling-events

Indeed inserting a lttng-modules uprobe within functions is not 
supported at the moment, mainly because we prefer to err towards safety 
and don't have the validation in place to prevent corrupting the 
program's instructions if an end user would try to insert a uprobe at an 
address which is not an instruction boundary.

So we only support inserting uprobe on functions and SDT probes at
the moment.

Thanks,

Mathieu


> Thanks,
> 
> Brian
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Trying to understand use of lttng enable-event --kernel --userspace-probe=

[Brian Hutchinson via lttng-dev]

On Wed, May 17, 2023 at 12:08 PM Mathieu Desnoyers
<mathieu.desnoyers@efficios.com> wrote:
>
> On 2023-05-16 22:11, Brian Hutchinson via lttng-dev wrote:
> > Hi,
> >
> > I'm trying to figure out how to use uprobes with lttng.
> >
> > I can't use a normal uprobe for a line number just using the address I
> > want to probe obtained from objdump?  As in:
> >
> > echo 'p /usr/local/bin/my_app:0x2c3a8' >>
> > /sys/kernel/debug/tracing/uprobe_events
> >
> > ... which isn't a function entry, it's just a line of code I want to probe on.
> >
> > This link says it has to be elf or sdt:
> > https://lttng.org/man/1/lttng-enable-event/v2.11/#doc-opt--userspace-probe
> >
> > So can I not probe on just a line of code by specifying an address???
> >
> > It doesn't look like these methods above will do what I'm wanting to
> > do.  I've tried to find examples of using enable-event --kernel
> > --userspace-probe= but there doesn't appear to be many.
> >
>
> There are examples here:
>
> https://lttng.org/docs/v2.13/#doc-enabling-disabling-events
>
> Indeed inserting a lttng-modules uprobe within functions is not
> supported at the moment, mainly because we prefer to err towards safety
> and don't have the validation in place to prevent corrupting the
> program's instructions if an end user would try to insert a uprobe at an
> address which is not an instruction boundary.

Hmm, was really hoping to be able to do dynamic tracing without having
to modify code.

I guess if I add a function call to a debug statement or something at
the point I want to probe then I could use the elf example.

>
> So we only support inserting uprobe on functions and SDT probes at
> the moment.

I've heard of system tap but never used it.  Will have to look into that.

I really want to get lttng-ust working but I'm getting pushback on the
time I'm spending trying to get it to work ... and would really like
to demonstrate something (was hoping kernel events and uprobes)
quickly to an audience that knows nothing about lttng or full stack
tracing to gain "buy in" for the effort.

You know, those pesky things called schedules.

Thanks,

Brian

>
> Thanks,
>
> Mathieu
>
>
> > Thanks,
> >
> > Brian
> > _______________________________________________
> > lttng-dev mailing list
> > lttng-dev@lists.lttng.org
> > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> https://www.efficios.com
>
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Trying to understand use of lttng enable-event --kernel --userspace-probe=

[Mathieu Desnoyers via lttng-dev]

On 2023-05-17 12:37, Brian Hutchinson wrote:
> On Wed, May 17, 2023 at 12:08 PM Mathieu Desnoyers
> <mathieu.desnoyers@efficios.com> wrote:
>>
>> On 2023-05-16 22:11, Brian Hutchinson via lttng-dev wrote:
>>> Hi,
>>>
>>> I'm trying to figure out how to use uprobes with lttng.
>>>
>>> I can't use a normal uprobe for a line number just using the address I
>>> want to probe obtained from objdump?  As in:
>>>
>>> echo 'p /usr/local/bin/my_app:0x2c3a8' >>
>>> /sys/kernel/debug/tracing/uprobe_events
>>>
>>> ... which isn't a function entry, it's just a line of code I want to probe on.
>>>
>>> This link says it has to be elf or sdt:
>>> https://lttng.org/man/1/lttng-enable-event/v2.11/#doc-opt--userspace-probe
>>>
>>> So can I not probe on just a line of code by specifying an address???
>>>
>>> It doesn't look like these methods above will do what I'm wanting to
>>> do.  I've tried to find examples of using enable-event --kernel
>>> --userspace-probe= but there doesn't appear to be many.
>>>
>>
>> There are examples here:
>>
>> https://lttng.org/docs/v2.13/#doc-enabling-disabling-events
>>
>> Indeed inserting a lttng-modules uprobe within functions is not
>> supported at the moment, mainly because we prefer to err towards safety
>> and don't have the validation in place to prevent corrupting the
>> program's instructions if an end user would try to insert a uprobe at an
>> address which is not an instruction boundary.
> 
> Hmm, was really hoping to be able to do dynamic tracing without having
> to modify code.

uprobes with the proper validations about instruction boundaries would 
eventually provide this. Another approach we want to invest time in is 
to integrate libpatch from Olivier Dion into lttng-ust. This would 
provide dynamic instrumentation with the performance of a purely 
userspace tracer.

But those are all things that were never prioritized by any of our 
customers, so they progress at a "back burner" pace.

> 
> I guess if I add a function call to a debug statement or something at
> the point I want to probe then I could use the elf example.

Yes.

> 
>>
>> So we only support inserting uprobe on functions and SDT probes at
>> the moment.
> 
> I've heard of system tap but never used it.  Will have to look into that.
> 
> I really want to get lttng-ust working but I'm getting pushback on the
> time I'm spending trying to get it to work ... and would really like
> to demonstrate something (was hoping kernel events and uprobes)
> quickly to an audience that knows nothing about lttng or full stack
> tracing to gain "buy in" for the effort.

Understood.

The main thing we are missing to help you on the UST front is a console 
log of the _application_ with LTTNG_UST_DEBUG=1. I suspect it is not 
collected in your tests.

Thanks,

Mathieu


> 
> You know, those pesky things called schedules.
> 
> Thanks,
> 
> Brian
> 
>>
>> Thanks,
>>
>> Mathieu
>>
>>
>>> Thanks,
>>>
>>> Brian
>>> _______________________________________________
>>> lttng-dev mailing list
>>> lttng-dev@lists.lttng.org
>>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>>
>> --
>> Mathieu Desnoyers
>> EfficiOS Inc.
>> https://www.efficios.com
>>

-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Trying to understand use of lttng enable-event --kernel --userspace-probe=

[Brian Hutchinson via lttng-dev]

On Wed, May 17, 2023 at 12:08 PM Mathieu Desnoyers
<mathieu.desnoyers@efficios.com> wrote:
>
> On 2023-05-16 22:11, Brian Hutchinson via lttng-dev wrote:
> > Hi,
> >
> > I'm trying to figure out how to use uprobes with lttng.
> >
> > I can't use a normal uprobe for a line number just using the address I
> > want to probe obtained from objdump?  As in:
> >
> > echo 'p /usr/local/bin/my_app:0x2c3a8' >>
> > /sys/kernel/debug/tracing/uprobe_events
> >
> > ... which isn't a function entry, it's just a line of code I want to probe on.
> >
> > This link says it has to be elf or sdt:
> > https://lttng.org/man/1/lttng-enable-event/v2.11/#doc-opt--userspace-probe
> >
> > So can I not probe on just a line of code by specifying an address???
> >
> > It doesn't look like these methods above will do what I'm wanting to
> > do.  I've tried to find examples of using enable-event --kernel
> > --userspace-probe= but there doesn't appear to be many.
> >
>
> There are examples here:
>
> https://lttng.org/docs/v2.13/#doc-enabling-disabling-events

I added a function that's called when the event I want to probe occurs.

I tried to enable the event with:
root@localhost:/usr/local/bin# lttng enable-event --kernel
--userspace-probe=/usr/local/bin/my_app:my_probe_event
Error: Missing event name(s).

... and:

root@localhost:/usr/local/bin# lttng enable-event --kernel
--userspace-probe=elf:/usr/local/bin/my_app:my_probe_event
Error: Missing event name(s).

I can do ${CROSS_COMPILE}objdump -d and verify that the symbol above
for the function I added is indeed there.

What am I doing wrong?

>
> Indeed inserting a lttng-modules uprobe within functions is not
> supported at the moment, mainly because we prefer to err towards safety
> and don't have the validation in place to prevent corrupting the
> program's instructions if an end user would try to insert a uprobe at an
> address which is not an instruction boundary.
>
> So we only support inserting uprobe on functions and SDT probes at
> the moment.
>
> Thanks,
>
> Mathieu
>
>
> > Thanks,
> >
> > Brian
> > _______________________________________________
> > lttng-dev mailing list
> > lttng-dev@lists.lttng.org
> > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> https://www.efficios.com
>
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Trying to understand use of lttng enable-event --kernel --userspace-probe=

[Brian Hutchinson via lttng-dev]

On Thu, May 18, 2023 at 9:17 AM Brian Hutchinson <b.hutchman@gmail.com> wrote:
>
> On Wed, May 17, 2023 at 12:08 PM Mathieu Desnoyers
> <mathieu.desnoyers@efficios.com> wrote:
> >
> > On 2023-05-16 22:11, Brian Hutchinson via lttng-dev wrote:
> > > Hi,
> > >
> > > I'm trying to figure out how to use uprobes with lttng.
> > >
> > > I can't use a normal uprobe for a line number just using the address I
> > > want to probe obtained from objdump?  As in:
> > >
> > > echo 'p /usr/local/bin/my_app:0x2c3a8' >>
> > > /sys/kernel/debug/tracing/uprobe_events
> > >
> > > ... which isn't a function entry, it's just a line of code I want to probe on.
> > >
> > > This link says it has to be elf or sdt:
> > > https://lttng.org/man/1/lttng-enable-event/v2.11/#doc-opt--userspace-probe
> > >
> > > So can I not probe on just a line of code by specifying an address???
> > >
> > > It doesn't look like these methods above will do what I'm wanting to
> > > do.  I've tried to find examples of using enable-event --kernel
> > > --userspace-probe= but there doesn't appear to be many.
> > >
> >
> > There are examples here:
> >
> > https://lttng.org/docs/v2.13/#doc-enabling-disabling-events
>
> I added a function that's called when the event I want to probe occurs.
>
> I tried to enable the event with:
> root@localhost:/usr/local/bin# lttng enable-event --kernel
> --userspace-probe=/usr/local/bin/my_app:my_probe_event
> Error: Missing event name(s).
>
> ... and:
>
> root@localhost:/usr/local/bin# lttng enable-event --kernel
> --userspace-probe=elf:/usr/local/bin/my_app:my_probe_event
> Error: Missing event name(s).
>
> I can do ${CROSS_COMPILE}objdump -d and verify that the symbol above
> for the function I added is indeed there.
>
> What am I doing wrong?
>

I updated my hello world to have a function I'd like to use the
--userspace-probe method on with the very original name of
'probe_function':

#include <stdio.h>
#include <lttng/tracef.h>

void probe_function(int i);

int main(int argc, char *argv[])
{
   unsigned int i;
   puts("Hello, World!\nPress Enter to continue...");
   /*
    * The following getchar() call only exists for the purpose of this
    * demonstration, to pause the application in order for you to have
    * time to list its tracepoints. You don't need it otherwise.
    */
   getchar();

   lttng_ust_tracef("Number %d, string %s", 23, "hi there!");
   printf("Number %d, string %s", 23, "hi there!");

   for (i = 0; i < argc; i++) {
       lttng_ust_tracef("Number %d, argv %s", i, argv[i]);
       printf("Number %d, argv %s", i, argv[i]);
   }

   puts("Quitting now!");

   probe_function(i);

   return 0;
}

void probe_function(int i) {

   lttng_ust_tracef("Number %d, string %s", i * i, "i^2");
   printf("Number %d, string %s", i * i, "i^2");

}

... and I get the same error as before when I try to enable the probe:
# lttng enable-event --kernel
--userspace-probe=/usr/local/bin/hello:probe_function
Error: Missing event name(s).

More system details below:


$ cat hello_objdump.txt

hello:     file format elf64-littleaarch64

SYMBOL TABLE:
0000000000000238 l    d  .interp        0000000000000000              .interp
0000000000000254 l    d  .note.gnu.build-id     0000000000000000
       .note.gnu.build-id
0000000000000278 l    d  .note.ABI-tag  0000000000000000
.note.ABI-tag
0000000000000298 l    d  .gnu.hash      0000000000000000              .gnu.hash
00000000000002b8 l    d  .dynsym        0000000000000000              .dynsym
00000000000004c8 l    d  .dynstr        0000000000000000              .dynstr
00000000000005ee l    d  .gnu.version   0000000000000000
.gnu.version
0000000000000620 l    d  .gnu.version_r 0000000000000000
.gnu.version_r
0000000000000660 l    d  .rela.dyn      0000000000000000              .rela.dyn
00000000000007b0 l    d  .rela.plt      0000000000000000              .rela.plt
0000000000000918 l    d  .init  0000000000000000              .init
0000000000000930 l    d  .plt   0000000000000000              .plt
0000000000000a40 l    d  .text  0000000000000000              .text
0000000000001194 l    d  .fini  0000000000000000              .fini
00000000000011a8 l    d  .rodata        0000000000000000              .rodata
00000000000013f0 l    d  .eh_frame_hdr  0000000000000000
.eh_frame_hdr
0000000000001468 l    d  .eh_frame      0000000000000000              .eh_frame
0000000000011d78 l    d  .init_array    0000000000000000
.init_array
0000000000011d88 l    d  .fini_array    0000000000000000
.fini_array
0000000000011d98 l    d  .dynamic       0000000000000000              .dynamic
0000000000011f98 l    d  .got   0000000000000000              .got
0000000000011fe8 l    d  .got.plt       0000000000000000              .got.plt
0000000000012078 l    d  .data  0000000000000000              .data
00000000000120d8 l    d  .bss   0000000000000000              .bss
0000000000000000 l    d  .comment       0000000000000000              .comment
0000000000000000 l    d  .debug_aranges 0000000000000000
.debug_aranges
0000000000000000 l    d  .debug_info    0000000000000000
.debug_info
0000000000000000 l    d  .debug_abbrev  0000000000000000
.debug_abbrev
0000000000000000 l    d  .debug_line    0000000000000000
.debug_line
0000000000000000 l    d  .debug_str     0000000000000000              .debug_str
0000000000000000 l    d  .debug_loc     0000000000000000              .debug_loc
0000000000000000 l    d  .debug_ranges  0000000000000000
.debug_ranges
0000000000000000 l    df *ABS*  0000000000000000              abi-note.S
0000000000000000 l    df *ABS*  0000000000000000              start.S
0000000000000000 l    df *ABS*  0000000000000000              init.c
0000000000000000 l    df *ABS*  0000000000000000              crti.S
0000000000000a78 l     F .text  0000000000000014              call_weak_fn
0000000000000000 l    df *ABS*  0000000000000000              crtn.S
0000000000000000 l    df *ABS*  0000000000000000              crtstuff.c
0000000000000a90 l     F .text  0000000000000000
deregister_tm_clones
0000000000000ac0 l     F .text  0000000000000000              register_tm_clones
0000000000000b00 l     F .text  0000000000000000
__do_global_dtors_aux
00000000000120d8 l     O .bss   0000000000000001              completed.9125
0000000000011d88 l     O .fini_array    0000000000000000
__do_global_dtors_aux_fini_array_entry
0000000000000b50 l     F .text  0000000000000000              frame_dummy
0000000000011d78 l     O .init_array    0000000000000000
__frame_dummy_init_array_entry
0000000000000000 l    df *ABS*  0000000000000000              hello.c
0000000000000b54 l     F .text  0000000000000120
lttng_ust_tracepoint__init_urcu_sym
0000000000000c74 l     F .text  0000000000000028
lttng_ust_tracepoint_logging_debug_enabled
0000000000000c9c l     F .text  000000000000007c
lttng_ust_tracepoints_print_disabled_message
00000000000013c0 l     O .rodata        000000000000002d
__func__.5512
0000000000000d18 l     F .text  00000000000000e8
lttng_ust__tracepoints__init
0000000000000e00 l     F .text  0000000000000170
lttng_ust__tracepoints__destroy
0000000000000000 l    df *ABS*  0000000000000000              elf-init.c
0000000000000000 l    df *ABS*  0000000000000000              crtstuff.c
0000000000001610 l     O .eh_frame      0000000000000000
__FRAME_END__
0000000000000000 l    df *ABS*  0000000000000000
0000000000011d88 l       .init_array    0000000000000000
__init_array_end
0000000000011d98 l     O *ABS*  0000000000000000              _DYNAMIC
0000000000011d78 l       .init_array    0000000000000000
__init_array_start
00000000000013f0 l       .eh_frame_hdr  0000000000000000
__GNU_EH_FRAME_HDR
0000000000011f98 l     O *ABS*  0000000000000000
_GLOBAL_OFFSET_TABLE_
0000000000000918 l     F .init  0000000000000000              _init
0000000000001190 g     F .text  0000000000000004              __libc_csu_fini
0000000000000000  w      *UND*  0000000000000000
_ITM_deregisterTMCloneTable
0000000000012078  w      .data  0000000000000000              data_start
00000000000120f0  w    O .bss   0000000000000008              .hidden
lttng_ust_tracepoint_destructors_syms_ptr
00000000000120d8 g       .bss   0000000000000000              __bss_start__
0000000000000000  w    F *UND*  0000000000000000
__cxa_finalize@@GLIBC_2.17
0000000000000000       O *UND*  0000000000000000              stderr@@GLIBC_2.17
0000000000000000       F *UND*  0000000000000000
dlclose@@GLIBC_2.17
00000000000120f8 g       .bss   0000000000000000              _bss_end__
00000000000120d8 g       .data  0000000000000000              _edata
0000000000000000       F *UND*  0000000000000000              dlopen@@GLIBC_2.17
0000000000001194 g     F .fini  0000000000000000              .hidden _fini
00000000000120f8 g       .bss   0000000000000000              __bss_end__
0000000000000000       F *UND*  0000000000000000              getpid@@GLIBC_2.17
0000000000000000       F *UND*  0000000000000000              dlsym@@GLIBC_2.17
0000000000000000       F *UND*  0000000000000000
__libc_start_main@@GLIBC_2.17
0000000000000000       F *UND*  0000000000000000              memset@@GLIBC_2.17
00000000000120e8  w    O .bss   0000000000000008              .hidden
lttng_ust_tracepoint_dlopen_ptr
0000000000012078 g       .data  0000000000000000              __data_start
0000000000000000  w      *UND*  0000000000000000              __gmon_start__
0000000000012080 g     O .data  0000000000000000              .hidden
__dso_handle
00000000000120c0  w    O .data  0000000000000018              .hidden
lttng_ust_tracepoint_destructors_syms
0000000000000000       F *UND*  0000000000000000              abort@@GLIBC_2.17
00000000000120e4  w    O .bss   0000000000000004              .hidden
lttng_ust_tracepoint_ptrs_registered
00000000000011a8 g     O .rodata        0000000000000004
_IO_stdin_used
0000000000012088  w    O .data  0000000000000038              .hidden
lttng_ust_tracepoint_dlopen
0000000000000000       F *UND*  0000000000000000              puts@@GLIBC_2.17
00000000000120e0  w    O .bss   0000000000000004              .hidden
lttng_ust_tracepoint_registered
0000000000001110 g     F .text  000000000000007c              __libc_csu_init
00000000000120f8 g       .bss   0000000000000000              _end
0000000000000a40 g     F .text  0000000000000000              _start
0000000000000000       F *UND*  0000000000000000
getchar@@GLIBC_2.17
00000000000120f8 g       .bss   0000000000000000              __end__
00000000000120d8 g       .bss   0000000000000000              __bss_start
0000000000000f70 g     F .text  000000000000012c              main
000000000000109c g     F .text  0000000000000074              probe_function
0000000000000000       O *UND*  0000000000000000
lttng_ust_tracepoint_lttng_ust_tracef___event
00000000000120d8 g     O .data  0000000000000000              .hidden
__TMC_END__
0000000000000000  w      *UND*  0000000000000000
_ITM_registerTMCloneTable
0000000000000000       F *UND*  0000000000000000              printf@@GLIBC_2.17
0000000000000000       F *UND*  0000000000000000              getenv@@GLIBC_2.17
0000000000000000       F *UND*  0000000000000000
fprintf@@GLIBC_2.17
0000000000000000       F *UND*  0000000000000000              lttng_ust__tracef

# zcat /proc/config.gz | grep -i probe
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_ARCH_SUPPORTS_UPROBES=y
CONFIG_KPROBES=y
CONFIG_UPROBES=y
CONFIG_KRETPROBES=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
# CONFIG_TEST_ASYNC_DRIVER_PROBE is not set
CONFIG_GENERIC_CPU_AUTOPROBE=y
# CONFIG_MTD_JEDECPROBE is not set
CONFIG_TIMER_PROBE=y
CONFIG_KPROBE_EVENTS=y
# CONFIG_KPROBE_EVENTS_ON_NOTRACE is not set
CONFIG_UPROBE_EVENTS=y
CONFIG_PROBE_EVENTS=y
# CONFIG_KPROBE_EVENT_GEN_TEST is not set
# CONFIG_KPROBES_SANITY_TEST is not set


> >
> > Indeed inserting a lttng-modules uprobe within functions is not
> > supported at the moment, mainly because we prefer to err towards safety
> > and don't have the validation in place to prevent corrupting the
> > program's instructions if an end user would try to insert a uprobe at an
> > address which is not an instruction boundary.
> >
> > So we only support inserting uprobe on functions and SDT probes at
> > the moment.
> >
> > Thanks,
> >
> > Mathieu
> >
> >
> > > Thanks,
> > >
> > > Brian
> > > _______________________________________________
> > > lttng-dev mailing list
> > > lttng-dev@lists.lttng.org
> > > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> >
> > --
> > Mathieu Desnoyers
> > EfficiOS Inc.
> > https://www.efficios.com
> >
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Trying to understand use of lttng enable-event --kernel --userspace-probe=

[Mathieu Desnoyers via lttng-dev]

On 2023-05-18 10:10, Brian Hutchinson wrote:
[...]
> I updated my hello world to have a function I'd like to use the
> --userspace-probe method on with the very original name of
> 'probe_function':
> 
> #include <stdio.h>
> #include <lttng/tracef.h>
> 
> void probe_function(int i);
> 
> int main(int argc, char *argv[])
> {
>     unsigned int i;
>     puts("Hello, World!\nPress Enter to continue...");
>     /*
>      * The following getchar() call only exists for the purpose of this
>      * demonstration, to pause the application in order for you to have
>      * time to list its tracepoints. You don't need it otherwise.
>      */
>     getchar();
> 
>     lttng_ust_tracef("Number %d, string %s", 23, "hi there!");
>     printf("Number %d, string %s", 23, "hi there!");
> 
>     for (i = 0; i < argc; i++) {
>         lttng_ust_tracef("Number %d, argv %s", i, argv[i]);
>         printf("Number %d, argv %s", i, argv[i]);
>     }
> 
>     puts("Quitting now!");
> 
>     probe_function(i);
> 
>     return 0;
> }
> 
> void probe_function(int i) {
> 
>     lttng_ust_tracef("Number %d, string %s", i * i, "i^2");
>     printf("Number %d, string %s", i * i, "i^2");
> 
> }
> 
> ... and I get the same error as before when I try to enable the probe:
> # lttng enable-event --kernel
> --userspace-probe=/usr/local/bin/hello:probe_function
> Error: Missing event name(s).

As the error states, you are missing the event name. See

man 1 lttng-enable-event

        lttng [GENERAL OPTIONS] enable-event --kernel
              [--probe=SOURCE | --function=SOURCE | --syscall |
               --userspace-probe=SOURCE]
              [--filter=EXPR] [--session=SESSION]
              [--channel=CHANNEL] EVENT[,EVENT]...


You will want something like:

lttng enable-event --kernel --userspace-probe=/usr/local/bin/hello:probe_function my_probe_function

Where "my_probe_function" is the event name that will appear in the collected traces.

Thanks,

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Trying to understand use of lttng enable-event --kernel --userspace-probe=

[Brian Hutchinson via lttng-dev]

On Thu, May 18, 2023 at 10:45 AM Mathieu Desnoyers
<mathieu.desnoyers@efficios.com> wrote:
>
> On 2023-05-18 10:10, Brian Hutchinson wrote:
> [...]
> > I updated my hello world to have a function I'd like to use the
> > --userspace-probe method on with the very original name of
> > 'probe_function':
> >
> > #include <stdio.h>
> > #include <lttng/tracef.h>
> >
> > void probe_function(int i);
> >
> > int main(int argc, char *argv[])
> > {
> >     unsigned int i;
> >     puts("Hello, World!\nPress Enter to continue...");
> >     /*
> >      * The following getchar() call only exists for the purpose of this
> >      * demonstration, to pause the application in order for you to have
> >      * time to list its tracepoints. You don't need it otherwise.
> >      */
> >     getchar();
> >
> >     lttng_ust_tracef("Number %d, string %s", 23, "hi there!");
> >     printf("Number %d, string %s", 23, "hi there!");
> >
> >     for (i = 0; i < argc; i++) {
> >         lttng_ust_tracef("Number %d, argv %s", i, argv[i]);
> >         printf("Number %d, argv %s", i, argv[i]);
> >     }
> >
> >     puts("Quitting now!");
> >
> >     probe_function(i);
> >
> >     return 0;
> > }
> >
> > void probe_function(int i) {
> >
> >     lttng_ust_tracef("Number %d, string %s", i * i, "i^2");
> >     printf("Number %d, string %s", i * i, "i^2");
> >
> > }
> >
> > ... and I get the same error as before when I try to enable the probe:
> > # lttng enable-event --kernel
> > --userspace-probe=/usr/local/bin/hello:probe_function
> > Error: Missing event name(s).
>
> As the error states, you are missing the event name. See
>
> man 1 lttng-enable-event
>
>         lttng [GENERAL OPTIONS] enable-event --kernel
>               [--probe=SOURCE | --function=SOURCE | --syscall |
>                --userspace-probe=SOURCE]
>               [--filter=EXPR] [--session=SESSION]
>               [--channel=CHANNEL] EVENT[,EVENT]...
>
>
> You will want something like:
>
> lttng enable-event --kernel --userspace-probe=/usr/local/bin/hello:probe_function my_probe_function
>
> Where "my_probe_function" is the event name that will appear in the collected traces.

Wow!  I must not have woken up this morning ha, ha.  Thanks for that!
The event is enabled now.  Hope to actually get tracing data now.

Regards,

Brian

>
> Thanks,
>
> Mathieu
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> https://www.efficios.com
>
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Trying to understand use of lttng enable-event --kernel --userspace-probe=

[Brian Hutchinson via lttng-dev]

On Thu, May 18, 2023 at 11:00 AM Brian Hutchinson <b.hutchman@gmail.com> wrote:
>
> On Thu, May 18, 2023 at 10:45 AM Mathieu Desnoyers
> <mathieu.desnoyers@efficios.com> wrote:
> >
> > On 2023-05-18 10:10, Brian Hutchinson wrote:
> > [...]
> > > I updated my hello world to have a function I'd like to use the
> > > --userspace-probe method on with the very original name of
> > > 'probe_function':
> > >
> > > #include <stdio.h>
> > > #include <lttng/tracef.h>
> > >
> > > void probe_function(int i);
> > >
> > > int main(int argc, char *argv[])
> > > {
> > >     unsigned int i;
> > >     puts("Hello, World!\nPress Enter to continue...");
> > >     /*
> > >      * The following getchar() call only exists for the purpose of this
> > >      * demonstration, to pause the application in order for you to have
> > >      * time to list its tracepoints. You don't need it otherwise.
> > >      */
> > >     getchar();
> > >
> > >     lttng_ust_tracef("Number %d, string %s", 23, "hi there!");
> > >     printf("Number %d, string %s", 23, "hi there!");
> > >
> > >     for (i = 0; i < argc; i++) {
> > >         lttng_ust_tracef("Number %d, argv %s", i, argv[i]);
> > >         printf("Number %d, argv %s", i, argv[i]);
> > >     }
> > >
> > >     puts("Quitting now!");
> > >
> > >     probe_function(i);
> > >
> > >     return 0;
> > > }
> > >
> > > void probe_function(int i) {
> > >
> > >     lttng_ust_tracef("Number %d, string %s", i * i, "i^2");
> > >     printf("Number %d, string %s", i * i, "i^2");
> > >
> > > }
> > >
> > > ... and I get the same error as before when I try to enable the probe:
> > > # lttng enable-event --kernel
> > > --userspace-probe=/usr/local/bin/hello:probe_function
> > > Error: Missing event name(s).
> >
> > As the error states, you are missing the event name. See
> >
> > man 1 lttng-enable-event
> >
> >         lttng [GENERAL OPTIONS] enable-event --kernel
> >               [--probe=SOURCE | --function=SOURCE | --syscall |
> >                --userspace-probe=SOURCE]
> >               [--filter=EXPR] [--session=SESSION]
> >               [--channel=CHANNEL] EVENT[,EVENT]...
> >
> >
> > You will want something like:
> >
> > lttng enable-event --kernel --userspace-probe=/usr/local/bin/hello:probe_function my_probe_function
> >
> > Where "my_probe_function" is the event name that will appear in the collected traces.
>
> Wow!  I must not have woken up this morning ha, ha.  Thanks for that!
> The event is enabled now.  Hope to actually get tracing data now.

Well, I guess we just have the app that thwarts all attempts at tracing.

I did a dynamic probe on several functions that should be getting
called like crazy and again I get no tracing data.

Tried it with my hello world example above after Mathieu set me
straight on the event syntax and it works.

I saw this comment in the documentation "As of this version, only USDT
probes that are not surrounded by a reference counter (semaphore) are
supported."

I don't know that I can say that this function I'm probing isn't
"surrounded" by a reference counter, it's in a large multi-threaded
application so I guess it's possible.

Sigh, I'm striking out every which way.

No offense (since this is lttng list - please don't flame me ... I
want/need lttng), but I think I'm going to try just straight kprobes
and uprobes and see if trace compass can show those traces in an
attempt to get "something/anything" working.

Regards,

Brian
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Trying to understand use of lttng enable-event --kernel --userspace-probe=

[Mathieu Desnoyers via lttng-dev]

On 2023-05-18 14:58, Brian Hutchinson wrote:
> On Thu, May 18, 2023 at 11:00 AM Brian Hutchinson <b.hutchman@gmail.com> wrote:
>>
>> On Thu, May 18, 2023 at 10:45 AM Mathieu Desnoyers
>> <mathieu.desnoyers@efficios.com> wrote:
>>>
>>> On 2023-05-18 10:10, Brian Hutchinson wrote:
>>> [...]
>>>> I updated my hello world to have a function I'd like to use the
>>>> --userspace-probe method on with the very original name of
>>>> 'probe_function':
>>>>
>>>> #include <stdio.h>
>>>> #include <lttng/tracef.h>
>>>>
>>>> void probe_function(int i);
>>>>
>>>> int main(int argc, char *argv[])
>>>> {
>>>>      unsigned int i;
>>>>      puts("Hello, World!\nPress Enter to continue...");
>>>>      /*
>>>>       * The following getchar() call only exists for the purpose of this
>>>>       * demonstration, to pause the application in order for you to have
>>>>       * time to list its tracepoints. You don't need it otherwise.
>>>>       */
>>>>      getchar();
>>>>
>>>>      lttng_ust_tracef("Number %d, string %s", 23, "hi there!");
>>>>      printf("Number %d, string %s", 23, "hi there!");
>>>>
>>>>      for (i = 0; i < argc; i++) {
>>>>          lttng_ust_tracef("Number %d, argv %s", i, argv[i]);
>>>>          printf("Number %d, argv %s", i, argv[i]);
>>>>      }
>>>>
>>>>      puts("Quitting now!");
>>>>
>>>>      probe_function(i);
>>>>
>>>>      return 0;
>>>> }
>>>>
>>>> void probe_function(int i) {
>>>>
>>>>      lttng_ust_tracef("Number %d, string %s", i * i, "i^2");
>>>>      printf("Number %d, string %s", i * i, "i^2");
>>>>
>>>> }
>>>>
>>>> ... and I get the same error as before when I try to enable the probe:
>>>> # lttng enable-event --kernel
>>>> --userspace-probe=/usr/local/bin/hello:probe_function
>>>> Error: Missing event name(s).
>>>
>>> As the error states, you are missing the event name. See
>>>
>>> man 1 lttng-enable-event
>>>
>>>          lttng [GENERAL OPTIONS] enable-event --kernel
>>>                [--probe=SOURCE | --function=SOURCE | --syscall |
>>>                 --userspace-probe=SOURCE]
>>>                [--filter=EXPR] [--session=SESSION]
>>>                [--channel=CHANNEL] EVENT[,EVENT]...
>>>
>>>
>>> You will want something like:
>>>
>>> lttng enable-event --kernel --userspace-probe=/usr/local/bin/hello:probe_function my_probe_function
>>>
>>> Where "my_probe_function" is the event name that will appear in the collected traces.
>>
>> Wow!  I must not have woken up this morning ha, ha.  Thanks for that!
>> The event is enabled now.  Hope to actually get tracing data now.
> 
> Well, I guess we just have the app that thwarts all attempts at tracing.
> 
> I did a dynamic probe on several functions that should be getting
> called like crazy and again I get no tracing data.
> 
> Tried it with my hello world example above after Mathieu set me
> straight on the event syntax and it works.
> 
> I saw this comment in the documentation "As of this version, only USDT
> probes that are not surrounded by a reference counter (semaphore) are
> supported."
> 
> I don't know that I can say that this function I'm probing isn't
> "surrounded" by a reference counter, it's in a large multi-threaded
> application so I guess it's possible.
> 
> Sigh, I'm striking out every which way.
> 
> No offense (since this is lttng list - please don't flame me ... I
> want/need lttng), but I think I'm going to try just straight kprobes
> and uprobes and see if trace compass can show those traces in an
> attempt to get "something/anything" working.

If you attach to an ELF symbol (function), then there is no USDT in 
play, so it should not be related to the issue you have.

But if your functions happen to be inlined, then there will be nothing 
to attach to. Perhaps this is what happens there ?

Mathieu

> 
> Regards,
> 
> Brian

-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Trying to understand use of lttng enable-event --kernel --userspace-probe=

[Brian Hutchinson via lttng-dev]

On Thu, May 18, 2023 at 3:03 PM Mathieu Desnoyers
<mathieu.desnoyers@efficios.com> wrote:
>
> On 2023-05-18 14:58, Brian Hutchinson wrote:
> > On Thu, May 18, 2023 at 11:00 AM Brian Hutchinson <b.hutchman@gmail.com> wrote:
> >>
> >> On Thu, May 18, 2023 at 10:45 AM Mathieu Desnoyers
> >> <mathieu.desnoyers@efficios.com> wrote:
> >>>
> >>> On 2023-05-18 10:10, Brian Hutchinson wrote:
> >>> [...]
> >>>> I updated my hello world to have a function I'd like to use the
> >>>> --userspace-probe method on with the very original name of
> >>>> 'probe_function':
> >>>>
> >>>> #include <stdio.h>
> >>>> #include <lttng/tracef.h>
> >>>>
> >>>> void probe_function(int i);
> >>>>
> >>>> int main(int argc, char *argv[])
> >>>> {
> >>>>      unsigned int i;
> >>>>      puts("Hello, World!\nPress Enter to continue...");
> >>>>      /*
> >>>>       * The following getchar() call only exists for the purpose of this
> >>>>       * demonstration, to pause the application in order for you to have
> >>>>       * time to list its tracepoints. You don't need it otherwise.
> >>>>       */
> >>>>      getchar();
> >>>>
> >>>>      lttng_ust_tracef("Number %d, string %s", 23, "hi there!");
> >>>>      printf("Number %d, string %s", 23, "hi there!");
> >>>>
> >>>>      for (i = 0; i < argc; i++) {
> >>>>          lttng_ust_tracef("Number %d, argv %s", i, argv[i]);
> >>>>          printf("Number %d, argv %s", i, argv[i]);
> >>>>      }
> >>>>
> >>>>      puts("Quitting now!");
> >>>>
> >>>>      probe_function(i);
> >>>>
> >>>>      return 0;
> >>>> }
> >>>>
> >>>> void probe_function(int i) {
> >>>>
> >>>>      lttng_ust_tracef("Number %d, string %s", i * i, "i^2");
> >>>>      printf("Number %d, string %s", i * i, "i^2");
> >>>>
> >>>> }
> >>>>
> >>>> ... and I get the same error as before when I try to enable the probe:
> >>>> # lttng enable-event --kernel
> >>>> --userspace-probe=/usr/local/bin/hello:probe_function
> >>>> Error: Missing event name(s).
> >>>
> >>> As the error states, you are missing the event name. See
> >>>
> >>> man 1 lttng-enable-event
> >>>
> >>>          lttng [GENERAL OPTIONS] enable-event --kernel
> >>>                [--probe=SOURCE | --function=SOURCE | --syscall |
> >>>                 --userspace-probe=SOURCE]
> >>>                [--filter=EXPR] [--session=SESSION]
> >>>                [--channel=CHANNEL] EVENT[,EVENT]...
> >>>
> >>>
> >>> You will want something like:
> >>>
> >>> lttng enable-event --kernel --userspace-probe=/usr/local/bin/hello:probe_function my_probe_function
> >>>
> >>> Where "my_probe_function" is the event name that will appear in the collected traces.
> >>
> >> Wow!  I must not have woken up this morning ha, ha.  Thanks for that!
> >> The event is enabled now.  Hope to actually get tracing data now.
> >
> > Well, I guess we just have the app that thwarts all attempts at tracing.
> >
> > I did a dynamic probe on several functions that should be getting
> > called like crazy and again I get no tracing data.
> >
> > Tried it with my hello world example above after Mathieu set me
> > straight on the event syntax and it works.
> >
> > I saw this comment in the documentation "As of this version, only USDT
> > probes that are not surrounded by a reference counter (semaphore) are
> > supported."
> >
> > I don't know that I can say that this function I'm probing isn't
> > "surrounded" by a reference counter, it's in a large multi-threaded
> > application so I guess it's possible.
> >
> > Sigh, I'm striking out every which way.
> >
> > No offense (since this is lttng list - please don't flame me ... I
> > want/need lttng), but I think I'm going to try just straight kprobes
> > and uprobes and see if trace compass can show those traces in an
> > attempt to get "something/anything" working.
>
> If you attach to an ELF symbol (function), then there is no USDT in
> play, so it should not be related to the issue you have.

That is what I was thinking which is why I wanted to try it.

>
> But if your functions happen to be inlined, then there will be nothing
> to attach to. Perhaps this is what happens there ?

I don't see any evidence of anything being inlined in this module.  I
grepped the code to verify.

Back to being stumped/stuck.

B
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Trying to understand use of lttng enable-event --kernel --userspace-probe=

[Mathieu Desnoyers via lttng-dev]

On 2023-05-18 15:07, Brian Hutchinson wrote:

[...]

>>
>> If you attach to an ELF symbol (function), then there is no USDT in
>> play, so it should not be related to the issue you have.
> 
> That is what I was thinking which is why I wanted to try it.
> 
>>
>> But if your functions happen to be inlined, then there will be nothing
>> to attach to. Perhaps this is what happens there ?
> 
> I don't see any evidence of anything being inlined in this module.  I
> grepped the code to verify.
> 
> Back to being stumped/stuck.

Make sure to check the resulting assembler and ELF symbol tables.

The compiler is free to inline various functions unless they are 
explicitly marked as __attribute__((noinline)). Also, if LTO is enabled, 
further optimization can be done at link-time.

One purpose of the UST tracepoints is to be less fragile with respect to 
specific optimizations done by the compiler and linker, thus 
guaranteeing that whatever is instrumented with a tracepoint is indeed 
available for tracing.

Also, double-check that the path you pass to --userspace-probe really 
targets your executable or .so binary file, and is not just a symbolic link.

Thanks,

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Trying to understand use of lttng enable-event --kernel --userspace-probe=

[Brian Hutchinson via lttng-dev]

On Thu, May 18, 2023 at 3:07 PM Brian Hutchinson <b.hutchman@gmail.com> wrote:
>
> On Thu, May 18, 2023 at 3:03 PM Mathieu Desnoyers
> <mathieu.desnoyers@efficios.com> wrote:
> >
> > On 2023-05-18 14:58, Brian Hutchinson wrote:
> > > On Thu, May 18, 2023 at 11:00 AM Brian Hutchinson <b.hutchman@gmail.com> wrote:
> > >>
> > >> On Thu, May 18, 2023 at 10:45 AM Mathieu Desnoyers
> > >> <mathieu.desnoyers@efficios.com> wrote:
> > >>>
> > >>> On 2023-05-18 10:10, Brian Hutchinson wrote:
> > >>> [...]
> > >>>> I updated my hello world to have a function I'd like to use the
> > >>>> --userspace-probe method on with the very original name of
> > >>>> 'probe_function':
> > >>>>
> > >>>> #include <stdio.h>
> > >>>> #include <lttng/tracef.h>
> > >>>>
> > >>>> void probe_function(int i);
> > >>>>
> > >>>> int main(int argc, char *argv[])
> > >>>> {
> > >>>>      unsigned int i;
> > >>>>      puts("Hello, World!\nPress Enter to continue...");
> > >>>>      /*
> > >>>>       * The following getchar() call only exists for the purpose of this
> > >>>>       * demonstration, to pause the application in order for you to have
> > >>>>       * time to list its tracepoints. You don't need it otherwise.
> > >>>>       */
> > >>>>      getchar();
> > >>>>
> > >>>>      lttng_ust_tracef("Number %d, string %s", 23, "hi there!");
> > >>>>      printf("Number %d, string %s", 23, "hi there!");
> > >>>>
> > >>>>      for (i = 0; i < argc; i++) {
> > >>>>          lttng_ust_tracef("Number %d, argv %s", i, argv[i]);
> > >>>>          printf("Number %d, argv %s", i, argv[i]);
> > >>>>      }
> > >>>>
> > >>>>      puts("Quitting now!");
> > >>>>
> > >>>>      probe_function(i);
> > >>>>
> > >>>>      return 0;
> > >>>> }
> > >>>>
> > >>>> void probe_function(int i) {
> > >>>>
> > >>>>      lttng_ust_tracef("Number %d, string %s", i * i, "i^2");
> > >>>>      printf("Number %d, string %s", i * i, "i^2");
> > >>>>
> > >>>> }
> > >>>>
> > >>>> ... and I get the same error as before when I try to enable the probe:
> > >>>> # lttng enable-event --kernel
> > >>>> --userspace-probe=/usr/local/bin/hello:probe_function
> > >>>> Error: Missing event name(s).
> > >>>
> > >>> As the error states, you are missing the event name. See
> > >>>
> > >>> man 1 lttng-enable-event
> > >>>
> > >>>          lttng [GENERAL OPTIONS] enable-event --kernel
> > >>>                [--probe=SOURCE | --function=SOURCE | --syscall |
> > >>>                 --userspace-probe=SOURCE]
> > >>>                [--filter=EXPR] [--session=SESSION]
> > >>>                [--channel=CHANNEL] EVENT[,EVENT]...
> > >>>
> > >>>
> > >>> You will want something like:
> > >>>
> > >>> lttng enable-event --kernel --userspace-probe=/usr/local/bin/hello:probe_function my_probe_function
> > >>>
> > >>> Where "my_probe_function" is the event name that will appear in the collected traces.
> > >>
> > >> Wow!  I must not have woken up this morning ha, ha.  Thanks for that!
> > >> The event is enabled now.  Hope to actually get tracing data now.
> > >
> > > Well, I guess we just have the app that thwarts all attempts at tracing.
> > >
> > > I did a dynamic probe on several functions that should be getting
> > > called like crazy and again I get no tracing data.
> > >
> > > Tried it with my hello world example above after Mathieu set me
> > > straight on the event syntax and it works.
> > >
> > > I saw this comment in the documentation "As of this version, only USDT
> > > probes that are not surrounded by a reference counter (semaphore) are
> > > supported."
> > >
> > > I don't know that I can say that this function I'm probing isn't
> > > "surrounded" by a reference counter, it's in a large multi-threaded
> > > application so I guess it's possible.
> > >
> > > Sigh, I'm striking out every which way.
> > >
> > > No offense (since this is lttng list - please don't flame me ... I
> > > want/need lttng), but I think I'm going to try just straight kprobes
> > > and uprobes and see if trace compass can show those traces in an
> > > attempt to get "something/anything" working.
> >
> > If you attach to an ELF symbol (function), then there is no USDT in
> > play, so it should not be related to the issue you have.
>
> That is what I was thinking which is why I wanted to try it.
>
> >
> > But if your functions happen to be inlined, then there will be nothing
> > to attach to. Perhaps this is what happens there ?
>
> I don't see any evidence of anything being inlined in this module.  I
> grepped the code to verify.
>
> Back to being stumped/stuck.

I can do trace-cmd stuff and it works.  The hello world above works so
I don't "think" this is a problem but again in full disclosure I'll
mention/ask about it.

Does any of the lttng tools/libs depend on kernel headers?  I ask
because old yocto (Dunfell) built lttng package against a 4.something
kernel and we're running a 5.10.69 kernel that lttng modules were
added to it with the "builtin" script and built that way.

Should probably have yocto build the local kernel too, but kernel is
being built stand alone due to vendor stuff that hasn't been mainlined
yet.

I'm running out of things to think about that could be the issue.

B
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Trying to understand use of lttng enable-event --kernel --userspace-probe=

[Mathieu Desnoyers via lttng-dev]

On 2023-05-18 15:20, Brian Hutchinson wrote:
> On Thu, May 18, 2023 at 3:07 PM Brian Hutchinson <b.hutchman@gmail.com> wrote:
>>
>> On Thu, May 18, 2023 at 3:03 PM Mathieu Desnoyers
>> <mathieu.desnoyers@efficios.com> wrote:
>>>
>>> On 2023-05-18 14:58, Brian Hutchinson wrote:
>>>> On Thu, May 18, 2023 at 11:00 AM Brian Hutchinson <b.hutchman@gmail.com> wrote:
>>>>>
>>>>> On Thu, May 18, 2023 at 10:45 AM Mathieu Desnoyers
>>>>> <mathieu.desnoyers@efficios.com> wrote:
>>>>>>
>>>>>> On 2023-05-18 10:10, Brian Hutchinson wrote:
>>>>>> [...]
>>>>>>> I updated my hello world to have a function I'd like to use the
>>>>>>> --userspace-probe method on with the very original name of
>>>>>>> 'probe_function':
>>>>>>>
>>>>>>> #include <stdio.h>
>>>>>>> #include <lttng/tracef.h>
>>>>>>>
>>>>>>> void probe_function(int i);
>>>>>>>
>>>>>>> int main(int argc, char *argv[])
>>>>>>> {
>>>>>>>       unsigned int i;
>>>>>>>       puts("Hello, World!\nPress Enter to continue...");
>>>>>>>       /*
>>>>>>>        * The following getchar() call only exists for the purpose of this
>>>>>>>        * demonstration, to pause the application in order for you to have
>>>>>>>        * time to list its tracepoints. You don't need it otherwise.
>>>>>>>        */
>>>>>>>       getchar();
>>>>>>>
>>>>>>>       lttng_ust_tracef("Number %d, string %s", 23, "hi there!");
>>>>>>>       printf("Number %d, string %s", 23, "hi there!");
>>>>>>>
>>>>>>>       for (i = 0; i < argc; i++) {
>>>>>>>           lttng_ust_tracef("Number %d, argv %s", i, argv[i]);
>>>>>>>           printf("Number %d, argv %s", i, argv[i]);
>>>>>>>       }
>>>>>>>
>>>>>>>       puts("Quitting now!");
>>>>>>>
>>>>>>>       probe_function(i);
>>>>>>>
>>>>>>>       return 0;
>>>>>>> }
>>>>>>>
>>>>>>> void probe_function(int i) {
>>>>>>>
>>>>>>>       lttng_ust_tracef("Number %d, string %s", i * i, "i^2");
>>>>>>>       printf("Number %d, string %s", i * i, "i^2");
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> ... and I get the same error as before when I try to enable the probe:
>>>>>>> # lttng enable-event --kernel
>>>>>>> --userspace-probe=/usr/local/bin/hello:probe_function
>>>>>>> Error: Missing event name(s).
>>>>>>
>>>>>> As the error states, you are missing the event name. See
>>>>>>
>>>>>> man 1 lttng-enable-event
>>>>>>
>>>>>>           lttng [GENERAL OPTIONS] enable-event --kernel
>>>>>>                 [--probe=SOURCE | --function=SOURCE | --syscall |
>>>>>>                  --userspace-probe=SOURCE]
>>>>>>                 [--filter=EXPR] [--session=SESSION]
>>>>>>                 [--channel=CHANNEL] EVENT[,EVENT]...
>>>>>>
>>>>>>
>>>>>> You will want something like:
>>>>>>
>>>>>> lttng enable-event --kernel --userspace-probe=/usr/local/bin/hello:probe_function my_probe_function
>>>>>>
>>>>>> Where "my_probe_function" is the event name that will appear in the collected traces.
>>>>>
>>>>> Wow!  I must not have woken up this morning ha, ha.  Thanks for that!
>>>>> The event is enabled now.  Hope to actually get tracing data now.
>>>>
>>>> Well, I guess we just have the app that thwarts all attempts at tracing.
>>>>
>>>> I did a dynamic probe on several functions that should be getting
>>>> called like crazy and again I get no tracing data.
>>>>
>>>> Tried it with my hello world example above after Mathieu set me
>>>> straight on the event syntax and it works.
>>>>
>>>> I saw this comment in the documentation "As of this version, only USDT
>>>> probes that are not surrounded by a reference counter (semaphore) are
>>>> supported."
>>>>
>>>> I don't know that I can say that this function I'm probing isn't
>>>> "surrounded" by a reference counter, it's in a large multi-threaded
>>>> application so I guess it's possible.
>>>>
>>>> Sigh, I'm striking out every which way.
>>>>
>>>> No offense (since this is lttng list - please don't flame me ... I
>>>> want/need lttng), but I think I'm going to try just straight kprobes
>>>> and uprobes and see if trace compass can show those traces in an
>>>> attempt to get "something/anything" working.
>>>
>>> If you attach to an ELF symbol (function), then there is no USDT in
>>> play, so it should not be related to the issue you have.
>>
>> That is what I was thinking which is why I wanted to try it.
>>
>>>
>>> But if your functions happen to be inlined, then there will be nothing
>>> to attach to. Perhaps this is what happens there ?
>>
>> I don't see any evidence of anything being inlined in this module.  I
>> grepped the code to verify.
>>
>> Back to being stumped/stuck.
> 
> I can do trace-cmd stuff and it works.  The hello world above works so
> I don't "think" this is a problem but again in full disclosure I'll
> mention/ask about it.
> 
> Does any of the lttng tools/libs depend on kernel headers?  I ask
> because old yocto (Dunfell) built lttng package against a 4.something
> kernel and we're running a 5.10.69 kernel that lttng modules were
> added to it with the "builtin" script and built that way.
> 
> Should probably have yocto build the local kernel too, but kernel is
> being built stand alone due to vendor stuff that hasn't been mainlined
> yet.
> 
> I'm running out of things to think about that could be the issue.

If lttng-modules can trace your smaller test application through 
uprobes, then the problem is likely elsewhere.

Only lttng-modules has dependencies on kernel headers. lttng-tools/ust 
don't depend on kernel headers.

Mathieu


-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Trying to understand use of lttng enable-event --kernel --userspace-probe=

[Brian Hutchinson via lttng-dev]

On Thu, May 18, 2023 at 3:16 PM Mathieu Desnoyers
<mathieu.desnoyers@efficios.com> wrote:
>
> On 2023-05-18 15:07, Brian Hutchinson wrote:
>
> [...]
>
> >>
> >> If you attach to an ELF symbol (function), then there is no USDT in
> >> play, so it should not be related to the issue you have.
> >
> > That is what I was thinking which is why I wanted to try it.
> >
> >>
> >> But if your functions happen to be inlined, then there will be nothing
> >> to attach to. Perhaps this is what happens there ?
> >
> > I don't see any evidence of anything being inlined in this module.  I
> > grepped the code to verify.
> >
> > Back to being stumped/stuck.
>
> Make sure to check the resulting assembler and ELF symbol tables.
>
> The compiler is free to inline various functions unless they are
> explicitly marked as __attribute__((noinline)). Also, if LTO is enabled,

I could put that in the code just to make sure.  It's a big function
so doubt it but I'll try anything at this point ha, ha.

I wondered if optimization could be doing it.

I can provide the standard yocto environment script that's being
source that was generated for the sdk:

$ cat environment-setup-aarch64-poky-linux


# Check for LD_LIBRARY_PATH being set, which can break SDK and
generally is a bad practice
# http://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html#AEN80
# http://xahlee.info/UnixResource_dir/_/ldpath.html
# Only disable this check if you are absolutely know what you are doing!
if [ ! -z "${LD_LIBRARY_PATH:-}" ]; then
   echo "Your environment is misconfigured, you probably need to
'unset LD_LIBRARY_PATH'"
   echo "but please check why this was set in the first place and that
it's safe to unset."
   echo "The SDK will not operate correctly in most cases when
LD_LIBRARY_PATH is set."
   echo "For more references see:"
   echo "  http://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html#AEN80"
   echo "  http://xahlee.info/UnixResource_dir/_/ldpath.html"
   return 1
fi
export SDKTARGETSYSROOT=/opt/poky/3.1.24/sysroots/aarch64-poky-linux
export PATH=/opt/poky/3.1.24/sysroots/x86_64-pokysdk-linux/usr/bin:/opt/poky/3.1.24/sysroots/x86_64-pokysdk-linux/usr/sbin:/opt/poky/3.1.24/sysroots/x86_64-pokysdk-linux/bin:/opt/poky/3.1.24/sysroots/x86_64-pokysdk-linux/sbin:/opt/poky/3.1.24/sysroots/x86_64-pokysdk-linux/usr/
bin/../x86_64-pokysdk-linux/bin:/opt/poky/3.1.24/sysroots/x86_64-pokysdk-linux/usr/bin/aarch64-poky-linux:/opt/poky/3.1.24/sysroots/x86_64-pokysdk-linux/usr/bin/aarch64-poky-linux-musl:"$PATH"
export PKG_CONFIG_SYSROOT_DIR=$SDKTARGETSYSROOT
export PKG_CONFIG_PATH=$SDKTARGETSYSROOT/usr/lib/pkgconfig:$SDKTARGETSYSROOT/usr/share/pkgconfig
export CONFIG_SITE=/opt/poky/3.1.24/site-config-aarch64-poky-linux
export OECORE_NATIVE_SYSROOT="/opt/poky/3.1.24/sysroots/x86_64-pokysdk-linux"
export OECORE_TARGET_SYSROOT="$SDKTARGETSYSROOT"
export OECORE_ACLOCAL_OPTS="-I
/opt/poky/3.1.24/sysroots/x86_64-pokysdk-linux/usr/share/aclocal"
export OECORE_BASELIB="lib"
export OECORE_TARGET_ARCH="aarch64"
export OECORE_TARGET_OS="linux"
unset command_not_found_handle
export CC="aarch64-poky-linux-gcc  -mcpu=cortex-a53+crc+crypto
-fstack-protector-strong  -D_FORTIFY_SOURCE=2 -Wformat
-Wformat-security -Werror=format-security --sysroot=$SDKTARGETSYSROOT"
export CXX="aarch64-poky-linux-g++  -mcpu=cortex-a53+crc+crypto
-fstack-protector-strong  -D_FORTIFY_SOURCE=2 -Wformat
-Wformat-security -Werror=format-security --sysroot=$SDKTARGETSYSROOT"
export CPP="aarch64-poky-linux-gcc -E  -mcpu=cortex-a53+crc+crypto
-fstack-protector-strong  -D_FORTIFY_SOURCE=2 -Wformat
-Wformat-security -Werror=format-security --sysroot=$SDKTARGETSYSROOT"
export AS="aarch64-poky-linux-as "
export LD="aarch64-poky-linux-ld  --sysroot=$SDKTARGETSYSROOT"
export GDB=aarch64-poky-linux-gdb
export STRIP=aarch64-poky-linux-strip
export RANLIB=aarch64-poky-linux-ranlib
export OBJCOPY=aarch64-poky-linux-objcopy
export OBJDUMP=aarch64-poky-linux-objdump
export READELF=aarch64-poky-linux-readelf
export AR=aarch64-poky-linux-ar
export NM=aarch64-poky-linux-nm
export M4=m4
export TARGET_PREFIX=aarch64-poky-linux-
export CONFIGURE_FLAGS="--target=aarch64-poky-linux
--host=aarch64-poky-linux --build=x86_64-linux
--with-libtool-sysroot=$SDKTARGETSYSROOT"
export CFLAGS=" -O2 -pipe -g -feliminate-unused-debug-types "
export CXXFLAGS=" -O2 -pipe -g -feliminate-unused-debug-types "
export LDFLAGS="-Wl,-O1 -Wl,--hash-style=gnu -Wl,--as-needed
-fstack-protector-strong -Wl,-z,relro,-z,now"
export CPPFLAGS=""
export KCFLAGS="--sysroot=$SDKTARGETSYSROOT"
export OECORE_DISTRO_VERSION="3.1.24"
export OECORE_SDK_VERSION="3.1.24"
export ARCH=arm64
export CROSS_COMPILE=aarch64-poky-linux-

# Append environment subscripts
if [ -d "$OECORE_TARGET_SYSROOT/environment-setup.d" ]; then
   for envfile in $OECORE_TARGET_SYSROOT/environment-setup.d/*.sh; do
           . $envfile
   done
fi
if [ -d "$OECORE_NATIVE_SYSROOT/environment-setup.d" ]; then
   for envfile in $OECORE_NATIVE_SYSROOT/environment-setup.d/*.sh; do
           . $envfile
   done
fi


> further optimization can be done at link-time.
>
> One purpose of the UST tracepoints is to be less fragile with respect to
> specific optimizations done by the compiler and linker, thus
> guaranteeing that whatever is instrumented with a tracepoint is indeed
> available for tracing.
>
> Also, double-check that the path you pass to --userspace-probe really
> targets your executable or .so binary file, and is not just a symbolic link.

It is full path.
>
> Thanks,
>
> Mathieu
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> https://www.efficios.com
>
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Trying to understand use of lttng enable-event --kernel --userspace-probe=

[Brian Hutchinson via lttng-dev]

On Thu, May 18, 2023 at 3:24 PM Mathieu Desnoyers
<mathieu.desnoyers@efficios.com> wrote:
>
> On 2023-05-18 15:20, Brian Hutchinson wrote:
> > On Thu, May 18, 2023 at 3:07 PM Brian Hutchinson <b.hutchman@gmail.com> wrote:
> >>
> >> On Thu, May 18, 2023 at 3:03 PM Mathieu Desnoyers
> >> <mathieu.desnoyers@efficios.com> wrote:
> >>>
> >>> On 2023-05-18 14:58, Brian Hutchinson wrote:
> >>>> On Thu, May 18, 2023 at 11:00 AM Brian Hutchinson <b.hutchman@gmail.com> wrote:
> >>>>>
> >>>>> On Thu, May 18, 2023 at 10:45 AM Mathieu Desnoyers
> >>>>> <mathieu.desnoyers@efficios.com> wrote:
> >>>>>>
> >>>>>> On 2023-05-18 10:10, Brian Hutchinson wrote:
> >>>>>> [...]
> >>>>>>> I updated my hello world to have a function I'd like to use the
> >>>>>>> --userspace-probe method on with the very original name of
> >>>>>>> 'probe_function':
> >>>>>>>
> >>>>>>> #include <stdio.h>
> >>>>>>> #include <lttng/tracef.h>
> >>>>>>>
> >>>>>>> void probe_function(int i);
> >>>>>>>
> >>>>>>> int main(int argc, char *argv[])
> >>>>>>> {
> >>>>>>>       unsigned int i;
> >>>>>>>       puts("Hello, World!\nPress Enter to continue...");
> >>>>>>>       /*
> >>>>>>>        * The following getchar() call only exists for the purpose of this
> >>>>>>>        * demonstration, to pause the application in order for you to have
> >>>>>>>        * time to list its tracepoints. You don't need it otherwise.
> >>>>>>>        */
> >>>>>>>       getchar();
> >>>>>>>
> >>>>>>>       lttng_ust_tracef("Number %d, string %s", 23, "hi there!");
> >>>>>>>       printf("Number %d, string %s", 23, "hi there!");
> >>>>>>>
> >>>>>>>       for (i = 0; i < argc; i++) {
> >>>>>>>           lttng_ust_tracef("Number %d, argv %s", i, argv[i]);
> >>>>>>>           printf("Number %d, argv %s", i, argv[i]);
> >>>>>>>       }
> >>>>>>>
> >>>>>>>       puts("Quitting now!");
> >>>>>>>
> >>>>>>>       probe_function(i);
> >>>>>>>
> >>>>>>>       return 0;
> >>>>>>> }
> >>>>>>>
> >>>>>>> void probe_function(int i) {
> >>>>>>>
> >>>>>>>       lttng_ust_tracef("Number %d, string %s", i * i, "i^2");
> >>>>>>>       printf("Number %d, string %s", i * i, "i^2");
> >>>>>>>
> >>>>>>> }
> >>>>>>>
> >>>>>>> ... and I get the same error as before when I try to enable the probe:
> >>>>>>> # lttng enable-event --kernel
> >>>>>>> --userspace-probe=/usr/local/bin/hello:probe_function
> >>>>>>> Error: Missing event name(s).
> >>>>>>
> >>>>>> As the error states, you are missing the event name. See
> >>>>>>
> >>>>>> man 1 lttng-enable-event
> >>>>>>
> >>>>>>           lttng [GENERAL OPTIONS] enable-event --kernel
> >>>>>>                 [--probe=SOURCE | --function=SOURCE | --syscall |
> >>>>>>                  --userspace-probe=SOURCE]
> >>>>>>                 [--filter=EXPR] [--session=SESSION]
> >>>>>>                 [--channel=CHANNEL] EVENT[,EVENT]...
> >>>>>>
> >>>>>>
> >>>>>> You will want something like:
> >>>>>>
> >>>>>> lttng enable-event --kernel --userspace-probe=/usr/local/bin/hello:probe_function my_probe_function
> >>>>>>
> >>>>>> Where "my_probe_function" is the event name that will appear in the collected traces.
> >>>>>
> >>>>> Wow!  I must not have woken up this morning ha, ha.  Thanks for that!
> >>>>> The event is enabled now.  Hope to actually get tracing data now.
> >>>>
> >>>> Well, I guess we just have the app that thwarts all attempts at tracing.
> >>>>
> >>>> I did a dynamic probe on several functions that should be getting
> >>>> called like crazy and again I get no tracing data.
> >>>>
> >>>> Tried it with my hello world example above after Mathieu set me
> >>>> straight on the event syntax and it works.
> >>>>
> >>>> I saw this comment in the documentation "As of this version, only USDT
> >>>> probes that are not surrounded by a reference counter (semaphore) are
> >>>> supported."
> >>>>
> >>>> I don't know that I can say that this function I'm probing isn't
> >>>> "surrounded" by a reference counter, it's in a large multi-threaded
> >>>> application so I guess it's possible.
> >>>>
> >>>> Sigh, I'm striking out every which way.
> >>>>
> >>>> No offense (since this is lttng list - please don't flame me ... I
> >>>> want/need lttng), but I think I'm going to try just straight kprobes
> >>>> and uprobes and see if trace compass can show those traces in an
> >>>> attempt to get "something/anything" working.
> >>>
> >>> If you attach to an ELF symbol (function), then there is no USDT in
> >>> play, so it should not be related to the issue you have.
> >>
> >> That is what I was thinking which is why I wanted to try it.
> >>
> >>>
> >>> But if your functions happen to be inlined, then there will be nothing
> >>> to attach to. Perhaps this is what happens there ?
> >>
> >> I don't see any evidence of anything being inlined in this module.  I
> >> grepped the code to verify.
> >>
> >> Back to being stumped/stuck.
> >
> > I can do trace-cmd stuff and it works.  The hello world above works so
> > I don't "think" this is a problem but again in full disclosure I'll
> > mention/ask about it.
> >
> > Does any of the lttng tools/libs depend on kernel headers?  I ask
> > because old yocto (Dunfell) built lttng package against a 4.something
> > kernel and we're running a 5.10.69 kernel that lttng modules were
> > added to it with the "builtin" script and built that way.
> >
> > Should probably have yocto build the local kernel too, but kernel is
> > being built stand alone due to vendor stuff that hasn't been mainlined
> > yet.
> >
> > I'm running out of things to think about that could be the issue.
>
> If lttng-modules can trace your smaller test application through
> uprobes, then the problem is likely elsewhere.
>
> Only lttng-modules has dependencies on kernel headers. lttng-tools/ust
> don't depend on kernel headers.
>
> Mathieu

Ok, that's what I thought but to cover all bases just wanted to make sure.

B
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev