[lttng-dev] Can't trace function calls

[lttng-dev] Can't trace function calls

[Ramesh Errabolu via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 1086 bytes --]

I am trying to capture the list of functions being called by an app. The
app relies on a couple of shared libraries (libhsa-runtime64.so and
libhsakmt.so).

My experiments all *FAIL *i.e. there are no log files of the output. The
link on lttng-ust-cyg-profile does not tell me where I can get them.
Instrument the build files for the two libraries to allow function tracing
by the gcc/clang flags -finstrument-functions and run workload as follows

   - Instrument the build files for the two libraries to allow function
   tracing by the gcc/clang flags *-finstrument-functions* and run workload
   as follows
      - LD_PRELOAD=liblttng-ust-cyg-profile.so   ./rocminfo
   - Instrument the application as well for the gcc/clang flag along with
   the two libs
   - LD_PRELOAD=liblttng-ust-cyg-profile.so   ./rocminfo

I am using LTTng version Ubuntu: LTTng Stable 2.12 PPA

   - apt-get install lttng-tools
   - apt-get install lttng-modules-dkms
   - apt-get install liblttng-ust-dev

I noticed no kernel modules with substring "ltt" in it.

Any help is appreciated.

Regards,
Ramesh

[-- Attachment #1.2: Type: text/html, Size: 1937 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Can't trace function calls

[Jonathan Rajotte-Julien via lttng-dev]

Hi,

On Wed, Mar 31, 2021 at 10:51:03AM -0500, Ramesh Errabolu via lttng-dev wrote:
> I am trying to capture the list of functions being called by an app. The
> app relies on a couple of shared libraries (libhsa-runtime64.so and
> libhsakmt.so).

ok.

> 
> My experiments all *FAIL *i.e. there are no log files of the output.

Not sure what you mean here. But let's continue first. 

> The
> link on lttng-ust-cyg-profile does not tell me where I can get them.

Could you share the "link" in question?

> Instrument the build files for the two libraries to allow function tracing
> by the gcc/clang flags -finstrument-functions and run workload as follows
> 
>    - Instrument the build files for the two libraries to allow function
>    tracing by the gcc/clang flags *-finstrument-functions* and run workload
>    as follows

Okai. I expect that you have read this:
https://lttng.org/docs/v2.12/#doc-liblttng-ust-cyg-profile 

and this
https://lttng.org/man/3/lttng-ust-cyg-profile/v2.12/

> I am using LTTng version Ubuntu: LTTng Stable 2.12 PPA
> 
>    - apt-get install lttng-tools
>    - apt-get install lttng-modules-dkms
>    - apt-get install liblttng-ust-dev
> 

Looks good.

> I noticed no kernel modules with substring "ltt" in it.

Okai so this is where we need much more information. 

First of all, what are the lttng commands used for your experiments? (lttng create, lttng enable-channel, lttng enable-event...) 
Please provide a sample script of your experiments. 

Second, you are mostly performing userspace tracing here, lttng-modules ( the
kernel tracer) might not even be needed here unless you also want to do kernel
tracing. 

What do you mean by "there are no log files of the output."? LTTng produces
traces encoded in CTF (Common Trace Format) that must be read by a reader
(Babeltrace most of the time).

Cheers

-- 
Jonathan Rajotte-Julien
EfficiOS
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Can't trace function calls

[Ramesh Errabolu via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 3892 bytes --]

Thanks for the quick response, appreciate it.

At a high level this is what I did.

   1. reboot
   2. login as sudo
   3. set LD_LIBRARY_PATH to point to right locations
   4. ldd of executable to show right library versions are picked up
   5. execute the workload by specifiy LD_PRELOAD env variable
   6. expect to see a trace file in my $HOME/ltt* directory

When I say *FAIL*, I mean that there was no trace file in my home directory
~/lttng*

I read - https://lttng.org/man/3/lttng-ust-cyg-profile/v2.12/
I didn't read - https://lttng.org/docs/v2.12/#doc-liblttng-ust-cyg-profile

I have not run any lttng commands besides what I already gave. If they are
needed, I don't know what they are.

I tried experimenting with lttng commands create and list. The command to
list "traceable" points returned an error as included below.

root@RocrLnx23:~/git/compute/out/ubuntu-18.04/18.04/bin# ls ~/ | grep -i ltt
root@RocrLnx23:~/git/compute/out/ubuntu-18.04/18.04/bin# lttng create
my-kernel-session --output=~/my-kernel-trace
Session my-kernel-session created.
Traces will be output to
/home/user1/git/compute/out/ubuntu-18.04/18.04/bin/~/my-kernel-trace
root@RocrLnx23:~/git/compute/out/ubuntu-18.04/18.04/bin# *lttng list
--kerne*l
*Error: Unable to list kernel events: Kernel tracer not available*
root@RocrLnx23:~/git/compute/out/ubuntu-18.04/18.04/bin#


A grep for lttng related processes shows me a daemon service running.

root@RocrLnx23:~/git/compute/out/ubuntu-18.04/18.04/bin# ps -ef | grep ltt
root      1002     1  0 12:16 ?        00:00:00 /usr/bin/lttng-sessiond
root      1054  1002  0 12:16 ?        00:00:00 */usr/bin/lttng-sessiond*
root      3145  2861  0 12:51 pts/0    00:00:00 grep --color=auto ltt
root@RocrLnx23:~/git/compute/out/ubuntu-18.04/18.04/bin#


Regards,
Ramesh


On Wed, Mar 31, 2021 at 12:42 PM Jonathan Rajotte-Julien <
jonathan.rajotte-julien@efficios.com> wrote:

> Hi,
>
> On Wed, Mar 31, 2021 at 10:51:03AM -0500, Ramesh Errabolu via lttng-dev
> wrote:
> > I am trying to capture the list of functions being called by an app. The
> > app relies on a couple of shared libraries (libhsa-runtime64.so and
> > libhsakmt.so).
>
> ok.
>
> >
> > My experiments all *FAIL *i.e. there are no log files of the output.
>
> Not sure what you mean here. But let's continue first.
>
> > The
> > link on lttng-ust-cyg-profile does not tell me where I can get them.
>
> Could you share the "link" in question?
>
> > Instrument the build files for the two libraries to allow function
> tracing
> > by the gcc/clang flags -finstrument-functions and run workload as follows
> >
> >    - Instrument the build files for the two libraries to allow function
> >    tracing by the gcc/clang flags *-finstrument-functions* and run
> workload
> >    as follows
>
> Okai. I expect that you have read this:
> https://lttng.org/docs/v2.12/#doc-liblttng-ust-cyg-profile
>
> and this
> https://lttng.org/man/3/lttng-ust-cyg-profile/v2.12/
>
> > I am using LTTng version Ubuntu: LTTng Stable 2.12 PPA
> >
> >    - apt-get install lttng-tools
> >    - apt-get install lttng-modules-dkms
> >    - apt-get install liblttng-ust-dev
> >
>
> Looks good.
>
> > I noticed no kernel modules with substring "ltt" in it.
>
> Okai so this is where we need much more information.
>
> First of all, what are the lttng commands used for your experiments?
> (lttng create, lttng enable-channel, lttng enable-event...)
> Please provide a sample script of your experiments.
>
> Second, you are mostly performing userspace tracing here, lttng-modules (
> the
> kernel tracer) might not even be needed here unless you also want to do
> kernel
> tracing.
>
> What do you mean by "there are no log files of the output."? LTTng produces
> traces encoded in CTF (Common Trace Format) that must be read by a reader
> (Babeltrace most of the time).
>
> Cheers
>
> --
> Jonathan Rajotte-Julien
> EfficiOS
>

[-- Attachment #1.2: Type: text/html, Size: 7616 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Can't trace function calls

[Jonathan Rajotte-Julien via lttng-dev]

On Wed, Mar 31, 2021 at 12:55:53PM -0500, Ramesh Errabolu wrote:
> root@RocrLnx23:~/git/compute/out/ubuntu-18.04/18.04/bin# ls ~/ | grep -i ltt
> root@RocrLnx23:~/git/compute/out/ubuntu-18.04/18.04/bin# lttng create
> my-kernel-session --output=~/my-kernel-trace
> Session my-kernel-session created.
> Traces will be output to
> /home/user1/git/compute/out/ubuntu-18.04/18.04/bin/~/my-kernel-trace
> root@RocrLnx23:~/git/compute/out/ubuntu-18.04/18.04/bin# *lttng list
> --kerne*l
> *Error: Unable to list kernel events: Kernel tracer not available*

Well this would be the first thing to look at.

First let's deactivate the lttng-sessiond.service installed by the packages.

   systemctl stop lttng-sessiond.service

You might want to reanable it later.

> 
> root@RocrLnx23:~/git/compute/out/ubuntu-18.04/18.04/bin# ps -ef | grep ltt
> root      1002     1  0 12:16 ?        00:00:00 /usr/bin/lttng-sessiond
> root      1054  1002  0 12:16 ?        00:00:00 */usr/bin/lttng-sessiond*
> root      3145  2861  0 12:51 pts/0    00:00:00 grep --color=auto ltt
> root@RocrLnx23:~/git/compute/out/ubuntu-18.04/18.04/bin#

Make sure after the systemctl call that no lttng-sessiond process is running.

Now let's launch a lttng-sessiond by hand with a bit more verbosity. For now
let's stick to the root user.

  # lttng-sessiond -vvv --verbose-consumer -b > /tmp/lttng-sessiond.log 2>&1 
  # pkill lttng-sessiond

Please share the content of /tmp/lttng-sessiond.log using a pasting service
(paste.ubuntu.com).

Also please provide the output of:

 # uname -a
 # find /lib/modules/$(uname -r)/ | grep lttng
 # dmesg | grep lttng


But again, the cyg-profile helper library is meant for Userspace tracing.

With the app running and having the LD_PRELOAD correctly set, and a sessiond
running.

 # lttng list -u 

If there is nothing, well you can start the application with the following and
share the output of it (make sure to remove any output from your application if
sensitive data is present)

 # LD_PRELOAD=.... LTTNG_UST_DEBUG=1 your_application_here

Note that debug log will be outputted on stderr.

Cheers

-- 
Jonathan Rajotte-Julien
EfficiOS
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev