Re: lttng-tracer missing

Re: lttng-tracer missing

[Nathan Lynch]

On 02/17/2015 11:02 AM, Neil Bryan wrote:
> Following the bitbake build I see these kernel modules in the build
> output directory:
> 
> ./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/lib/lib-ring-buffer.ko
> ./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/ltt-ring-buffer-client-overwrite.ko
> ./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/probes/lttng-probe-irq.ko
> ./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/probes/lttng-probe-lttng.ko
> ./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/probes/lttng-probe-block.ko
> ./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/probes/lttng-probe-sched.ko
> ./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/probes/lttng-types.ko
> ./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/ltt-ring-buffer-metadata-mmap-client.ko
> ./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/ltt-relay.ko
> ./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/ltt-ring-buffer-metadata-client.ko

I'm guessing you have a rather old version of lttng-modules?  The "ltt-"
prefix was replaced with "lttng-" about three years ago:

commit a90917c3f8c4
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Date:   Sun Jan 22 16:13:41 2012 -0500

    Mass rename: ltt_*/ltt-* to LTTNG_*/LTTNG-*

$ git describe --all  a90917c3f8c4
tags/v2.0-pre11-5-ga90917c3f8c4

Make sure you're not inadvertently using an old unsupported 0.x version
of the modules.

Re: lttng-tracer missing

[Neil Bryan]

On 02/19/2015 03:02 AM, Neil Bryan wrote:
> Hi Nathan,
> 
> Thank you for the prompt response. I have fixed the kernel tracing problem and indeed your diagnosis was accurate.

I'm glad to have helped, but...

> Step 4) Within the recipe 
> ~/yocto/meta/recipes-kernel/lttng-2.0/lttng-modules_2.0.5.bb
> 
> Change the following lines:
> 
> SRCREV = "4d3e89e379fc66480d729abe8daa5c86eb585400"
> PV = "2.0.pre11+git${SRCREV}"
> 
> To:
> 
> SRCREV = "497051bdaa2b09c77860eb3be4f93fc4975e388e"
> PV = "2.0.pre12+git${SRCREV}"
> 
> The SRCREV is now pointing at the commit checksum and the package revision is now 2.0.pre12.

This is still very old... I should have been more explicit earlier: you should try to use version 2.5.x or later (both modules and tools) in order to get help with runtime issues here.

I see. My concern is that, at this stage, I am stuck with version 3.7 of the kernel. I shall try modifying the bitbake recipe to pick up more recent versions of modules and tools, but this could send me into a different world of pain (I am new to Yocto and LTTng integration).

My plan is to migrate to a much more recent version of kernel and LTTng when my confidence increases, but Altera provide a snapshot of the Yocto sources and at the moment I am not brave enough to wander too far away from that.

Re: lttng-tracer missing

[Nathan Lynch]

On 02/19/2015 03:02 AM, Neil Bryan wrote:
> Hi Nathan,
> 
> Thank you for the prompt response. I have fixed the kernel tracing problem and indeed your diagnosis was accurate.

I'm glad to have helped, but...

> Step 4) Within the recipe ~/yocto/meta/recipes-kernel/lttng-2.0/lttng-modules_2.0.5.bb
> 
> Change the following lines:
> 
> SRCREV = "4d3e89e379fc66480d729abe8daa5c86eb585400"
> PV = "2.0.pre11+git${SRCREV}"
> 
> To:
> 
> SRCREV = "497051bdaa2b09c77860eb3be4f93fc4975e388e"
> PV = "2.0.pre12+git${SRCREV}"
> 
> The SRCREV is now pointing at the commit checksum and the package revision is now 2.0.pre12.

This is still very old... I should have been more explicit earlier: you
should try to use version 2.5.x or later (both modules and tools) in
order to get help with runtime issues here.

Re: lttng-tracer missing

[Neil Bryan]

Hi Nathan,

Thank you for the prompt response. I have fixed the kernel tracing problem and indeed your diagnosis was accurate.
I shall post the changes I made to solve the problem so that there is a searchable record for people in a similar position using the Poky component of the Yocto project for the Altera SoCFPGA.

Step 1) Find the recipe responsible for building lttng-modules.

On my installation the recipe can be found here:

~/yocto/meta/recipes-kernel/lttng-2.0

Step 2) Go to the LTTng git repository and look for the tag described by Nathan (v2.0-pre11):

http://git.lttng.org/?p=lttng-modules.git;a=tags

I diff'ed the next tagged commit (v2.0_pre12) and the changes looked minimal so I took a chance that it would work with my kernel/RFS.

Step 3) Examine the changelog (click on tag link) and obtain the commit MD5 checksum, which in this instance is:

497051bdaa2b09c77860eb3be4f93fc4975e388e

Step 4) Within the recipe ~/yocto/meta/recipes-kernel/lttng-2.0/lttng-modules_2.0.5.bb

Change the following lines:

SRCREV = "4d3e89e379fc66480d729abe8daa5c86eb585400"
PV = "2.0.pre11+git${SRCREV}"

To:

SRCREV = "497051bdaa2b09c77860eb3be4f93fc4975e388e"
PV = "2.0.pre12+git${SRCREV}"

The SRCREV is now pointing at the commit checksum and the package revision is now 2.0.pre12.

Step 5) Rebuild the project:

~/yocto/build$ bitbake altera-image

Now I can see the kernel-=tracer.ko module:

~/yocto/build/tmp/work/socfpga_cyclone5-poky-linux-gnueabi/lttng-modules-2.0.pre12+git497051bdaa2b09c77860eb3be4f93fc4975e388e-r0$ find -name *.ko <SNIP> ./sysroot-destdir/lib/modules/3.7.0/kernel/lttng-modules/lib/lttng-lib-ring-buffer.ko
./sysroot-destdir/lib/modules/3.7.0/kernel/lttng-modules/lttng-ring-buffer-metadata-client.ko
./sysroot-destdir/lib/modules/3.7.0/kernel/lttng-modules/lttng-ring-buffer-client-discard.ko
./sysroot-destdir/lib/modules/3.7.0/kernel/lttng-modules/lttng-ring-buffer-client-mmap-overwrite.ko
./sysroot-destdir/lib/modules/3.7.0/kernel/lttng-modules/probes/lttng-probe-irq.ko
./sysroot-destdir/lib/modules/3.7.0/kernel/lttng-modules/probes/lttng-probe-lttng.ko
./sysroot-destdir/lib/modules/3.7.0/kernel/lttng-modules/probes/lttng-probe-block.ko
./sysroot-destdir/lib/modules/3.7.0/kernel/lttng-modules/probes/lttng-probe-sched.ko
./sysroot-destdir/lib/modules/3.7.0/kernel/lttng-modules/probes/lttng-types.ko
./sysroot-destdir/lib/modules/3.7.0/kernel/lttng-modules/lttng-tracer.ko
./sysroot-destdir/lib/modules/3.7.0/kernel/lttng-modules/lttng-ring-buffer-client-mmap-discard.ko
./sysroot-destdir/lib/modules/3.7.0/kernel/lttng-modules/lttng-ring-buffer-metadata-mmap-client.ko
./sysroot-destdir/lib/modules/3.7.0/kernel/lttng-modules/lttng-ring-buffer-client-overwrite.ko

Also the names are all lttng-* rather than a couple of ltt-*

Step 6) Recreate the SD card or update the touched components on the SD card (RFS and kernel).

Step 7) Cross fingers, boot and test.

Thanks again and regards,

Neil.

-----Original Message-----
From: Nathan Lynch [mailto:Nathan_Lynch@mentor.com]
Sent: Tuesday, February 17, 2015 7:00 PM
To: Neil Bryan
Cc: lttng-dev@lists.lttng.org
Subject: Re: [lttng-dev] lttng-tracer missing

On 02/17/2015 11:02 AM, Neil Bryan wrote:
> Following the bitbake build I see these kernel modules in the build 
> output directory:
> 
> ./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/li
> b/lib-ring-buffer.ko
> ./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/lt
> t-ring-buffer-client-overwrite.ko
> ./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/pr
> obes/lttng-probe-irq.ko
> ./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/pr
> obes/lttng-probe-lttng.ko
> ./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/pr
> obes/lttng-probe-block.ko
> ./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/pr
> obes/lttng-probe-sched.ko
> ./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/pr
> obes/lttng-types.ko
> ./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/lt
> t-ring-buffer-metadata-mmap-client.ko
> ./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/lt
> t-relay.ko
> ./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/lt
> t-ring-buffer-metadata-client.ko

I'm guessing you have a rather old version of lttng-modules?  The "ltt-"
prefix was replaced with "lttng-" about three years ago:

commit a90917c3f8c4
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Date:   Sun Jan 22 16:13:41 2012 -0500

    Mass rename: ltt_*/ltt-* to LTTNG_*/LTTNG-*

$ git describe --all  a90917c3f8c4
tags/v2.0-pre11-5-ga90917c3f8c4

Make sure you're not inadvertently using an old unsupported 0.x version of the modules.

lttng-tracer missing

[Neil Bryan]

[-- Attachment #1.1: Type: text/plain, Size: 9916 bytes --]

Hello Forum,

I am having a few problems enabling LTTng kernel tracing. I am using Yocto on an Alter SOC FPGA platform.
I have added the following to local.conf to bring in the required packages:
<SNIP>
EXTRA_IMAGE_FEATURES = "debug-tweaks tools-debug tools-profile"
PACKAGE_DEBUG_SPLIT_STYLE = 'debug-file-directory'

INHIBIT_PACKAGE_STRIP = "1"

KERNEL_CONFIG_TRACEPOINTS="y"
KERNEL_CONFIG_HAVE_ARCH_TRACEHOOK="y"
KERNEL_CONFIG_TRACE_IRQFLAGS="y"
KERNEL_CONFIG_STACKTRACE="y"
KERNEL_CONFIG_FRAME_POINTER="y"
KERNEL_CONFIG_NOP_TRACER="y"
KERNEL_CONFIG_TRACER_MAX_TRACE="y"
KERNEL_CONFIG_EVENT_TRACING="y"
KERNEL_CONFIG_EVENT_POWER_TRACING_DEPRECATED="y"
KERNEL_CONFIG_CONTEXT_SWITCH_TRACER="y"
KERNEL_CONFIG_TRACING="y"
KERNEL_CONFIG_GENERIC_TRACER="y"
KERNEL_CONFIG_FTRACE="y"
KERNEL_CONFIG_FUNCTION_TRACER="y"
KERNEL_CONFIG_FUNCTION_GRAPH_TRACER="y"
KERNEL_CONFIG_IRQSOFF_TRACER="y"
KERNEL_CONFIG_PREEMPT_TRACER="y"
KERNEL_CONFIG_SCHED_TRACER="y"
KERNEL_CONFIG_FTRACE_SYSCALLS="y"
KERNEL_CONFIG_BRANCH_PROFILE_NONE="y"
KERNEL_CONFIG_STACK_TRACER="y"
KERNEL_CONFIG_BLK_DEV_IO_TRACE="y"
KERNEL_CONFIG_KPROBES="y"
KERNEL_CONFIG_KPROBE_EVENT="y"
KERNEL_CONFIG_DYNAMIC_FTRACE="y"
KERNEL_CONFIG_FUNCTION_PROFILER="y"
KERNEL_CONFIG_FTRACE_MCOUNT_RECORD="y"
KERNEL_CONFIG_FTRACE_SELFTEST="y"
KERNEL_CONFIG_FTRACE_STARTUP_TEST="y"
KERNEL_CONFIG_EVENT_TRACE_TEST_SYSCALLS="y"
KERNEL_CONFIG_RING_BUFFER_BENCHMARK="y"
<SNIP>

Following the bitbake build I see these kernel modules in the build output directory:

./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/lib/lib-ring-buffer.ko
./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/ltt-ring-buffer-client-overwrite.ko
./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/probes/lttng-probe-irq.ko
./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/probes/lttng-probe-lttng.ko
./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/probes/lttng-probe-block.ko
./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/probes/lttng-probe-sched.ko
./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/probes/lttng-types.ko
./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/ltt-ring-buffer-metadata-mmap-client.ko
./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/ltt-relay.ko
./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/ltt-ring-buffer-metadata-client.ko
./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/ltt-ring-buffer-client-mmap-discard.ko
./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/ltt-ring-buffer-client-discard.ko
./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/ltt-ring-buffer-client-mmap-overwrite.ko

When I try to create an lttng session on the target I see the following (not surprisingly as the lttng-tracer module is not present above):

root@socfpga_cyclone5:/lib/modules/3.7.0# lttng-sessiond -vvv
DEBUG3: Creating LTTng run directory: /var/run/lttng [in create_lttng_rundir() at main.c:4315]
DEBUG2: Kernel consumer err path: /var/run/lttng/kconsumerd/error [in main() at main.c:4543]
DEBUG2: Kernel consumer cmd path: /var/run/lttng/kconsumerd/command [in main() at main.c:4545]
DEBUG1: Client socket path /var/run/lttng/client-lttng-sessiond [in main() at main.c:4592]
DEBUG1: Application socket path /var/run/lttng/apps-lttng-sessiond [in main() at main.c:4593]
DEBUG1: LTTng run directory path: /var/run/lttng [in main() at main.c:4594]
DEBUG2: UST consumer 32 bits err path: /var/run/lttng/ustconsumerd32/error [in main() at main.c:4603]
DEBUG2: UST consumer 32 bits cmd path: /var/run/lttng/ustconsumerd32/command [in main() at main.c:4605]
DEBUG2: UST consumer 64 bits err path: /var/run/lttng/ustconsumerd64/error [in main() at main.c:4614]
DEBUG2: UST consumer 64 bits cmd path: /var/run/lttng/ustconsumerd64/command [in main() at main.c:4616]
DEBUG3: Created hashtable size 4 at 0x37080 of type 1 [in lttng_ht_new() at hashtable.c:96]
DEBUG3: Created hashtable size 4 at 0x37168 of type 1 [in lttng_ht_new() at hashtable.c:96]
DEBUG2: Creating consumer directory: /var/run/lttng/kconsumerd [in set_consumer_sockets() at main.c:4357]
FATAL: Module lttng-tracer not found.
Error: Unable to load module lttng-tracer
DEBUG1: Failed to open /proc/lttng [in init_kernel_tracer() at main.c:1871]
Error: Unable to remove module lttng-tracer
Warning: No kernel tracer available
DEBUG2: Creating consumer directory: /var/run/lttng/ustconsumerd64 [in set_consumer_sockets() at main.c:4357]
DEBUG2: Creating consumer directory: /var/run/lttng/ustconsumerd32 [in set_consumer_sockets() at main.c:4357]
DEBUG1: Signal handler set for SIGTERM, SIGPIPE and SIGINT [in set_signal_handler() at main.c:4449]
DEBUG1: All permissions are set [in set_permissions() at main.c:4250]
DEBUG1: epoll set max size is 89794 [in compat_epoll_set_max_size() at compat-epoll.c:224]
DEBUG1: [thread] Manage client started [in thread_manage_clients() at main.c:3794]
DEBUG1: [thread] Dispatch UST command started [in thread_dispatch_ust_registration() at main.c:1324]
DEBUG1: Futex n to 1 prepare done [in futex_nto1_prepare() at futex.c:73]
DEBUG1: Accepting client command ... [in thread_manage_clients() at main.c:3826]
DEBUG1: Woken up but nothing in the UST command queue [in thread_dispatch_ust_registration() at main.c:1334]
DEBUG1: Thread manage kernel started [in thread_manage_kernel() at main.c:876]
DEBUG1: [thread] Manage application started [in thread_manage_apps() at main.c:1179]
DEBUG1: [thread] Manage application registration started [in thread_registration_apps() at main.c:1392]
DEBUG1: Notifying applications of session daemon state: 1 [in notify_ust_apps() at main.c:687]
DEBUG1: Updating kernel poll set [in update_kernel_poll() at main.c:748]
DEBUG1: Apps thread polling on 2 fds [in thread_manage_apps() at main.c:1200]
DEBUG1: Thread kernel polling on 2 fds [in thread_manage_kernel() at main.c:905]
DEBUG1: Got the wait shm fd 18 [in get_wait_shm() at shm.c:117]
DEBUG1: Futex wait update active 1 [in futex_wait_update() at futex.c:62]
DEBUG1: Accepting application registration [in thread_registration_apps() at main.c:1423]

DEBUG1: SIGINT caught [in sighandler() at main.c:4404]
DEBUG1: Terminating all threads [in stop_threads() at main.c:412]
DEBUG1: Futex n to 1 wake done [in futex_nto1_wake() at futex.c:100]
DEBUG1: Kernel thread dying [in thread_manage_kernel() at main.c:967]
DEBUG1: Application communication apps thread cleanup complete [in thread_manage_apps() at main.c:1308]
DEBUG1: Futex n to 1 wait done [in futex_nto1_wait() at futex.c:87]
DEBUG1: Notifying applications of session daemon state: 0 [in notify_ust_apps() at main.c:687]
DEBUG1: Dispatch thread dying [in thread_dispatch_ust_registration() at main.c:1374]
DEBUG1: Got the wait shm fd 15 [in get_wait_shm() at shm.c:117]
DEBUG1: Client thread dying [in thread_manage_clients() at main.c:3953]
DEBUG1: Futex wait update active 0 [in futex_wait_update() at futex.c:62]
DEBUG1: Clean command context structure [in clean_command_ctx() at main.c:534]
DEBUG1: UST Registration thread cleanup complete [in thread_registration_apps() at main.c:1551]
DEBUG1: Cleaning up [in cleanup() at main.c:432]
DEBUG1: Removing /var/run/lttng directory [in cleanup() at main.c:434]
DEBUG1: Cleaning up all sessions [in cleanup() at main.c:447]
DEBUG1: Closing all UST sockets [in cleanup() at main.c:462]
DEBUG2: UST app cleaning registered apps hash table [in ust_app_clean_list() at ust-app.c:1509]
DEBUG2: Closing kernel fd [in cleanup() at main.c:468]
DEBUG1: Unloading kernel modules [in cleanup() at main.c:475]
DEBUG1: Modprobe removal successful lttng-probe-timer [in modprobe_remove_lttng_data() at modprobe.c:118]
DEBUG1: Modprobe removal successful lttng-probe-statedump [in modprobe_remove_lttng_data() at modprobe.c:118]
DEBUG1: Modprobe removal successful lttng-probe-signal [in modprobe_remove_lttng_data() at modprobe.c:118]
DEBUG1: Modprobe removal successful lttng-probe-sched [in modprobe_remove_lttng_data() at modprobe.c:118]
DEBUG1: Modprobe removal successful lttng-probe-kvm [in modprobe_remove_lttng_data() at modprobe.c:118]
DEBUG1: Modprobe removal successful lttng-probe-irq [in modprobe_remove_lttng_data() at modprobe.c:118]
DEBUG1: Modprobe removal successful lttng-probe-block [in modprobe_remove_lttng_data() at modprobe.c:118]
DEBUG1: Modprobe removal successful lttng-types [in modprobe_remove_lttng_data() at modprobe.c:118]
DEBUG1: Modprobe removal successful lttng-probe-lttng [in modprobe_remove_lttng_data() at modprobe.c:118]
Error: Unable to remove module lttng-ring-buffer-metadata-mmap-client
Error: Unable to remove module lttng-ring-buffer-client-mmap-overwrite
Error: Unable to remove module lttng-ring-buffer-client-mmap-discard
Error: Unable to remove module lttng-ring-buffer-metadata-client
Error: Unable to remove module lttng-ring-buffer-client-overwrite
Error: Unable to remove module lttng-ring-buffer-client-discard
Error: Unable to remove module lttng-lib-ring-buffer
DEBUG1: Modprobe removal successful lttng-kretprobes [in modprobe_remove_lttng_data() at modprobe.c:118]
DEBUG1: Modprobe removal successful lttng-kprobes [in modprobe_remove_lttng_data() at modprobe.c:118]
DEBUG1: Modprobe removal successful lttng-ftrace [in modprobe_remove_lttng_data() at modprobe.c:118]
Error: Unable to remove module lttng-tracer
DEBUG1: *** assert failed :-) *** ==> Matthew, BEET driven development works! [in cleanup()

I am running an early(ish) version of the kernel (3.7) but this is because the Altera Yocto source package is quite an old vintage.

Any clues as to what am I doing wrong?

Thanks for any help anyone can offer.

Best regards,

Neil.




[-- Attachment #1.2: Type: text/html, Size: 17515 bytes --]

[-- Attachment #2: Type: text/plain, Size: 155 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev