* Re: compiling LTTng for android
[not found] <CAHg4Oy_bQHi7q6+Deb7AO_ZtJZj=9i5h+WyPTnaFO4uLxAwQXA@mail.gmail.com>
@ 2013-02-21 13:18 ` Pierre-Luc St-Charles
[not found] ` <CA+umk=rPAR7uiHQjL9FRE1UtVOEQ7J5xM46rfb_be7ZXoNC2MA@mail.gmail.com>
1 sibling, 0 replies; 8+ messages in thread
From: Pierre-Luc St-Charles @ 2013-02-21 13:18 UTC (permalink / raw)
To: Amit Balboul; +Cc: lttng-dev
[-- Attachment #1.1: Type: text/plain, Size: 4801 bytes --]
Forwarded to mailing list:
Hey Amil,
Sorry we kept you out of the loop, we've been struggling on some issues in
the later projects; as for liburcu, we've come across the same problems you
cited, except the last one:
- syscall.h / lpthread / gettid : your solutions, or a #ifdef
__ANDROID__ barrier, seems to work as a temp fix.
- rand_r : after substituting for the regular (not-thread-safe) rand,
we decide to go with our own implementation, but the one you found might
actually be better
- rpl_malloc : same fix here, but we also found
that ac_cv_func_realloc_0_nonnull *might* also help in some cases (along
with ac_cv_func_malloc_0_nonnull)
As for the __sync_synchronize error, we haven't encountered those and we
managed to successfully build the library.
Possible differences:
We opted for AOSP's own pre-built & packaged NDK toolchain, (which can be
found here: http://developer.android.com/tools/sdk/ndk/index.html) and
pointed the sysroot variable to its platforms/android-14/arch-arm/
directory (latest). By default, it doesn't include some
module/kernel headers, but it should be enough for liburcu. We are
currently considering adding the liburcu project itself (along with the
others) directly in the android external modules, so that it may be
compiled with exactly the SAME settings as the rest of the
libraries/binaries/modules, using the same NDK (but for that, we'll need to
drop the autoconf/automake stuff, and that's another story).
The rest should be identical, we might have used some extra parameters here
and there that might be based on our environment in the build process, but
here they are anyway:
in ./configure, add : --target=arm-linux-androideabi
--prefix=$SYSROOT/usr
make sure you provide $SYSROOT/usr/lib instead of $SYSROOT/lib as the
default library path
Keep us informed if the problem still happens, we'll try to debug your
issue further.
On Thu, Feb 21, 2013 at 5:13 AM, Amit Balboul <amit.balboul@shinesec.com>wrote:
> Hi all,
>
> In continue to posts from
> http://www.mail-archive.com/lttng-dev@lists.lttng.org/msg02542.html,
>
> I'm trying to build the LTTng kernel-tracer for Android on ARM:
> (JB4.2.1 - kernel 3.0.31 - on Galaxy Nexus).
> I'm having some trouble completing the process, didn't find any usable
> information so far, maybe u can help.
>
> I want to build the: (I've managed in the past to build the modules for
> goldfish's 2.6.29 but this required patching the kernel).
> (1) userspace-rcu
> (2) modules
> (3) tools
> I'm currently stuck in the first phase (building userspace-rcu).
>
> My steps and the issues I've encountered are:
> 1. Clone the userspace-rcu git repository
> 2. Generate a sysroot for arm-linux-androideabi from google's NDK (using
> its "make-standalone-toolchain.sh" script with arm-linux-androideabi-4.4.3
> toolchain).
> 3. Run ./bootstrap - creating config.h.in, Makefile.in, configure, ...
> 4. Run ./configure with --host=arm-linux-androideabi and the relevant
> environment variables (CC, LD, CPPFLAGS, CFLAGS, LDFLAGS, LIB), pointing to
> the NDK ARM SYSROOT.
> 5. Run make
>
> Some errors I encoutered and their fixes (workarounds), while trying as
> much as I can leave the sources untouched:
> 1. syscall.h not found. In some files, the <syscall.h> is included. In NDK
> SYSROOT the syscall.h lays in sys/
> FIX: created a link: SYSROOT/include/syscall.h ->
> SYSROOT/include/sys/syscall.h
> 2. Android's libraries includes the pthread lib automatically, thus not
> having a stand-alone pthread library.
> FIX: removed "-lpthread" from Makefile.am, tests/Makefile.am
> 3. static declaration of "gettid" follows non-static declaration (in
> SYSROOT/include/unistd.h, "gettid" is declared as "external")
> For the mean time I commented this declaration... Maybe adding "#ifndef
> gettid" before the declaration of "static gettid" in the sources will fix
> it ?
> 4. rand_r not declared - added a module that implements rand_r (as seen in
> https://github.com/xbmc/xbmc/tree/master/xbmc/android/bionic_supplement)
> 5. undefined reference to rpl_malloc
> FIX: setting ac_cv_func_malloc_0_nonnull=yes before launching ./configure.
>
> now, after running ./configure and make, still not completed. I get many
> "undefined reference to __sync_syncronize" messages. the
> urcu/arch/generic.h defines the memory barriers functions as cmm_* but
> their implementation is not taken from urcu/arch/arm.h but from Line 46 in
> the same file (generic.h).
> Can u help with this?
> Am I missing something here ?
>
> Also, how does the build system use the kernel 3.0.31 (or any other)
> sysroot ?
>
> Thank you
>
> Amit.
>
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>
>
[-- Attachment #1.2: Type: text/html, Size: 8158 bytes --]
[-- Attachment #2: Type: text/plain, Size: 155 bytes --]
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: compiling LTTng for android
[not found] ` <CAHg4Oy-=fuTYU=5p0MPWJ9wQ+71NpoAWvzcHuB7rw4UesWUe-Q@mail.gmail.com>
@ 2013-02-21 18:37 ` Pierre-Luc St-Charles
[not found] ` <CA+umk=qr5ZDCJ8S7RoHRPgE0Q4RgYBGNOULKs3GYYCEunyshdA@mail.gmail.com>
1 sibling, 0 replies; 8+ messages in thread
From: Pierre-Luc St-Charles @ 2013-02-21 18:37 UTC (permalink / raw)
To: Amit Balboul; +Cc: lttng-dev
[-- Attachment #1.1: Type: text/plain, Size: 8562 bytes --]
Hey Amit,
For the gettid issue, we were considering adding a #ifndef __ANDROID__ (all
mentions of gettid in urcu) #endif type of solution (which seems to work
for us at the moment), which also brings us to the last point: __ANDROID__
should be defined by default whenever you compile with the NDK's toolchain
(we have not added this variable anywhere ourselves and so far, it's been
very useful in all the projects). As for the patch itself, I personally
have no clue if this would break any convention already in place for
liburcu/LTTng in general; we have not been pushing any part of our
progression so far as nothing as been tested, and some changes we made
locally are extremely device/version specific (and would most likely
break functionalities in other environments).
For the syscall.h issue, we avoided including directly
"$SYSROOT/usr/include/sys",
since some files in that folder are not safe to use directly (see errno.h
and statfs.h for example). This seems to be the 'agreed-upon' logic used in
the AOSP source tree; possible alternatives are setting up new Android
#ifdef barriers in liburcu itself, or trying to push an empty, logically
linked syscall.h file (identical the above errno.h and statfs.h) to
"$SYSROOT/usr/include"
(which would probably not be accepted in the AOSP source).
If your next plan after liburcu is porting the LTTng modules to Android,
you'll most likely hit the missing kernel config options (syscall/hook
related) in 3.0.x like we did; this will require some specific patching we
have yet to complete. This work has been done in Torvalds's kernel
(post-3.5 I believe, see his git for more info), but considering how
different the current Android device kernels still are from the official
ones, we might still have a lot to do.
On another note, we peeked a little bit into porting UST to Android, but
here again, replacing the SysV shmem implementation by Android's ashmem
proved a lot more complicated than we had anticipated. Currently, we are
only focusing on kernel tracing, so LTTng-modules + liburcu + LTTng-tools.
So far, we successfully built the modules (with limited support), liburcu
and the tools (also with limited support), but we have yet to start tracing.
Feel free to ask away if you have any more questions, although other people
on the mailing list might have better answers for LTTng-related questions.
-PL
On Thu, Feb 21, 2013 at 12:08 PM, Amit Balboul <amit.balboul@shinesec.com>wrote:
> Hey Pierre,
> Thank you for the fast response.
>
> I'm also using the pre-built NDK environment (r8d),
> I continue to work on it, I'll let you know if I'm still having those
> errors.
> About the "static inline pid_t gettid(void)" issue:
> I cannot find any better solution than modifying the userspace-rcu sources
> to avoid static declaration.
> I dont see the connection to building to Android, because unistd.h
> "extern"s gettid anyway, so it cannot be defined as static in any of the
> files (i just removed the "static" from the inline definition).
>
> How do you want me to make this patch ?
>
> About the syscall.h issue:
> applying the SYSROOT/usr/include/sys folder AFTER the SYSROOT/usr/include
> in the CPPFLAGS solves the issue (looks like it is better than the link
> solution).
>
> About the __ANDROID__ definition:
> May I add this symbol to configure.ac and use it in the sources, ?
>
>
> Amit.
>
>
>
>
> On Thu, Feb 21, 2013 at 3:18 PM, Pierre-Luc St-Charles <
> pierre-luc.st-charles@polymtl.ca> wrote:
>
>> Forwarded to mailing list:
>>
>> Hey Amil,
>>
>> Sorry we kept you out of the loop, we've been struggling on some issues
>> in the later projects; as for liburcu, we've come across the same problems
>> you cited, except the last one:
>> - syscall.h / lpthread / gettid : your solutions, or a #ifdef
>> __ANDROID__ barrier, seems to work as a temp fix.
>> - rand_r : after substituting for the regular (not-thread-safe) rand,
>> we decide to go with our own implementation, but the one you found might
>> actually be better
>> - rpl_malloc : same fix here, but we also found
>> that ac_cv_func_realloc_0_nonnull *might* also help in some cases (along
>> with ac_cv_func_malloc_0_nonnull)
>>
>> As for the __sync_synchronize error, we haven't encountered those and we
>> managed to successfully build the library.
>>
>> Possible differences:
>> We opted for AOSP's own pre-built & packaged NDK toolchain, (which can be
>> found here: http://developer.android.com/tools/sdk/ndk/index.html) and
>> pointed the sysroot variable to its platforms/android-14/arch-arm/
>> directory (latest). By default, it doesn't include some
>> module/kernel headers, but it should be enough for liburcu. We are
>> currently considering adding the liburcu project itself (along with the
>> others) directly in the android external modules, so that it may be
>> compiled with exactly the SAME settings as the rest of the
>> libraries/binaries/modules, using the same NDK (but for that, we'll need to
>> drop the autoconf/automake stuff, and that's another story).
>>
>> The rest should be identical, we might have used some extra parameters
>> here and there that might be based on our environment in the build process,
>> but here they are anyway:
>> in ./configure, add : --target=arm-linux-androideabi
>> --prefix=$SYSROOT/usr
>> make sure you provide $SYSROOT/usr/lib instead of $SYSROOT/lib as the
>> default library path
>>
>> Keep us informed if the problem still happens, we'll try to debug your
>> issue further.
>>
>> On Thu, Feb 21, 2013 at 5:13 AM, Amit Balboul <amit.balboul@shinesec.com>wrote:
>>
>>> Hi all,
>>>
>>> In continue to posts from
>>> http://www.mail-archive.com/lttng-dev@lists.lttng.org/msg02542.html,
>>>
>>> I'm trying to build the LTTng kernel-tracer for Android on ARM:
>>> (JB4.2.1 - kernel 3.0.31 - on Galaxy Nexus).
>>> I'm having some trouble completing the process, didn't find any usable
>>> information so far, maybe u can help.
>>>
>>> I want to build the: (I've managed in the past to build the modules for
>>> goldfish's 2.6.29 but this required patching the kernel).
>>> (1) userspace-rcu
>>> (2) modules
>>> (3) tools
>>> I'm currently stuck in the first phase (building userspace-rcu).
>>>
>>> My steps and the issues I've encountered are:
>>> 1. Clone the userspace-rcu git repository
>>> 2. Generate a sysroot for arm-linux-androideabi from google's NDK (using
>>> its "make-standalone-toolchain.sh" script with arm-linux-androideabi-4.4.3
>>> toolchain).
>>> 3. Run ./bootstrap - creating config.h.in, Makefile.in, configure, ...
>>> 4. Run ./configure with --host=arm-linux-androideabi and the relevant
>>> environment variables (CC, LD, CPPFLAGS, CFLAGS, LDFLAGS, LIB), pointing to
>>> the NDK ARM SYSROOT.
>>> 5. Run make
>>>
>>> Some errors I encoutered and their fixes (workarounds), while trying as
>>> much as I can leave the sources untouched:
>>> 1. syscall.h not found. In some files, the <syscall.h> is included. In
>>> NDK SYSROOT the syscall.h lays in sys/
>>> FIX: created a link: SYSROOT/include/syscall.h ->
>>> SYSROOT/include/sys/syscall.h
>>> 2. Android's libraries includes the pthread lib automatically, thus not
>>> having a stand-alone pthread library.
>>> FIX: removed "-lpthread" from Makefile.am, tests/Makefile.am
>>> 3. static declaration of "gettid" follows non-static declaration (in
>>> SYSROOT/include/unistd.h, "gettid" is declared as "external")
>>> For the mean time I commented this declaration... Maybe adding "#ifndef
>>> gettid" before the declaration of "static gettid" in the sources will fix
>>> it ?
>>> 4. rand_r not declared - added a module that implements rand_r (as seen
>>> in
>>> https://github.com/xbmc/xbmc/tree/master/xbmc/android/bionic_supplement)
>>> 5. undefined reference to rpl_malloc
>>> FIX: setting ac_cv_func_malloc_0_nonnull=yes before launching
>>> ./configure.
>>>
>>> now, after running ./configure and make, still not completed. I get many
>>> "undefined reference to __sync_syncronize" messages. the
>>> urcu/arch/generic.h defines the memory barriers functions as cmm_* but
>>> their implementation is not taken from urcu/arch/arm.h but from Line 46 in
>>> the same file (generic.h).
>>> Can u help with this?
>>> Am I missing something here ?
>>>
>>> Also, how does the build system use the kernel 3.0.31 (or any other)
>>> sysroot ?
>>>
>>> Thank you
>>>
>>> Amit.
>>>
>>> _______________________________________________
>>> lttng-dev mailing list
>>> lttng-dev@lists.lttng.org
>>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>>>
>>>
>>
>
[-- Attachment #1.2: Type: text/html, Size: 13150 bytes --]
[-- Attachment #2: Type: text/plain, Size: 155 bytes --]
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
^ permalink raw reply [flat|nested] 8+ messages in thread
* Fwd: compiling LTTng for android
[not found] ` <CAHg4Oy8LbkQT3yLeWMq7myty8t06brtrr9yyYwvDk8cjsmFx9A@mail.gmail.com>
@ 2013-03-06 23:09 ` Amit Balboul
[not found] ` <CAHg4Oy_X-NhH5_JFXCBg-HsoZxyh+equV1O8vDnqAVv1QCroRw@mail.gmail.com>
1 sibling, 0 replies; 8+ messages in thread
From: Amit Balboul @ 2013-03-06 23:09 UTC (permalink / raw)
To: lttng-dev, Pierre-Luc St-Charles
[-- Attachment #1.1: Type: text/plain, Size: 17925 bytes --]
Hi All,
Regarding the last mail about tracing system calls on Android using LTTng,
The issue of not receiving enabled event to trace from the kernel can be
fixed patching src/bin/lttng-sessiond/kernel.c (patch attached)
In the function *kernel_list_events,* fscanf-ing the events exported by the
kernel using the fp must not use the syntax *%m* as it is obsolete (in
Android),
thus the fscanf returns 0 and won't read the kernel events (first condition
false).
Instead I use a pre-allocated buffer for the string and use the
*%[^;]*syntax, and omitting the memory free (attached is the patch).
The resulting output is (sessiond is run with verbose debug prints):
root@android:/data/lttng/bin # *./lttng list -k*
DEBUG1 [10239/10290]: Wait for client response (in thread_manage_clients()
at main.c:3242)
DEBUG1 [10239/10290]: Receiving data from client ... (in
thread_manage_clients() at main.c:3287)
DEBUG1 [10239/10290]: Nothing recv() from client... continuing (in
thread_manage_clients() at main.c:3291)
DEBUG1 [10239/10290]: Clean command context structure (in
clean_command_ctx() at main.c:482)
DEBUG1 [10239/10290]: Accepting client command ... (in
thread_manage_clients() at main.c:3200)
DEBUG1 [10239/10290]: Wait for client response (in thread_manage_clients()
at main.c:3242)
DEBUG1 [10239/10290]: Receiving data from client ... (in
thread_manage_clients() at main.c:3287)
DEBUG1 [10239/10290]: Processing client command 14 (in process_client_msg()
at main.c:2227)
DEBUG1 [10239/10290]: Kernel list events done (61 events) (in
kernel_list_events() at kernel.c:653)
DEBUG1 [10239/10290]: Sending response (size: 35640, retcode: Success) (in
thread_manage_clients() at main.c:3338)
DEBUG1 [10239/10290]: Clean command context structure (in
clean_command_ctx() at main.c:482)
DEBUG1 [10239/10290]: Accepting client command ... (in
thread_manage_clients() at main.c:3200)
Kernel events:
-------------
block_rq_abort (loglevel: TRACE_EMERG (0)) (type: tracepoint)
block_rq_requeue (loglevel: TRACE_EMERG (0)) (type: tracepoint)
block_rq_complete (loglevel: TRACE_EMERG (0)) (type: tracepoint)
block_rq_insert (loglevel: TRACE_EMERG (0)) (type: tracepoint)
block_rq_issue (loglevel: TRACE_EMERG (0)) (type: tracepoint)
block_bio_bounce (loglevel: TRACE_EMERG (0)) (type: tracepoint)
block_bio_complete (loglevel: TRACE_EMERG (0)) (type: tracepoint)
block_bio_backmerge (loglevel: TRACE_EMERG (0)) (type: tracepoint)
block_bio_frontmerge (loglevel: TRACE_EMERG (0)) (type: tracepoint)
block_bio_queue (loglevel: TRACE_EMERG (0)) (type: tracepoint)
block_getrq (loglevel: TRACE_EMERG (0)) (type: tracepoint)
block_sleeprq (loglevel: TRACE_EMERG (0)) (type: tracepoint)
block_plug (loglevel: TRACE_EMERG (0)) (type: tracepoint)
block_unplug (loglevel: TRACE_EMERG (0)) (type: tracepoint)
block_split (loglevel: TRACE_EMERG (0)) (type: tracepoint)
block_bio_remap (loglevel: TRACE_EMERG (0)) (type: tracepoint)
block_rq_remap (loglevel: TRACE_EMERG (0)) (type: tracepoint)
irq_handler_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint)
irq_handler_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint)
softirq_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint)
softirq_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint)
softirq_raise (loglevel: TRACE_EMERG (0)) (type: tracepoint)
sched_kthread_stop (loglevel: TRACE_EMERG (0)) (type: tracepoint)
sched_kthread_stop_ret (loglevel: TRACE_EMERG (0)) (type: tracepoint)
sched_wakeup (loglevel: TRACE_EMERG (0)) (type: tracepoint)
sched_wakeup_new (loglevel: TRACE_EMERG (0)) (type: tracepoint)
sched_switch (loglevel: TRACE_EMERG (0)) (type: tracepoint)
sched_migrate_task (loglevel: TRACE_EMERG (0)) (type: tracepoint)
sched_process_free (loglevel: TRACE_EMERG (0)) (type: tracepoint)
sched_process_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint)
sched_wait_task (loglevel: TRACE_EMERG (0)) (type: tracepoint)
sched_process_wait (loglevel: TRACE_EMERG (0)) (type: tracepoint)
sched_process_fork (loglevel: TRACE_EMERG (0)) (type: tracepoint)
sched_stat_wait (loglevel: TRACE_EMERG (0)) (type: tracepoint)
sched_stat_sleep (loglevel: TRACE_EMERG (0)) (type: tracepoint)
sched_stat_iowait (loglevel: TRACE_EMERG (0)) (type: tracepoint)
sched_stat_runtime (loglevel: TRACE_EMERG (0)) (type: tracepoint)
sched_pi_setprio (loglevel: TRACE_EMERG (0)) (type: tracepoint)
signal_generate (loglevel: TRACE_EMERG (0)) (type: tracepoint)
signal_deliver (loglevel: TRACE_EMERG (0)) (type: tracepoint)
signal_overflow_fail (loglevel: TRACE_EMERG (0)) (type: tracepoint)
signal_lose_info (loglevel: TRACE_EMERG (0)) (type: tracepoint)
lttng_statedump_start (loglevel: TRACE_EMERG (0)) (type: tracepoint)
lttng_statedump_end (loglevel: TRACE_EMERG (0)) (type: tracepoint)
lttng_statedump_process_state (loglevel: TRACE_EMERG (0)) (type:
tracepoint)
lttng_statedump_file_descriptor (loglevel: TRACE_EMERG (0)) (type:
tracepoint)
lttng_statedump_vm_map (loglevel: TRACE_EMERG (0)) (type: tracepoint)
lttng_statedump_network_interface (loglevel: TRACE_EMERG (0)) (type:
tracepoint)
lttng_statedump_interrupt (loglevel: TRACE_EMERG (0)) (type:
tracepoint)
timer_init (loglevel: TRACE_EMERG (0)) (type: tracepoint)
timer_start (loglevel: TRACE_EMERG (0)) (type: tracepoint)
timer_expire_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint)
timer_expire_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint)
timer_cancel (loglevel: TRACE_EMERG (0)) (type: tracepoint)
hrtimer_init (loglevel: TRACE_EMERG (0)) (type: tracepoint)
hrtimer_start (loglevel: TRACE_EMERG (0)) (type: tracepoint)
hrtimer_expire_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint)
hrtimer_expire_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint)
hrtimer_cancel (loglevel: TRACE_EMERG (0)) (type: tracepoint)
itimer_state (loglevel: TRACE_EMERG (0)) (type: tracepoint)
itimer_expire (loglevel: TRACE_EMERG (0)) (type: tracepoint)
root@android:/data/lttng/bin #
Now I'm facing the issue of starting the traces:
debugging the sessiond, consumerd and lttng I concluded that the issue is
due to splice system call failure.
even using a MMAP output type channel, the metadata channel is SPLICE type
(hard-coded).
Problem persists also when using relayd to pass the traces through the
network.
Ignoring the splice failure means closing of the metadata channel (maybe??)
but need not affect the traces channel (right ?)
Attached is a debug log of each thread of the sessiond, right after
invoking "*./lttng start*". (with extra informative my prints)
Please keep me in touch with any progress in this issue,
Thank you,
Amit.
---------- Forwarded message ----------
From: Amit Balboul <amit.balboul@shinesec.com>
Date: Wed, Feb 27, 2013 at 4:37 PM
Subject: Re: [lttng-dev] compiling LTTng for android
To: PLSTC <b0mb00z.it@gmail.com>
Hey Pierre,
Some major progress has been achieved, following is the summary:
1. Modules issue:
I built (for ARM/Android) and installed latest stable version of busybox on
the device.
Seeing the code tries to run (hardcoded) /sbin/modprobe, made a soft link
to busybox's modeprobe.
Also, created /lib/modules/<version+build> directory and pushed the
<version+build> directory from /lib/modules on the machine.
Now session daemon modprobes the desired modules successfully, (some are
not present though as mentioned before) - no need to use insmod.
2. epoll_create1 issue:
epoll_create1 patch was OK, I accidentally defined EPOLL_CLOEXEC as
0x02000000 instead of 02000000 (fixed).
The running scenario:
1. Run ./lttng-sessiond -d -vvv
Then: (commands are in bold, output is in blue for readability)
root@android:/data/lttng/install_sysroot/bin # *./lttng create s1*
Session s1 created.
Traces will be written in /data/lttng-traces/s1-20130227-161145
root@android:/data/lttng/install_sysroot/bin # *./lttng enable-channel ch -k
*
Kernel channel ch enabled for session s1
root@android:/data/lttng/install_sysroot/bin # *./lttng enable-event ev -k
-a*
All kernel events are enabled in channel channel0
root@android:/data/lttng/install_sysroot/bin # *./lttng list s1*
Tracing session s1: [inactive]
Trace path: /data/lttng-traces/s1-20130227-161145
=== Domain: Kernel ===
Channels:
-------------
- channel0: [enabled]
Attributes:
overwrite mode: 0
subbufers size: 262144
number of subbufers: 4
switch timer interval: 0
read timer interval: 200
output: splice()
Events:
None
- ch: [enabled]
Attributes:
overwrite mode: 0
subbufers size: 262144
number of subbufers: 4
switch timer interval: 0
read timer interval: 200
output: splice()
Events:
None
root@android:/data/lttng/install_sysroot/bin # *./lttng list -k *
Kernel events:
-------------
root@android:/data/lttng/install_sysroot/bin # *./lttng start*
Tracing started for session s1
root@android:/data/lttng/install_sysroot/bin # *./lttng stop *
Waiting for data availability
Tracing stopped for session s1
root@android:/data/lttng/install_sysroot/bin #
*Now: (this is not surprising because the kernel events list is empty...)*
the folder /data/lttng-traces/s1-20130227-161145/kernel contains:
total 0
-rwxrwxrwx 1 0 0 0 Feb 27 14:18 ch_0
-rwxrwxrwx 1 0 0 0 Feb 27 14:18 ch_1
-rwxrwxrwx 1 0 0 0 Feb 27 14:18 channel0_0
-rwxrwxrwx 1 0 0 0 Feb 27 14:18 channel0_1
-rwxrwxrwx 1 0 0 0 Feb 27 14:18 metadata
*By the way:*
When running the same scenario but creating the channel with *--output mmap*,
still no data but the size of the first two files in the traces directory
are 4Kb (I didn't bother to investigate them as I assume it is just the
header - not the traces payload).
Another issue: at this point, listing the sessions gives:
root@android:/data/lttng/install_sysroot/bin # *./lttng list s1 *
Tracing session s1: [inactive]
Trace path: /data/lttng-traces/s1-20130227-161145
=== Domain: Kernel ===
Error: No kernel consumer detected
99|root@android:/data/lttng/install_sysroot/bin # *./lttng list -k *
Error: Unable to list kernel events
Error: Command error
1|root@android:/data/lttng/install_sysroot/bin #
So currently I'm facing these issues:
1. Kernel events list is empty.
2. Listing the session/ kernel events after creating session, channel,
event, start, stop - results in "no kernel consumer detected"
If you have some insights please share them with me..
Thanks
Amit.
On Tue, Feb 26, 2013 at 7:35 PM, PLSTC <b0mb00z.it@gmail.com> wrote:
> The modules you can't find are the same I am also missing I believe; I
> think they are only compiled when certain kernel config defines are met
> (and those might simply be unavailable under ARM).
>
> As for what you pointed in your first mail, I believe modprobe isn't
> available on ARM, did you mean that some modules were not found when using
> insmod?
>
> For the epoll_create1 problem, what kind of patch did you apply to add
> support for the epoll_create1 function?
>
>
> On Tue, Feb 26, 2013 at 11:29 AM, Amit Balboul <amit.balboul@shinesec.com>wrote:
>
>> Ok I got :
>> lttng-kretprobes.ko
>> lttng-kprobes.ko
>> , the other two are missing.
>>
>>
>> ---------- Forwarded message ----------
>> From: Amit Balboul <amit.balboul@shinesec.com>
>> Date: Tue, Feb 26, 2013 at 5:49 PM
>> Subject: Re: [lttng-dev] compiling LTTng for android
>> To: PLSTC <b0mb00z.it@gmail.com>
>>
>>
>> One more thing:
>>
>> The list of modules you gave my is not identical to mine:
>> a. I pushed the modules from my machine to my folder in the device (not
>> to /system/lib) as you directed me, and insmoded them manually.
>> b. I pushed the whole tree under /lib/modules/<KERNEL_VER_BUILD>/ which
>> includes the modules.* and the sub directories:
>>
>> /lib/modules/3.0.31-gd5a18e0/extra$ ll
>> total 2464
>> drwxr-xr-x 4 root root 4096 Feb 26 15:58 ./
>> drwxr-xr-x 3 root root 4096 Feb 26 15:58 ../
>> drwxr-xr-x 2 root root 4096 Feb 26 15:58 lib/
>> -rw-r--r-- 1 root root 167184 Feb 26 15:58
>> lttng-ring-buffer-client-discard.ko
>> -rw-r--r-- 1 root root 167206 Feb 26 15:58
>> lttng-ring-buffer-client-mmap-discard.ko
>> -rw-r--r-- 1 root root 178586 Feb 26 15:58
>> lttng-ring-buffer-client-mmap-overwrite.ko
>> -rw-r--r-- 1 root root 178564 Feb 26 15:58
>> lttng-ring-buffer-client-overwrite.ko
>> -rw-r--r-- 1 root root 137179 Feb 26 15:58
>> lttng-ring-buffer-metadata-client.ko
>> -rw-r--r-- 1 root root 137233 Feb 26 15:58
>> lttng-ring-buffer-metadata-mmap-client.ko
>> -rw-r--r-- 1 root root 213451 Feb 26 15:58 lttng-statedump.ko
>> -rw-r--r-- 1 root root 1314235 Feb 26 15:58 lttng-tracer.ko
>> drwxr-xr-x 2 root root 4096 Feb 26 15:58 probes/
>>
>> /lib/modules/3.0.31-gd5a18e0/extra/lib$ ll
>> total 732
>> drwxr-xr-x 2 root root 4096 Feb 26 15:58 ./
>> drwxr-xr-x 4 root root 4096 Feb 26 15:58 ../
>> -rw-r--r-- 1 root root 738964 Feb 26 15:58 lttng-lib-ring-buffer.ko
>>
>> /lib/modules/3.0.31-gd5a18e0/extra/probes$ ll
>> total 2872
>> drwxr-xr-x 2 root root 4096 Feb 26 15:58 ./
>> drwxr-xr-x 4 root root 4096 Feb 26 15:58 ../
>> -rw-r--r-- 1 root root 162872 Feb 26 15:58 lttng-probe-asoc.ko
>> -rw-r--r-- 1 root root 171049 Feb 26 15:58 lttng-probe-block.ko
>> -rw-r--r-- 1 root root 81165 Feb 26 15:58 lttng-probe-compaction.ko
>> -rw-r--r-- 1 root root 333215 Feb 26 15:58 lttng-probe-ext4.ko
>> -rw-r--r-- 1 root root 80413 Feb 26 15:58 lttng-probe-gpio.ko
>> -rw-r--r-- 1 root root 84256 Feb 26 15:58 lttng-probe-irq.ko
>> -rw-r--r-- 1 root root 135453 Feb 26 15:58 lttng-probe-jbd2.ko
>> -rw-r--r-- 1 root root 105399 Feb 26 15:58 lttng-probe-kmem.ko
>> -rw-r--r-- 1 root root 76240 Feb 26 15:58 lttng-probe-lttng.ko
>> -rw-r--r-- 1 root root 88059 Feb 26 15:58 lttng-probe-module.ko
>> -rw-r--r-- 1 root root 154032 Feb 26 15:58 lttng-probe-napi.ko
>> -rw-r--r-- 1 root root 142038 Feb 26 15:58 lttng-probe-net.ko
>> -rw-r--r-- 1 root root 93564 Feb 26 15:58 lttng-probe-power.ko
>> -rw-r--r-- 1 root root 85530 Feb 26 15:58 lttng-probe-regulator.ko
>> -rw-r--r-- 1 root root 138121 Feb 26 15:58 lttng-probe-sched.ko
>> -rw-r--r-- 1 root root 152909 Feb 26 15:58 lttng-probe-scsi.ko
>> -rw-r--r-- 1 root root 107491 Feb 26 15:58 lttng-probe-signal.ko
>> -rw-r--r-- 1 root root 142711 Feb 26 15:58 lttng-probe-skb.ko
>> -rw-r--r-- 1 root root 181983 Feb 26 15:58 lttng-probe-statedump.ko
>> -rw-r--r-- 1 root root 126731 Feb 26 15:58 lttng-probe-timer.ko
>> -rw-r--r-- 1 root root 86591 Feb 26 15:58 lttng-probe-workqueue.ko
>> -rw-r--r-- 1 root root 133886 Feb 26 15:58 lttng-probe-writeback.ko
>> -rw-r--r-- 1 root root 30863 Feb 26 15:58 lttng-types.ko
>>
>> Looking at the files shows I miss:
>> lttng-kretprobes.ko
>> lttng-kprobes.ko
>> lttng-ftrace.ko
>> lttng-probe-kvm.ko
>>
>>
>> On Tue, Feb 26, 2013 at 3:56 PM, PLSTC <b0mb00z.it@gmail.com> wrote:
>>
>>> Back on the last email:
>>>
>>> Here's the insmod list I ended up building; you might not have compiled
>>> all those modules, depending on your kernel config and on the architecture
>>> used, so some might simply be absent from your output directory.
>>>
>>> insmod /system/lib/modules/lttng-types.ko
>>> insmod /system/lib/modules/lttng-kretprobes.ko
>>> insmod /system/lib/modules/lttng-kprobes.ko
>>> insmod /system/lib/modules/lttng-ftrace.ko
>>> insmod /system/lib/modules/lttng-lib-ring-buffer.ko
>>> insmod /system/lib/modules/lttng-statedump.ko
>>> insmod /system/lib/modules/lttng-tracer.ko
>>> insmod /system/lib/modules/lttng-probe-timer.ko
>>> insmod /system/lib/modules/lttng-probe-statedump.ko
>>> insmod /system/lib/modules/lttng-probe-signal.ko
>>> insmod /system/lib/modules/lttng-probe-sched.ko
>>> insmod /system/lib/modules/lttng-probe-lttng.ko
>>> insmod /system/lib/modules/lttng-probe-kvm.ko
>>> insmod /system/lib/modules/lttng-probe-irq.ko
>>> insmod /system/lib/modules/lttng-probe-block.ko
>>> insmod /system/lib/modules/lttng-ring-buffer-metadata-mmap-client.ko
>>> insmod /system/lib/modules/lttng-ring-buffer-metadata-client.ko
>>> insmod /system/lib/modules/lttng-ring-buffer-client-overwrite.ko
>>> insmod /system/lib/modules/lttng-ring-buffer-client-mmap-overwrite.ko
>>> insmod /system/lib/modules/lttng-ring-buffer-client-mmap-discard.ko
>>> insmod /system/lib/modules/lttng-ring-buffer-client-discard.ko
>>>
>>>
>>> We also haven't had time recently to test if the kprobes/kretprobes
>>> worked, but they probably do; we decided to focus a bit more on the
>>> 'tracepoints' aspect instead.
>>>
>>> Anyway, good luck with your exploration, and if you find something
>>> interesting, I'd love to know more about it!
>>>
>>> -PL
>>>
>>> On Tue, Feb 26, 2013 at 8:06 AM, PLSTC <b0mb00z.it@gmail.com> wrote:
>>>
>>>> Hey Amit,
>>>>
>>>> Good news indeed it seems!
>>>>
>>>> When it comes to pushing the libs/bin, we haven't found any better way
>>>> than using adb push yet; of course, if all the projects were included
>>>> directly in Android's external, we would simply flash the device after each
>>>> build, but currently we're still having trouble generating the correct
>>>> Android makefiles.
>>>>
>>>> As for the modules, I might still have a dep load list somewhere I
>>>> could send you in a few hours, I remember fighting to get the order right...
>>>>
>>>> Also, once everything loads, could you tell me what kind of output
>>>> 'lttng list -k' provides you? Here, we're having trouble listing the
>>>> 'default' kernel trace points provided by HAVE_TRACEPOINTS, and the
>>>> returned list is simply empty (which is a bit strange).
>>>>
>>>> -PL
>>>>
>>>
>>>
>>
>>
>
[-- Attachment #1.2: Type: text/html, Size: 34214 bytes --]
[-- Attachment #2: kernel.c.patch --]
[-- Type: application/octet-stream, Size: 1286 bytes --]
diff --git a/src/bin/lttng-sessiond/kernel.c b/src/bin/lttng-sessiond/kernel.c
index d3a6453..3d21be9 100644
--- a/src/bin/lttng-sessiond/kernel.c
+++ b/src/bin/lttng-sessiond/kernel.c
@@ -587,8 +587,8 @@ error:
*/
ssize_t kernel_list_events(int tracer_fd, struct lttng_event **events)
{
- int fd, pos, ret;
- char *event;
+ int fd, ret;
+ char event[LTTNG_SYMBOL_NAME_LEN];
size_t nbmem, count = 0;
FILE *fp;
struct lttng_event *elist;
@@ -619,7 +619,7 @@ ssize_t kernel_list_events(int tracer_fd, struct lttng_event **events)
goto end;
}
- while (fscanf(fp, "event { name = %m[^;]; };%n\n", &event, &pos) == 1) {
+ while (fscanf(fp, "event { name = %[^;]; };\n", event) == 1) {
if (count >= nbmem) {
struct lttng_event *new_elist;
@@ -630,7 +630,6 @@ ssize_t kernel_list_events(int tracer_fd, struct lttng_event **events)
new_elist = realloc(elist, nbmem * sizeof(struct lttng_event));
if (new_elist == NULL) {
PERROR("realloc list events");
- free(event);
free(elist);
count = -ENOMEM;
goto end;
@@ -641,7 +640,6 @@ ssize_t kernel_list_events(int tracer_fd, struct lttng_event **events)
elist[count].name[LTTNG_SYMBOL_NAME_LEN - 1] = '\0';
elist[count].enabled = -1;
count++;
- free(event);
}
*events = elist;
[-- Attachment #3: l_after_start_0.sessiond --]
[-- Type: application/octet-stream, Size: 2956 bytes --]
DEBUG1 [5340/5391]: Wait for client response (in thread_manage_clients() at main.c:3242)
DEBUG1 [5340/5391]: Receiving data from client ... (in thread_manage_clients() at main.c:3287)
DEBUG1 [5340/5391]: Nothing recv() from client... continuing (in thread_manage_clients() at main.c:3291)
DEBUG1 [5340/5391]: Clean command context structure (in clean_command_ctx() at main.c:482)
DEBUG1 [5340/5391]: Accepting client command ... (in thread_manage_clients() at main.c:3200)
DEBUG1 [5340/5391]: Wait for client response (in thread_manage_clients() at main.c:3242)
DEBUG1 [5340/5391]: Receiving data from client ... (in thread_manage_clients() at main.c:3287)
DEBUG1 [5340/5391]: Processing client command 16 (in process_client_msg() at main.c:2227)
DEBUG1 [5340/5391]: Getting session s2 by name (in process_client_msg() at main.c:2298)
DEBUG2 [5340/5391]: Trying to find session by name s2 (in session_find_by_name() at session.c:132)
DEBUG1 [5340/5391]: amitHANDLING COMMAND TO START TRACING..............
(in process_client_msg() at main.c:2693)
DEBUG1 [5340/5391]: Kernel metadata opened (fd: 90) (in kernel_open_metadata() at kernel.c:381)
DEBUG1 [5340/5391]: Kernel metadata stream created (fd: 91) (in kernel_open_metadata_stream() at kernel.c:572)
DEBUG1 [5340/5391]: Sending session stream to kernel consumer (in kernel_consumer_send_session() at kernel-consumer.c:319)
DEBUG1 [5340/5391]: Sending metadata 91 to kernel consumer (in kernel_consumer_add_metadata() at kernel-consumer.c:128)
DEBUG3 [5340/5391]: mkdir() recursive /data/lttng-traces/s2-20130306-180938//kernel with mode 504 for uid 0 and gid 0 (in run_as_mkdir_recursive() at runas.c:306)
DEBUG1 [5340/5391]: Using run_as_clone (in run_as() at runas.c:289)
DEBUG3 [5340/5391]: Kernel local consumer tracefile path: /data/lttng-traces/s2-20130306-180938//kernel (in kernel_consumer_add_metadata() at kernel-consumer.c:153)
DEBUG1 [5340/5391]: Sending streams of channel c1 to kernel consumer (in kernel_consumer_send_channel_stream() at kernel-consumer.c:275)
DEBUG1 [5340/5391]: Kernel consumer adding channel c1 to kernel consumer (in kernel_consumer_add_channel() at kernel-consumer.c:51)
DEBUG3 [5340/5391]: mkdir() recursive /data/lttng-traces/s2-20130306-180938//kernel with mode 504 for uid 0 and gid 0 (in run_as_mkdir_recursive() at runas.c:306)
DEBUG1 [5340/5391]: Using run_as_clone (in run_as() at runas.c:289)
DEBUG3 [5340/5391]: Kernel local consumer tracefile path: /data/lttng-traces/s2-20130306-180938//kernel (in kernel_consumer_add_channel() at kernel-consumer.c:73)
DEBUG1 [5340/5391]: Sending stream 28 of channel c1 to kernel consumer (in kernel_consumer_add_stream() at kernel-consumer.c:227)
DEBUG1 [5340/5391]: Sending stream 27 of channel c1 to kernel consumer (in kernel_consumer_add_stream() at kernel-consumer.c:227)
DEBUG1 [5340/5391]: Kernel consumer FDs of metadata and channel streams sent (in kernel_consumer_send_session() at kernel-consumer.c:339)
[-- Attachment #4: l_after_start_1.sessiond --]
[-- Type: application/octet-stream, Size: 2881 bytes --]
DEBUG1 [5448/5452]: Incoming command on sock (in consumer_thread_sessiond_poll() at consumer.c:2513)
DEBUG1 [5448/5452]: consumer_add_channel 90 (in lttng_kconsumer_recv_cmd() at kernel-consumer.c:131)
DEBUG1 [5448/5452]: Allocated channel (key 90) (in consumer_allocate_channel() at consumer.c:807)
DEBUG1 [5448/5452]: received command on sock (in consumer_thread_sessiond_poll() at consumer.c:2531)
DEBUG1 [5448/5452]: Incoming command on sock (in consumer_thread_sessiond_poll() at consumer.c:2513)
DEBUG3 [5448/5452]: Allocated stream metadata (key 19, relayd_id -1, session_id 0 (in consumer_allocate_stream() at consumer.c:549)
DEBUG3 [5448/5452]: open() /data/lttng-traces/s2-20130306-180938//kernel/metadata with flags 241 mode 511 for uid 0 and gid 0 (in run_as_open() at runas.c:334)
DEBUG1 [5448/5452]: Using run_as_clone (in run_as() at runas.c:289)
DEBUG1 [5448/5452]: Kernel consumer ADD_STREAM metadata (fd: 19) with relayd id 0 (in lttng_kconsumer_recv_cmd() at kernel-consumer.c:302)
DEBUG1 [5448/5452]: received command on sock (in consumer_thread_sessiond_poll() at consumer.c:2531)
DEBUG1 [5448/5452]: Incoming command on sock (in consumer_thread_sessiond_poll() at consumer.c:2513)
DEBUG1 [5448/5452]: consumer_add_channel 26 (in lttng_kconsumer_recv_cmd() at kernel-consumer.c:131)
DEBUG1 [5448/5452]: Allocated channel (key 26) (in consumer_allocate_channel() at consumer.c:807)
DEBUG1 [5448/5452]: received command on sock (in consumer_thread_sessiond_poll() at consumer.c:2531)
DEBUG1 [5448/5452]: Incoming command on sock (in consumer_thread_sessiond_poll() at consumer.c:2513)
DEBUG3 [5448/5452]: Allocated stream c1_1 (key 21, relayd_id -1, session_id 0 (in consumer_allocate_stream() at consumer.c:549)
DEBUG3 [5448/5452]: open() /data/lttng-traces/s2-20130306-180938//kernel/c1_1 with flags 241 mode 511 for uid 0 and gid 0 (in run_as_open() at runas.c:334)
DEBUG1 [5448/5452]: Using run_as_clone (in run_as() at runas.c:289)
DEBUG1 [5448/5452]: Kernel consumer ADD_STREAM c1_1 (fd: 21) with relayd id 0 (in lttng_kconsumer_recv_cmd() at kernel-consumer.c:302)
DEBUG1 [5448/5452]: received command on sock (in consumer_thread_sessiond_poll() at consumer.c:2531)
DEBUG1 [5448/5452]: Incoming command on sock (in consumer_thread_sessiond_poll() at consumer.c:2513)
DEBUG3 [5448/5452]: Allocated stream c1_0 (key 22, relayd_id -1, session_id 0 (in consumer_allocate_stream() at consumer.c:549)
DEBUG3 [5448/5452]: open() /data/lttng-traces/s2-20130306-180938//kernel/c1_0 with flags 241 mode 511 for uid 0 and gid 0 (in run_as_open() at runas.c:334)
DEBUG1 [5448/5452]: Using run_as_clone (in run_as() at runas.c:289)
DEBUG1 [5448/5452]: Kernel consumer ADD_STREAM c1_0 (fd: 22) with relayd id 0 (in lttng_kconsumer_recv_cmd() at kernel-consumer.c:302)
DEBUG1 [5448/5452]: received command on sock (in consumer_thread_sessiond_poll() at consumer.c:2531)
[-- Attachment #5: l_after_start_2.sessiond --]
[-- Type: application/octet-stream, Size: 2106 bytes --]
DEBUG1 [5448/5450]: Metadata event catched in thread (in consumer_thread_metadata_poll() at consumer.c:2030)
DEBUG1 [5448/5450]: Adding metadata stream 19 to poll set (in consumer_thread_metadata_poll() at consumer.c:2087)
DEBUG3 [5448/5450]: Adding metadata stream 19 to hash table (in add_metadata_stream() at consumer.c:1872)
DEBUG1 [5448/5450]: Metadata poll wait with 2 fd(s) (in consumer_thread_metadata_poll() at consumer.c:2028)
DEBUG1 [5448/5450]: Metadata event catched in thread (in consumer_thread_metadata_poll() at consumer.c:2030)
DEBUG1 [5448/5450]: Metadata available on fd 19 (in consumer_thread_metadata_poll() at consumer.c:2144)
DEBUG1 [5448/5450]: In read_subbuffer (infd : 19) (in lttng_kconsumer_read_subbuffer() at kernel-consumer.c:392)
DEBUG1 [5448/5450]: amit output type is : SPLICE ! (in lttng_kconsumer_read_subbuffer() at kernel-consumer.c:417)
DEBUG1 [5448/5450]: amit ########################## (in lttng_consumer_on_read_subbuffer_splice() at consumer.c:1438)
DEBUG1 [5448/5450]: amit ok !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! (in lttng_consumer_on_read_subbuffer_splice() at consumer.c:1453)
DEBUG1 [5448/5450]: splice chan to pipe offset 0 of len 4096 (fd : 19, pipe: 16) (in lttng_consumer_on_read_subbuffer_splice() at consumer.c:1524)
DEBUG1 [5448/5450]: amit $$ return from splice, ret=-38, errno=0 (in lttng_consumer_on_read_subbuffer_splice() at consumer.c:1527)
DEBUG1 [5448/5450]: splice chan to pipe, ret -38 (in lttng_consumer_on_read_subbuffer_splice() at consumer.c:1528)
PERROR [5448/5450]: Error in relay splice: (null) (in lttng_consumer_on_read_subbuffer_splice() at consumer.c:1530)
Error: Error splicing to tracefile (ret: -38 != len: 4096)
DEBUG3 [5448/5450]: Consumer delete metadata stream 19 (in consumer_del_metadata_stream() at consumer.c:1761)
DEBUG1 [5448/5450]: Consumer delete channel key 90 (in consumer_del_channel() at consumer.c:236)
DEBUG1 [5448/5450]: Metadata poll wait with 1 fd(s) (in consumer_thread_metadata_poll() at consumer.c:2028)
Error: consumer err socket second poll error
Error: Health error occurred in thread_manage_consumer
[-- Attachment #6: l_after_start_3.sessiond --]
[-- Type: application/octet-stream, Size: 1116 bytes --]
DEBUG1 [5448/5451]: poll num_rdy : 1 (in consumer_thread_data_poll() at consumer.c:2252)
DEBUG1 [5448/5451]: consumer_data_pipe wake up (in consumer_thread_data_poll() at consumer.c:2276)
DEBUG3 [5448/5451]: Adding consumer stream 21 (in add_stream() at consumer.c:576)
DEBUG1 [5448/5451]: Updating poll fd array (in update_poll_array() at consumer.c:864)
DEBUG1 [5448/5451]: Active FD 21 (in update_poll_array() at consumer.c:880)
DEBUG1 [5448/5451]: polling on 2 fd (in consumer_thread_data_poll() at consumer.c:2250)
DEBUG1 [5448/5451]: poll num_rdy : 1 (in consumer_thread_data_poll() at consumer.c:2252)
DEBUG1 [5448/5451]: consumer_data_pipe wake up (in consumer_thread_data_poll() at consumer.c:2276)
DEBUG3 [5448/5451]: Adding consumer stream 22 (in add_stream() at consumer.c:576)
DEBUG1 [5448/5451]: Updating poll fd array (in update_poll_array() at consumer.c:864)
DEBUG1 [5448/5451]: Active FD 22 (in update_poll_array() at consumer.c:880)
DEBUG1 [5448/5451]: Active FD 21 (in update_poll_array() at consumer.c:880)
DEBUG1 [5448/5451]: polling on 3 fd (in consumer_thread_data_poll() at consumer.c:2250)
[-- Attachment #7: l_after_start_4.sessiond --]
[-- Type: application/octet-stream, Size: 98 bytes --]
DEBUG1 [5340/5449]: consumer thread cleanup completed (in thread_manage_consumer() at main.c:1084)
[-- Attachment #8: Type: text/plain, Size: 155 bytes --]
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: Fwd: compiling LTTng for android
[not found] ` <CAHg4Oy_X-NhH5_JFXCBg-HsoZxyh+equV1O8vDnqAVv1QCroRw@mail.gmail.com>
@ 2013-03-07 1:04 ` Pierre-Luc St-Charles
[not found] ` <CA+umk=qjGDo4O7NmhfKHQRM96pTz3X8TnveaShtWQJRBZ82X3A@mail.gmail.com>
1 sibling, 0 replies; 8+ messages in thread
From: Pierre-Luc St-Charles @ 2013-03-07 1:04 UTC (permalink / raw)
To: Amit Balboul; +Cc: lttng-dev
[-- Attachment #1.1: Type: text/plain, Size: 18829 bytes --]
Great findings!
I'll also try to look into this new problem by next monday; I'm falling a
bit behind on my 'investigations', but I'll keep you updated if anything
comes up.
-PL
On Mar 6, 2013 6:09 PM, "Amit Balboul" <amit.balboul@shinesec.com> wrote:
> Hi All,
>
> Regarding the last mail about tracing system calls on Android using LTTng,
> The issue of not receiving enabled event to trace from the kernel can be
> fixed patching src/bin/lttng-sessiond/kernel.c (patch attached)
> In the function *kernel_list_events,* fscanf-ing the events exported by
> the kernel using the fp must not use the syntax *%m* as it is obsolete
> (in Android),
> thus the fscanf returns 0 and won't read the kernel events (first
> condition false).
> Instead I use a pre-allocated buffer for the string and use the *%[^;]*syntax, and omitting the memory free (attached is the patch).
>
> The resulting output is (sessiond is run with verbose debug prints):
>
> root@android:/data/lttng/bin # *./lttng list -k*
> DEBUG1 [10239/10290]: Wait for client response (in thread_manage_clients()
> at main.c:3242)
> DEBUG1 [10239/10290]: Receiving data from client ... (in
> thread_manage_clients() at main.c:3287)
> DEBUG1 [10239/10290]: Nothing recv() from client... continuing (in
> thread_manage_clients() at main.c:3291)
> DEBUG1 [10239/10290]: Clean command context structure (in
> clean_command_ctx() at main.c:482)
> DEBUG1 [10239/10290]: Accepting client command ... (in
> thread_manage_clients() at main.c:3200)
> DEBUG1 [10239/10290]: Wait for client response (in thread_manage_clients()
> at main.c:3242)
> DEBUG1 [10239/10290]: Receiving data from client ... (in
> thread_manage_clients() at main.c:3287)
> DEBUG1 [10239/10290]: Processing client command 14 (in
> process_client_msg() at main.c:2227)
> DEBUG1 [10239/10290]: Kernel list events done (61 events) (in
> kernel_list_events() at kernel.c:653)
> DEBUG1 [10239/10290]: Sending response (size: 35640, retcode: Success) (in
> thread_manage_clients() at main.c:3338)
> DEBUG1 [10239/10290]: Clean command context structure (in
> clean_command_ctx() at main.c:482)
> DEBUG1 [10239/10290]: Accepting client command ... (in
> thread_manage_clients() at main.c:3200)
> Kernel events:
> -------------
> block_rq_abort (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> block_rq_requeue (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> block_rq_complete (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> block_rq_insert (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> block_rq_issue (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> block_bio_bounce (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> block_bio_complete (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> block_bio_backmerge (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> block_bio_frontmerge (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> block_bio_queue (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> block_getrq (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> block_sleeprq (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> block_plug (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> block_unplug (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> block_split (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> block_bio_remap (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> block_rq_remap (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> irq_handler_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> irq_handler_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> softirq_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> softirq_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> softirq_raise (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> sched_kthread_stop (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> sched_kthread_stop_ret (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> sched_wakeup (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> sched_wakeup_new (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> sched_switch (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> sched_migrate_task (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> sched_process_free (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> sched_process_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> sched_wait_task (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> sched_process_wait (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> sched_process_fork (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> sched_stat_wait (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> sched_stat_sleep (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> sched_stat_iowait (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> sched_stat_runtime (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> sched_pi_setprio (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> signal_generate (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> signal_deliver (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> signal_overflow_fail (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> signal_lose_info (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> lttng_statedump_start (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> lttng_statedump_end (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> lttng_statedump_process_state (loglevel: TRACE_EMERG (0)) (type:
> tracepoint)
> lttng_statedump_file_descriptor (loglevel: TRACE_EMERG (0)) (type:
> tracepoint)
> lttng_statedump_vm_map (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> lttng_statedump_network_interface (loglevel: TRACE_EMERG (0)) (type:
> tracepoint)
> lttng_statedump_interrupt (loglevel: TRACE_EMERG (0)) (type:
> tracepoint)
> timer_init (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> timer_start (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> timer_expire_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> timer_expire_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> timer_cancel (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> hrtimer_init (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> hrtimer_start (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> hrtimer_expire_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> hrtimer_expire_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> hrtimer_cancel (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> itimer_state (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> itimer_expire (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>
> root@android:/data/lttng/bin #
>
> Now I'm facing the issue of starting the traces:
> debugging the sessiond, consumerd and lttng I concluded that the issue is
> due to splice system call failure.
> even using a MMAP output type channel, the metadata channel is SPLICE type
> (hard-coded).
> Problem persists also when using relayd to pass the traces through the
> network.
> Ignoring the splice failure means closing of the metadata channel
> (maybe??) but need not affect the traces channel (right ?)
> Attached is a debug log of each thread of the sessiond, right after
> invoking "*./lttng start*". (with extra informative my prints)
>
> Please keep me in touch with any progress in this issue,
> Thank you,
> Amit.
>
>
>
>
>
>
> ---------- Forwarded message ----------
> From: Amit Balboul <amit.balboul@shinesec.com>
> Date: Wed, Feb 27, 2013 at 4:37 PM
> Subject: Re: [lttng-dev] compiling LTTng for android
> To: PLSTC <b0mb00z.it@gmail.com>
>
>
> Hey Pierre,
>
> Some major progress has been achieved, following is the summary:
>
> 1. Modules issue:
> I built (for ARM/Android) and installed latest stable version of busybox
> on the device.
> Seeing the code tries to run (hardcoded) /sbin/modprobe, made a soft link
> to busybox's modeprobe.
> Also, created /lib/modules/<version+build> directory and pushed the
> <version+build> directory from /lib/modules on the machine.
> Now session daemon modprobes the desired modules successfully, (some are
> not present though as mentioned before) - no need to use insmod.
>
> 2. epoll_create1 issue:
> epoll_create1 patch was OK, I accidentally defined EPOLL_CLOEXEC as
> 0x02000000 instead of 02000000 (fixed).
>
> The running scenario:
>
> 1. Run ./lttng-sessiond -d -vvv
>
> Then: (commands are in bold, output is in blue for readability)
>
> root@android:/data/lttng/install_sysroot/bin # *./lttng create s1*
>
> Session s1 created.
> Traces will be written in /data/lttng-traces/s1-20130227-161145
> root@android:/data/lttng/install_sysroot/bin # *./lttng enable-channel ch
> -k*
> Kernel channel ch enabled for session s1
> root@android:/data/lttng/install_sysroot/bin # *./lttng enable-event ev
> -k -a*
> All kernel events are enabled in channel channel0
> root@android:/data/lttng/install_sysroot/bin # *./lttng list s1*
> Tracing session s1: [inactive]
> Trace path: /data/lttng-traces/s1-20130227-161145
>
> === Domain: Kernel ===
>
> Channels:
> -------------
> - channel0: [enabled]
>
> Attributes:
> overwrite mode: 0
> subbufers size: 262144
> number of subbufers: 4
> switch timer interval: 0
> read timer interval: 200
> output: splice()
>
> Events:
> None
>
> - ch: [enabled]
>
> Attributes:
> overwrite mode: 0
> subbufers size: 262144
> number of subbufers: 4
> switch timer interval: 0
> read timer interval: 200
> output: splice()
>
> Events:
> None
>
> root@android:/data/lttng/install_sysroot/bin # *./lttng list -k *
>
> Kernel events:
> -------------
>
> root@android:/data/lttng/install_sysroot/bin # *./lttng start*
> Tracing started for session s1
> root@android:/data/lttng/install_sysroot/bin # *./lttng stop *
>
> Waiting for data availability
> Tracing stopped for session s1
> root@android:/data/lttng/install_sysroot/bin #
>
> *Now: (this is not surprising because the kernel events list is empty...)*
> the folder /data/lttng-traces/s1-20130227-161145/kernel contains:
> total 0
> -rwxrwxrwx 1 0 0 0 Feb 27 14:18 ch_0
> -rwxrwxrwx 1 0 0 0 Feb 27 14:18 ch_1
> -rwxrwxrwx 1 0 0 0 Feb 27 14:18 channel0_0
> -rwxrwxrwx 1 0 0 0 Feb 27 14:18 channel0_1
> -rwxrwxrwx 1 0 0 0 Feb 27 14:18 metadata
>
> *By the way:*
> When running the same scenario but creating the channel with *--output
> mmap*, still no data but the size of the first two files in the traces
> directory are 4Kb (I didn't bother to investigate them as I assume it is
> just the header - not the traces payload).
>
> Another issue: at this point, listing the sessions gives:
> root@android:/data/lttng/install_sysroot/bin # *./lttng list s1 *
>
> Tracing session s1: [inactive]
> Trace path: /data/lttng-traces/s1-20130227-161145
>
> === Domain: Kernel ===
>
> Error: No kernel consumer detected
> 99|root@android:/data/lttng/install_sysroot/bin # *./lttng list -k *
>
> Error: Unable to list kernel events
> Error: Command error
> 1|root@android:/data/lttng/install_sysroot/bin #
>
> So currently I'm facing these issues:
> 1. Kernel events list is empty.
> 2. Listing the session/ kernel events after creating session, channel,
> event, start, stop - results in "no kernel consumer detected"
>
> If you have some insights please share them with me..
>
> Thanks
>
> Amit.
>
>
> On Tue, Feb 26, 2013 at 7:35 PM, PLSTC <b0mb00z.it@gmail.com> wrote:
>
>> The modules you can't find are the same I am also missing I believe; I
>> think they are only compiled when certain kernel config defines are met
>> (and those might simply be unavailable under ARM).
>>
>> As for what you pointed in your first mail, I believe modprobe isn't
>> available on ARM, did you mean that some modules were not found when using
>> insmod?
>>
>> For the epoll_create1 problem, what kind of patch did you apply to add
>> support for the epoll_create1 function?
>>
>>
>> On Tue, Feb 26, 2013 at 11:29 AM, Amit Balboul <amit.balboul@shinesec.com
>> > wrote:
>>
>>> Ok I got :
>>> lttng-kretprobes.ko
>>> lttng-kprobes.ko
>>> , the other two are missing.
>>>
>>>
>>> ---------- Forwarded message ----------
>>> From: Amit Balboul <amit.balboul@shinesec.com>
>>> Date: Tue, Feb 26, 2013 at 5:49 PM
>>> Subject: Re: [lttng-dev] compiling LTTng for android
>>> To: PLSTC <b0mb00z.it@gmail.com>
>>>
>>>
>>> One more thing:
>>>
>>> The list of modules you gave my is not identical to mine:
>>> a. I pushed the modules from my machine to my folder in the device (not
>>> to /system/lib) as you directed me, and insmoded them manually.
>>> b. I pushed the whole tree under /lib/modules/<KERNEL_VER_BUILD>/ which
>>> includes the modules.* and the sub directories:
>>>
>>> /lib/modules/3.0.31-gd5a18e0/extra$ ll
>>> total 2464
>>> drwxr-xr-x 4 root root 4096 Feb 26 15:58 ./
>>> drwxr-xr-x 3 root root 4096 Feb 26 15:58 ../
>>> drwxr-xr-x 2 root root 4096 Feb 26 15:58 lib/
>>> -rw-r--r-- 1 root root 167184 Feb 26 15:58
>>> lttng-ring-buffer-client-discard.ko
>>> -rw-r--r-- 1 root root 167206 Feb 26 15:58
>>> lttng-ring-buffer-client-mmap-discard.ko
>>> -rw-r--r-- 1 root root 178586 Feb 26 15:58
>>> lttng-ring-buffer-client-mmap-overwrite.ko
>>> -rw-r--r-- 1 root root 178564 Feb 26 15:58
>>> lttng-ring-buffer-client-overwrite.ko
>>> -rw-r--r-- 1 root root 137179 Feb 26 15:58
>>> lttng-ring-buffer-metadata-client.ko
>>> -rw-r--r-- 1 root root 137233 Feb 26 15:58
>>> lttng-ring-buffer-metadata-mmap-client.ko
>>> -rw-r--r-- 1 root root 213451 Feb 26 15:58 lttng-statedump.ko
>>> -rw-r--r-- 1 root root 1314235 Feb 26 15:58 lttng-tracer.ko
>>> drwxr-xr-x 2 root root 4096 Feb 26 15:58 probes/
>>>
>>> /lib/modules/3.0.31-gd5a18e0/extra/lib$ ll
>>> total 732
>>> drwxr-xr-x 2 root root 4096 Feb 26 15:58 ./
>>> drwxr-xr-x 4 root root 4096 Feb 26 15:58 ../
>>> -rw-r--r-- 1 root root 738964 Feb 26 15:58 lttng-lib-ring-buffer.ko
>>>
>>> /lib/modules/3.0.31-gd5a18e0/extra/probes$ ll
>>> total 2872
>>> drwxr-xr-x 2 root root 4096 Feb 26 15:58 ./
>>> drwxr-xr-x 4 root root 4096 Feb 26 15:58 ../
>>> -rw-r--r-- 1 root root 162872 Feb 26 15:58 lttng-probe-asoc.ko
>>> -rw-r--r-- 1 root root 171049 Feb 26 15:58 lttng-probe-block.ko
>>> -rw-r--r-- 1 root root 81165 Feb 26 15:58 lttng-probe-compaction.ko
>>> -rw-r--r-- 1 root root 333215 Feb 26 15:58 lttng-probe-ext4.ko
>>> -rw-r--r-- 1 root root 80413 Feb 26 15:58 lttng-probe-gpio.ko
>>> -rw-r--r-- 1 root root 84256 Feb 26 15:58 lttng-probe-irq.ko
>>> -rw-r--r-- 1 root root 135453 Feb 26 15:58 lttng-probe-jbd2.ko
>>> -rw-r--r-- 1 root root 105399 Feb 26 15:58 lttng-probe-kmem.ko
>>> -rw-r--r-- 1 root root 76240 Feb 26 15:58 lttng-probe-lttng.ko
>>> -rw-r--r-- 1 root root 88059 Feb 26 15:58 lttng-probe-module.ko
>>> -rw-r--r-- 1 root root 154032 Feb 26 15:58 lttng-probe-napi.ko
>>> -rw-r--r-- 1 root root 142038 Feb 26 15:58 lttng-probe-net.ko
>>> -rw-r--r-- 1 root root 93564 Feb 26 15:58 lttng-probe-power.ko
>>> -rw-r--r-- 1 root root 85530 Feb 26 15:58 lttng-probe-regulator.ko
>>> -rw-r--r-- 1 root root 138121 Feb 26 15:58 lttng-probe-sched.ko
>>> -rw-r--r-- 1 root root 152909 Feb 26 15:58 lttng-probe-scsi.ko
>>> -rw-r--r-- 1 root root 107491 Feb 26 15:58 lttng-probe-signal.ko
>>> -rw-r--r-- 1 root root 142711 Feb 26 15:58 lttng-probe-skb.ko
>>> -rw-r--r-- 1 root root 181983 Feb 26 15:58 lttng-probe-statedump.ko
>>> -rw-r--r-- 1 root root 126731 Feb 26 15:58 lttng-probe-timer.ko
>>> -rw-r--r-- 1 root root 86591 Feb 26 15:58 lttng-probe-workqueue.ko
>>> -rw-r--r-- 1 root root 133886 Feb 26 15:58 lttng-probe-writeback.ko
>>> -rw-r--r-- 1 root root 30863 Feb 26 15:58 lttng-types.ko
>>>
>>> Looking at the files shows I miss:
>>> lttng-kretprobes.ko
>>> lttng-kprobes.ko
>>> lttng-ftrace.ko
>>> lttng-probe-kvm.ko
>>>
>>>
>>> On Tue, Feb 26, 2013 at 3:56 PM, PLSTC <b0mb00z.it@gmail.com> wrote:
>>>
>>>> Back on the last email:
>>>>
>>>> Here's the insmod list I ended up building; you might not have compiled
>>>> all those modules, depending on your kernel config and on the architecture
>>>> used, so some might simply be absent from your output directory.
>>>>
>>>> insmod /system/lib/modules/lttng-types.ko
>>>> insmod /system/lib/modules/lttng-kretprobes.ko
>>>> insmod /system/lib/modules/lttng-kprobes.ko
>>>> insmod /system/lib/modules/lttng-ftrace.ko
>>>> insmod /system/lib/modules/lttng-lib-ring-buffer.ko
>>>> insmod /system/lib/modules/lttng-statedump.ko
>>>> insmod /system/lib/modules/lttng-tracer.ko
>>>> insmod /system/lib/modules/lttng-probe-timer.ko
>>>> insmod /system/lib/modules/lttng-probe-statedump.ko
>>>> insmod /system/lib/modules/lttng-probe-signal.ko
>>>> insmod /system/lib/modules/lttng-probe-sched.ko
>>>> insmod /system/lib/modules/lttng-probe-lttng.ko
>>>> insmod /system/lib/modules/lttng-probe-kvm.ko
>>>> insmod /system/lib/modules/lttng-probe-irq.ko
>>>> insmod /system/lib/modules/lttng-probe-block.ko
>>>> insmod /system/lib/modules/lttng-ring-buffer-metadata-mmap-client.ko
>>>> insmod /system/lib/modules/lttng-ring-buffer-metadata-client.ko
>>>> insmod /system/lib/modules/lttng-ring-buffer-client-overwrite.ko
>>>> insmod /system/lib/modules/lttng-ring-buffer-client-mmap-overwrite.ko
>>>> insmod /system/lib/modules/lttng-ring-buffer-client-mmap-discard.ko
>>>> insmod /system/lib/modules/lttng-ring-buffer-client-discard.ko
>>>>
>>>>
>>>> We also haven't had time recently to test if the kprobes/kretprobes
>>>> worked, but they probably do; we decided to focus a bit more on the
>>>> 'tracepoints' aspect instead.
>>>>
>>>> Anyway, good luck with your exploration, and if you find something
>>>> interesting, I'd love to know more about it!
>>>>
>>>> -PL
>>>>
>>>> On Tue, Feb 26, 2013 at 8:06 AM, PLSTC <b0mb00z.it@gmail.com> wrote:
>>>>
>>>>> Hey Amit,
>>>>>
>>>>> Good news indeed it seems!
>>>>>
>>>>> When it comes to pushing the libs/bin, we haven't found any better way
>>>>> than using adb push yet; of course, if all the projects were included
>>>>> directly in Android's external, we would simply flash the device after each
>>>>> build, but currently we're still having trouble generating the correct
>>>>> Android makefiles.
>>>>>
>>>>> As for the modules, I might still have a dep load list somewhere I
>>>>> could send you in a few hours, I remember fighting to get the order right...
>>>>>
>>>>> Also, once everything loads, could you tell me what kind of output
>>>>> 'lttng list -k' provides you? Here, we're having trouble listing the
>>>>> 'default' kernel trace points provided by HAVE_TRACEPOINTS, and the
>>>>> returned list is simply empty (which is a bit strange).
>>>>>
>>>>> -PL
>>>>>
>>>>
>>>>
>>>
>>>
>>
>
>
[-- Attachment #1.2: Type: text/html, Size: 34857 bytes --]
[-- Attachment #2: Type: text/plain, Size: 155 bytes --]
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Fwd: compiling LTTng for android
[not found] ` <CA+umk=qjGDo4O7NmhfKHQRM96pTz3X8TnveaShtWQJRBZ82X3A@mail.gmail.com>
@ 2013-03-12 22:51 ` Pierre-Luc St-Charles
[not found] ` <CA+umk=pgM9BVebkXj2hrgi_LDBdwjm2ZGpS=PmT9DapPJ5kD0A@mail.gmail.com>
1 sibling, 0 replies; 8+ messages in thread
From: Pierre-Luc St-Charles @ 2013-03-12 22:51 UTC (permalink / raw)
To: Amit Balboul; +Cc: lttng-dev
[-- Attachment #1.1: Type: text/plain, Size: 20842 bytes --]
Hey Amit,
Here's a small update on my latest findings regarding the broken *splice* /
bad Bionic impl problem:
Although it didn't really cross my mind earlier, we had seen this exact
problem before, but in another project; we had also investigated a bit at
the time, but to no avail. It seems the issue is known by the
Android/Bionic community, but it's simply not on their immediate todo list
(and hasn't been for at least the past two years -- see
https://code.google.com/p/android/issues/detail?id=11330). The solutions we
are now considering are: adding splice support (and whatever else might
also be needed) to Bionic ourselves (saving BILLIONS of broken lives in the
process, IF the changes are accepted), or simply inserting glibc (and
building against it) in our device images.
In the next few days, we'll be studying how much time the first solution
would actually take (as we're not really experts in that domain at all),
but considering the nature/objective of this project, it might be the
wisest decision (as it would not require too much swerving away from the
default images people have on their devices for LTTng to actually work). If
we deem it 'impossible/out of our league', we'll simply try to get a
working tool asap using glibc, which should not be very complicated itself
(many people have actually successfully done it before).
In any case, I'll keep you updated, and I'll also post any big updates on
the mailing list.
-PL
On Wed, Mar 6, 2013 at 8:04 PM, Pierre-Luc St-Charles <
pierre-luc.st-charles@polymtl.ca> wrote:
> Great findings!
>
> I'll also try to look into this new problem by next monday; I'm falling a
> bit behind on my 'investigations', but I'll keep you updated if anything
> comes up.
>
> -PL
> On Mar 6, 2013 6:09 PM, "Amit Balboul" <amit.balboul@shinesec.com> wrote:
>
>> Hi All,
>>
>> Regarding the last mail about tracing system calls on Android using LTTng,
>> The issue of not receiving enabled event to trace from the kernel can be
>> fixed patching src/bin/lttng-sessiond/kernel.c (patch attached)
>> In the function *kernel_list_events,* fscanf-ing the events exported by
>> the kernel using the fp must not use the syntax *%m* as it is obsolete
>> (in Android),
>> thus the fscanf returns 0 and won't read the kernel events (first
>> condition false).
>> Instead I use a pre-allocated buffer for the string and use the *%[^;]*syntax, and omitting the memory free (attached is the patch).
>>
>> The resulting output is (sessiond is run with verbose debug prints):
>>
>> root@android:/data/lttng/bin # *./lttng list -k*
>> DEBUG1 [10239/10290]: Wait for client response (in
>> thread_manage_clients() at main.c:3242)
>> DEBUG1 [10239/10290]: Receiving data from client ... (in
>> thread_manage_clients() at main.c:3287)
>> DEBUG1 [10239/10290]: Nothing recv() from client... continuing (in
>> thread_manage_clients() at main.c:3291)
>> DEBUG1 [10239/10290]: Clean command context structure (in
>> clean_command_ctx() at main.c:482)
>> DEBUG1 [10239/10290]: Accepting client command ... (in
>> thread_manage_clients() at main.c:3200)
>> DEBUG1 [10239/10290]: Wait for client response (in
>> thread_manage_clients() at main.c:3242)
>> DEBUG1 [10239/10290]: Receiving data from client ... (in
>> thread_manage_clients() at main.c:3287)
>> DEBUG1 [10239/10290]: Processing client command 14 (in
>> process_client_msg() at main.c:2227)
>> DEBUG1 [10239/10290]: Kernel list events done (61 events) (in
>> kernel_list_events() at kernel.c:653)
>> DEBUG1 [10239/10290]: Sending response (size: 35640, retcode: Success)
>> (in thread_manage_clients() at main.c:3338)
>> DEBUG1 [10239/10290]: Clean command context structure (in
>> clean_command_ctx() at main.c:482)
>> DEBUG1 [10239/10290]: Accepting client command ... (in
>> thread_manage_clients() at main.c:3200)
>> Kernel events:
>> -------------
>> block_rq_abort (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> block_rq_requeue (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> block_rq_complete (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> block_rq_insert (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> block_rq_issue (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> block_bio_bounce (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> block_bio_complete (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> block_bio_backmerge (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> block_bio_frontmerge (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> block_bio_queue (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> block_getrq (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> block_sleeprq (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> block_plug (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> block_unplug (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> block_split (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> block_bio_remap (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> block_rq_remap (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> irq_handler_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> irq_handler_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> softirq_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> softirq_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> softirq_raise (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> sched_kthread_stop (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> sched_kthread_stop_ret (loglevel: TRACE_EMERG (0)) (type:
>> tracepoint)
>> sched_wakeup (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> sched_wakeup_new (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> sched_switch (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> sched_migrate_task (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> sched_process_free (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> sched_process_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> sched_wait_task (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> sched_process_wait (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> sched_process_fork (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> sched_stat_wait (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> sched_stat_sleep (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> sched_stat_iowait (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> sched_stat_runtime (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> sched_pi_setprio (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> signal_generate (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> signal_deliver (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> signal_overflow_fail (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> signal_lose_info (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> lttng_statedump_start (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> lttng_statedump_end (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> lttng_statedump_process_state (loglevel: TRACE_EMERG (0)) (type:
>> tracepoint)
>> lttng_statedump_file_descriptor (loglevel: TRACE_EMERG (0)) (type:
>> tracepoint)
>> lttng_statedump_vm_map (loglevel: TRACE_EMERG (0)) (type:
>> tracepoint)
>> lttng_statedump_network_interface (loglevel: TRACE_EMERG (0))
>> (type: tracepoint)
>> lttng_statedump_interrupt (loglevel: TRACE_EMERG (0)) (type:
>> tracepoint)
>> timer_init (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> timer_start (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> timer_expire_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> timer_expire_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> timer_cancel (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> hrtimer_init (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> hrtimer_start (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> hrtimer_expire_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> hrtimer_expire_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> hrtimer_cancel (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> itimer_state (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>> itimer_expire (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>
>> root@android:/data/lttng/bin #
>>
>> Now I'm facing the issue of starting the traces:
>> debugging the sessiond, consumerd and lttng I concluded that the issue is
>> due to splice system call failure.
>> even using a MMAP output type channel, the metadata channel is SPLICE
>> type (hard-coded).
>> Problem persists also when using relayd to pass the traces through the
>> network.
>> Ignoring the splice failure means closing of the metadata channel
>> (maybe??) but need not affect the traces channel (right ?)
>> Attached is a debug log of each thread of the sessiond, right after
>> invoking "*./lttng start*". (with extra informative my prints)
>>
>> Please keep me in touch with any progress in this issue,
>> Thank you,
>> Amit.
>>
>>
>>
>>
>>
>>
>> ---------- Forwarded message ----------
>> From: Amit Balboul <amit.balboul@shinesec.com>
>> Date: Wed, Feb 27, 2013 at 4:37 PM
>> Subject: Re: [lttng-dev] compiling LTTng for android
>> To: PLSTC <b0mb00z.it@gmail.com>
>>
>>
>> Hey Pierre,
>>
>> Some major progress has been achieved, following is the summary:
>>
>> 1. Modules issue:
>> I built (for ARM/Android) and installed latest stable version of busybox
>> on the device.
>> Seeing the code tries to run (hardcoded) /sbin/modprobe, made a soft link
>> to busybox's modeprobe.
>> Also, created /lib/modules/<version+build> directory and pushed the
>> <version+build> directory from /lib/modules on the machine.
>> Now session daemon modprobes the desired modules successfully, (some are
>> not present though as mentioned before) - no need to use insmod.
>>
>> 2. epoll_create1 issue:
>> epoll_create1 patch was OK, I accidentally defined EPOLL_CLOEXEC as
>> 0x02000000 instead of 02000000 (fixed).
>>
>> The running scenario:
>>
>> 1. Run ./lttng-sessiond -d -vvv
>>
>> Then: (commands are in bold, output is in blue for readability)
>>
>> root@android:/data/lttng/install_sysroot/bin # *./lttng create s1*
>>
>> Session s1 created.
>> Traces will be written in /data/lttng-traces/s1-20130227-161145
>> root@android:/data/lttng/install_sysroot/bin # *./lttng enable-channel
>> ch -k*
>> Kernel channel ch enabled for session s1
>> root@android:/data/lttng/install_sysroot/bin # *./lttng enable-event ev
>> -k -a*
>> All kernel events are enabled in channel channel0
>> root@android:/data/lttng/install_sysroot/bin # *./lttng list s1*
>> Tracing session s1: [inactive]
>> Trace path: /data/lttng-traces/s1-20130227-161145
>>
>> === Domain: Kernel ===
>>
>> Channels:
>> -------------
>> - channel0: [enabled]
>>
>> Attributes:
>> overwrite mode: 0
>> subbufers size: 262144
>> number of subbufers: 4
>> switch timer interval: 0
>> read timer interval: 200
>> output: splice()
>>
>> Events:
>> None
>>
>> - ch: [enabled]
>>
>> Attributes:
>> overwrite mode: 0
>> subbufers size: 262144
>> number of subbufers: 4
>> switch timer interval: 0
>> read timer interval: 200
>> output: splice()
>>
>> Events:
>> None
>>
>> root@android:/data/lttng/install_sysroot/bin # *./lttng list -k *
>>
>> Kernel events:
>> -------------
>>
>> root@android:/data/lttng/install_sysroot/bin # *./lttng start*
>> Tracing started for session s1
>> root@android:/data/lttng/install_sysroot/bin # *./lttng stop *
>>
>> Waiting for data availability
>> Tracing stopped for session s1
>> root@android:/data/lttng/install_sysroot/bin #
>>
>> *Now: (this is not surprising because the kernel events list is empty...)
>> *
>> the folder /data/lttng-traces/s1-20130227-161145/kernel contains:
>> total 0
>> -rwxrwxrwx 1 0 0 0 Feb 27 14:18 ch_0
>> -rwxrwxrwx 1 0 0 0 Feb 27 14:18 ch_1
>> -rwxrwxrwx 1 0 0 0 Feb 27 14:18 channel0_0
>> -rwxrwxrwx 1 0 0 0 Feb 27 14:18 channel0_1
>> -rwxrwxrwx 1 0 0 0 Feb 27 14:18 metadata
>>
>> *By the way:*
>> When running the same scenario but creating the channel with *--output
>> mmap*, still no data but the size of the first two files in the traces
>> directory are 4Kb (I didn't bother to investigate them as I assume it is
>> just the header - not the traces payload).
>>
>> Another issue: at this point, listing the sessions gives:
>> root@android:/data/lttng/install_sysroot/bin # *./lttng list s1 *
>>
>> Tracing session s1: [inactive]
>> Trace path: /data/lttng-traces/s1-20130227-161145
>>
>> === Domain: Kernel ===
>>
>> Error: No kernel consumer detected
>> 99|root@android:/data/lttng/install_sysroot/bin # *./lttng list -k *
>>
>> Error: Unable to list kernel events
>> Error: Command error
>> 1|root@android:/data/lttng/install_sysroot/bin #
>>
>> So currently I'm facing these issues:
>> 1. Kernel events list is empty.
>> 2. Listing the session/ kernel events after creating session, channel,
>> event, start, stop - results in "no kernel consumer detected"
>>
>> If you have some insights please share them with me..
>>
>> Thanks
>>
>> Amit.
>>
>>
>> On Tue, Feb 26, 2013 at 7:35 PM, PLSTC <b0mb00z.it@gmail.com> wrote:
>>
>>> The modules you can't find are the same I am also missing I believe; I
>>> think they are only compiled when certain kernel config defines are met
>>> (and those might simply be unavailable under ARM).
>>>
>>> As for what you pointed in your first mail, I believe modprobe isn't
>>> available on ARM, did you mean that some modules were not found when using
>>> insmod?
>>>
>>> For the epoll_create1 problem, what kind of patch did you apply to add
>>> support for the epoll_create1 function?
>>>
>>>
>>> On Tue, Feb 26, 2013 at 11:29 AM, Amit Balboul <
>>> amit.balboul@shinesec.com> wrote:
>>>
>>>> Ok I got :
>>>> lttng-kretprobes.ko
>>>> lttng-kprobes.ko
>>>> , the other two are missing.
>>>>
>>>>
>>>> ---------- Forwarded message ----------
>>>> From: Amit Balboul <amit.balboul@shinesec.com>
>>>> Date: Tue, Feb 26, 2013 at 5:49 PM
>>>> Subject: Re: [lttng-dev] compiling LTTng for android
>>>> To: PLSTC <b0mb00z.it@gmail.com>
>>>>
>>>>
>>>> One more thing:
>>>>
>>>> The list of modules you gave my is not identical to mine:
>>>> a. I pushed the modules from my machine to my folder in the device (not
>>>> to /system/lib) as you directed me, and insmoded them manually.
>>>> b. I pushed the whole tree under /lib/modules/<KERNEL_VER_BUILD>/ which
>>>> includes the modules.* and the sub directories:
>>>>
>>>> /lib/modules/3.0.31-gd5a18e0/extra$ ll
>>>> total 2464
>>>> drwxr-xr-x 4 root root 4096 Feb 26 15:58 ./
>>>> drwxr-xr-x 3 root root 4096 Feb 26 15:58 ../
>>>> drwxr-xr-x 2 root root 4096 Feb 26 15:58 lib/
>>>> -rw-r--r-- 1 root root 167184 Feb 26 15:58
>>>> lttng-ring-buffer-client-discard.ko
>>>> -rw-r--r-- 1 root root 167206 Feb 26 15:58
>>>> lttng-ring-buffer-client-mmap-discard.ko
>>>> -rw-r--r-- 1 root root 178586 Feb 26 15:58
>>>> lttng-ring-buffer-client-mmap-overwrite.ko
>>>> -rw-r--r-- 1 root root 178564 Feb 26 15:58
>>>> lttng-ring-buffer-client-overwrite.ko
>>>> -rw-r--r-- 1 root root 137179 Feb 26 15:58
>>>> lttng-ring-buffer-metadata-client.ko
>>>> -rw-r--r-- 1 root root 137233 Feb 26 15:58
>>>> lttng-ring-buffer-metadata-mmap-client.ko
>>>> -rw-r--r-- 1 root root 213451 Feb 26 15:58 lttng-statedump.ko
>>>> -rw-r--r-- 1 root root 1314235 Feb 26 15:58 lttng-tracer.ko
>>>> drwxr-xr-x 2 root root 4096 Feb 26 15:58 probes/
>>>>
>>>> /lib/modules/3.0.31-gd5a18e0/extra/lib$ ll
>>>> total 732
>>>> drwxr-xr-x 2 root root 4096 Feb 26 15:58 ./
>>>> drwxr-xr-x 4 root root 4096 Feb 26 15:58 ../
>>>> -rw-r--r-- 1 root root 738964 Feb 26 15:58 lttng-lib-ring-buffer.ko
>>>>
>>>> /lib/modules/3.0.31-gd5a18e0/extra/probes$ ll
>>>> total 2872
>>>> drwxr-xr-x 2 root root 4096 Feb 26 15:58 ./
>>>> drwxr-xr-x 4 root root 4096 Feb 26 15:58 ../
>>>> -rw-r--r-- 1 root root 162872 Feb 26 15:58 lttng-probe-asoc.ko
>>>> -rw-r--r-- 1 root root 171049 Feb 26 15:58 lttng-probe-block.ko
>>>> -rw-r--r-- 1 root root 81165 Feb 26 15:58 lttng-probe-compaction.ko
>>>> -rw-r--r-- 1 root root 333215 Feb 26 15:58 lttng-probe-ext4.ko
>>>> -rw-r--r-- 1 root root 80413 Feb 26 15:58 lttng-probe-gpio.ko
>>>> -rw-r--r-- 1 root root 84256 Feb 26 15:58 lttng-probe-irq.ko
>>>> -rw-r--r-- 1 root root 135453 Feb 26 15:58 lttng-probe-jbd2.ko
>>>> -rw-r--r-- 1 root root 105399 Feb 26 15:58 lttng-probe-kmem.ko
>>>> -rw-r--r-- 1 root root 76240 Feb 26 15:58 lttng-probe-lttng.ko
>>>> -rw-r--r-- 1 root root 88059 Feb 26 15:58 lttng-probe-module.ko
>>>> -rw-r--r-- 1 root root 154032 Feb 26 15:58 lttng-probe-napi.ko
>>>> -rw-r--r-- 1 root root 142038 Feb 26 15:58 lttng-probe-net.ko
>>>> -rw-r--r-- 1 root root 93564 Feb 26 15:58 lttng-probe-power.ko
>>>> -rw-r--r-- 1 root root 85530 Feb 26 15:58 lttng-probe-regulator.ko
>>>> -rw-r--r-- 1 root root 138121 Feb 26 15:58 lttng-probe-sched.ko
>>>> -rw-r--r-- 1 root root 152909 Feb 26 15:58 lttng-probe-scsi.ko
>>>> -rw-r--r-- 1 root root 107491 Feb 26 15:58 lttng-probe-signal.ko
>>>> -rw-r--r-- 1 root root 142711 Feb 26 15:58 lttng-probe-skb.ko
>>>> -rw-r--r-- 1 root root 181983 Feb 26 15:58 lttng-probe-statedump.ko
>>>> -rw-r--r-- 1 root root 126731 Feb 26 15:58 lttng-probe-timer.ko
>>>> -rw-r--r-- 1 root root 86591 Feb 26 15:58 lttng-probe-workqueue.ko
>>>> -rw-r--r-- 1 root root 133886 Feb 26 15:58 lttng-probe-writeback.ko
>>>> -rw-r--r-- 1 root root 30863 Feb 26 15:58 lttng-types.ko
>>>>
>>>> Looking at the files shows I miss:
>>>> lttng-kretprobes.ko
>>>> lttng-kprobes.ko
>>>> lttng-ftrace.ko
>>>> lttng-probe-kvm.ko
>>>>
>>>>
>>>> On Tue, Feb 26, 2013 at 3:56 PM, PLSTC <b0mb00z.it@gmail.com> wrote:
>>>>
>>>>> Back on the last email:
>>>>>
>>>>> Here's the insmod list I ended up building; you might not have
>>>>> compiled all those modules, depending on your kernel config and on the
>>>>> architecture used, so some might simply be absent from your output
>>>>> directory.
>>>>>
>>>>> insmod /system/lib/modules/lttng-types.ko
>>>>> insmod /system/lib/modules/lttng-kretprobes.ko
>>>>> insmod /system/lib/modules/lttng-kprobes.ko
>>>>> insmod /system/lib/modules/lttng-ftrace.ko
>>>>> insmod /system/lib/modules/lttng-lib-ring-buffer.ko
>>>>> insmod /system/lib/modules/lttng-statedump.ko
>>>>> insmod /system/lib/modules/lttng-tracer.ko
>>>>> insmod /system/lib/modules/lttng-probe-timer.ko
>>>>> insmod /system/lib/modules/lttng-probe-statedump.ko
>>>>> insmod /system/lib/modules/lttng-probe-signal.ko
>>>>> insmod /system/lib/modules/lttng-probe-sched.ko
>>>>> insmod /system/lib/modules/lttng-probe-lttng.ko
>>>>> insmod /system/lib/modules/lttng-probe-kvm.ko
>>>>> insmod /system/lib/modules/lttng-probe-irq.ko
>>>>> insmod /system/lib/modules/lttng-probe-block.ko
>>>>> insmod /system/lib/modules/lttng-ring-buffer-metadata-mmap-client.ko
>>>>> insmod /system/lib/modules/lttng-ring-buffer-metadata-client.ko
>>>>> insmod /system/lib/modules/lttng-ring-buffer-client-overwrite.ko
>>>>> insmod /system/lib/modules/lttng-ring-buffer-client-mmap-overwrite.ko
>>>>> insmod /system/lib/modules/lttng-ring-buffer-client-mmap-discard.ko
>>>>> insmod /system/lib/modules/lttng-ring-buffer-client-discard.ko
>>>>>
>>>>>
>>>>> We also haven't had time recently to test if the kprobes/kretprobes
>>>>> worked, but they probably do; we decided to focus a bit more on the
>>>>> 'tracepoints' aspect instead.
>>>>>
>>>>> Anyway, good luck with your exploration, and if you find something
>>>>> interesting, I'd love to know more about it!
>>>>>
>>>>> -PL
>>>>>
>>>>> On Tue, Feb 26, 2013 at 8:06 AM, PLSTC <b0mb00z.it@gmail.com> wrote:
>>>>>
>>>>>> Hey Amit,
>>>>>>
>>>>>> Good news indeed it seems!
>>>>>>
>>>>>> When it comes to pushing the libs/bin, we haven't found any better
>>>>>> way than using adb push yet; of course, if all the projects were included
>>>>>> directly in Android's external, we would simply flash the device after each
>>>>>> build, but currently we're still having trouble generating the correct
>>>>>> Android makefiles.
>>>>>>
>>>>>> As for the modules, I might still have a dep load list somewhere I
>>>>>> could send you in a few hours, I remember fighting to get the order right...
>>>>>>
>>>>>> Also, once everything loads, could you tell me what kind of output
>>>>>> 'lttng list -k' provides you? Here, we're having trouble listing the
>>>>>> 'default' kernel trace points provided by HAVE_TRACEPOINTS, and the
>>>>>> returned list is simply empty (which is a bit strange).
>>>>>>
>>>>>> -PL
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
[-- Attachment #1.2: Type: text/html, Size: 37263 bytes --]
[-- Attachment #2: Type: text/plain, Size: 155 bytes --]
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Fwd: compiling LTTng for android
[not found] ` <CA+umk=pgM9BVebkXj2hrgi_LDBdwjm2ZGpS=PmT9DapPJ5kD0A@mail.gmail.com>
@ 2013-03-15 14:33 ` Pierre-Luc St-Charles
[not found] ` <CA+umk=rVfxiSdSAkUCy2yDKbqdgtWW9zJ8O=ivSgOe828br+oQ@mail.gmail.com>
1 sibling, 0 replies; 8+ messages in thread
From: Pierre-Luc St-Charles @ 2013-03-15 14:33 UTC (permalink / raw)
To: Amit Balboul, michel.dagenais, yannick.brosseau; +Cc: lttng-dev
[-- Attachment #1.1: Type: text/plain, Size: 23246 bytes --]
Good news!
After peeking yesterday through the different glibc/SysV implementations
(wrappers...) of splice I could find, I realized that those are not
actually necessary for what we intend to do: from what I found out, they
only redefine/wrap/redeclare the sys_splice system call that's implemented
directly in the kernel (in any architecture). Considering we don't need
anything more than the default (kernel-space) splice, we can simply rewire
the 'splice' declaration we use/have to syscall(__NR_splice,...), just like
what was needed for epoll_create1 earlier.
For example : (doing this from memory... I don't have access to the code
atm)
> in src/common/compat/fcntl.h, inside the linux ifdef, top of the file:
#ifdef __ANDROID__
#define SPLICE_F_MOVE 0x01
#define SPLICE_F_NONBLOCK 0x02
#define SPLICE_F_MORE 0x04
#define SPLICE_F_GIFT 0x08
static inline ssize_t splice(int fd_in, loff_t *off_in, int fd_out,
loff_t *off_out,
size_t len, unsigned int flags)
{
return syscall(__NR_splice,fd_in,loff_in,fd_out,loff_out,len,flags);
}
#endif
Result? Boom, traces.
The rest of the project seems to work without any other visible problem,
and we can finally obtain traces when asking the tools nicely. In fact, we
can obtain traces from any device without even having to modify the
kernel/system image at all (except to add module and tracepoints support,
which should already be there), which is pretty surprising, but also very
useful, considering we can just ship the required binaries anywhere and
start tracing.
Once we confirm that the traces are actually good and we haven't broken
anything else (we're still not sure liburcu works 100%, we're trying to run
all the tests), we'll start sending in results for anyone who might be
interested.
- PL
On Tue, Mar 12, 2013 at 6:51 PM, Pierre-Luc St-Charles <
pierre-luc.st-charles@polymtl.ca> wrote:
> Hey Amit,
>
> Here's a small update on my latest findings regarding the broken *splice*
> / bad Bionic impl problem:
>
> Although it didn't really cross my mind earlier, we had seen this exact
> problem before, but in another project; we had also investigated a bit at
> the time, but to no avail. It seems the issue is known by the
> Android/Bionic community, but it's simply not on their immediate todo list
> (and hasn't been for at least the past two years -- see
> https://code.google.com/p/android/issues/detail?id=11330). The solutions
> we are now considering are: adding splice support (and whatever else might
> also be needed) to Bionic ourselves (saving BILLIONS of broken lives in the
> process, IF the changes are accepted), or simply inserting glibc (and
> building against it) in our device images.
>
> In the next few days, we'll be studying how much time the first solution
> would actually take (as we're not really experts in that domain at all),
> but considering the nature/objective of this project, it might be the
> wisest decision (as it would not require too much swerving away from the
> default images people have on their devices for LTTng to actually work). If
> we deem it 'impossible/out of our league', we'll simply try to get a
> working tool asap using glibc, which should not be very complicated itself
> (many people have actually successfully done it before).
>
> In any case, I'll keep you updated, and I'll also post any big updates on
> the mailing list.
>
> -PL
>
> On Wed, Mar 6, 2013 at 8:04 PM, Pierre-Luc St-Charles <
> pierre-luc.st-charles@polymtl.ca> wrote:
>
>> Great findings!
>>
>> I'll also try to look into this new problem by next monday; I'm falling a
>> bit behind on my 'investigations', but I'll keep you updated if anything
>> comes up.
>>
>> -PL
>> On Mar 6, 2013 6:09 PM, "Amit Balboul" <amit.balboul@shinesec.com> wrote:
>>
>>> Hi All,
>>>
>>> Regarding the last mail about tracing system calls on Android using
>>> LTTng,
>>> The issue of not receiving enabled event to trace from the kernel can be
>>> fixed patching src/bin/lttng-sessiond/kernel.c (patch attached)
>>> In the function *kernel_list_events,* fscanf-ing the events exported by
>>> the kernel using the fp must not use the syntax *%m* as it is obsolete
>>> (in Android),
>>> thus the fscanf returns 0 and won't read the kernel events (first
>>> condition false).
>>> Instead I use a pre-allocated buffer for the string and use the *%[^;]*syntax, and omitting the memory free (attached is the patch).
>>>
>>> The resulting output is (sessiond is run with verbose debug prints):
>>>
>>> root@android:/data/lttng/bin # *./lttng list -k*
>>> DEBUG1 [10239/10290]: Wait for client response (in
>>> thread_manage_clients() at main.c:3242)
>>> DEBUG1 [10239/10290]: Receiving data from client ... (in
>>> thread_manage_clients() at main.c:3287)
>>> DEBUG1 [10239/10290]: Nothing recv() from client... continuing (in
>>> thread_manage_clients() at main.c:3291)
>>> DEBUG1 [10239/10290]: Clean command context structure (in
>>> clean_command_ctx() at main.c:482)
>>> DEBUG1 [10239/10290]: Accepting client command ... (in
>>> thread_manage_clients() at main.c:3200)
>>> DEBUG1 [10239/10290]: Wait for client response (in
>>> thread_manage_clients() at main.c:3242)
>>> DEBUG1 [10239/10290]: Receiving data from client ... (in
>>> thread_manage_clients() at main.c:3287)
>>> DEBUG1 [10239/10290]: Processing client command 14 (in
>>> process_client_msg() at main.c:2227)
>>> DEBUG1 [10239/10290]: Kernel list events done (61 events) (in
>>> kernel_list_events() at kernel.c:653)
>>> DEBUG1 [10239/10290]: Sending response (size: 35640, retcode: Success)
>>> (in thread_manage_clients() at main.c:3338)
>>> DEBUG1 [10239/10290]: Clean command context structure (in
>>> clean_command_ctx() at main.c:482)
>>> DEBUG1 [10239/10290]: Accepting client command ... (in
>>> thread_manage_clients() at main.c:3200)
>>> Kernel events:
>>> -------------
>>> block_rq_abort (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> block_rq_requeue (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> block_rq_complete (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> block_rq_insert (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> block_rq_issue (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> block_bio_bounce (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> block_bio_complete (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> block_bio_backmerge (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> block_bio_frontmerge (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> block_bio_queue (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> block_getrq (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> block_sleeprq (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> block_plug (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> block_unplug (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> block_split (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> block_bio_remap (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> block_rq_remap (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> irq_handler_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> irq_handler_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> softirq_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> softirq_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> softirq_raise (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> sched_kthread_stop (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> sched_kthread_stop_ret (loglevel: TRACE_EMERG (0)) (type:
>>> tracepoint)
>>> sched_wakeup (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> sched_wakeup_new (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> sched_switch (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> sched_migrate_task (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> sched_process_free (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> sched_process_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> sched_wait_task (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> sched_process_wait (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> sched_process_fork (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> sched_stat_wait (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> sched_stat_sleep (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> sched_stat_iowait (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> sched_stat_runtime (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> sched_pi_setprio (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> signal_generate (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> signal_deliver (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> signal_overflow_fail (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> signal_lose_info (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> lttng_statedump_start (loglevel: TRACE_EMERG (0)) (type:
>>> tracepoint)
>>> lttng_statedump_end (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> lttng_statedump_process_state (loglevel: TRACE_EMERG (0)) (type:
>>> tracepoint)
>>> lttng_statedump_file_descriptor (loglevel: TRACE_EMERG (0)) (type:
>>> tracepoint)
>>> lttng_statedump_vm_map (loglevel: TRACE_EMERG (0)) (type:
>>> tracepoint)
>>> lttng_statedump_network_interface (loglevel: TRACE_EMERG (0))
>>> (type: tracepoint)
>>> lttng_statedump_interrupt (loglevel: TRACE_EMERG (0)) (type:
>>> tracepoint)
>>> timer_init (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> timer_start (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> timer_expire_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> timer_expire_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> timer_cancel (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> hrtimer_init (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> hrtimer_start (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> hrtimer_expire_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> hrtimer_expire_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> hrtimer_cancel (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> itimer_state (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>> itimer_expire (loglevel: TRACE_EMERG (0)) (type: tracepoint)
>>>
>>> root@android:/data/lttng/bin #
>>>
>>> Now I'm facing the issue of starting the traces:
>>> debugging the sessiond, consumerd and lttng I concluded that the issue
>>> is due to splice system call failure.
>>> even using a MMAP output type channel, the metadata channel is SPLICE
>>> type (hard-coded).
>>> Problem persists also when using relayd to pass the traces through the
>>> network.
>>> Ignoring the splice failure means closing of the metadata channel
>>> (maybe??) but need not affect the traces channel (right ?)
>>> Attached is a debug log of each thread of the sessiond, right after
>>> invoking "*./lttng start*". (with extra informative my prints)
>>>
>>> Please keep me in touch with any progress in this issue,
>>> Thank you,
>>> Amit.
>>>
>>>
>>>
>>>
>>>
>>>
>>> ---------- Forwarded message ----------
>>> From: Amit Balboul <amit.balboul@shinesec.com>
>>> Date: Wed, Feb 27, 2013 at 4:37 PM
>>> Subject: Re: [lttng-dev] compiling LTTng for android
>>> To: PLSTC <b0mb00z.it@gmail.com>
>>>
>>>
>>> Hey Pierre,
>>>
>>> Some major progress has been achieved, following is the summary:
>>>
>>> 1. Modules issue:
>>> I built (for ARM/Android) and installed latest stable version of busybox
>>> on the device.
>>> Seeing the code tries to run (hardcoded) /sbin/modprobe, made a soft
>>> link to busybox's modeprobe.
>>> Also, created /lib/modules/<version+build> directory and pushed the
>>> <version+build> directory from /lib/modules on the machine.
>>> Now session daemon modprobes the desired modules successfully, (some are
>>> not present though as mentioned before) - no need to use insmod.
>>>
>>> 2. epoll_create1 issue:
>>> epoll_create1 patch was OK, I accidentally defined EPOLL_CLOEXEC as
>>> 0x02000000 instead of 02000000 (fixed).
>>>
>>> The running scenario:
>>>
>>> 1. Run ./lttng-sessiond -d -vvv
>>>
>>> Then: (commands are in bold, output is in blue for readability)
>>>
>>> root@android:/data/lttng/install_sysroot/bin # *./lttng create s1*
>>>
>>> Session s1 created.
>>> Traces will be written in /data/lttng-traces/s1-20130227-161145
>>> root@android:/data/lttng/install_sysroot/bin # *./lttng enable-channel
>>> ch -k*
>>> Kernel channel ch enabled for session s1
>>> root@android:/data/lttng/install_sysroot/bin # *./lttng enable-event ev
>>> -k -a*
>>> All kernel events are enabled in channel channel0
>>> root@android:/data/lttng/install_sysroot/bin # *./lttng list s1*
>>> Tracing session s1: [inactive]
>>> Trace path: /data/lttng-traces/s1-20130227-161145
>>>
>>> === Domain: Kernel ===
>>>
>>> Channels:
>>> -------------
>>> - channel0: [enabled]
>>>
>>> Attributes:
>>> overwrite mode: 0
>>> subbufers size: 262144
>>> number of subbufers: 4
>>> switch timer interval: 0
>>> read timer interval: 200
>>> output: splice()
>>>
>>> Events:
>>> None
>>>
>>> - ch: [enabled]
>>>
>>> Attributes:
>>> overwrite mode: 0
>>> subbufers size: 262144
>>> number of subbufers: 4
>>> switch timer interval: 0
>>> read timer interval: 200
>>> output: splice()
>>>
>>> Events:
>>> None
>>>
>>> root@android:/data/lttng/install_sysroot/bin # *./lttng list -k *
>>>
>>> Kernel events:
>>> -------------
>>>
>>> root@android:/data/lttng/install_sysroot/bin # *./lttng start*
>>> Tracing started for session s1
>>> root@android:/data/lttng/install_sysroot/bin # *./lttng stop *
>>>
>>> Waiting for data availability
>>> Tracing stopped for session s1
>>> root@android:/data/lttng/install_sysroot/bin #
>>>
>>> *Now: (this is not surprising because the kernel events list is
>>> empty...)*
>>> the folder /data/lttng-traces/s1-20130227-161145/kernel contains:
>>> total 0
>>> -rwxrwxrwx 1 0 0 0 Feb 27 14:18 ch_0
>>> -rwxrwxrwx 1 0 0 0 Feb 27 14:18 ch_1
>>> -rwxrwxrwx 1 0 0 0 Feb 27 14:18 channel0_0
>>> -rwxrwxrwx 1 0 0 0 Feb 27 14:18 channel0_1
>>> -rwxrwxrwx 1 0 0 0 Feb 27 14:18 metadata
>>>
>>> *By the way:*
>>> When running the same scenario but creating the channel with *--output
>>> mmap*, still no data but the size of the first two files in the traces
>>> directory are 4Kb (I didn't bother to investigate them as I assume it is
>>> just the header - not the traces payload).
>>>
>>> Another issue: at this point, listing the sessions gives:
>>> root@android:/data/lttng/install_sysroot/bin # *./lttng list s1 *
>>>
>>> Tracing session s1: [inactive]
>>> Trace path: /data/lttng-traces/s1-20130227-161145
>>>
>>> === Domain: Kernel ===
>>>
>>> Error: No kernel consumer detected
>>> 99|root@android:/data/lttng/install_sysroot/bin # *./lttng list -k *
>>>
>>> Error: Unable to list kernel events
>>> Error: Command error
>>> 1|root@android:/data/lttng/install_sysroot/bin #
>>>
>>> So currently I'm facing these issues:
>>> 1. Kernel events list is empty.
>>> 2. Listing the session/ kernel events after creating session, channel,
>>> event, start, stop - results in "no kernel consumer detected"
>>>
>>> If you have some insights please share them with me..
>>>
>>> Thanks
>>>
>>> Amit.
>>>
>>>
>>> On Tue, Feb 26, 2013 at 7:35 PM, PLSTC <b0mb00z.it@gmail.com> wrote:
>>>
>>>> The modules you can't find are the same I am also missing I believe; I
>>>> think they are only compiled when certain kernel config defines are met
>>>> (and those might simply be unavailable under ARM).
>>>>
>>>> As for what you pointed in your first mail, I believe modprobe isn't
>>>> available on ARM, did you mean that some modules were not found when using
>>>> insmod?
>>>>
>>>> For the epoll_create1 problem, what kind of patch did you apply to add
>>>> support for the epoll_create1 function?
>>>>
>>>>
>>>> On Tue, Feb 26, 2013 at 11:29 AM, Amit Balboul <
>>>> amit.balboul@shinesec.com> wrote:
>>>>
>>>>> Ok I got :
>>>>> lttng-kretprobes.ko
>>>>> lttng-kprobes.ko
>>>>> , the other two are missing.
>>>>>
>>>>>
>>>>> ---------- Forwarded message ----------
>>>>> From: Amit Balboul <amit.balboul@shinesec.com>
>>>>> Date: Tue, Feb 26, 2013 at 5:49 PM
>>>>> Subject: Re: [lttng-dev] compiling LTTng for android
>>>>> To: PLSTC <b0mb00z.it@gmail.com>
>>>>>
>>>>>
>>>>> One more thing:
>>>>>
>>>>> The list of modules you gave my is not identical to mine:
>>>>> a. I pushed the modules from my machine to my folder in the device
>>>>> (not to /system/lib) as you directed me, and insmoded them manually.
>>>>> b. I pushed the whole tree under /lib/modules/<KERNEL_VER_BUILD>/
>>>>> which includes the modules.* and the sub directories:
>>>>>
>>>>> /lib/modules/3.0.31-gd5a18e0/extra$ ll
>>>>> total 2464
>>>>> drwxr-xr-x 4 root root 4096 Feb 26 15:58 ./
>>>>> drwxr-xr-x 3 root root 4096 Feb 26 15:58 ../
>>>>> drwxr-xr-x 2 root root 4096 Feb 26 15:58 lib/
>>>>> -rw-r--r-- 1 root root 167184 Feb 26 15:58
>>>>> lttng-ring-buffer-client-discard.ko
>>>>> -rw-r--r-- 1 root root 167206 Feb 26 15:58
>>>>> lttng-ring-buffer-client-mmap-discard.ko
>>>>> -rw-r--r-- 1 root root 178586 Feb 26 15:58
>>>>> lttng-ring-buffer-client-mmap-overwrite.ko
>>>>> -rw-r--r-- 1 root root 178564 Feb 26 15:58
>>>>> lttng-ring-buffer-client-overwrite.ko
>>>>> -rw-r--r-- 1 root root 137179 Feb 26 15:58
>>>>> lttng-ring-buffer-metadata-client.ko
>>>>> -rw-r--r-- 1 root root 137233 Feb 26 15:58
>>>>> lttng-ring-buffer-metadata-mmap-client.ko
>>>>> -rw-r--r-- 1 root root 213451 Feb 26 15:58 lttng-statedump.ko
>>>>> -rw-r--r-- 1 root root 1314235 Feb 26 15:58 lttng-tracer.ko
>>>>> drwxr-xr-x 2 root root 4096 Feb 26 15:58 probes/
>>>>>
>>>>> /lib/modules/3.0.31-gd5a18e0/extra/lib$ ll
>>>>> total 732
>>>>> drwxr-xr-x 2 root root 4096 Feb 26 15:58 ./
>>>>> drwxr-xr-x 4 root root 4096 Feb 26 15:58 ../
>>>>> -rw-r--r-- 1 root root 738964 Feb 26 15:58 lttng-lib-ring-buffer.ko
>>>>>
>>>>> /lib/modules/3.0.31-gd5a18e0/extra/probes$ ll
>>>>> total 2872
>>>>> drwxr-xr-x 2 root root 4096 Feb 26 15:58 ./
>>>>> drwxr-xr-x 4 root root 4096 Feb 26 15:58 ../
>>>>> -rw-r--r-- 1 root root 162872 Feb 26 15:58 lttng-probe-asoc.ko
>>>>> -rw-r--r-- 1 root root 171049 Feb 26 15:58 lttng-probe-block.ko
>>>>> -rw-r--r-- 1 root root 81165 Feb 26 15:58 lttng-probe-compaction.ko
>>>>> -rw-r--r-- 1 root root 333215 Feb 26 15:58 lttng-probe-ext4.ko
>>>>> -rw-r--r-- 1 root root 80413 Feb 26 15:58 lttng-probe-gpio.ko
>>>>> -rw-r--r-- 1 root root 84256 Feb 26 15:58 lttng-probe-irq.ko
>>>>> -rw-r--r-- 1 root root 135453 Feb 26 15:58 lttng-probe-jbd2.ko
>>>>> -rw-r--r-- 1 root root 105399 Feb 26 15:58 lttng-probe-kmem.ko
>>>>> -rw-r--r-- 1 root root 76240 Feb 26 15:58 lttng-probe-lttng.ko
>>>>> -rw-r--r-- 1 root root 88059 Feb 26 15:58 lttng-probe-module.ko
>>>>> -rw-r--r-- 1 root root 154032 Feb 26 15:58 lttng-probe-napi.ko
>>>>> -rw-r--r-- 1 root root 142038 Feb 26 15:58 lttng-probe-net.ko
>>>>> -rw-r--r-- 1 root root 93564 Feb 26 15:58 lttng-probe-power.ko
>>>>> -rw-r--r-- 1 root root 85530 Feb 26 15:58 lttng-probe-regulator.ko
>>>>> -rw-r--r-- 1 root root 138121 Feb 26 15:58 lttng-probe-sched.ko
>>>>> -rw-r--r-- 1 root root 152909 Feb 26 15:58 lttng-probe-scsi.ko
>>>>> -rw-r--r-- 1 root root 107491 Feb 26 15:58 lttng-probe-signal.ko
>>>>> -rw-r--r-- 1 root root 142711 Feb 26 15:58 lttng-probe-skb.ko
>>>>> -rw-r--r-- 1 root root 181983 Feb 26 15:58 lttng-probe-statedump.ko
>>>>> -rw-r--r-- 1 root root 126731 Feb 26 15:58 lttng-probe-timer.ko
>>>>> -rw-r--r-- 1 root root 86591 Feb 26 15:58 lttng-probe-workqueue.ko
>>>>> -rw-r--r-- 1 root root 133886 Feb 26 15:58 lttng-probe-writeback.ko
>>>>> -rw-r--r-- 1 root root 30863 Feb 26 15:58 lttng-types.ko
>>>>>
>>>>> Looking at the files shows I miss:
>>>>> lttng-kretprobes.ko
>>>>> lttng-kprobes.ko
>>>>> lttng-ftrace.ko
>>>>> lttng-probe-kvm.ko
>>>>>
>>>>>
>>>>> On Tue, Feb 26, 2013 at 3:56 PM, PLSTC <b0mb00z.it@gmail.com> wrote:
>>>>>
>>>>>> Back on the last email:
>>>>>>
>>>>>> Here's the insmod list I ended up building; you might not have
>>>>>> compiled all those modules, depending on your kernel config and on the
>>>>>> architecture used, so some might simply be absent from your output
>>>>>> directory.
>>>>>>
>>>>>> insmod /system/lib/modules/lttng-types.ko
>>>>>> insmod /system/lib/modules/lttng-kretprobes.ko
>>>>>> insmod /system/lib/modules/lttng-kprobes.ko
>>>>>> insmod /system/lib/modules/lttng-ftrace.ko
>>>>>> insmod /system/lib/modules/lttng-lib-ring-buffer.ko
>>>>>> insmod /system/lib/modules/lttng-statedump.ko
>>>>>> insmod /system/lib/modules/lttng-tracer.ko
>>>>>> insmod /system/lib/modules/lttng-probe-timer.ko
>>>>>> insmod /system/lib/modules/lttng-probe-statedump.ko
>>>>>> insmod /system/lib/modules/lttng-probe-signal.ko
>>>>>> insmod /system/lib/modules/lttng-probe-sched.ko
>>>>>> insmod /system/lib/modules/lttng-probe-lttng.ko
>>>>>> insmod /system/lib/modules/lttng-probe-kvm.ko
>>>>>> insmod /system/lib/modules/lttng-probe-irq.ko
>>>>>> insmod /system/lib/modules/lttng-probe-block.ko
>>>>>> insmod /system/lib/modules/lttng-ring-buffer-metadata-mmap-client.ko
>>>>>> insmod /system/lib/modules/lttng-ring-buffer-metadata-client.ko
>>>>>> insmod /system/lib/modules/lttng-ring-buffer-client-overwrite.ko
>>>>>> insmod /system/lib/modules/lttng-ring-buffer-client-mmap-overwrite.ko
>>>>>> insmod /system/lib/modules/lttng-ring-buffer-client-mmap-discard.ko
>>>>>> insmod /system/lib/modules/lttng-ring-buffer-client-discard.ko
>>>>>>
>>>>>>
>>>>>> We also haven't had time recently to test if the kprobes/kretprobes
>>>>>> worked, but they probably do; we decided to focus a bit more on the
>>>>>> 'tracepoints' aspect instead.
>>>>>>
>>>>>> Anyway, good luck with your exploration, and if you find something
>>>>>> interesting, I'd love to know more about it!
>>>>>>
>>>>>> -PL
>>>>>>
>>>>>> On Tue, Feb 26, 2013 at 8:06 AM, PLSTC <b0mb00z.it@gmail.com> wrote:
>>>>>>
>>>>>>> Hey Amit,
>>>>>>>
>>>>>>> Good news indeed it seems!
>>>>>>>
>>>>>>> When it comes to pushing the libs/bin, we haven't found any better
>>>>>>> way than using adb push yet; of course, if all the projects were included
>>>>>>> directly in Android's external, we would simply flash the device after each
>>>>>>> build, but currently we're still having trouble generating the correct
>>>>>>> Android makefiles.
>>>>>>>
>>>>>>> As for the modules, I might still have a dep load list somewhere I
>>>>>>> could send you in a few hours, I remember fighting to get the order right...
>>>>>>>
>>>>>>> Also, once everything loads, could you tell me what kind of output
>>>>>>> 'lttng list -k' provides you? Here, we're having trouble listing the
>>>>>>> 'default' kernel trace points provided by HAVE_TRACEPOINTS, and the
>>>>>>> returned list is simply empty (which is a bit strange).
>>>>>>>
>>>>>>> -PL
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>
[-- Attachment #1.2: Type: text/html, Size: 40153 bytes --]
[-- Attachment #2: Type: text/plain, Size: 155 bytes --]
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Fwd: compiling LTTng for android
[not found] ` <CA+umk=rVfxiSdSAkUCy2yDKbqdgtWW9zJ8O=ivSgOe828br+oQ@mail.gmail.com>
@ 2013-03-15 15:08 ` Mathieu Desnoyers
[not found] ` <20130315150806.GA1467@Krystal>
1 sibling, 0 replies; 8+ messages in thread
From: Mathieu Desnoyers @ 2013-03-15 15:08 UTC (permalink / raw)
To: Pierre-Luc St-Charles; +Cc: Amit Balboul, lttng-dev, yannick.brosseau
* Pierre-Luc St-Charles (pierre-luc.st-charles@polymtl.ca) wrote:
> Good news!
>
> After peeking yesterday through the different glibc/SysV implementations
> (wrappers...) of splice I could find, I realized that those are not
> actually necessary for what we intend to do: from what I found out, they
> only redefine/wrap/redeclare the sys_splice system call that's implemented
> directly in the kernel (in any architecture). Considering we don't need
> anything more than the default (kernel-space) splice, we can simply rewire
> the 'splice' declaration we use/have to syscall(__NR_splice,...), just like
> what was needed for epoll_create1 earlier.
>
> For example : (doing this from memory... I don't have access to the code
> atm)
> > in src/common/compat/fcntl.h, inside the linux ifdef, top of the file:
>
> #ifdef __ANDROID__
> #define SPLICE_F_MOVE 0x01
> #define SPLICE_F_NONBLOCK 0x02
> #define SPLICE_F_MORE 0x04
> #define SPLICE_F_GIFT 0x08
> static inline ssize_t splice(int fd_in, loff_t *off_in, int fd_out,
> loff_t *off_out,
> size_t len, unsigned int flags)
> {
> return syscall(__NR_splice,fd_in,loff_in,fd_out,loff_out,len,flags);
> }
> #endif
>
>
> Result? Boom, traces.
>
> The rest of the project seems to work without any other visible problem,
> and we can finally obtain traces when asking the tools nicely. In fact, we
> can obtain traces from any device without even having to modify the
> kernel/system image at all (except to add module and tracepoints support,
> which should already be there), which is pretty surprising, but also very
> useful, considering we can just ship the required binaries anywhere and
> start tracing.
>
> Once we confirm that the traces are actually good and we haven't broken
> anything else (we're still not sure liburcu works 100%, we're trying to run
> all the tests), we'll start sending in results for anyone who might be
> interested.
That's great news! So I understand that you got lttng-modules Linux
kernel tracing to work on Android. I look forward to see patches we
could merge into lttng-tools.
I guess the next step will be to look into LTTng-UST user-space tracing.
Thanks,
Mathieu
>
> - PL
>
> On Tue, Mar 12, 2013 at 6:51 PM, Pierre-Luc St-Charles <
> pierre-luc.st-charles@polymtl.ca> wrote:
>
> > Hey Amit,
> >
> > Here's a small update on my latest findings regarding the broken *splice*
> > / bad Bionic impl problem:
> >
> > Although it didn't really cross my mind earlier, we had seen this exact
> > problem before, but in another project; we had also investigated a bit at
> > the time, but to no avail. It seems the issue is known by the
> > Android/Bionic community, but it's simply not on their immediate todo list
> > (and hasn't been for at least the past two years -- see
> > https://code.google.com/p/android/issues/detail?id=11330). The solutions
> > we are now considering are: adding splice support (and whatever else might
> > also be needed) to Bionic ourselves (saving BILLIONS of broken lives in the
> > process, IF the changes are accepted), or simply inserting glibc (and
> > building against it) in our device images.
> >
> > In the next few days, we'll be studying how much time the first solution
> > would actually take (as we're not really experts in that domain at all),
> > but considering the nature/objective of this project, it might be the
> > wisest decision (as it would not require too much swerving away from the
> > default images people have on their devices for LTTng to actually work). If
> > we deem it 'impossible/out of our league', we'll simply try to get a
> > working tool asap using glibc, which should not be very complicated itself
> > (many people have actually successfully done it before).
> >
> > In any case, I'll keep you updated, and I'll also post any big updates on
> > the mailing list.
> >
> > -PL
> >
> > On Wed, Mar 6, 2013 at 8:04 PM, Pierre-Luc St-Charles <
> > pierre-luc.st-charles@polymtl.ca> wrote:
> >
> >> Great findings!
> >>
> >> I'll also try to look into this new problem by next monday; I'm falling a
> >> bit behind on my 'investigations', but I'll keep you updated if anything
> >> comes up.
> >>
> >> -PL
> >> On Mar 6, 2013 6:09 PM, "Amit Balboul" <amit.balboul@shinesec.com> wrote:
> >>
> >>> Hi All,
> >>>
> >>> Regarding the last mail about tracing system calls on Android using
> >>> LTTng,
> >>> The issue of not receiving enabled event to trace from the kernel can be
> >>> fixed patching src/bin/lttng-sessiond/kernel.c (patch attached)
> >>> In the function *kernel_list_events,* fscanf-ing the events exported by
> >>> the kernel using the fp must not use the syntax *%m* as it is obsolete
> >>> (in Android),
> >>> thus the fscanf returns 0 and won't read the kernel events (first
> >>> condition false).
> >>> Instead I use a pre-allocated buffer for the string and use the *%[^;]*syntax, and omitting the memory free (attached is the patch).
> >>>
> >>> The resulting output is (sessiond is run with verbose debug prints):
> >>>
> >>> root@android:/data/lttng/bin # *./lttng list -k*
> >>> DEBUG1 [10239/10290]: Wait for client response (in
> >>> thread_manage_clients() at main.c:3242)
> >>> DEBUG1 [10239/10290]: Receiving data from client ... (in
> >>> thread_manage_clients() at main.c:3287)
> >>> DEBUG1 [10239/10290]: Nothing recv() from client... continuing (in
> >>> thread_manage_clients() at main.c:3291)
> >>> DEBUG1 [10239/10290]: Clean command context structure (in
> >>> clean_command_ctx() at main.c:482)
> >>> DEBUG1 [10239/10290]: Accepting client command ... (in
> >>> thread_manage_clients() at main.c:3200)
> >>> DEBUG1 [10239/10290]: Wait for client response (in
> >>> thread_manage_clients() at main.c:3242)
> >>> DEBUG1 [10239/10290]: Receiving data from client ... (in
> >>> thread_manage_clients() at main.c:3287)
> >>> DEBUG1 [10239/10290]: Processing client command 14 (in
> >>> process_client_msg() at main.c:2227)
> >>> DEBUG1 [10239/10290]: Kernel list events done (61 events) (in
> >>> kernel_list_events() at kernel.c:653)
> >>> DEBUG1 [10239/10290]: Sending response (size: 35640, retcode: Success)
> >>> (in thread_manage_clients() at main.c:3338)
> >>> DEBUG1 [10239/10290]: Clean command context structure (in
> >>> clean_command_ctx() at main.c:482)
> >>> DEBUG1 [10239/10290]: Accepting client command ... (in
> >>> thread_manage_clients() at main.c:3200)
> >>> Kernel events:
> >>> -------------
> >>> block_rq_abort (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> block_rq_requeue (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> block_rq_complete (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> block_rq_insert (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> block_rq_issue (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> block_bio_bounce (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> block_bio_complete (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> block_bio_backmerge (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> block_bio_frontmerge (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> block_bio_queue (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> block_getrq (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> block_sleeprq (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> block_plug (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> block_unplug (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> block_split (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> block_bio_remap (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> block_rq_remap (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> irq_handler_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> irq_handler_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> softirq_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> softirq_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> softirq_raise (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> sched_kthread_stop (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> sched_kthread_stop_ret (loglevel: TRACE_EMERG (0)) (type:
> >>> tracepoint)
> >>> sched_wakeup (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> sched_wakeup_new (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> sched_switch (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> sched_migrate_task (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> sched_process_free (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> sched_process_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> sched_wait_task (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> sched_process_wait (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> sched_process_fork (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> sched_stat_wait (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> sched_stat_sleep (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> sched_stat_iowait (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> sched_stat_runtime (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> sched_pi_setprio (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> signal_generate (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> signal_deliver (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> signal_overflow_fail (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> signal_lose_info (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> lttng_statedump_start (loglevel: TRACE_EMERG (0)) (type:
> >>> tracepoint)
> >>> lttng_statedump_end (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> lttng_statedump_process_state (loglevel: TRACE_EMERG (0)) (type:
> >>> tracepoint)
> >>> lttng_statedump_file_descriptor (loglevel: TRACE_EMERG (0)) (type:
> >>> tracepoint)
> >>> lttng_statedump_vm_map (loglevel: TRACE_EMERG (0)) (type:
> >>> tracepoint)
> >>> lttng_statedump_network_interface (loglevel: TRACE_EMERG (0))
> >>> (type: tracepoint)
> >>> lttng_statedump_interrupt (loglevel: TRACE_EMERG (0)) (type:
> >>> tracepoint)
> >>> timer_init (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> timer_start (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> timer_expire_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> timer_expire_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> timer_cancel (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> hrtimer_init (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> hrtimer_start (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> hrtimer_expire_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> hrtimer_expire_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> hrtimer_cancel (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> itimer_state (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>> itimer_expire (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> >>>
> >>> root@android:/data/lttng/bin #
> >>>
> >>> Now I'm facing the issue of starting the traces:
> >>> debugging the sessiond, consumerd and lttng I concluded that the issue
> >>> is due to splice system call failure.
> >>> even using a MMAP output type channel, the metadata channel is SPLICE
> >>> type (hard-coded).
> >>> Problem persists also when using relayd to pass the traces through the
> >>> network.
> >>> Ignoring the splice failure means closing of the metadata channel
> >>> (maybe??) but need not affect the traces channel (right ?)
> >>> Attached is a debug log of each thread of the sessiond, right after
> >>> invoking "*./lttng start*". (with extra informative my prints)
> >>>
> >>> Please keep me in touch with any progress in this issue,
> >>> Thank you,
> >>> Amit.
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> ---------- Forwarded message ----------
> >>> From: Amit Balboul <amit.balboul@shinesec.com>
> >>> Date: Wed, Feb 27, 2013 at 4:37 PM
> >>> Subject: Re: [lttng-dev] compiling LTTng for android
> >>> To: PLSTC <b0mb00z.it@gmail.com>
> >>>
> >>>
> >>> Hey Pierre,
> >>>
> >>> Some major progress has been achieved, following is the summary:
> >>>
> >>> 1. Modules issue:
> >>> I built (for ARM/Android) and installed latest stable version of busybox
> >>> on the device.
> >>> Seeing the code tries to run (hardcoded) /sbin/modprobe, made a soft
> >>> link to busybox's modeprobe.
> >>> Also, created /lib/modules/<version+build> directory and pushed the
> >>> <version+build> directory from /lib/modules on the machine.
> >>> Now session daemon modprobes the desired modules successfully, (some are
> >>> not present though as mentioned before) - no need to use insmod.
> >>>
> >>> 2. epoll_create1 issue:
> >>> epoll_create1 patch was OK, I accidentally defined EPOLL_CLOEXEC as
> >>> 0x02000000 instead of 02000000 (fixed).
> >>>
> >>> The running scenario:
> >>>
> >>> 1. Run ./lttng-sessiond -d -vvv
> >>>
> >>> Then: (commands are in bold, output is in blue for readability)
> >>>
> >>> root@android:/data/lttng/install_sysroot/bin # *./lttng create s1*
> >>>
> >>> Session s1 created.
> >>> Traces will be written in /data/lttng-traces/s1-20130227-161145
> >>> root@android:/data/lttng/install_sysroot/bin # *./lttng enable-channel
> >>> ch -k*
> >>> Kernel channel ch enabled for session s1
> >>> root@android:/data/lttng/install_sysroot/bin # *./lttng enable-event ev
> >>> -k -a*
> >>> All kernel events are enabled in channel channel0
> >>> root@android:/data/lttng/install_sysroot/bin # *./lttng list s1*
> >>> Tracing session s1: [inactive]
> >>> Trace path: /data/lttng-traces/s1-20130227-161145
> >>>
> >>> === Domain: Kernel ===
> >>>
> >>> Channels:
> >>> -------------
> >>> - channel0: [enabled]
> >>>
> >>> Attributes:
> >>> overwrite mode: 0
> >>> subbufers size: 262144
> >>> number of subbufers: 4
> >>> switch timer interval: 0
> >>> read timer interval: 200
> >>> output: splice()
> >>>
> >>> Events:
> >>> None
> >>>
> >>> - ch: [enabled]
> >>>
> >>> Attributes:
> >>> overwrite mode: 0
> >>> subbufers size: 262144
> >>> number of subbufers: 4
> >>> switch timer interval: 0
> >>> read timer interval: 200
> >>> output: splice()
> >>>
> >>> Events:
> >>> None
> >>>
> >>> root@android:/data/lttng/install_sysroot/bin # *./lttng list -k *
> >>>
> >>> Kernel events:
> >>> -------------
> >>>
> >>> root@android:/data/lttng/install_sysroot/bin # *./lttng start*
> >>> Tracing started for session s1
> >>> root@android:/data/lttng/install_sysroot/bin # *./lttng stop *
> >>>
> >>> Waiting for data availability
> >>> Tracing stopped for session s1
> >>> root@android:/data/lttng/install_sysroot/bin #
> >>>
> >>> *Now: (this is not surprising because the kernel events list is
> >>> empty...)*
> >>> the folder /data/lttng-traces/s1-20130227-161145/kernel contains:
> >>> total 0
> >>> -rwxrwxrwx 1 0 0 0 Feb 27 14:18 ch_0
> >>> -rwxrwxrwx 1 0 0 0 Feb 27 14:18 ch_1
> >>> -rwxrwxrwx 1 0 0 0 Feb 27 14:18 channel0_0
> >>> -rwxrwxrwx 1 0 0 0 Feb 27 14:18 channel0_1
> >>> -rwxrwxrwx 1 0 0 0 Feb 27 14:18 metadata
> >>>
> >>> *By the way:*
> >>> When running the same scenario but creating the channel with *--output
> >>> mmap*, still no data but the size of the first two files in the traces
> >>> directory are 4Kb (I didn't bother to investigate them as I assume it is
> >>> just the header - not the traces payload).
> >>>
> >>> Another issue: at this point, listing the sessions gives:
> >>> root@android:/data/lttng/install_sysroot/bin # *./lttng list s1 *
> >>>
> >>> Tracing session s1: [inactive]
> >>> Trace path: /data/lttng-traces/s1-20130227-161145
> >>>
> >>> === Domain: Kernel ===
> >>>
> >>> Error: No kernel consumer detected
> >>> 99|root@android:/data/lttng/install_sysroot/bin # *./lttng list -k *
> >>>
> >>> Error: Unable to list kernel events
> >>> Error: Command error
> >>> 1|root@android:/data/lttng/install_sysroot/bin #
> >>>
> >>> So currently I'm facing these issues:
> >>> 1. Kernel events list is empty.
> >>> 2. Listing the session/ kernel events after creating session, channel,
> >>> event, start, stop - results in "no kernel consumer detected"
> >>>
> >>> If you have some insights please share them with me..
> >>>
> >>> Thanks
> >>>
> >>> Amit.
> >>>
> >>>
> >>> On Tue, Feb 26, 2013 at 7:35 PM, PLSTC <b0mb00z.it@gmail.com> wrote:
> >>>
> >>>> The modules you can't find are the same I am also missing I believe; I
> >>>> think they are only compiled when certain kernel config defines are met
> >>>> (and those might simply be unavailable under ARM).
> >>>>
> >>>> As for what you pointed in your first mail, I believe modprobe isn't
> >>>> available on ARM, did you mean that some modules were not found when using
> >>>> insmod?
> >>>>
> >>>> For the epoll_create1 problem, what kind of patch did you apply to add
> >>>> support for the epoll_create1 function?
> >>>>
> >>>>
> >>>> On Tue, Feb 26, 2013 at 11:29 AM, Amit Balboul <
> >>>> amit.balboul@shinesec.com> wrote:
> >>>>
> >>>>> Ok I got :
> >>>>> lttng-kretprobes.ko
> >>>>> lttng-kprobes.ko
> >>>>> , the other two are missing.
> >>>>>
> >>>>>
> >>>>> ---------- Forwarded message ----------
> >>>>> From: Amit Balboul <amit.balboul@shinesec.com>
> >>>>> Date: Tue, Feb 26, 2013 at 5:49 PM
> >>>>> Subject: Re: [lttng-dev] compiling LTTng for android
> >>>>> To: PLSTC <b0mb00z.it@gmail.com>
> >>>>>
> >>>>>
> >>>>> One more thing:
> >>>>>
> >>>>> The list of modules you gave my is not identical to mine:
> >>>>> a. I pushed the modules from my machine to my folder in the device
> >>>>> (not to /system/lib) as you directed me, and insmoded them manually.
> >>>>> b. I pushed the whole tree under /lib/modules/<KERNEL_VER_BUILD>/
> >>>>> which includes the modules.* and the sub directories:
> >>>>>
> >>>>> /lib/modules/3.0.31-gd5a18e0/extra$ ll
> >>>>> total 2464
> >>>>> drwxr-xr-x 4 root root 4096 Feb 26 15:58 ./
> >>>>> drwxr-xr-x 3 root root 4096 Feb 26 15:58 ../
> >>>>> drwxr-xr-x 2 root root 4096 Feb 26 15:58 lib/
> >>>>> -rw-r--r-- 1 root root 167184 Feb 26 15:58
> >>>>> lttng-ring-buffer-client-discard.ko
> >>>>> -rw-r--r-- 1 root root 167206 Feb 26 15:58
> >>>>> lttng-ring-buffer-client-mmap-discard.ko
> >>>>> -rw-r--r-- 1 root root 178586 Feb 26 15:58
> >>>>> lttng-ring-buffer-client-mmap-overwrite.ko
> >>>>> -rw-r--r-- 1 root root 178564 Feb 26 15:58
> >>>>> lttng-ring-buffer-client-overwrite.ko
> >>>>> -rw-r--r-- 1 root root 137179 Feb 26 15:58
> >>>>> lttng-ring-buffer-metadata-client.ko
> >>>>> -rw-r--r-- 1 root root 137233 Feb 26 15:58
> >>>>> lttng-ring-buffer-metadata-mmap-client.ko
> >>>>> -rw-r--r-- 1 root root 213451 Feb 26 15:58 lttng-statedump.ko
> >>>>> -rw-r--r-- 1 root root 1314235 Feb 26 15:58 lttng-tracer.ko
> >>>>> drwxr-xr-x 2 root root 4096 Feb 26 15:58 probes/
> >>>>>
> >>>>> /lib/modules/3.0.31-gd5a18e0/extra/lib$ ll
> >>>>> total 732
> >>>>> drwxr-xr-x 2 root root 4096 Feb 26 15:58 ./
> >>>>> drwxr-xr-x 4 root root 4096 Feb 26 15:58 ../
> >>>>> -rw-r--r-- 1 root root 738964 Feb 26 15:58 lttng-lib-ring-buffer.ko
> >>>>>
> >>>>> /lib/modules/3.0.31-gd5a18e0/extra/probes$ ll
> >>>>> total 2872
> >>>>> drwxr-xr-x 2 root root 4096 Feb 26 15:58 ./
> >>>>> drwxr-xr-x 4 root root 4096 Feb 26 15:58 ../
> >>>>> -rw-r--r-- 1 root root 162872 Feb 26 15:58 lttng-probe-asoc.ko
> >>>>> -rw-r--r-- 1 root root 171049 Feb 26 15:58 lttng-probe-block.ko
> >>>>> -rw-r--r-- 1 root root 81165 Feb 26 15:58 lttng-probe-compaction.ko
> >>>>> -rw-r--r-- 1 root root 333215 Feb 26 15:58 lttng-probe-ext4.ko
> >>>>> -rw-r--r-- 1 root root 80413 Feb 26 15:58 lttng-probe-gpio.ko
> >>>>> -rw-r--r-- 1 root root 84256 Feb 26 15:58 lttng-probe-irq.ko
> >>>>> -rw-r--r-- 1 root root 135453 Feb 26 15:58 lttng-probe-jbd2.ko
> >>>>> -rw-r--r-- 1 root root 105399 Feb 26 15:58 lttng-probe-kmem.ko
> >>>>> -rw-r--r-- 1 root root 76240 Feb 26 15:58 lttng-probe-lttng.ko
> >>>>> -rw-r--r-- 1 root root 88059 Feb 26 15:58 lttng-probe-module.ko
> >>>>> -rw-r--r-- 1 root root 154032 Feb 26 15:58 lttng-probe-napi.ko
> >>>>> -rw-r--r-- 1 root root 142038 Feb 26 15:58 lttng-probe-net.ko
> >>>>> -rw-r--r-- 1 root root 93564 Feb 26 15:58 lttng-probe-power.ko
> >>>>> -rw-r--r-- 1 root root 85530 Feb 26 15:58 lttng-probe-regulator.ko
> >>>>> -rw-r--r-- 1 root root 138121 Feb 26 15:58 lttng-probe-sched.ko
> >>>>> -rw-r--r-- 1 root root 152909 Feb 26 15:58 lttng-probe-scsi.ko
> >>>>> -rw-r--r-- 1 root root 107491 Feb 26 15:58 lttng-probe-signal.ko
> >>>>> -rw-r--r-- 1 root root 142711 Feb 26 15:58 lttng-probe-skb.ko
> >>>>> -rw-r--r-- 1 root root 181983 Feb 26 15:58 lttng-probe-statedump.ko
> >>>>> -rw-r--r-- 1 root root 126731 Feb 26 15:58 lttng-probe-timer.ko
> >>>>> -rw-r--r-- 1 root root 86591 Feb 26 15:58 lttng-probe-workqueue.ko
> >>>>> -rw-r--r-- 1 root root 133886 Feb 26 15:58 lttng-probe-writeback.ko
> >>>>> -rw-r--r-- 1 root root 30863 Feb 26 15:58 lttng-types.ko
> >>>>>
> >>>>> Looking at the files shows I miss:
> >>>>> lttng-kretprobes.ko
> >>>>> lttng-kprobes.ko
> >>>>> lttng-ftrace.ko
> >>>>> lttng-probe-kvm.ko
> >>>>>
> >>>>>
> >>>>> On Tue, Feb 26, 2013 at 3:56 PM, PLSTC <b0mb00z.it@gmail.com> wrote:
> >>>>>
> >>>>>> Back on the last email:
> >>>>>>
> >>>>>> Here's the insmod list I ended up building; you might not have
> >>>>>> compiled all those modules, depending on your kernel config and on the
> >>>>>> architecture used, so some might simply be absent from your output
> >>>>>> directory.
> >>>>>>
> >>>>>> insmod /system/lib/modules/lttng-types.ko
> >>>>>> insmod /system/lib/modules/lttng-kretprobes.ko
> >>>>>> insmod /system/lib/modules/lttng-kprobes.ko
> >>>>>> insmod /system/lib/modules/lttng-ftrace.ko
> >>>>>> insmod /system/lib/modules/lttng-lib-ring-buffer.ko
> >>>>>> insmod /system/lib/modules/lttng-statedump.ko
> >>>>>> insmod /system/lib/modules/lttng-tracer.ko
> >>>>>> insmod /system/lib/modules/lttng-probe-timer.ko
> >>>>>> insmod /system/lib/modules/lttng-probe-statedump.ko
> >>>>>> insmod /system/lib/modules/lttng-probe-signal.ko
> >>>>>> insmod /system/lib/modules/lttng-probe-sched.ko
> >>>>>> insmod /system/lib/modules/lttng-probe-lttng.ko
> >>>>>> insmod /system/lib/modules/lttng-probe-kvm.ko
> >>>>>> insmod /system/lib/modules/lttng-probe-irq.ko
> >>>>>> insmod /system/lib/modules/lttng-probe-block.ko
> >>>>>> insmod /system/lib/modules/lttng-ring-buffer-metadata-mmap-client.ko
> >>>>>> insmod /system/lib/modules/lttng-ring-buffer-metadata-client.ko
> >>>>>> insmod /system/lib/modules/lttng-ring-buffer-client-overwrite.ko
> >>>>>> insmod /system/lib/modules/lttng-ring-buffer-client-mmap-overwrite.ko
> >>>>>> insmod /system/lib/modules/lttng-ring-buffer-client-mmap-discard.ko
> >>>>>> insmod /system/lib/modules/lttng-ring-buffer-client-discard.ko
> >>>>>>
> >>>>>>
> >>>>>> We also haven't had time recently to test if the kprobes/kretprobes
> >>>>>> worked, but they probably do; we decided to focus a bit more on the
> >>>>>> 'tracepoints' aspect instead.
> >>>>>>
> >>>>>> Anyway, good luck with your exploration, and if you find something
> >>>>>> interesting, I'd love to know more about it!
> >>>>>>
> >>>>>> -PL
> >>>>>>
> >>>>>> On Tue, Feb 26, 2013 at 8:06 AM, PLSTC <b0mb00z.it@gmail.com> wrote:
> >>>>>>
> >>>>>>> Hey Amit,
> >>>>>>>
> >>>>>>> Good news indeed it seems!
> >>>>>>>
> >>>>>>> When it comes to pushing the libs/bin, we haven't found any better
> >>>>>>> way than using adb push yet; of course, if all the projects were included
> >>>>>>> directly in Android's external, we would simply flash the device after each
> >>>>>>> build, but currently we're still having trouble generating the correct
> >>>>>>> Android makefiles.
> >>>>>>>
> >>>>>>> As for the modules, I might still have a dep load list somewhere I
> >>>>>>> could send you in a few hours, I remember fighting to get the order right...
> >>>>>>>
> >>>>>>> Also, once everything loads, could you tell me what kind of output
> >>>>>>> 'lttng list -k' provides you? Here, we're having trouble listing the
> >>>>>>> 'default' kernel trace points provided by HAVE_TRACEPOINTS, and the
> >>>>>>> returned list is simply empty (which is a bit strange).
> >>>>>>>
> >>>>>>> -PL
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Fwd: compiling LTTng for android
[not found] ` <20130315150806.GA1467@Krystal>
@ 2013-03-17 7:17 ` Amit Balboul
0 siblings, 0 replies; 8+ messages in thread
From: Amit Balboul @ 2013-03-17 7:17 UTC (permalink / raw)
To: Mathieu Desnoyers; +Cc: lttng-dev, yannick.brosseau
[-- Attachment #1.1: Type: text/plain, Size: 26277 bytes --]
That's great news !!!
Glad to hear that this is the last hurdle to pass,
Thanks,
Amit.
On Fri, Mar 15, 2013 at 5:08 PM, Mathieu Desnoyers <
mathieu.desnoyers@efficios.com> wrote:
> * Pierre-Luc St-Charles (pierre-luc.st-charles@polymtl.ca) wrote:
> > Good news!
> >
> > After peeking yesterday through the different glibc/SysV implementations
> > (wrappers...) of splice I could find, I realized that those are not
> > actually necessary for what we intend to do: from what I found out, they
> > only redefine/wrap/redeclare the sys_splice system call that's
> implemented
> > directly in the kernel (in any architecture). Considering we don't need
> > anything more than the default (kernel-space) splice, we can simply
> rewire
> > the 'splice' declaration we use/have to syscall(__NR_splice,...), just
> like
> > what was needed for epoll_create1 earlier.
> >
> > For example : (doing this from memory... I don't have access to the code
> > atm)
> > > in src/common/compat/fcntl.h, inside the linux ifdef, top of the file:
> >
> > #ifdef __ANDROID__
> > #define SPLICE_F_MOVE 0x01
> > #define SPLICE_F_NONBLOCK 0x02
> > #define SPLICE_F_MORE 0x04
> > #define SPLICE_F_GIFT 0x08
> > static inline ssize_t splice(int fd_in, loff_t *off_in, int fd_out,
> > loff_t *off_out,
> > size_t len, unsigned int flags)
> > {
> > return syscall(__NR_splice,fd_in,loff_in,fd_out,loff_out,len,flags);
> > }
> > #endif
> >
> >
> > Result? Boom, traces.
> >
> > The rest of the project seems to work without any other visible problem,
> > and we can finally obtain traces when asking the tools nicely. In fact,
> we
> > can obtain traces from any device without even having to modify the
> > kernel/system image at all (except to add module and tracepoints support,
> > which should already be there), which is pretty surprising, but also very
> > useful, considering we can just ship the required binaries anywhere and
> > start tracing.
> >
> > Once we confirm that the traces are actually good and we haven't broken
> > anything else (we're still not sure liburcu works 100%, we're trying to
> run
> > all the tests), we'll start sending in results for anyone who might be
> > interested.
>
> That's great news! So I understand that you got lttng-modules Linux
> kernel tracing to work on Android. I look forward to see patches we
> could merge into lttng-tools.
>
> I guess the next step will be to look into LTTng-UST user-space tracing.
>
> Thanks,
>
> Mathieu
>
> >
> > - PL
> >
> > On Tue, Mar 12, 2013 at 6:51 PM, Pierre-Luc St-Charles <
> > pierre-luc.st-charles@polymtl.ca> wrote:
> >
> > > Hey Amit,
> > >
> > > Here's a small update on my latest findings regarding the broken
> *splice*
> > > / bad Bionic impl problem:
> > >
> > > Although it didn't really cross my mind earlier, we had seen this exact
> > > problem before, but in another project; we had also investigated a bit
> at
> > > the time, but to no avail. It seems the issue is known by the
> > > Android/Bionic community, but it's simply not on their immediate todo
> list
> > > (and hasn't been for at least the past two years -- see
> > > https://code.google.com/p/android/issues/detail?id=11330). The
> solutions
> > > we are now considering are: adding splice support (and whatever else
> might
> > > also be needed) to Bionic ourselves (saving BILLIONS of broken lives
> in the
> > > process, IF the changes are accepted), or simply inserting glibc (and
> > > building against it) in our device images.
> > >
> > > In the next few days, we'll be studying how much time the first
> solution
> > > would actually take (as we're not really experts in that domain at
> all),
> > > but considering the nature/objective of this project, it might be the
> > > wisest decision (as it would not require too much swerving away from
> the
> > > default images people have on their devices for LTTng to actually
> work). If
> > > we deem it 'impossible/out of our league', we'll simply try to get a
> > > working tool asap using glibc, which should not be very complicated
> itself
> > > (many people have actually successfully done it before).
> > >
> > > In any case, I'll keep you updated, and I'll also post any big updates
> on
> > > the mailing list.
> > >
> > > -PL
> > >
> > > On Wed, Mar 6, 2013 at 8:04 PM, Pierre-Luc St-Charles <
> > > pierre-luc.st-charles@polymtl.ca> wrote:
> > >
> > >> Great findings!
> > >>
> > >> I'll also try to look into this new problem by next monday; I'm
> falling a
> > >> bit behind on my 'investigations', but I'll keep you updated if
> anything
> > >> comes up.
> > >>
> > >> -PL
> > >> On Mar 6, 2013 6:09 PM, "Amit Balboul" <amit.balboul@shinesec.com>
> wrote:
> > >>
> > >>> Hi All,
> > >>>
> > >>> Regarding the last mail about tracing system calls on Android using
> > >>> LTTng,
> > >>> The issue of not receiving enabled event to trace from the kernel
> can be
> > >>> fixed patching src/bin/lttng-sessiond/kernel.c (patch attached)
> > >>> In the function *kernel_list_events,* fscanf-ing the events exported
> by
> > >>> the kernel using the fp must not use the syntax *%m* as it is
> obsolete
> > >>> (in Android),
> > >>> thus the fscanf returns 0 and won't read the kernel events (first
> > >>> condition false).
> > >>> Instead I use a pre-allocated buffer for the string and use the
> *%[^;]*syntax, and omitting the memory free (attached is the patch).
> > >>>
> > >>> The resulting output is (sessiond is run with verbose debug prints):
> > >>>
> > >>> root@android:/data/lttng/bin # *./lttng list -k*
> > >>> DEBUG1 [10239/10290]: Wait for client response (in
> > >>> thread_manage_clients() at main.c:3242)
> > >>> DEBUG1 [10239/10290]: Receiving data from client ... (in
> > >>> thread_manage_clients() at main.c:3287)
> > >>> DEBUG1 [10239/10290]: Nothing recv() from client... continuing (in
> > >>> thread_manage_clients() at main.c:3291)
> > >>> DEBUG1 [10239/10290]: Clean command context structure (in
> > >>> clean_command_ctx() at main.c:482)
> > >>> DEBUG1 [10239/10290]: Accepting client command ... (in
> > >>> thread_manage_clients() at main.c:3200)
> > >>> DEBUG1 [10239/10290]: Wait for client response (in
> > >>> thread_manage_clients() at main.c:3242)
> > >>> DEBUG1 [10239/10290]: Receiving data from client ... (in
> > >>> thread_manage_clients() at main.c:3287)
> > >>> DEBUG1 [10239/10290]: Processing client command 14 (in
> > >>> process_client_msg() at main.c:2227)
> > >>> DEBUG1 [10239/10290]: Kernel list events done (61 events) (in
> > >>> kernel_list_events() at kernel.c:653)
> > >>> DEBUG1 [10239/10290]: Sending response (size: 35640, retcode:
> Success)
> > >>> (in thread_manage_clients() at main.c:3338)
> > >>> DEBUG1 [10239/10290]: Clean command context structure (in
> > >>> clean_command_ctx() at main.c:482)
> > >>> DEBUG1 [10239/10290]: Accepting client command ... (in
> > >>> thread_manage_clients() at main.c:3200)
> > >>> Kernel events:
> > >>> -------------
> > >>> block_rq_abort (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> > >>> block_rq_requeue (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> > >>> block_rq_complete (loglevel: TRACE_EMERG (0)) (type:
> tracepoint)
> > >>> block_rq_insert (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> > >>> block_rq_issue (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> > >>> block_bio_bounce (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> > >>> block_bio_complete (loglevel: TRACE_EMERG (0)) (type:
> tracepoint)
> > >>> block_bio_backmerge (loglevel: TRACE_EMERG (0)) (type:
> tracepoint)
> > >>> block_bio_frontmerge (loglevel: TRACE_EMERG (0)) (type:
> tracepoint)
> > >>> block_bio_queue (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> > >>> block_getrq (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> > >>> block_sleeprq (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> > >>> block_plug (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> > >>> block_unplug (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> > >>> block_split (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> > >>> block_bio_remap (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> > >>> block_rq_remap (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> > >>> irq_handler_entry (loglevel: TRACE_EMERG (0)) (type:
> tracepoint)
> > >>> irq_handler_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> > >>> softirq_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> > >>> softirq_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> > >>> softirq_raise (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> > >>> sched_kthread_stop (loglevel: TRACE_EMERG (0)) (type:
> tracepoint)
> > >>> sched_kthread_stop_ret (loglevel: TRACE_EMERG (0)) (type:
> > >>> tracepoint)
> > >>> sched_wakeup (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> > >>> sched_wakeup_new (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> > >>> sched_switch (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> > >>> sched_migrate_task (loglevel: TRACE_EMERG (0)) (type:
> tracepoint)
> > >>> sched_process_free (loglevel: TRACE_EMERG (0)) (type:
> tracepoint)
> > >>> sched_process_exit (loglevel: TRACE_EMERG (0)) (type:
> tracepoint)
> > >>> sched_wait_task (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> > >>> sched_process_wait (loglevel: TRACE_EMERG (0)) (type:
> tracepoint)
> > >>> sched_process_fork (loglevel: TRACE_EMERG (0)) (type:
> tracepoint)
> > >>> sched_stat_wait (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> > >>> sched_stat_sleep (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> > >>> sched_stat_iowait (loglevel: TRACE_EMERG (0)) (type:
> tracepoint)
> > >>> sched_stat_runtime (loglevel: TRACE_EMERG (0)) (type:
> tracepoint)
> > >>> sched_pi_setprio (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> > >>> signal_generate (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> > >>> signal_deliver (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> > >>> signal_overflow_fail (loglevel: TRACE_EMERG (0)) (type:
> tracepoint)
> > >>> signal_lose_info (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> > >>> lttng_statedump_start (loglevel: TRACE_EMERG (0)) (type:
> > >>> tracepoint)
> > >>> lttng_statedump_end (loglevel: TRACE_EMERG (0)) (type:
> tracepoint)
> > >>> lttng_statedump_process_state (loglevel: TRACE_EMERG (0))
> (type:
> > >>> tracepoint)
> > >>> lttng_statedump_file_descriptor (loglevel: TRACE_EMERG (0))
> (type:
> > >>> tracepoint)
> > >>> lttng_statedump_vm_map (loglevel: TRACE_EMERG (0)) (type:
> > >>> tracepoint)
> > >>> lttng_statedump_network_interface (loglevel: TRACE_EMERG (0))
> > >>> (type: tracepoint)
> > >>> lttng_statedump_interrupt (loglevel: TRACE_EMERG (0)) (type:
> > >>> tracepoint)
> > >>> timer_init (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> > >>> timer_start (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> > >>> timer_expire_entry (loglevel: TRACE_EMERG (0)) (type:
> tracepoint)
> > >>> timer_expire_exit (loglevel: TRACE_EMERG (0)) (type:
> tracepoint)
> > >>> timer_cancel (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> > >>> hrtimer_init (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> > >>> hrtimer_start (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> > >>> hrtimer_expire_entry (loglevel: TRACE_EMERG (0)) (type:
> tracepoint)
> > >>> hrtimer_expire_exit (loglevel: TRACE_EMERG (0)) (type:
> tracepoint)
> > >>> hrtimer_cancel (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> > >>> itimer_state (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> > >>> itimer_expire (loglevel: TRACE_EMERG (0)) (type: tracepoint)
> > >>>
> > >>> root@android:/data/lttng/bin #
> > >>>
> > >>> Now I'm facing the issue of starting the traces:
> > >>> debugging the sessiond, consumerd and lttng I concluded that the
> issue
> > >>> is due to splice system call failure.
> > >>> even using a MMAP output type channel, the metadata channel is SPLICE
> > >>> type (hard-coded).
> > >>> Problem persists also when using relayd to pass the traces through
> the
> > >>> network.
> > >>> Ignoring the splice failure means closing of the metadata channel
> > >>> (maybe??) but need not affect the traces channel (right ?)
> > >>> Attached is a debug log of each thread of the sessiond, right after
> > >>> invoking "*./lttng start*". (with extra informative my prints)
> > >>>
> > >>> Please keep me in touch with any progress in this issue,
> > >>> Thank you,
> > >>> Amit.
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> ---------- Forwarded message ----------
> > >>> From: Amit Balboul <amit.balboul@shinesec.com>
> > >>> Date: Wed, Feb 27, 2013 at 4:37 PM
> > >>> Subject: Re: [lttng-dev] compiling LTTng for android
> > >>> To: PLSTC <b0mb00z.it@gmail.com>
> > >>>
> > >>>
> > >>> Hey Pierre,
> > >>>
> > >>> Some major progress has been achieved, following is the summary:
> > >>>
> > >>> 1. Modules issue:
> > >>> I built (for ARM/Android) and installed latest stable version of
> busybox
> > >>> on the device.
> > >>> Seeing the code tries to run (hardcoded) /sbin/modprobe, made a soft
> > >>> link to busybox's modeprobe.
> > >>> Also, created /lib/modules/<version+build> directory and pushed the
> > >>> <version+build> directory from /lib/modules on the machine.
> > >>> Now session daemon modprobes the desired modules successfully, (some
> are
> > >>> not present though as mentioned before) - no need to use insmod.
> > >>>
> > >>> 2. epoll_create1 issue:
> > >>> epoll_create1 patch was OK, I accidentally defined EPOLL_CLOEXEC as
> > >>> 0x02000000 instead of 02000000 (fixed).
> > >>>
> > >>> The running scenario:
> > >>>
> > >>> 1. Run ./lttng-sessiond -d -vvv
> > >>>
> > >>> Then: (commands are in bold, output is in blue for readability)
> > >>>
> > >>> root@android:/data/lttng/install_sysroot/bin # *./lttng create s1*
> > >>>
> > >>> Session s1 created.
> > >>> Traces will be written in /data/lttng-traces/s1-20130227-161145
> > >>> root@android:/data/lttng/install_sysroot/bin # *./lttng
> enable-channel
> > >>> ch -k*
> > >>> Kernel channel ch enabled for session s1
> > >>> root@android:/data/lttng/install_sysroot/bin # *./lttng
> enable-event ev
> > >>> -k -a*
> > >>> All kernel events are enabled in channel channel0
> > >>> root@android:/data/lttng/install_sysroot/bin # *./lttng list s1*
> > >>> Tracing session s1: [inactive]
> > >>> Trace path: /data/lttng-traces/s1-20130227-161145
> > >>>
> > >>> === Domain: Kernel ===
> > >>>
> > >>> Channels:
> > >>> -------------
> > >>> - channel0: [enabled]
> > >>>
> > >>> Attributes:
> > >>> overwrite mode: 0
> > >>> subbufers size: 262144
> > >>> number of subbufers: 4
> > >>> switch timer interval: 0
> > >>> read timer interval: 200
> > >>> output: splice()
> > >>>
> > >>> Events:
> > >>> None
> > >>>
> > >>> - ch: [enabled]
> > >>>
> > >>> Attributes:
> > >>> overwrite mode: 0
> > >>> subbufers size: 262144
> > >>> number of subbufers: 4
> > >>> switch timer interval: 0
> > >>> read timer interval: 200
> > >>> output: splice()
> > >>>
> > >>> Events:
> > >>> None
> > >>>
> > >>> root@android:/data/lttng/install_sysroot/bin # *./lttng list -k *
> > >>>
> > >>> Kernel events:
> > >>> -------------
> > >>>
> > >>> root@android:/data/lttng/install_sysroot/bin # *./lttng start*
> > >>> Tracing started for session s1
> > >>> root@android:/data/lttng/install_sysroot/bin # *./lttng stop *
> > >>>
> > >>> Waiting for data availability
> > >>> Tracing stopped for session s1
> > >>> root@android:/data/lttng/install_sysroot/bin #
> > >>>
> > >>> *Now: (this is not surprising because the kernel events list is
> > >>> empty...)*
> > >>> the folder /data/lttng-traces/s1-20130227-161145/kernel contains:
> > >>> total 0
> > >>> -rwxrwxrwx 1 0 0 0 Feb 27 14:18 ch_0
> > >>> -rwxrwxrwx 1 0 0 0 Feb 27 14:18 ch_1
> > >>> -rwxrwxrwx 1 0 0 0 Feb 27 14:18 channel0_0
> > >>> -rwxrwxrwx 1 0 0 0 Feb 27 14:18 channel0_1
> > >>> -rwxrwxrwx 1 0 0 0 Feb 27 14:18 metadata
> > >>>
> > >>> *By the way:*
> > >>> When running the same scenario but creating the channel with
> *--output
> > >>> mmap*, still no data but the size of the first two files in the
> traces
> > >>> directory are 4Kb (I didn't bother to investigate them as I assume
> it is
> > >>> just the header - not the traces payload).
> > >>>
> > >>> Another issue: at this point, listing the sessions gives:
> > >>> root@android:/data/lttng/install_sysroot/bin # *./lttng list s1 *
> > >>>
> > >>> Tracing session s1: [inactive]
> > >>> Trace path: /data/lttng-traces/s1-20130227-161145
> > >>>
> > >>> === Domain: Kernel ===
> > >>>
> > >>> Error: No kernel consumer detected
> > >>> 99|root@android:/data/lttng/install_sysroot/bin # *./lttng list -k
> *
> > >>>
> > >>> Error: Unable to list kernel events
> > >>> Error: Command error
> > >>> 1|root@android:/data/lttng/install_sysroot/bin #
> > >>>
> > >>> So currently I'm facing these issues:
> > >>> 1. Kernel events list is empty.
> > >>> 2. Listing the session/ kernel events after creating session,
> channel,
> > >>> event, start, stop - results in "no kernel consumer detected"
> > >>>
> > >>> If you have some insights please share them with me..
> > >>>
> > >>> Thanks
> > >>>
> > >>> Amit.
> > >>>
> > >>>
> > >>> On Tue, Feb 26, 2013 at 7:35 PM, PLSTC <b0mb00z.it@gmail.com> wrote:
> > >>>
> > >>>> The modules you can't find are the same I am also missing I
> believe; I
> > >>>> think they are only compiled when certain kernel config defines are
> met
> > >>>> (and those might simply be unavailable under ARM).
> > >>>>
> > >>>> As for what you pointed in your first mail, I believe modprobe isn't
> > >>>> available on ARM, did you mean that some modules were not found
> when using
> > >>>> insmod?
> > >>>>
> > >>>> For the epoll_create1 problem, what kind of patch did you apply to
> add
> > >>>> support for the epoll_create1 function?
> > >>>>
> > >>>>
> > >>>> On Tue, Feb 26, 2013 at 11:29 AM, Amit Balboul <
> > >>>> amit.balboul@shinesec.com> wrote:
> > >>>>
> > >>>>> Ok I got :
> > >>>>> lttng-kretprobes.ko
> > >>>>> lttng-kprobes.ko
> > >>>>> , the other two are missing.
> > >>>>>
> > >>>>>
> > >>>>> ---------- Forwarded message ----------
> > >>>>> From: Amit Balboul <amit.balboul@shinesec.com>
> > >>>>> Date: Tue, Feb 26, 2013 at 5:49 PM
> > >>>>> Subject: Re: [lttng-dev] compiling LTTng for android
> > >>>>> To: PLSTC <b0mb00z.it@gmail.com>
> > >>>>>
> > >>>>>
> > >>>>> One more thing:
> > >>>>>
> > >>>>> The list of modules you gave my is not identical to mine:
> > >>>>> a. I pushed the modules from my machine to my folder in the device
> > >>>>> (not to /system/lib) as you directed me, and insmoded them
> manually.
> > >>>>> b. I pushed the whole tree under /lib/modules/<KERNEL_VER_BUILD>/
> > >>>>> which includes the modules.* and the sub directories:
> > >>>>>
> > >>>>> /lib/modules/3.0.31-gd5a18e0/extra$ ll
> > >>>>> total 2464
> > >>>>> drwxr-xr-x 4 root root 4096 Feb 26 15:58 ./
> > >>>>> drwxr-xr-x 3 root root 4096 Feb 26 15:58 ../
> > >>>>> drwxr-xr-x 2 root root 4096 Feb 26 15:58 lib/
> > >>>>> -rw-r--r-- 1 root root 167184 Feb 26 15:58
> > >>>>> lttng-ring-buffer-client-discard.ko
> > >>>>> -rw-r--r-- 1 root root 167206 Feb 26 15:58
> > >>>>> lttng-ring-buffer-client-mmap-discard.ko
> > >>>>> -rw-r--r-- 1 root root 178586 Feb 26 15:58
> > >>>>> lttng-ring-buffer-client-mmap-overwrite.ko
> > >>>>> -rw-r--r-- 1 root root 178564 Feb 26 15:58
> > >>>>> lttng-ring-buffer-client-overwrite.ko
> > >>>>> -rw-r--r-- 1 root root 137179 Feb 26 15:58
> > >>>>> lttng-ring-buffer-metadata-client.ko
> > >>>>> -rw-r--r-- 1 root root 137233 Feb 26 15:58
> > >>>>> lttng-ring-buffer-metadata-mmap-client.ko
> > >>>>> -rw-r--r-- 1 root root 213451 Feb 26 15:58 lttng-statedump.ko
> > >>>>> -rw-r--r-- 1 root root 1314235 Feb 26 15:58 lttng-tracer.ko
> > >>>>> drwxr-xr-x 2 root root 4096 Feb 26 15:58 probes/
> > >>>>>
> > >>>>> /lib/modules/3.0.31-gd5a18e0/extra/lib$ ll
> > >>>>> total 732
> > >>>>> drwxr-xr-x 2 root root 4096 Feb 26 15:58 ./
> > >>>>> drwxr-xr-x 4 root root 4096 Feb 26 15:58 ../
> > >>>>> -rw-r--r-- 1 root root 738964 Feb 26 15:58 lttng-lib-ring-buffer.ko
> > >>>>>
> > >>>>> /lib/modules/3.0.31-gd5a18e0/extra/probes$ ll
> > >>>>> total 2872
> > >>>>> drwxr-xr-x 2 root root 4096 Feb 26 15:58 ./
> > >>>>> drwxr-xr-x 4 root root 4096 Feb 26 15:58 ../
> > >>>>> -rw-r--r-- 1 root root 162872 Feb 26 15:58 lttng-probe-asoc.ko
> > >>>>> -rw-r--r-- 1 root root 171049 Feb 26 15:58 lttng-probe-block.ko
> > >>>>> -rw-r--r-- 1 root root 81165 Feb 26 15:58
> lttng-probe-compaction.ko
> > >>>>> -rw-r--r-- 1 root root 333215 Feb 26 15:58 lttng-probe-ext4.ko
> > >>>>> -rw-r--r-- 1 root root 80413 Feb 26 15:58 lttng-probe-gpio.ko
> > >>>>> -rw-r--r-- 1 root root 84256 Feb 26 15:58 lttng-probe-irq.ko
> > >>>>> -rw-r--r-- 1 root root 135453 Feb 26 15:58 lttng-probe-jbd2.ko
> > >>>>> -rw-r--r-- 1 root root 105399 Feb 26 15:58 lttng-probe-kmem.ko
> > >>>>> -rw-r--r-- 1 root root 76240 Feb 26 15:58 lttng-probe-lttng.ko
> > >>>>> -rw-r--r-- 1 root root 88059 Feb 26 15:58 lttng-probe-module.ko
> > >>>>> -rw-r--r-- 1 root root 154032 Feb 26 15:58 lttng-probe-napi.ko
> > >>>>> -rw-r--r-- 1 root root 142038 Feb 26 15:58 lttng-probe-net.ko
> > >>>>> -rw-r--r-- 1 root root 93564 Feb 26 15:58 lttng-probe-power.ko
> > >>>>> -rw-r--r-- 1 root root 85530 Feb 26 15:58 lttng-probe-regulator.ko
> > >>>>> -rw-r--r-- 1 root root 138121 Feb 26 15:58 lttng-probe-sched.ko
> > >>>>> -rw-r--r-- 1 root root 152909 Feb 26 15:58 lttng-probe-scsi.ko
> > >>>>> -rw-r--r-- 1 root root 107491 Feb 26 15:58 lttng-probe-signal.ko
> > >>>>> -rw-r--r-- 1 root root 142711 Feb 26 15:58 lttng-probe-skb.ko
> > >>>>> -rw-r--r-- 1 root root 181983 Feb 26 15:58 lttng-probe-statedump.ko
> > >>>>> -rw-r--r-- 1 root root 126731 Feb 26 15:58 lttng-probe-timer.ko
> > >>>>> -rw-r--r-- 1 root root 86591 Feb 26 15:58 lttng-probe-workqueue.ko
> > >>>>> -rw-r--r-- 1 root root 133886 Feb 26 15:58 lttng-probe-writeback.ko
> > >>>>> -rw-r--r-- 1 root root 30863 Feb 26 15:58 lttng-types.ko
> > >>>>>
> > >>>>> Looking at the files shows I miss:
> > >>>>> lttng-kretprobes.ko
> > >>>>> lttng-kprobes.ko
> > >>>>> lttng-ftrace.ko
> > >>>>> lttng-probe-kvm.ko
> > >>>>>
> > >>>>>
> > >>>>> On Tue, Feb 26, 2013 at 3:56 PM, PLSTC <b0mb00z.it@gmail.com>
> wrote:
> > >>>>>
> > >>>>>> Back on the last email:
> > >>>>>>
> > >>>>>> Here's the insmod list I ended up building; you might not have
> > >>>>>> compiled all those modules, depending on your kernel config and
> on the
> > >>>>>> architecture used, so some might simply be absent from your output
> > >>>>>> directory.
> > >>>>>>
> > >>>>>> insmod /system/lib/modules/lttng-types.ko
> > >>>>>> insmod /system/lib/modules/lttng-kretprobes.ko
> > >>>>>> insmod /system/lib/modules/lttng-kprobes.ko
> > >>>>>> insmod /system/lib/modules/lttng-ftrace.ko
> > >>>>>> insmod /system/lib/modules/lttng-lib-ring-buffer.ko
> > >>>>>> insmod /system/lib/modules/lttng-statedump.ko
> > >>>>>> insmod /system/lib/modules/lttng-tracer.ko
> > >>>>>> insmod /system/lib/modules/lttng-probe-timer.ko
> > >>>>>> insmod /system/lib/modules/lttng-probe-statedump.ko
> > >>>>>> insmod /system/lib/modules/lttng-probe-signal.ko
> > >>>>>> insmod /system/lib/modules/lttng-probe-sched.ko
> > >>>>>> insmod /system/lib/modules/lttng-probe-lttng.ko
> > >>>>>> insmod /system/lib/modules/lttng-probe-kvm.ko
> > >>>>>> insmod /system/lib/modules/lttng-probe-irq.ko
> > >>>>>> insmod /system/lib/modules/lttng-probe-block.ko
> > >>>>>> insmod
> /system/lib/modules/lttng-ring-buffer-metadata-mmap-client.ko
> > >>>>>> insmod /system/lib/modules/lttng-ring-buffer-metadata-client.ko
> > >>>>>> insmod /system/lib/modules/lttng-ring-buffer-client-overwrite.ko
> > >>>>>> insmod
> /system/lib/modules/lttng-ring-buffer-client-mmap-overwrite.ko
> > >>>>>> insmod
> /system/lib/modules/lttng-ring-buffer-client-mmap-discard.ko
> > >>>>>> insmod /system/lib/modules/lttng-ring-buffer-client-discard.ko
> > >>>>>>
> > >>>>>>
> > >>>>>> We also haven't had time recently to test if the
> kprobes/kretprobes
> > >>>>>> worked, but they probably do; we decided to focus a bit more on
> the
> > >>>>>> 'tracepoints' aspect instead.
> > >>>>>>
> > >>>>>> Anyway, good luck with your exploration, and if you find something
> > >>>>>> interesting, I'd love to know more about it!
> > >>>>>>
> > >>>>>> -PL
> > >>>>>>
> > >>>>>> On Tue, Feb 26, 2013 at 8:06 AM, PLSTC <b0mb00z.it@gmail.com>
> wrote:
> > >>>>>>
> > >>>>>>> Hey Amit,
> > >>>>>>>
> > >>>>>>> Good news indeed it seems!
> > >>>>>>>
> > >>>>>>> When it comes to pushing the libs/bin, we haven't found any
> better
> > >>>>>>> way than using adb push yet; of course, if all the projects were
> included
> > >>>>>>> directly in Android's external, we would simply flash the device
> after each
> > >>>>>>> build, but currently we're still having trouble generating the
> correct
> > >>>>>>> Android makefiles.
> > >>>>>>>
> > >>>>>>> As for the modules, I might still have a dep load list somewhere
> I
> > >>>>>>> could send you in a few hours, I remember fighting to get the
> order right...
> > >>>>>>>
> > >>>>>>> Also, once everything loads, could you tell me what kind of
> output
> > >>>>>>> 'lttng list -k' provides you? Here, we're having trouble listing
> the
> > >>>>>>> 'default' kernel trace points provided by HAVE_TRACEPOINTS, and
> the
> > >>>>>>> returned list is simply empty (which is a bit strange).
> > >>>>>>>
> > >>>>>>> -PL
> > >>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>>
> > >
>
> > _______________________________________________
> > lttng-dev mailing list
> > lttng-dev@lists.lttng.org
> > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
>
[-- Attachment #1.2: Type: text/html, Size: 36691 bytes --]
[-- Attachment #2: Type: text/plain, Size: 155 bytes --]
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2013-03-17 7:17 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <CAHg4Oy_bQHi7q6+Deb7AO_ZtJZj=9i5h+WyPTnaFO4uLxAwQXA@mail.gmail.com>
2013-02-21 13:18 ` compiling LTTng for android Pierre-Luc St-Charles
[not found] ` <CA+umk=rPAR7uiHQjL9FRE1UtVOEQ7J5xM46rfb_be7ZXoNC2MA@mail.gmail.com>
[not found] ` <CAHg4Oy-=fuTYU=5p0MPWJ9wQ+71NpoAWvzcHuB7rw4UesWUe-Q@mail.gmail.com>
2013-02-21 18:37 ` Pierre-Luc St-Charles
[not found] ` <CA+umk=qr5ZDCJ8S7RoHRPgE0Q4RgYBGNOULKs3GYYCEunyshdA@mail.gmail.com>
[not found] ` <CAHg4Oy975gWBAUyZg36TLP4Ngd8-WB-zqJQKcPYn+JUkcWWnJg@mail.gmail.com>
[not found] ` <CA+umk=r5X+gpaawrHQB_H71RyWw0vLW47b5dFeL2Vqt0jATB-Q@mail.gmail.com>
[not found] ` <CAHg4Oy-qpAJ+RT52vyK4_Tx7RAih-U=ZLoY1a4ZXqENqfcKc2w@mail.gmail.com>
[not found] ` <CA+umk=rhbBhGrJgw4PziJSHrNDbdfhdLEvBYLvrLpqMohBz10A@mail.gmail.com>
[not found] ` <CA+umk=p6s+sJPrMMqpBVFj3MdK1fy1rA=Y+LFFsWHd34TzhEVA@mail.gmail.com>
[not found] ` <CAHg4Oy96WTKqyV3CJOXvcWOW2n6J=k3jkxGMdYO6LjfHV0Kd9Q@mail.gmail.com>
[not found] ` <CAHg4Oy8Z1WhrYJW6oqq_2NU7BBO57E7YSMdG_HUx5H8+kCQB9Q@mail.gmail.com>
[not found] ` <CAHg4Oy9CZvRBsLoUkOZ2Tg2mF1rDZbr4bMwQUQ1WwqmeWVGEgg@mail.gmail.com>
[not found] ` <CA+umk=q_7M4JEwBXryWEpoCE9tHpQYdJrXq5gV9QsmRy198rAA@mail.gmail.com>
[not found] ` <CAHg4Oy_hD8CYiBvvhd0vQhW=NgQ0g8uwpbyUUv=2+HkDH6cdeQ@mail.gmail.com>
[not found] ` <CA+umk=p=cc2r7P1dW3yAmwZWB1gW_d_zB+Ce-JS2RJoHObdjCw@mail.gmail.com>
[not found] ` <CA+umk=qs6cR+LPuDpALtLpAUWL9dSmRzn5k-CdW0-vwDr+6S=Q@mail.gmail.com>
[not found] ` <CAHg4Oy--R1jHhHAbQzTKgDNgTi7cnqPAdYyPoANct+sjVJP35A@mail.gmail.com>
[not found] ` <CAHg4Oy875Z0T=nW=MQ3_xm-b3rhDDYfQFemjeVR68r9mqUu_VQ@mail.gmail.com>
[not found] ` <CA+umk=r3F7C8zKvF2Bvjoh9FmuOU_Htf3P+VngRKyvo+PHbsQw@mail.gmail.com>
[not found] ` <CAHg4Oy8LbkQT3yLeWMq7myty8t06brtrr9yyYwvDk8cjsmFx9A@mail.gmail.com>
2013-03-06 23:09 ` Fwd: " Amit Balboul
[not found] ` <CAHg4Oy_X-NhH5_JFXCBg-HsoZxyh+equV1O8vDnqAVv1QCroRw@mail.gmail.com>
2013-03-07 1:04 ` Pierre-Luc St-Charles
[not found] ` <CA+umk=qjGDo4O7NmhfKHQRM96pTz3X8TnveaShtWQJRBZ82X3A@mail.gmail.com>
2013-03-12 22:51 ` Pierre-Luc St-Charles
[not found] ` <CA+umk=pgM9BVebkXj2hrgi_LDBdwjm2ZGpS=PmT9DapPJ5kD0A@mail.gmail.com>
2013-03-15 14:33 ` Pierre-Luc St-Charles
[not found] ` <CA+umk=rVfxiSdSAkUCy2yDKbqdgtWW9zJ8O=ivSgOe828br+oQ@mail.gmail.com>
2013-03-15 15:08 ` Mathieu Desnoyers
[not found] ` <20130315150806.GA1467@Krystal>
2013-03-17 7:17 ` Amit Balboul
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.