lttng-tools-2.10.6: Spawning consumerd failed

lttng-tools-2.10.6: Spawning consumerd failed

[Zvi Vered]

[-- Attachment #1.1: Type: text/plain, Size: 1095 bytes --]

Hello,

I ran the following command on the target (where my kernel is running):

~ # ./lttng create my-kernel-session --output=/tmp/my-kernel-trace

and then:


~ # ./lttng enable-event --kernel --syscall open,close

and got:

Error: Spawning consumerd failed
Error: Event open: Kernel consumer start failed (channel channel0,
session my-kernel-session)
PERROR - 00:25:59.397308194 [1930/1930]: listen: Invalid argument (in
lttcomm_listen_unix_sock() at unix.c:173)
PERROR - 00:25:59.397344426 [1930/1930]: sendmsg: Bad file descriptor
(in lttcomm_send_unix_sock() at unix.c:292)
Error: Failed to send data back to client
PERROR - 00:25:59.397353233 [1930/1930]: close: Bad file descriptor
(in thread_manage_clients() at main.c:4530)
Error: Event close: Kernel consumer start failed (channel channel0,
session my-kernel-session)
~ #


Can you please explain the cause to this error ?


I'm running under a 32 bit (x86) vanilla, 4.9.20 kernel with busybox
with the following lttng versions:


lttng-tools-2.10.6.tar.bz2

lttng-ust-2.10.3.tar.bz2

lttng-modules-2.10.9.tar.bz2


Thank you,

Zvika

[-- Attachment #1.2: Type: text/html, Size: 6143 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: lttng-tools-2.10.6: Spawning consumerd failed

[Zvi Vered]

[-- Attachment #1.1: Type: text/plain, Size: 2677 bytes --]

Hi Jonathan,

I had to run:
export LTTNG_CONSUMERD32_BIN=/lib/lttng/libexec/lttng-consumerd

Thank you very much !
Zvika

On Fri, May 17, 2019 at 10:48 PM Jonathan Rajotte-Julien <
jonathan.rajotte-julien@efficios.com> wrote:

> Hi,
>
> You forgot to cc the mailing list. Adding it.
>
> On Fri, May 17, 2019 at 04:50:02AM +0300, Zvi Vered wrote:
> > Hi Janathan, All,
> >
> > I solved the path issue with
> >
> > export LTTNG_CONSUMERD32_BIN=/lib/lttng/libexec
> > export LTTNG_CONSUMERD32_LIBDIR =/lib
> > export LTTNG_SESSION_CONFIG_XSD_PATH=/share/lttng/...
>
> This is a way to solve it, I would still recommend you fix it at the source
> (proper cross build).
>
> ...
>
> > ....
> > DEBUG3 - 00:13:48.436226116 [1980/1980]: Looking for a kernel consumer at
> > these locations: (in spawn_consumerd() at main.c:2429)
> > DEBUG3 - 00:13:48.436252139 [1980/1980]:        1) NULL (in
> > spawn_consumerd() at main.c:2430)
> > DEBUG3 - 00:13:48.436256414 [1980/1980]:        2)
> >
> /64_sw/git/zvi_vered/MY_PROJ/linux4.9.20/boot/rootfs/lib/lttng/libexec/lttng-consumerd
> > (in spawn_consumerd() at main.c:2431)
> > DEBUG3 - 00:13:48.436260543 [1980/1980]:        3) /lib/lttng/libexec (in
> > spawn_consumerd() at main.c:2432)
> > DEBUG3 - 00:13:48.436313492 [1980/1980]: Found location #3 (in
> > spawn_consumerd() at main.c:2441)
> > DEBUG1 - 00:13:48.436319027 [1980/1980]: Using kernel consumer at:
> > /lib/lttng/libexec (in spawn_consumerd() at main.c:2448)
> > PERROR - 00:13:48.436341261 [1980/1980]: Consumer execl(): Permission
> > denied (in spawn_consumerd() at main.c:2539)
>
> We found the consumerd in /lib/lttng/libexec/lttng-consumerd. Good.
>
> We now try to execl the following:
>
> /lib/lttng/libexec/lttng-consumerd -k --consumerd-cmd-sock <a_path>
> --consumerd-error-sock <b_path> --group root
>
> And execl returns "Permission Denied".
>
> Clearly, /lib/lttng/libexec/lttng-consumerd have some permissions problem.
>
> "Permission denied" is linked to EACCES on my system.
>   asm-generic/errno-base.h
>   17:#define    EACCES          13     /* Permission denied */
>
> "man execve" list all the reasons why execl might return EACCES per "man
> execl":
>
>   EACCES Search permission is denied on a component of the path prefix of
> filename or the name of a script interpreter.  (See also
> path_resolution(7).)
>   EACCES The file or a script interpreter is not a regular file.
>   EACCES Execute permission is denied for the file or a script or ELF
> interpreter.
>   EACCES The filesystem is mounted noexec.
>
> One of them probably fit your current scenario.
>
> Let us know how it goes.
>
> Cheers
>
> --
> Jonathan Rajotte-Julien
> EfficiOS
>

[-- Attachment #1.2: Type: text/html, Size: 3479 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: lttng-tools-2.10.6: Spawning consumerd failed

[Jonathan Rajotte-Julien]

Hi,

You forgot to cc the mailing list. Adding it.

On Fri, May 17, 2019 at 04:50:02AM +0300, Zvi Vered wrote:
> Hi Janathan, All,
> 
> I solved the path issue with
> 
> export LTTNG_CONSUMERD32_BIN=/lib/lttng/libexec
> export LTTNG_CONSUMERD32_LIBDIR =/lib
> export LTTNG_SESSION_CONFIG_XSD_PATH=/share/lttng/...

This is a way to solve it, I would still recommend you fix it at the source
(proper cross build).

...

> ....
> DEBUG3 - 00:13:48.436226116 [1980/1980]: Looking for a kernel consumer at
> these locations: (in spawn_consumerd() at main.c:2429)
> DEBUG3 - 00:13:48.436252139 [1980/1980]:        1) NULL (in
> spawn_consumerd() at main.c:2430)
> DEBUG3 - 00:13:48.436256414 [1980/1980]:        2)
> /64_sw/git/zvi_vered/MY_PROJ/linux4.9.20/boot/rootfs/lib/lttng/libexec/lttng-consumerd
> (in spawn_consumerd() at main.c:2431)
> DEBUG3 - 00:13:48.436260543 [1980/1980]:        3) /lib/lttng/libexec (in
> spawn_consumerd() at main.c:2432)
> DEBUG3 - 00:13:48.436313492 [1980/1980]: Found location #3 (in
> spawn_consumerd() at main.c:2441)
> DEBUG1 - 00:13:48.436319027 [1980/1980]: Using kernel consumer at:
> /lib/lttng/libexec (in spawn_consumerd() at main.c:2448)
> PERROR - 00:13:48.436341261 [1980/1980]: Consumer execl(): Permission
> denied (in spawn_consumerd() at main.c:2539)

We found the consumerd in /lib/lttng/libexec/lttng-consumerd. Good.

We now try to execl the following:

/lib/lttng/libexec/lttng-consumerd -k --consumerd-cmd-sock <a_path> --consumerd-error-sock <b_path> --group root

And execl returns "Permission Denied".

Clearly, /lib/lttng/libexec/lttng-consumerd have some permissions problem.

"Permission denied" is linked to EACCES on my system.
  asm-generic/errno-base.h
  17:#define    EACCES          13     /* Permission denied */

"man execve" list all the reasons why execl might return EACCES per "man execl":

  EACCES Search permission is denied on a component of the path prefix of filename or the name of a script interpreter.  (See also path_resolution(7).)
  EACCES The file or a script interpreter is not a regular file.
  EACCES Execute permission is denied for the file or a script or ELF interpreter.
  EACCES The filesystem is mounted noexec.

One of them probably fit your current scenario.

Let us know how it goes.

Cheers

-- 
Jonathan Rajotte-Julien
EfficiOS

Re: lttng-tools-2.10.6: Spawning consumerd failed

[Jonathan Rajotte-Julien]

On Wed, May 15, 2019 at 04:42:57PM +0300, Zvi Vered wrote:
> Hi Jonathan,
> 
> I'm not using buildroot nor Yokto.
> In order to build lttng-tools I ran:
> ./configure --prefix=/64_sw/git/zvi_vered/MY_PROJ/linux4.9.20/boot/rootfs

I'm no expert on building/cross compiling but I think you are missing some
options here. with-sysroot is probably one of them.

I think this should do the trick:

./configure --with-sysroot=/64_sw/git/zvi_vered/MY_PROJ/linux4.9.20/boot/rootfs
--prefix /

Another possibility would be using DESTDIR on make install.

> make
> make install
> 
> This path contains a complete root file system (based on busybox) that is
> later copied to my target.
> 
> On the target I ran:
> lttng-sessiond -vvv --verbose-consumer
> and got:
> ....
> I/O warning : failed to load external entity
> "/64_sw/git/zvi_vered/MY_PROJ/linux4.9.20/boot/rootfs/share/xml/lttng/session.xsd"
> XML Error: Failed to locate the main schema resource at
> '/64_sw/git/zvi_vered/MY_PROJ/linux4.9.20/boot/rootfs/share/xml/lttng/session.xsd'.
> Error: XSD parsing failed
> Error: Session load failed: Invalid session configuration
> DEBUG1 - 00:10:36.746330272 [1476/1618]: Accepting client command ... (in
> thread_manage_clients() at main.c:4374)
> 
> Then I ran:
> lttng create my-kernel-session
> and got:
> ....
> DEBUG3 - 00:11:41.869875832 [1628/1628]: Looking for a kernel consumer at
> these locations: (in spawn_consumerd() at main.c:2429)
> DEBUG3 - 00:11:41.869901726 [1628/1628]:        1) NULL (in
> spawn_consumerd() at main.c:2430)
> DEBUG3 - 00:11:41.869908520 [1628/1628]:        2)
> /64_sw/git/zvi_vered/MY_PROJ/linux4.9.20/boot/rootfs/lib/lttng/libexec/lttng-consumerd
> (in spawn_consumerd() at main.c:2431)
> DEBUG3 - 00:11:41.869915959 [1628/1628]:        3)
> /64_sw/git/zvi_vered/MY_PROJ/linux4.9.20/boot/rootfs/lib/lttng/libexec/lttng-consumerd
> (in spawn_consumerd() at main.c:2432)
> DEBUG1 - 00:11:41.869971468 [1628/1628]: Could not find any valid consumerd
> executable (in spawn_consumerd() at main.c:2444)
> Error: Spawning consumerd failed
> 
> I double checked: The target has /lib/lttng-libexec/lttng-consumerd
> But it looks for it in:
>  /64_sw/git/zvi_vered/MY_PROJ/linux4.9.20/boot/rootfs/lib/lttng/libexec ]
> 
> What is missing in my configuration ?

The prefix is used to bake in the path used by lttng-sessiond for
consumerd lookup.

We use libdir as the base path.

  --libdir=DIR            object code libraries [EPREFIX/lib]

Cheers

-- 
Jonathan Rajotte-Julien
EfficiOS

Re: lttng-tools-2.10.6: Spawning consumerd failed

[Zvi Vered]

[-- Attachment #1.1: Type: text/plain, Size: 2407 bytes --]

Hi Jonathan,

I'm not using buildroot nor Yokto.
In order to build lttng-tools I ran:
./configure --prefix=/64_sw/git/zvi_vered/MY_PROJ/linux4.9.20/boot/rootfs
make
make install

This path contains a complete root file system (based on busybox) that is
later copied to my target.

On the target I ran:
lttng-sessiond -vvv --verbose-consumer
and got:
....
I/O warning : failed to load external entity
"/64_sw/git/zvi_vered/MY_PROJ/linux4.9.20/boot/rootfs/share/xml/lttng/session.xsd"
XML Error: Failed to locate the main schema resource at
'/64_sw/git/zvi_vered/MY_PROJ/linux4.9.20/boot/rootfs/share/xml/lttng/session.xsd'.
Error: XSD parsing failed
Error: Session load failed: Invalid session configuration
DEBUG1 - 00:10:36.746330272 [1476/1618]: Accepting client command ... (in
thread_manage_clients() at main.c:4374)

Then I ran:
lttng create my-kernel-session
and got:
....
DEBUG3 - 00:11:41.869875832 [1628/1628]: Looking for a kernel consumer at
these locations: (in spawn_consumerd() at main.c:2429)
DEBUG3 - 00:11:41.869901726 [1628/1628]:        1) NULL (in
spawn_consumerd() at main.c:2430)
DEBUG3 - 00:11:41.869908520 [1628/1628]:        2)
/64_sw/git/zvi_vered/MY_PROJ/linux4.9.20/boot/rootfs/lib/lttng/libexec/lttng-consumerd
(in spawn_consumerd() at main.c:2431)
DEBUG3 - 00:11:41.869915959 [1628/1628]:        3)
/64_sw/git/zvi_vered/MY_PROJ/linux4.9.20/boot/rootfs/lib/lttng/libexec/lttng-consumerd
(in spawn_consumerd() at main.c:2432)
DEBUG1 - 00:11:41.869971468 [1628/1628]: Could not find any valid consumerd
executable (in spawn_consumerd() at main.c:2444)
Error: Spawning consumerd failed

I double checked: The target has /lib/lttng-libexec/lttng-consumerd
But it looks for it in:
 /64_sw/git/zvi_vered/MY_PROJ/linux4.9.20/boot/rootfs/lib/lttng/libexec ]

What is missing in my configuration ?

Thank you,
Zvika

On Tue, May 14, 2019 at 9:04 PM Jonathan Rajotte-Julien <
jonathan.rajotte-julien@efficios.com> wrote:

> >
> > >
> > >
> > > I'm running under a 32 bit (x86) vanilla, 4.9.20 kernel with busybox
> > > with the following lttng versions:
>
> I took some time to at least test on a 32 bit system (buildroot) just in
> case
> and everything seems to be working fine on a base installation.
>
> The log from lttng-sessiond should provides more information on why we
> fail to
> spawn the consumerd (wrong path, etc.).
>
> Cheers
>
> --
> Jonathan Rajotte-Julien
> EfficiOS
>

[-- Attachment #1.2: Type: text/html, Size: 3354 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: lttng-tools-2.10.6: Spawning consumerd failed

[Jonathan Rajotte-Julien]

> 
> > 
> > 
> > I'm running under a 32 bit (x86) vanilla, 4.9.20 kernel with busybox
> > with the following lttng versions:

I took some time to at least test on a 32 bit system (buildroot) just in case
and everything seems to be working fine on a base installation.

The log from lttng-sessiond should provides more information on why we fail to
spawn the consumerd (wrong path, etc.).

Cheers

-- 
Jonathan Rajotte-Julien
EfficiOS

Re: lttng-tools-2.10.6: Spawning consumerd failed

[Jonathan Rajotte-Julien]

Hi,

On Tue, May 14, 2019 at 03:20:31PM +0300, Zvi Vered wrote:
> Hello,
> 
> I ran the following command on the target (where my kernel is running):
> 
> ~ # ./lttng create my-kernel-session --output=/tmp/my-kernel-trace
> 
> and then:
> 
> 
> ~ # ./lttng enable-event --kernel --syscall open,close
> 
> and got:
> 
> Error: Spawning consumerd failed
> Error: Event open: Kernel consumer start failed (channel channel0,
> session my-kernel-session)
> PERROR - 00:25:59.397308194 [1930/1930]: listen: Invalid argument (in
> lttcomm_listen_unix_sock() at unix.c:173)
> PERROR - 00:25:59.397344426 [1930/1930]: sendmsg: Bad file descriptor
> (in lttcomm_send_unix_sock() at unix.c:292)
> Error: Failed to send data back to client
> PERROR - 00:25:59.397353233 [1930/1930]: close: Bad file descriptor
> (in thread_manage_clients() at main.c:4530)
> Error: Event close: Kernel consumer start failed (channel channel0,
> session my-kernel-session)
> ~ #
> 
> 
> Can you please explain the cause to this error ?

Clearly, lttng is having problem spawning the kernel consumer:

   Error: Spawning consumerd failed

Now for the why, we will need more information from your end.

> 
> 
> I'm running under a 32 bit (x86) vanilla, 4.9.20 kernel with busybox
> with the following lttng versions:

Are you using a distro (yocto/buildroot)?

How can we reproduce the exact environment for this? This will greatly speedup
this, especially if we can spawn such env in qemu.

Also, could you run lttng-sessiond in verbose mode and provide the output
(pastebinit)?

  lttng-sessiond -vvv --verbose-consumer

  ./lttng create my-kernel-session --output=/tmp/my-kernel-trace
  ./lttng enable-event --kernel --syscall open,close

Cheers

-- 
Jonathan Rajotte-Julien
EfficiOS