Correctly using callstack-user context

Correctly using callstack-user context

[Christophe Bédard via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 1266 bytes --]

Hi,

As part of a big software safety certification effort, we are looking into
making sure that some functions of our API do not allocate memory and do
not use any blocking syscalls.

This part is done and is working (using kmem_mm_page_{alloc,free} for
memory allocations for now). However, if we do get memory allocations, we
want to know where they're from. To do this, I've been looking at using the
callstack-user context. However, I have a hard time getting more than 1
callstack-user address.

I'm on 5.3.0-46-generic, i.e. CONFIG_ARCH_STACKWALK-compatible, from what I
could find. I tried creating a minimal version using
-fno-omit-frame-pointer & without any optimization, but I still get only
one address in the callstack_user context when I know there should be at
least a few.

Note that this is running inside a Docker container, but lttng-modules is
installed on the host, and a root session-daemon is running inside the
Docker container. Also, I installed LTTng using the stable-2.11 Ubuntu PPA.

Is there something I might be doing wrong? Am I supposed to compile
lttng-modules from source in order to use the callstack-* contexts? I
couldn't find much documentation about this, other than the brief mention
in the 2.11 docs.

Regards,


Christophe

[-- Attachment #1.2: Type: text/html, Size: 1494 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

[lttng-dev] Correctly using callstack-user context

[Christophe Bédard via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 1266 bytes --]

Hi,

As part of a big software safety certification effort, we are looking into
making sure that some functions of our API do not allocate memory and do
not use any blocking syscalls.

This part is done and is working (using kmem_mm_page_{alloc,free} for
memory allocations for now). However, if we do get memory allocations, we
want to know where they're from. To do this, I've been looking at using the
callstack-user context. However, I have a hard time getting more than 1
callstack-user address.

I'm on 5.3.0-46-generic, i.e. CONFIG_ARCH_STACKWALK-compatible, from what I
could find. I tried creating a minimal version using
-fno-omit-frame-pointer & without any optimization, but I still get only
one address in the callstack_user context when I know there should be at
least a few.

Note that this is running inside a Docker container, but lttng-modules is
installed on the host, and a root session-daemon is running inside the
Docker container. Also, I installed LTTng using the stable-2.11 Ubuntu PPA.

Is there something I might be doing wrong? Am I supposed to compile
lttng-modules from source in order to use the callstack-* contexts? I
couldn't find much documentation about this, other than the brief mention
in the 2.11 docs.

Regards,


Christophe

[-- Attachment #1.2: Type: text/html, Size: 1494 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Correctly using callstack-user context

[Mathieu Desnoyers via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 1799 bytes --]

----- On May 11, 2020, at 5:09 PM, lttng-dev <lttng-dev@lists.lttng.org> wrote: 

> Hi,
> As part of a big software safety certification effort, we are looking into
> making sure that some functions of our API do not allocate memory and do not
> use any blocking syscalls.

> This part is done and is working (using kmem_mm_page_{alloc,free} for memory
> allocations for now). However, if we do get memory allocations, we want to know
> where they're from. To do this, I've been looking at using the callstack-user
> context. However, I have a hard time getting more than 1 callstack-user
> address.

> I'm on 5.3.0-46-generic, i.e. CONFIG_ARCH_STACKWALK-compatible, from what I
> could find. I tried creating a minimal version using -fno-omit-frame-pointer &
> without any optimization, but I still get only one address in the
> callstack_user context when I know there should be at least a few.

> Note that this is running inside a Docker container, but lttng-modules is
> installed on the host, and a root session-daemon is running inside the Docker
> container. Also, I installed LTTng using the stable-2.11 Ubuntu PPA.

> Is there something I might be doing wrong? Am I supposed to compile
> lttng-modules from source in order to use the callstack-* contexts? I couldn't
> find much documentation about this, other than the brief mention in the 2.11
> docs.

How does your test program issue the system call ? Is it directly with syscall() (see syscall(2) man page), or 
does it call into libc ? Is your entire libc compiled with -fno-omit-frame-pointer ? 

As soon as one library in the chain to the system call is compiled without frame pointers, this is where the 
stack walk stops. This is usually libc. 

Thanks, 

Mathieu 

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 

[-- Attachment #1.2: Type: text/html, Size: 2687 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Correctly using callstack-user context

[Mathieu Desnoyers via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 1799 bytes --]

----- On May 11, 2020, at 5:09 PM, lttng-dev <lttng-dev@lists.lttng.org> wrote: 

> Hi,
> As part of a big software safety certification effort, we are looking into
> making sure that some functions of our API do not allocate memory and do not
> use any blocking syscalls.

> This part is done and is working (using kmem_mm_page_{alloc,free} for memory
> allocations for now). However, if we do get memory allocations, we want to know
> where they're from. To do this, I've been looking at using the callstack-user
> context. However, I have a hard time getting more than 1 callstack-user
> address.

> I'm on 5.3.0-46-generic, i.e. CONFIG_ARCH_STACKWALK-compatible, from what I
> could find. I tried creating a minimal version using -fno-omit-frame-pointer &
> without any optimization, but I still get only one address in the
> callstack_user context when I know there should be at least a few.

> Note that this is running inside a Docker container, but lttng-modules is
> installed on the host, and a root session-daemon is running inside the Docker
> container. Also, I installed LTTng using the stable-2.11 Ubuntu PPA.

> Is there something I might be doing wrong? Am I supposed to compile
> lttng-modules from source in order to use the callstack-* contexts? I couldn't
> find much documentation about this, other than the brief mention in the 2.11
> docs.

How does your test program issue the system call ? Is it directly with syscall() (see syscall(2) man page), or 
does it call into libc ? Is your entire libc compiled with -fno-omit-frame-pointer ? 

As soon as one library in the chain to the system call is compiled without frame pointers, this is where the 
stack walk stops. This is usually libc. 

Thanks, 

Mathieu 

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 

[-- Attachment #1.2: Type: text/html, Size: 2687 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Correctly using callstack-user context

[Christophe Bédard via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 838 bytes --]

On Tue, 12 May 2020 at 08:27, Mathieu Desnoyers <
mathieu.desnoyers@efficios.com> wrote:

> How does your test program issue the system call ? Is it directly with
> syscall() (see syscall(2) man page), or
> does it call into libc ? Is your entire libc compiled with
> -fno-omit-frame-pointer ?
>

Indirectly, through libc/libstdc++. We use the default system libs (which I
assume use -O, which in turn implies -fomit-frame-pointer).


> As soon as one library in the chain to the system call is compiled without
> frame pointers, this is where the
> stack walk stops. This is usually libc.
>

I see. Thanks for the information. Not sure we want to ship a custom libc
unfortunately.

Related question: how feasible would adding the callstack-user context to
userspace events be? If it is feasible, is such a feature planned?



Christophe

[-- Attachment #1.2: Type: text/html, Size: 1677 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Correctly using callstack-user context

[Christophe Bédard via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 838 bytes --]

On Tue, 12 May 2020 at 08:27, Mathieu Desnoyers <
mathieu.desnoyers@efficios.com> wrote:

> How does your test program issue the system call ? Is it directly with
> syscall() (see syscall(2) man page), or
> does it call into libc ? Is your entire libc compiled with
> -fno-omit-frame-pointer ?
>

Indirectly, through libc/libstdc++. We use the default system libs (which I
assume use -O, which in turn implies -fomit-frame-pointer).


> As soon as one library in the chain to the system call is compiled without
> frame pointers, this is where the
> stack walk stops. This is usually libc.
>

I see. Thanks for the information. Not sure we want to ship a custom libc
unfortunately.

Related question: how feasible would adding the callstack-user context to
userspace events be? If it is feasible, is such a feature planned?



Christophe

[-- Attachment #1.2: Type: text/html, Size: 1677 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Correctly using callstack-user context

[Genevieve Bastien via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 1456 bytes --]

Hi Christophe,

On 5/13/20 9:42 AM, Christophe Bédard via lttng-dev wrote:
>
> On Tue, 12 May 2020 at 08:27, Mathieu Desnoyers
> <mathieu.desnoyers@efficios.com
> <mailto:mathieu.desnoyers@efficios.com>> wrote:
>
>     How does your test program issue the system call ? Is it directly
>     with syscall() (see syscall(2) man page), or
>     does it call into libc ? Is your entire libc compiled with
>     -fno-omit-frame-pointer ?
>
>
> Indirectly, through libc/libstdc++. We use the default system libs
> (which I assume use -O, which in turn implies -fomit-frame-pointer).
>  
>
>     As soon as one library in the chain to the system call is compiled
>     without frame pointers, this is where the
>     stack walk stops. This is usually libc.
>
>
> I see. Thanks for the information. Not sure we want to ship a custom
> libc unfortunately.
>
> Related question: how feasible would adding the callstack-user context
> to userspace events be? If it is feasible, is such a feature planned?

We had a very very experimental branch with it, using backtrace first,
then unwind. But it is not suggested, not performant. It's far from the
correct desired implementation, for which Mathieu could give more details.

It works, but not to be used in production. Students have used it in the
lab for some specific use case prototyping. I won't share the branch
here, but if you wish to try, let me know.

Geneviève



[-- Attachment #1.2: Type: text/html, Size: 3276 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Correctly using callstack-user context

[Genevieve Bastien via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 1456 bytes --]

Hi Christophe,

On 5/13/20 9:42 AM, Christophe Bédard via lttng-dev wrote:
>
> On Tue, 12 May 2020 at 08:27, Mathieu Desnoyers
> <mathieu.desnoyers@efficios.com
> <mailto:mathieu.desnoyers@efficios.com>> wrote:
>
>     How does your test program issue the system call ? Is it directly
>     with syscall() (see syscall(2) man page), or
>     does it call into libc ? Is your entire libc compiled with
>     -fno-omit-frame-pointer ?
>
>
> Indirectly, through libc/libstdc++. We use the default system libs
> (which I assume use -O, which in turn implies -fomit-frame-pointer).
>  
>
>     As soon as one library in the chain to the system call is compiled
>     without frame pointers, this is where the
>     stack walk stops. This is usually libc.
>
>
> I see. Thanks for the information. Not sure we want to ship a custom
> libc unfortunately.
>
> Related question: how feasible would adding the callstack-user context
> to userspace events be? If it is feasible, is such a feature planned?

We had a very very experimental branch with it, using backtrace first,
then unwind. But it is not suggested, not performant. It's far from the
correct desired implementation, for which Mathieu could give more details.

It works, but not to be used in production. Students have used it in the
lab for some specific use case prototyping. I won't share the branch
here, but if you wish to try, let me know.

Geneviève



[-- Attachment #1.2: Type: text/html, Size: 3276 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Correctly using callstack-user context

[Christophe Bédard via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 572 bytes --]

On Wed, 13 May 2020 at 10:18, Genevieve Bastien via lttng-dev <
lttng-dev@lists.lttng.org> wrote:

> We had a very very experimental branch with it, using backtrace first,
> then unwind. But it is not suggested, not performant. It's far from the
> correct desired implementation, for which Mathieu could give more details.
>
> It works, but not to be used in production. Students have used it in the
> lab for some specific use case prototyping. I won't share the branch here,
> but if you wish to try, let me know.
>
> Geneviève
>
Sure, I'd be interested!

[-- Attachment #1.2: Type: text/html, Size: 1069 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Correctly using callstack-user context

[Christophe Bédard via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 572 bytes --]

On Wed, 13 May 2020 at 10:18, Genevieve Bastien via lttng-dev <
lttng-dev@lists.lttng.org> wrote:

> We had a very very experimental branch with it, using backtrace first,
> then unwind. But it is not suggested, not performant. It's far from the
> correct desired implementation, for which Mathieu could give more details.
>
> It works, but not to be used in production. Students have used it in the
> lab for some specific use case prototyping. I won't share the branch here,
> but if you wish to try, let me know.
>
> Geneviève
>
Sure, I'd be interested!

[-- Attachment #1.2: Type: text/html, Size: 1069 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev