LTTng user-space callstacks

LTTng user-space callstacks

[Valentin Grigorev via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 298 bytes --]

Hello!

Currently, callstack collection in LTTng is only available for kernel-space
events with context fields *callstack-kernel *and *callstack-user*.

Is it expected that callstack collection for LTTng-UST will be added too?
And if it is expected, then how soon?

Best regards,
Valentin Grigorev

[-- Attachment #1.2: Type: text/html, Size: 491 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: LTTng user-space callstacks

[Mathieu Desnoyers via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 1030 bytes --]

----- On Mar 26, 2020, at 1:39 PM, lttng-dev <lttng-dev@lists.lttng.org> wrote: 

> Hello!

> Currently, callstack collection in LTTng is only available for kernel-space
> events with context fields callstack-kernel and callstack-user .

> Is it expected that callstack collection for LTTng-UST will be added too? And if
> it is expected, then how soon?

Hi Valentin, 

It is something that would be interesting and useful, but a lot of work would 
be needed to have stack-walking this is fast enough and reentrant wrt signal 
handlers. Unfortunately, the backtrace(3) functions do not meet those 
requirements. 

Also, for gathering user-space callstacks from the kernel tracer, it only works if 
all user-space is compiled with frame pointers. This is also a limitation that would 
require a lot of work to overcome neatly. 

None of those features are currently on any roadmap due to lack of customers 
showing interest in getting this done. 

Thanks, 

Mathieu 

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 

[-- Attachment #1.2: Type: text/html, Size: 1917 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: LTTng user-space callstacks

[Milian Wolff via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 1480 bytes --]

On Donnerstag, 26. März 2020 20:41:17 CET Mathieu Desnoyers via lttng-dev 
wrote:
> ----- On Mar 26, 2020, at 1:39 PM, lttng-dev <lttng-dev@lists.lttng.org> 
wrote:
> > Hello!
> > 
> > Currently, callstack collection in LTTng is only available for
> > kernel-space
> > events with context fields callstack-kernel and callstack-user .
> > 
> > Is it expected that callstack collection for LTTng-UST will be added too?
> > And if it is expected, then how soon?
> 
> Hi Valentin,
> 
> It is something that would be interesting and useful, but a lot of work
> would be needed to have stack-walking this is fast enough and reentrant wrt
> signal handlers. Unfortunately, the backtrace(3) functions do not meet
> those requirements.
> 
> Also, for gathering user-space callstacks from the kernel tracer, it only
> works if all user-space is compiled with frame pointers. This is also a
> limitation that would require a lot of work to overcome neatly.
> 
> None of those features are currently on any roadmap due to lack of customers
> showing interest in getting this done.

libunwind [1] should meet the requirements you need. It is high performance 
thanks to extensive caching, signal safe and does not rely on frame pointers.

[1]: https://github.com/libunwind/libunwind

-- 
Milian Wolff | milian.wolff@kdab.com | Senior Software Engineer
KDAB (Deutschland) GmbH, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt, C++ and OpenGL Experts

[-- Attachment #1.2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 3826 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: LTTng user-space callstacks

[Valentin Grigorev via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 1915 bytes --]

Yeah, as far as I know, perf [1] is successfully using libunwind to provide
callstacks. Probably, LTTng team should think about it too, if there are no
serious obstacles.

[1]: https://perf.wiki.kernel.org/index.php/Main_Page


Best regards,
Valentin Grigorev


On Thu, Mar 26, 2020 at 11:53 PM Milian Wolff <milian.wolff@kdab.com> wrote:

> On Donnerstag, 26. März 2020 20:41:17 CET Mathieu Desnoyers via lttng-dev
> wrote:
> > ----- On Mar 26, 2020, at 1:39 PM, lttng-dev <lttng-dev@lists.lttng.org>
>
> wrote:
> > > Hello!
> > >
> > > Currently, callstack collection in LTTng is only available for
> > > kernel-space
> > > events with context fields callstack-kernel and callstack-user .
> > >
> > > Is it expected that callstack collection for LTTng-UST will be added
> too?
> > > And if it is expected, then how soon?
> >
> > Hi Valentin,
> >
> > It is something that would be interesting and useful, but a lot of work
> > would be needed to have stack-walking this is fast enough and reentrant
> wrt
> > signal handlers. Unfortunately, the backtrace(3) functions do not meet
> > those requirements.
> >
> > Also, for gathering user-space callstacks from the kernel tracer, it only
> > works if all user-space is compiled with frame pointers. This is also a
> > limitation that would require a lot of work to overcome neatly.
> >
> > None of those features are currently on any roadmap due to lack of
> customers
> > showing interest in getting this done.
>
> libunwind [1] should meet the requirements you need. It is high
> performance
> thanks to extensive caching, signal safe and does not rely on frame
> pointers.
>
> [1]: https://github.com/libunwind/libunwind
>
> --
> Milian Wolff | milian.wolff@kdab.com | Senior Software Engineer
> KDAB (Deutschland) GmbH, a KDAB Group company
> Tel: +49-30-521325470
> KDAB - The Qt, C++ and OpenGL Experts

[-- Attachment #1.2: Type: text/html, Size: 2821 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: LTTng user-space callstacks

[Mathieu Desnoyers via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 2739 bytes --]

Last time I checked, the main issue with perf's handling of cases that lack frame pointers 
is that they grab a ~2-4kB worth of stack data into the trace, and then post-process it 
with libunwind. This might be OK for sampling use-cases, but is not something I consider 
appropriate for tracing due to considerably increased use of tracing throughput. 

Has the situation changed on the perf side ? 

Thanks, 

Mathieu 

----- On Mar 27, 2020, at 9:44 AM, Valentin Grigorev <valentin.grigorev@jetbrains.com> wrote: 

> Yeah, as far as I know, perf [1] is successfully using libunwind to provide
> callstacks. Probably, LTTng team should think about it too, if there are no
> serious obstacles.

> [1]: [ https://perf.wiki.kernel.org/index.php/Main_Page |
> https://perf.wiki.kernel.org/index.php/Main_Page ]

> Best regards,
> Valentin Grigorev

> On Thu, Mar 26, 2020 at 11:53 PM Milian Wolff < [ mailto:milian.wolff@kdab.com |
> milian.wolff@kdab.com ] > wrote:

>> On Donnerstag, 26. März 2020 20:41:17 CET Mathieu Desnoyers via lttng-dev
>> wrote:
>>> ----- On Mar 26, 2020, at 1:39 PM, lttng-dev < [
>> > mailto:lttng-dev@lists.lttng.org | lttng-dev@lists.lttng.org ] >
>> wrote:
>> > > Hello!

>> > > Currently, callstack collection in LTTng is only available for
>> > > kernel-space
>> > > events with context fields callstack-kernel and callstack-user .

>> > > Is it expected that callstack collection for LTTng-UST will be added too?
>> > > And if it is expected, then how soon?

>> > Hi Valentin,

>> > It is something that would be interesting and useful, but a lot of work
>> > would be needed to have stack-walking this is fast enough and reentrant wrt
>> > signal handlers. Unfortunately, the backtrace(3) functions do not meet
>> > those requirements.

>> > Also, for gathering user-space callstacks from the kernel tracer, it only
>> > works if all user-space is compiled with frame pointers. This is also a
>> > limitation that would require a lot of work to overcome neatly.

>> > None of those features are currently on any roadmap due to lack of customers
>> > showing interest in getting this done.

>> libunwind [1] should meet the requirements you need. It is high performance
>> thanks to extensive caching, signal safe and does not rely on frame pointers.

>> [1]: [ https://github.com/libunwind/libunwind |
>> https://github.com/libunwind/libunwind ]

>> --
>> Milian Wolff | [ mailto:milian.wolff@kdab.com | milian.wolff@kdab.com ] | Senior
>> Software Engineer
>> KDAB (Deutschland) GmbH, a KDAB Group company
>> Tel: +49-30-521325470
>> KDAB - The Qt, C++ and OpenGL Experts
-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 

[-- Attachment #1.2: Type: text/html, Size: 4388 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: LTTng user-space callstacks

[Valentin Grigorev via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 2817 bytes --]

Hmm, I don't know unfortunately, it should be checked

Best regards,
Valentin Grigorev


On Fri, Mar 27, 2020 at 4:55 PM Mathieu Desnoyers <
mathieu.desnoyers@efficios.com> wrote:

> Last time I checked, the main issue with perf's handling of cases that
> lack frame pointers
> is that they grab a ~2-4kB worth of stack data into the trace, and then
> post-process it
> with libunwind. This might be OK for sampling use-cases, but is not
> something I consider
> appropriate for tracing due to considerably increased use of tracing
> throughput.
>
> Has the situation changed on the perf side ?
>
> Thanks,
>
> Mathieu
>
> ----- On Mar 27, 2020, at 9:44 AM, Valentin Grigorev <
> valentin.grigorev@jetbrains.com> wrote:
>
> Yeah, as far as I know, perf [1] is successfully using libunwind to
> provide callstacks. Probably, LTTng team should think about it too, if
> there are no serious obstacles.
>
> [1]: https://perf.wiki.kernel.org/index.php/Main_Page
>
>
> Best regards,
> Valentin Grigorev
>
>
> On Thu, Mar 26, 2020 at 11:53 PM Milian Wolff <milian.wolff@kdab.com>
> wrote:
>
>> On Donnerstag, 26. März 2020 20:41:17 CET Mathieu Desnoyers via lttng-dev
>> wrote:
>> > ----- On Mar 26, 2020, at 1:39 PM, lttng-dev <lttng-dev@lists.lttng.org>
>>
>> wrote:
>> > > Hello!
>> > >
>> > > Currently, callstack collection in LTTng is only available for
>> > > kernel-space
>> > > events with context fields callstack-kernel and callstack-user .
>> > >
>> > > Is it expected that callstack collection for LTTng-UST will be added
>> too?
>> > > And if it is expected, then how soon?
>> >
>> > Hi Valentin,
>> >
>> > It is something that would be interesting and useful, but a lot of work
>> > would be needed to have stack-walking this is fast enough and reentrant
>> wrt
>> > signal handlers. Unfortunately, the backtrace(3) functions do not meet
>> > those requirements.
>> >
>> > Also, for gathering user-space callstacks from the kernel tracer, it
>> only
>> > works if all user-space is compiled with frame pointers. This is also a
>> > limitation that would require a lot of work to overcome neatly.
>> >
>> > None of those features are currently on any roadmap due to lack of
>> customers
>> > showing interest in getting this done.
>>
>> libunwind [1] should meet the requirements you need. It is high
>> performance
>> thanks to extensive caching, signal safe and does not rely on frame
>> pointers.
>>
>> [1]: https://github.com/libunwind/libunwind
>>
>> --
>> Milian Wolff | milian.wolff@kdab.com | Senior Software Engineer
>> KDAB (Deutschland) GmbH, a KDAB Group company
>> Tel: +49-30-521325470
>> KDAB - The Qt, C++ and OpenGL Experts
>
>
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
>

[-- Attachment #1.2: Type: text/html, Size: 4919 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: LTTng user-space callstacks

[Milian Wolff via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 1071 bytes --]

On Freitag, 27. März 2020 14:55:08 CET Mathieu Desnoyers wrote:
> Last time I checked, the main issue with perf's handling of cases that lack
> frame pointers is that they grab a ~2-4kB worth of stack data into the
> trace, and then post-process it with libunwind. This might be OK for
> sampling use-cases, but is not something I consider appropriate for tracing
> due to considerably increased use of tracing throughput.
> 
> Has the situation changed on the perf side ?

Perf still does this. But heaptrack, of which I'm the main author, uses 
libunwind to unwind on every heap allocation and it's plenty fast. Obviously 
the impact is measureable and it's far away from even close to zero overhead. 
But it's super useful. I guess having that capability in LTTng would be 
similarly useful and if it's documented to have an overhead people can use it 
with care.

Cheers

-- 
Milian Wolff | milian.wolff@kdab.com | Senior Software Engineer
KDAB (Deutschland) GmbH, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt, C++ and OpenGL Experts

[-- Attachment #1.2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 3826 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev