Re: Tracing directly into shared memory

Re: Tracing directly into shared memory

[Mathieu Desnoyers]

[-- Attachment #1.1: Type: text/plain, Size: 2602 bytes --]

----- On Mar 4, 2019, at 4:28 AM, Thomas Schickentanz <thomas.schickentanz@harman.com> wrote: 

> Hi all,

> My idea is to pre-allocate memory on system startup which can later be used by
> lttng for tracing. It should work like flight recorder mode without extracting
> data from ringbuffers to trace file while recording data. Data extraction of
> shm ringbuffers should be done when recording has been finished.

> As per documention with option `--shm-path‘ ringbuffer for tracing can be
> created in shared memory. With lttng-crash data can be extracted from
> ringbuffer.

> Can this option be used with shared memory in volatile memory (default ram),
> too? And is it possible to record kernel space traces with this option?

> I tried

> lttng create testshm --shm-path=/dev/shm --snapshot

> lttng enable-channel c0 -s testshm -k

> lttng enable-event -s testshm -c c0 -k -a

> lttng start

> lttng stop

> lttng-crash -extract=/home/test /dev/shm

> With lttng and lttng-crash everything seems to work fine (no error messages).

> However the extract folder keeps empty.

> Has someone researched into this option and can help to clarify the points?

Hi Thomas, 

About "--shm-path" and lttng-crash, those are discussed here: [ https://lttng.org/docs/v2.10/#doc-persistent-memory-file-systems | https://lttng.org/docs/v2.10/#doc-persistent-memory-file-systems ] 

I'm not sure I completely understand your use-case and whether your intent is to extract trace data after a 
system crash or if the "snapshot" mode is enough for your needs. 

Note that the "shm-path" option only applies to UST (user-space) traces at the moment. It has not been implemented 
for kernel tracing yet. 

The way we typically use it for extraction of trace data from user-space when a system crashes is to use the pmem driver to 
create a block device out of a range of the system memory, and then use ext2 or ext4 with the DAX mount option to ensure 
the filesystem bypasses the page cache. 

Then we ensure that kexec is used to boot a new kernel on crash, or make sure the BIOS does not erase the 
memory content on soft reboot, which gives access to the shm-path buffers after crash/reboot. 

We don't usually use /dev/shm as an argument to --shm-path. This path is normally used to hold POSIX shared memory 
files. I would expect a different mount point path as argument to --shm-path which points to a DAX-enabled filesystem 
backed by a PMEM block device. 

Thanks, 

Mathieu 

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 

[-- Attachment #1.2: Type: text/html, Size: 6329 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Tracing directly into shared memory

[Schickentanz, Thomas]

[-- Attachment #1.1: Type: text/plain, Size: 4047 bytes --]

Hi Mathieu,

Thanks for clarifying the restrictions of ‘shm-path’ option!
Our purpose is to use exclusively reserved memory for tracing. Our customers request a certain amount of free memory on the system for future development. In development stage chunks of this “free” memory are grabbed by the applications and tracing lacks memory. We want to allocate this “free” memory in an early boot stage to ensure that we get the amount of memory we need for tracing. E.g. if we have reserved 512 Mbyte of memory, we know that we can start lttng at a later point of time with 16 buffers of 4 Mbyte on a 8 core system. But this works only, if we can provide the previously reserved memory to lttng for buffer allocation.
Is there some general concern about using POSIX shared memory for buffers in kernel tracing mode (apart from performance reasons)?

I used lttng-crash only for testing, because I wanted to verify, if tracing data is available from provided shared memory. If this is ensured, I agree with you, that ‘lttng snapshot’ can do the work.

Regards,
Thomas

From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Sent: Montag, 4. März 2019 17:01
To: Schickentanz, Thomas <thomas.schickentanz@harman.com>
Cc: lttng-dev <lttng-dev@lists.lttng.org>
Subject: [EXTERNAL] Re: [lttng-dev] Tracing directly into shared memory


----- On Mar 4, 2019, at 4:28 AM, Thomas Schickentanz <thomas.schickentanz@harman.com<mailto:thomas.schickentanz@harman.com>> wrote:

Hi all,
My idea is to pre-allocate memory on system startup which can later be used by lttng for tracing. It should work like flight recorder mode without extracting data from ringbuffers to trace file while recording data. Data extraction of shm ringbuffers should be done when recording has been finished.
As per documention with option `--shm-path‘ ringbuffer for tracing can be created in shared memory. With lttng-crash data can be extracted from ringbuffer.
Can this option be used with shared memory in volatile memory (default ram), too? And is it possible to record kernel space traces with this option?
I tried
lttng create testshm --shm-path=/dev/shm --snapshot
lttng enable-channel c0 -s testshm -k
lttng enable-event -s testshm -c c0 -k -a
lttng start
lttng stop
lttng-crash -extract=/home/test /dev/shm
With lttng and lttng-crash everything seems to work fine (no error messages).
However the extract folder keeps empty.
Has someone researched into this option and can help to clarify the points?
Hi Thomas,

About "--shm-path" and lttng-crash, those are discussed here: https://lttng.org/docs/v2.10/#doc-persistent-memory-file-systems<https://clicktime.symantec.com/3YZ15pbJVSD47yeXxhXiwqE7Vc?u=https%3A%2F%2Flttng.org%2Fdocs%2Fv2.10%2F%23doc-persistent-memory-file-systems>

I'm not sure I completely understand your use-case and whether your intent is to extract trace data after a
system crash or if the "snapshot" mode is enough for your needs.

Note that the "shm-path" option only applies to UST (user-space) traces at the moment. It has not been implemented
for kernel tracing yet.

The way we typically use it for extraction of trace data from user-space when a system crashes is to use the pmem driver to
create a block device out of a range of the system memory, and then use ext2 or ext4 with the DAX mount option to ensure
the filesystem bypasses the page cache.

Then we ensure that kexec is used to boot a new kernel on crash, or make sure the BIOS does not erase the
memory content on soft reboot, which gives access to the shm-path buffers after crash/reboot.

We don't usually use /dev/shm as an argument to --shm-path. This path is normally used to hold POSIX shared memory
files. I would expect a different mount point path as argument to --shm-path which points to a DAX-enabled filesystem
backed by a PMEM block device.

Thanks,

Mathieu


--
Mathieu Desnoyers
EfficiOS Inc.
https://clicktime.symantec.com/32vnSPUsVCfHCkkwoyPymVx7Vc?u=http%3A%2F%2Fwww.efficios.com

[-- Attachment #1.2: Type: text/html, Size: 15517 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Tracing directly into shared memory

[Mathieu Desnoyers]

[-- Attachment #1.1: Type: text/plain, Size: 5771 bytes --]

----- On Mar 5, 2019, at 4:43 AM, Thomas Schickentanz <thomas.schickentanz@harman.com> wrote: 

> Hi Mathieu,

> Thanks for clarifying the restrictions of ‘shm-path’ option!

> Our purpose is to use exclusively reserved memory for tracing. Our customers
> request a certain amount of free memory on the system for future development.
> In development stage chunks of this “free” memory are grabbed by the
> applications and tracing lacks memory. We want to allocate this “free” memory
> in an early boot stage to ensure that we get the amount of memory we need for
> tracing. E.g. if we have reserved 512 Mbyte of memory, we know that we can
> start lttng at a later point of time with 16 buffers of 4 Mbyte on a 8 core
> system. But this works only, if we can provide the previously reserved memory
> to lttng for buffer allocation.

> Is there some general concern about using POSIX shared memory for buffers in
> kernel tracing mode (apart from performance reasons)?

The kernel tracer does not use POSIX shared memory at all at the moment. Only lttng-ust does. lttng-modules allocates memory for its buffers 
directly from the kernel page allocator all by itself. 

Switching to a design where an application process (kernel consumerd) would map memory associated to files on a filesystem, and then 
hand over that memory to the kernel so it can trace there would be a rather large design change. 

For the needs you have, I think a simpler option would be to implement a new setup argument to lttng-modules so it can reserve some memory 
from the page allocator when the lttng modules are loaded (e.g. at system start), and then use that pre-reserved pool to allocate the buffers 
when a tracing session is created. All this would need to be NUMA-aware, with e.g. one pool per NUMA node. 

This would be a new feature request against lttng-modules. 

In order to reserve memory for lttng-ust, you could ensure your /dev/shm mountpoint is a tmpfs filesystem which contains enough memory 
to hold the lttng-ust buffers. An alternative there would be to use pmem + dax to hold the buffers with the --shm-path option. 

Thanks, 

Mathieu 

> I used lttng-crash only for testing, because I wanted to verify, if tracing data
> is available from provided shared memory. If this is ensured, I agree with you,
> that ‘lttng snapshot’ can do the work.

> Regards,

> Thomas

> From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> Sent: Montag, 4. März 2019 17:01
> To: Schickentanz, Thomas <thomas.schickentanz@harman.com>
> Cc: lttng-dev <lttng-dev@lists.lttng.org>
> Subject: [EXTERNAL] Re: [lttng-dev] Tracing directly into shared memory

> ----- On Mar 4, 2019, at 4:28 AM, Thomas Schickentanz < [
> mailto:thomas.schickentanz@harman.com | thomas.schickentanz@harman.com ] >
> wrote:

>> Hi all,

>> My idea is to pre-allocate memory on system startup which can later be used by
>> lttng for tracing. It should work like flight recorder mode without extracting
>> data from ringbuffers to trace file while recording data. Data extraction of
>> shm ringbuffers should be done when recording has been finished.

>> As per documention with option `--shm-path‘ ringbuffer for tracing can be
>> created in shared memory. With lttng-crash data can be extracted from
>> ringbuffer.

>> Can this option be used with shared memory in volatile memory (default ram),
>> too? And is it possible to record kernel space traces with this option?

>> I tried

>> lttng create testshm --shm-path=/dev/shm --snapshot

>> lttng enable-channel c0 -s testshm -k

>> lttng enable-event -s testshm -c c0 -k -a

>> lttng start

>> lttng stop

>> lttng-crash -extract=/home/test /dev/shm

>> With lttng and lttng-crash everything seems to work fine (no error messages).

>> However the extract folder keeps empty.

>> Has someone researched into this option and can help to clarify the points?
> Hi Thomas,

> About "--shm-path" and lttng-crash, those are discussed here: [
> https://clicktime.symantec.com/3YZ15pbJVSD47yeXxhXiwqE7Vc?u=https%3A%2F%2Flttng.org%2Fdocs%2Fv2.10%2F%23doc-persistent-memory-file-systems
> |
> https://lttng.org/docs/v2.10/#doc-persistent-memory-file-systems ]

> I'm not sure I completely understand your use-case and whether your intent is to
> extract trace data after a

> system crash or if the "snapshot" mode is enough for your needs.

> Note that the "shm-path" option only applies to UST (user-space) traces at the
> moment. It has not been implemented

> for kernel tracing yet.

> The way we typically use it for extraction of trace data from user-space when a
> system crashes is to use the pmem driver to

> create a block device out of a range of the system memory, and then use ext2 or
> ext4 with the DAX mount option to ensure

> the filesystem bypasses the page cache.

> Then we ensure that kexec is used to boot a new kernel on crash, or make sure
> the BIOS does not erase the

> memory content on soft reboot, which gives access to the shm-path buffers after
> crash/reboot.

> We don't usually use /dev/shm as an argument to --shm-path. This path is
> normally used to hold POSIX shared memory

> files. I would expect a different mount point path as argument to --shm-path
> which points to a DAX-enabled filesystem

> backed by a PMEM block device.

> Thanks,

> Mathieu

> --

> Mathieu Desnoyers
> EfficiOS Inc.
> [
> https://clicktime.symantec.com/32vnSPUsVCfHCkkwoyPymVx7Vc?u=http%3A%2F%2Fwww.efficios.com
> |
> https://clicktime.symantec.com/32vnSPUsVCfHCkkwoyPymVx7Vc?u=http%3A%2F%2Fwww.efficios.com
> ]

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 

[-- Attachment #1.2: Type: text/html, Size: 14407 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Tracing directly into shared memory

[Schickentanz, Thomas]

[-- Attachment #1.1: Type: text/plain, Size: 1050 bytes --]

Hi all,

My idea is to pre-allocate memory on system startup which can later be used by lttng for tracing. It should work like flight recorder mode without extracting data from ringbuffers to trace file while recording data. Data extraction of shm ringbuffers should be done when recording has been finished.
As per documention with option `--shm-path' ringbuffer for tracing can be created in shared memory. With lttng-crash data can be extracted from ringbuffer.
Can this option be used with shared memory in volatile memory (default ram), too? And is it possible to record kernel space traces with this option?
I tried

lttng create testshm --shm-path=/dev/shm --snapshot
lttng enable-channel c0 -s testshm -k
lttng enable-event -s testshm -c c0 -k -a
lttng start
lttng stop
lttng-crash -extract=/home/test /dev/shm

With lttng and lttng-crash everything seems to work fine (no error messages).
However the extract folder keeps empty.
Has someone researched into this option and can help to clarify the points?

Thanks,
Thomas

[-- Attachment #1.2: Type: text/html, Size: 5024 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev