LTTng user mode performance observation

LTTng user mode performance observation

[Richard Schmitt]

[-- Attachment #1.1: Type: text/plain, Size: 796 bytes --]

We are trying to use user space LTTng to perform some performance analysis of fast-path layer 3 forwarding.  We have seen that with the tracepoints enabled, we experience about a 30% increase in service time.  This is spread out over two tracepoints so each one is still contributing a good amount.

This fast path is fast, about 300 cycles so we are asking a lot of efficiency of the tracepoint but for comparison, we implemented a simple ring buffer.  When we run with this type of capture, we only experience a 5% increase in service time.

I’m looking for pointers.  We’d rather use LTTng.  Are there ways to tune LTTng ust for ultra high performance requirements.  Perhaps there is  a way we could write directly to ust’s ring buffers, or perhaps preallocate the buffers?

Rich

[-- Attachment #1.2: Type: text/html, Size: 2879 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: LTTng user mode performance observation

[Mathieu Desnoyers]

[-- Attachment #1.1: Type: text/plain, Size: 1636 bytes --]

---- On Jun 2, 2016, at 1:29 AM, Richard Schmitt <richard.schmitt@nxp.com> wrote: 

> We are trying to use user space LTTng to perform some performance analysis of
> fast-path layer 3 forwarding. We have seen that with the tracepoints enabled,
> we experience about a 30% increase in service time. This is spread out over two
> tracepoints so each one is still contributing a good amount.

> This fast path is fast, about 300 cycles so we are asking a lot of efficiency of
> the tracepoint but for comparison, we implemented a simple ring buffer. When we
> run with this type of capture, we only experience a 5% increase in service
> time.

> I’m looking for pointers. We’d rather use LTTng. Are there ways to tune LTTng
> ust for ultra high performance requirements. Perhaps there is a way we could
> write directly to ust’s ring buffers, or perhaps preallocate the buffers?

What is the target architecture ? 

I'm currently working on proposing restartable sequence system call to the kernel to 
improve the speed of getcpu and remove atomic operations from the fast path 
on arm32, intel 32/64. This will be portable to other architectures too. 

Make sure you run a kernel with sys_membarrier available, and that you have 
a recent liburcu 0.9.x. This should take care of removing memory barriers from 
the tracing fast path. 

Thanks, 

Mathieu 

> Rich

> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 

[-- Attachment #1.2: Type: text/html, Size: 4255 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: LTTng user mode performance observation

[Michel Dagenais]

[-- Attachment #1.1: Type: text/plain, Size: 1740 bytes --]

Here is a short answer to start. Others will have more precise suggestions. 

Something that can seriously affect the performance is what is used to obtain information such as the timestamp and possibly the CPU id. More details on your platform would help in that respect. The second thing is that UST is flexible (many parameters can be changed / activated at run time) and robust (reentrant, can be used from within signals) which adds a cost in terms of structure and memory barriers. A high performance mode with restrictions on its use has been discussed in the past but the speed difference is not necessarily that great, typically much lower than the impact of selecting an efficient source for timestamps. 

----- Mail original -----

> We are trying to use user space LTTng to perform some performance analysis of
> fast-path layer 3 forwarding. We have seen that with the tracepoints
> enabled, we experience about a 30% increase in service time. This is spread
> out over two tracepoints so each one is still contributing a good amount.

> This fast path is fast, about 300 cycles so we are asking a lot of efficiency
> of the tracepoint but for comparison, we implemented a simple ring buffer.
> When we run with this type of capture, we only experience a 5% increase in
> service time.

> I’m looking for pointers. We’d rather use LTTng. Are there ways to tune LTTng
> ust for ultra high performance requirements. Perhaps there is a way we could
> write directly to ust’s ring buffers, or perhaps preallocate the buffers?

> Rich

> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

[-- Attachment #1.2: Type: text/html, Size: 4005 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev