Re: [dpdk-dev] [RFC] DPDK Trace support

["Mattias Rönnblom" <mattias.ronnblom@ericsson.com>]

On 2020-01-17 10:52, Jerin Jacob wrote:
> On Fri, Jan 17, 2020 at 1:35 PM David Marchand
> <david.marchand@redhat.com> wrote:
>> On Fri, Jan 17, 2020 at 5:41 AM Jerin Jacob <jerinjacobk@gmail.com> wrote:
>>>>>>> Yes this is when trace is enabled. If the trace is disabled then it
>>>>>>> will be the only a handful of cycles.
>>>>>>>
>>>>>> Two follow-on questions:
>>>>>> 1. Is the trace enable/disable dynamic at runtime?
>>>>> Yes. See the requirement section.
>>>>>
>>>>>> 2. Have you investigated how low the "handful of cycles" actually is?
>>>>> Yes. it is around 1 to 3 cycles based on the arch. it boils down to
>>>>> mostly a branch hit/miss on a memory location
>>>>> embedded in a C macro.
>>>>>
>>>> That seems impressively low, which is great news!
>>> Does anyone have an objection to have
>>> 1) Use CTF as trace format to reuse the opensource tracing tools and
>>> compatibility wth LTTng
>>> https://diamon.org/ctf/
>>> 2) Have native DPDK CTF trace emitter for better performance for DPDK
>>> fast path tracing and Non-Linux support.
>>>
>>> I would like to avoid the situation where once code gets completed and
>>> then starts our basic discussion
>>> on the design decisions.
>>>
>>> If someone needs more time to think through or any clarification is
>>> required then please discuss.
>> I did not find the time to look at this.
>> Some quick questions:
>> - is LTTng coming with out-of-tree kmod? making it hard to support in
>> distributions?
> LTTng kernel tracing only needs kmod support.
> For the userspace tracing at minium following libraries are required.
>
> a) LTTng-UST
> b) LTTng-tools
> c) liburcu
> d) libpopt-dev

This "DPDK CTF trace emitter" would make DPDK interoperate with, but 
without any build-time dependencies to, LTTng. Correct?

Do you have any idea of what the performance benefits one would receive 
from having something DPDK native, compared to just depending on LTTng UST?

Would this work also include moving over the DPDK trace macros to using 
this new CTF trace emitter? If so, we would retain the current 
printf()-style pattern, or move to a more LTTng-native like approach, 
with trace event type declarations and binary-format trace events?

> Based on the https://lttng.org/docs/v2.11/#doc-installing-lttng
> -------------------------- 8<----------------------------------
> Important:As of 22 October 2019, LTTng 2.11 is not available as
> distribution packages, except for Arch Linux.
> You can build LTTng 2.11 from source to install and use it.
> -------------------------- >8----------------------------------
>
>> - I have been playing with perf those days to track live processes and
>> gathering informations/stats at key point of a dpdk app without adding
>> anything in the binary. What does LTTng provide that scripting around
>> perf would not solve?
> Profiler and Tracer are two different things: Perf is a profiler.
>
> Definitions from https://lttng.org/docs/v2.11/#doc-what-is-tracing
> -------------------------- 8<----------------------------------
> A profiler is often the tool of choice to identify performance
> bottlenecks. Profiling is suitable to identify where performance is
> lost in a given software. The profiler outputs a profile, a
> statistical summary of observed events, which you may use to discover
> which functions took the most time to execute. However, a profiler
> won’t report why some identified functions are the bottleneck.
> Bottlenecks might only occur when specific conditions are met,
> conditions that are sometimes impossible to capture by a statistical
> profiler, or impossible to reproduce with an application altered by
> the overhead of an event-based profiler. For a thorough investigation
> of software performance issues, a history of execution is essential,
> with the recorded values of variables and context fields you choose,
> and with as little influence as possible on the instrumented software.
> This is where tracing comes in handy.
>
> Tracing is a technique used to understand what goes on in a running
> software system. The software used for tracing is called a tracer,
> which is conceptually similar to a tape recorder. When recording,
> specific instrumentation points placed in the software source code
> generate events that are saved on a giant tape: a trace file. You can
> trace user applications and the operating system at the same time,
> opening the possibility of resolving a wide range of problems that
> would otherwise be extremely challenging.
> -------------------------- >8----------------------------------
>
> Once tracing infrastructure is in place, we can add tracepoints in the
> dpdk functions such as rte_eth_dev_configure(), rx_burst, etc so
> that one can trace the flow of the program and debug. The use case
> details from the RFC:
>
> -------------------------- 8<----------------------------------
> Use-cases
> ---------
> - Most of the cases, The DPDK provider will not have access to the
> DPDK customer applications.
> To debug/analyze the slow path and fast path DPDK API usage from the field,
> we need to have integrated trace support in DPDK.
>
> - Need a low overhead Fast path multi-core PMD driver debugging/analysis
> infrastructure in DPDK to fix the functional and performance issue(s) of PMD.
>
> - Post trace analysis tools can provide various status across the system such
> as cpu_idle() using the timestamp added in the trace.
> -------------------------- >8----------------------------------
>
> Here is more details on viewing Traces using Trace compass(An
> opensource CTF trace viewer)
>
> https://www.renesas.com/cn/zh/doc/products/tool/doc/014/r20ut4479ej0000-lttng.pdf
>
>
>>
>> --
>> David Marchand
>>