DPDK-dev Archive on lore.kernel.org
 help / color / Atom feed
From: Jerin Jacob <jerinjacobk@gmail.com>
To: David Marchand <david.marchand@redhat.com>
Cc: "Bruce Richardson" <bruce.richardson@intel.com>,
	"Jerin Jacob Kollanukkaran" <jerinj@marvell.com>,
	"dev@dpdk.org" <dev@dpdk.org>,
	"Thomas Monjalon" <thomas@monjalon.net>,
	"Ferruh Yigit" <ferruh.yigit@intel.com>,
	"Andrew Rybchenko" <arybchenko@solarflare.com>,
	"Ajit Khaparde" <ajit.khaparde@broadcom.com>,
	"Qi Zhang" <qi.z.zhang@intel.com>,
	"Xiaolong Ye" <xiaolong.ye@intel.com>,
	"Raslan Darawsheh" <rasland@mellanox.com>,
	"Maxime Coquelin" <maxime.coquelin@redhat.com>,
	"Tiwei Bie" <tiwei.bie@intel.com>,
	"Akhil Goyal" <akhil.goyal@nxp.com>,
	"Luca Boccassi" <bluca@debian.org>,
	"Kevin Traynor" <ktraynor@redhat.com>,
	"maintainers@dpdk.org" <maintainers@dpdk.org>,
	"John McNamara" <john.mcnamara@intel.com>,
	"Marko Kovacevic" <marko.kovacevic@intel.com>,
	"Ray Kinsella" <mdr@ashroe.eu>,
	"Aaron Conole" <aconole@redhat.com>,
	"Michael Santana" <maicolgabriel@hotmail.com>,
	"Harry van Haaren" <harry.van.haaren@intel.com>,
	"Cristian Dumitrescu" <cristian.dumitrescu@intel.com>,
	"Phil Yang" <phil.yang@arm.com>,
	"Joyce Kong" <joyce.kong@arm.com>,
	"Mattias Rönnblom" <mattias.ronnblom@ericsson.com>,
	"Jan Viktorin" <viktorin@rehivetech.com>,
	"Gavin Hu" <gavin.hu@arm.com>,
	"David Christensen" <drc@linux.vnet.ibm.com>,
	"Konstantin Ananyev" <konstantin.ananyev@intel.com>,
	"Anatoly Burakov" <anatoly.burakov@intel.com>,
	"Harini Ramakrishnan" <harini.ramakrishnan@microsoft.com>,
	"Omar Cardona" <ocardona@microsoft.com>,
	"Anand Rawat" <anand.rawat@intel.com>,
	"Olivier Matz" <olivier.matz@6wind.com>,
	"Gage Eads" <gage.eads@intel.com>,
	"Adrien Mazarguil" <adrien.mazarguil@6wind.com>,
	"Nicolas Chautru" <nicolas.chautru@intel.com>,
	"Declan Doherty" <declan.doherty@intel.com>,
	"Fiona Trahe" <fiona.trahe@intel.com>,
	"Ashish Gupta" <ashishg@marvell.com>,
	"Erik Gabriel Carrillo" <erik.g.carrillo@intel.com>,
	"Abhinandan Gujjar" <abhinandan.gujjar@intel.com>,
	"Hemant Agrawal" <hemant.agrawal@nxp.com>,
	"Artem V. Andreev" <artem.andreev@oktetlabs.ru>,
	"Nithin Kumar Dabilpuram" <ndabilpuram@marvell.com>,
	"Vamsi Krishna Attunuru" <vattunuru@marvell.com>,
	"Rosen Xu" <rosen.xu@intel.com>,
	"Sachin Saxena" <sachin.saxena@nxp.com>,
	"Stephen Hemminger" <sthemmin@microsoft.com>,
	"Chas Williams" <chas3@att.com>,
	"John W. Linville" <linville@tuxdriver.com>,
	"Prasun Kapoor" <pkapoor@marvell.com>,
	"Marcin Wojtas" <mw@semihalf.com>,
	"Michal Krawczyk" <mk@semihalf.com>,
	"Guy Tzalik" <gtzalik@amazon.com>,
	"Evgeny Schemeilin" <evgenys@amazon.com>,
	"Igor Chauskin" <igorch@amazon.com>,
	"Ravi Kumar" <ravi1.kumar@amd.com>,
	"Igor Russkikh" <igor.russkikh@aquantia.com>,
	"Pavel Belous" <pavel.belous@aquantia.com>,
	"Shepard Siegel" <shepard.siegel@atomicrules.com>,
	"Ed Czeck" <ed.czeck@atomicrules.com>,
	"John Miller" <john.miller@atomicrules.com>,
	"Somnath Kotur" <somnath.kotur@broadcom.com>,
	"Maciej Czekaj" <mczekaj@marvell.com>,
	"Shijith Thotton" <sthotton@marvell.com>,
	"Srisivasubramanian Srinivasan" <srinivasan@marvell.com>,
	"Rahul Lakkireddy" <rahul.lakkireddy@chelsio.com>,
	"John Daley" <johndale@cisco.com>,
	"Hyong Youb Kim" <hyonkim@cisco.com>,
	"Wei Hu (Xavier" <xavier.huwei@huawei.com>,
	"Min Hu (Connor" <humin29@huawei.com>,
	"Yisen Zhuang" <yisen.zhuang@huawei.com>,
	"Ziyang Xuan" <xuanziyang2@huawei.com>,
	"Xiaoyun Wang" <cloud.wangxiaoyun@huawei.com>,
	"Guoyang Zhou" <zhouguoyang@huawei.com>,
	"Beilei Xing" <beilei.xing@intel.com>,
	"Xiao Wang" <xiao.w.wang@intel.com>,
	"Jingjing Wu" <jingjing.wu@intel.com>,
	"Wenzhuo Lu" <wenzhuo.lu@intel.com>,
	"Qiming Yang" <qiming.yang@intel.com>,
	"Tomasz Duszynski" <tdu@semihalf.com>,
	"Liron Himi" <lironh@marvell.com>, "Zyta Szpak" <zr@semihalf.com>,
	"Kiran Kumar Kokkilagadda" <kirankumark@marvell.com>,
	"Matan Azrad" <matan@mellanox.com>,
	"Shahaf Shuler" <shahafs@mellanox.com>,
	"Viacheslav Ovsiienko" <viacheslavo@mellanox.com>,
	"K. Y. Srinivasan" <kys@microsoft.com>,
	"Haiyang Zhang" <haiyangz@microsoft.com>,
	"Jan Remes" <remes@netcope.com>,
	"Heinrich Kuhn" <heinrich.kuhn@netronome.com>,
	"Jan Gutter" <jan.gutter@netronome.com>,
	"Gagandeep Singh" <g.singh@nxp.com>,
	"Rasesh Mody" <rmody@marvell.com>,
	"Shahed Shaikh" <shshaikh@marvell.com>,
	"Yong Wang" <yongwang@vmware.com>,
	"Zhihong Wang" <zhihong.wang@intel.com>,
	"Steven Webster" <steven.webster@windriver.com>,
	"Matt Peters" <matt.peters@windriver.com>,
	"Keith Wiles" <keith.wiles@intel.com>,
	"Tetsuya Mukawa" <mtetsuyah@gmail.com>,
	"Jasvinder Singh" <jasvinder.singh@intel.com>,
	"Jakub Grajciar" <jgrajcia@cisco.com>,
	"Ruifeng Wang" <ruifeng.wang@arm.com>,
	"Anoob Joseph" <anoobj@marvell.com>,
	"Fan Zhang" <roy.fan.zhang@intel.com>,
	"Pablo de Lara" <pablo.de.lara.guarch@intel.com>,
	"John Griffin" <john.griffin@intel.com>,
	"Deepak Kumar Jain" <deepak.k.jain@intel.com>,
	"Michael Shamis" <michaelsh@marvell.com>,
	"Nagadheeraj Rottela" <rnagadheeraj@marvell.com>,
	"Srikanth Jampala" <jsrikanth@marvell.com>,
	"Ankur Dwivedi" <adwivedi@marvell.com>,
	"Jay Zhou" <jianjay.zhou@huawei.com>,
	"Lee Daly" <lee.daly@intel.com>,
	"Sunila Sahu" <ssahu@marvell.com>,
	"Nipun Gupta" <nipun.gupta@nxp.com>,
	"Liang Ma" <liang.j.ma@intel.com>,
	"Peter Mccarthy" <peter.mccarthy@intel.com>,
	"Tianfei zhang" <tianfei.zhang@intel.com>,
	"Satha Koteswara Rao Kottidi" <skoteshwar@marvell.com>,
	"Xiaoyun Li" <xiaoyun.li@intel.com>,
	"Bernard Iremonger" <bernard.iremonger@intel.com>,
	"Vladimir Medvedkin" <vladimir.medvedkin@intel.com>,
	"David Hunt" <david.hunt@intel.com>,
	"Reshma Pattan" <reshma.pattan@intel.com>,
	"Byron Marohn" <byron.marohn@intel.com>,
	"Sameh Gobriel" <sameh.gobriel@intel.com>,
	"Yipeng Wang" <yipeng1.wang@intel.com>,
	"Honnappa Nagarahalli" <honnappa.nagarahalli@arm.com>,
	"Robert Sanford" <rsanford@akamai.com>,
	"Kevin Laatz" <kevin.laatz@intel.com>,
	"Maryam Tahhan" <maryam.tahhan@intel.com>,
	"Ori Kam" <orika@mellanox.com>,
	"Radu Nicolau" <radu.nicolau@intel.com>,
	"Tomasz Kantecki" <tomasz.kantecki@intel.com>,
	"Sunil Kumar Kori" <skori@marvell.com>,
	"Pavan Nikhilesh Bhagavatula" <pbhagavatula@marvell.com>,
	"Kirill Rybalchenko" <kirill.rybalchenko@intel.com>,
	"Kadam, Pallavi" <pallavi.kadam@intel.com>
Subject: Re: [dpdk-dev] [RFC] DPDK Trace support
Date: Fri, 17 Jan 2020 15:22:05 +0530
Message-ID: <CALBAE1Ps0gfK+FS3j_ZBWVpC+VyzLZXd9Xj=r9x1oEoNrtc87A@mail.gmail.com> (raw)
In-Reply-To: <CAJFAV8y-nbJTz3o6ipz76fKg7417NqESxvrVEWPs+vEkyeLHTQ@mail.gmail.com>

On Fri, Jan 17, 2020 at 1:35 PM David Marchand
<david.marchand@redhat.com> wrote:
>
> On Fri, Jan 17, 2020 at 5:41 AM Jerin Jacob <jerinjacobk@gmail.com> wrote:
> >
> > > > > >
> > > > > > Yes this is when trace is enabled. If the trace is disabled then it
> > > > > > will be the only a handful of cycles.
> > > > > >
> > > > > Two follow-on questions:
> > > > > 1. Is the trace enable/disable dynamic at runtime?
> > > >
> > > > Yes. See the requirement section.
> > > >
> > > > > 2. Have you investigated how low the "handful of cycles" actually is?
> > > >
> > > > Yes. it is around 1 to 3 cycles based on the arch. it boils down to
> > > > mostly a branch hit/miss on a memory location
> > > > embedded in a C macro.
> > > >
> > > That seems impressively low, which is great news!
> >
> > Does anyone have an objection to have
> > 1) Use CTF as trace format to reuse the opensource tracing tools and
> > compatibility wth LTTng
> > https://diamon.org/ctf/
> > 2) Have native DPDK CTF trace emitter for better performance for DPDK
> > fast path tracing and Non-Linux support.
> >
> > I would like to avoid the situation where once code gets completed and
> > then starts our basic discussion
> > on the design decisions.
> >
> > If someone needs more time to think through or any clarification is
> > required then please discuss.
>
> I did not find the time to look at this.
> Some quick questions:
> - is LTTng coming with out-of-tree kmod? making it hard to support in
> distributions?

LTTng kernel tracing only needs kmod support.
For the userspace tracing at minium following libraries are required.

a) LTTng-UST
b) LTTng-tools
c) liburcu
d) libpopt-dev

Based on the https://lttng.org/docs/v2.11/#doc-installing-lttng
-------------------------- 8<----------------------------------
Important:As of 22 October 2019, LTTng 2.11 is not available as
distribution packages, except for Arch Linux.
You can build LTTng 2.11 from source to install and use it.
-------------------------- >8----------------------------------

> - I have been playing with perf those days to track live processes and
> gathering informations/stats at key point of a dpdk app without adding
> anything in the binary. What does LTTng provide that scripting around
> perf would not solve?

Profiler and Tracer are two different things: Perf is a profiler.

Definitions from https://lttng.org/docs/v2.11/#doc-what-is-tracing
-------------------------- 8<----------------------------------
A profiler is often the tool of choice to identify performance
bottlenecks. Profiling is suitable to identify where performance is
lost in a given software. The profiler outputs a profile, a
statistical summary of observed events, which you may use to discover
which functions took the most time to execute. However, a profiler
won’t report why some identified functions are the bottleneck.
Bottlenecks might only occur when specific conditions are met,
conditions that are sometimes impossible to capture by a statistical
profiler, or impossible to reproduce with an application altered by
the overhead of an event-based profiler. For a thorough investigation
of software performance issues, a history of execution is essential,
with the recorded values of variables and context fields you choose,
and with as little influence as possible on the instrumented software.
This is where tracing comes in handy.

Tracing is a technique used to understand what goes on in a running
software system. The software used for tracing is called a tracer,
which is conceptually similar to a tape recorder. When recording,
specific instrumentation points placed in the software source code
generate events that are saved on a giant tape: a trace file. You can
trace user applications and the operating system at the same time,
opening the possibility of resolving a wide range of problems that
would otherwise be extremely challenging.
-------------------------- >8----------------------------------

Once tracing infrastructure is in place, we can add tracepoints in the
dpdk functions such as rte_eth_dev_configure(), rx_burst, etc so
that one can trace the flow of the program and debug. The use case
details from the RFC:

-------------------------- 8<----------------------------------
Use-cases
---------
- Most of the cases, The DPDK provider will not have access to the
DPDK customer applications.
To debug/analyze the slow path and fast path DPDK API usage from the field,
we need to have integrated trace support in DPDK.

- Need a low overhead Fast path multi-core PMD driver debugging/analysis
infrastructure in DPDK to fix the functional and performance issue(s) of PMD.

- Post trace analysis tools can provide various status across the system such
as cpu_idle() using the timestamp added in the trace.
-------------------------- >8----------------------------------

Here is more details on viewing Traces using Trace compass(An
opensource CTF trace viewer)

https://www.renesas.com/cn/zh/doc/products/tool/doc/014/r20ut4479ej0000-lttng.pdf


>
>
> --
> David Marchand
>

  reply index

Thread overview: 24+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-01-13 10:40 Jerin Jacob Kollanukkaran
2020-01-13 11:00 ` Ray Kinsella
2020-01-13 12:04   ` [dpdk-dev] [EXT] " Jerin Jacob Kollanukkaran
2020-01-18 15:14   ` [dpdk-dev] " dave
2020-01-20 16:51     ` Stephen Hemminger
2020-01-13 13:05 ` Bruce Richardson
2020-01-13 14:46   ` Jerin Jacob
2020-01-13 14:58     ` Bruce Richardson
2020-01-13 15:13       ` Jerin Jacob
2020-01-13 16:12         ` Bruce Richardson
2020-01-17  4:41           ` Jerin Jacob
2020-01-17  8:04             ` David Marchand
2020-01-17  9:52               ` Jerin Jacob [this message]
2020-01-17 10:30                 ` Mattias Rönnblom
2020-01-17 10:54                   ` Jerin Jacob
2020-02-15 10:21                     ` Jerin Jacob
2020-02-17  9:35                       ` Mattias Rönnblom
2020-02-17 10:23                         ` Jerin Jacob
2020-01-17 10:43                 ` David Marchand
2020-01-17 11:08                   ` Jerin Jacob
2020-01-27 16:12 ` Aaron Conole
2020-01-27 17:23   ` Jerin Jacob
2020-01-20  4:48 Jerin Jacob Kollanukkaran
2020-01-20 12:08 ` Ray Kinsella

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CALBAE1Ps0gfK+FS3j_ZBWVpC+VyzLZXd9Xj=r9x1oEoNrtc87A@mail.gmail.com' \
    --to=jerinjacobk@gmail.com \
    --cc=abhinandan.gujjar@intel.com \
    --cc=aconole@redhat.com \
    --cc=adrien.mazarguil@6wind.com \
    --cc=adwivedi@marvell.com \
    --cc=ajit.khaparde@broadcom.com \
    --cc=akhil.goyal@nxp.com \
    --cc=anand.rawat@intel.com \
    --cc=anatoly.burakov@intel.com \
    --cc=anoobj@marvell.com \
    --cc=artem.andreev@oktetlabs.ru \
    --cc=arybchenko@solarflare.com \
    --cc=ashishg@marvell.com \
    --cc=beilei.xing@intel.com \
    --cc=bernard.iremonger@intel.com \
    --cc=bluca@debian.org \
    --cc=bruce.richardson@intel.com \
    --cc=byron.marohn@intel.com \
    --cc=chas3@att.com \
    --cc=cloud.wangxiaoyun@huawei.com \
    --cc=cristian.dumitrescu@intel.com \
    --cc=david.hunt@intel.com \
    --cc=david.marchand@redhat.com \
    --cc=declan.doherty@intel.com \
    --cc=deepak.k.jain@intel.com \
    --cc=dev@dpdk.org \
    --cc=drc@linux.vnet.ibm.com \
    --cc=ed.czeck@atomicrules.com \
    --cc=erik.g.carrillo@intel.com \
    --cc=evgenys@amazon.com \
    --cc=ferruh.yigit@intel.com \
    --cc=fiona.trahe@intel.com \
    --cc=g.singh@nxp.com \
    --cc=gage.eads@intel.com \
    --cc=gavin.hu@arm.com \
    --cc=gtzalik@amazon.com \
    --cc=haiyangz@microsoft.com \
    --cc=harini.ramakrishnan@microsoft.com \
    --cc=harry.van.haaren@intel.com \
    --cc=heinrich.kuhn@netronome.com \
    --cc=hemant.agrawal@nxp.com \
    --cc=honnappa.nagarahalli@arm.com \
    --cc=humin29@huawei.com \
    --cc=hyonkim@cisco.com \
    --cc=igor.russkikh@aquantia.com \
    --cc=igorch@amazon.com \
    --cc=jan.gutter@netronome.com \
    --cc=jasvinder.singh@intel.com \
    --cc=jerinj@marvell.com \
    --cc=jgrajcia@cisco.com \
    --cc=jianjay.zhou@huawei.com \
    --cc=jingjing.wu@intel.com \
    --cc=john.griffin@intel.com \
    --cc=john.mcnamara@intel.com \
    --cc=john.miller@atomicrules.com \
    --cc=johndale@cisco.com \
    --cc=joyce.kong@arm.com \
    --cc=jsrikanth@marvell.com \
    --cc=keith.wiles@intel.com \
    --cc=kevin.laatz@intel.com \
    --cc=kirankumark@marvell.com \
    --cc=kirill.rybalchenko@intel.com \
    --cc=konstantin.ananyev@intel.com \
    --cc=ktraynor@redhat.com \
    --cc=kys@microsoft.com \
    --cc=lee.daly@intel.com \
    --cc=liang.j.ma@intel.com \
    --cc=linville@tuxdriver.com \
    --cc=lironh@marvell.com \
    --cc=maicolgabriel@hotmail.com \
    --cc=maintainers@dpdk.org \
    --cc=marko.kovacevic@intel.com \
    --cc=maryam.tahhan@intel.com \
    --cc=matan@mellanox.com \
    --cc=matt.peters@windriver.com \
    --cc=mattias.ronnblom@ericsson.com \
    --cc=maxime.coquelin@redhat.com \
    --cc=mczekaj@marvell.com \
    --cc=mdr@ashroe.eu \
    --cc=michaelsh@marvell.com \
    --cc=mk@semihalf.com \
    --cc=mtetsuyah@gmail.com \
    --cc=mw@semihalf.com \
    --cc=ndabilpuram@marvell.com \
    --cc=nicolas.chautru@intel.com \
    --cc=nipun.gupta@nxp.com \
    --cc=ocardona@microsoft.com \
    --cc=olivier.matz@6wind.com \
    --cc=orika@mellanox.com \
    --cc=pablo.de.lara.guarch@intel.com \
    --cc=pallavi.kadam@intel.com \
    --cc=pavel.belous@aquantia.com \
    --cc=pbhagavatula@marvell.com \
    --cc=peter.mccarthy@intel.com \
    --cc=phil.yang@arm.com \
    --cc=pkapoor@marvell.com \
    --cc=qi.z.zhang@intel.com \
    --cc=qiming.yang@intel.com \
    --cc=radu.nicolau@intel.com \
    --cc=rahul.lakkireddy@chelsio.com \
    --cc=rasland@mellanox.com \
    --cc=ravi1.kumar@amd.com \
    --cc=remes@netcope.com \
    --cc=reshma.pattan@intel.com \
    --cc=rmody@marvell.com \
    --cc=rnagadheeraj@marvell.com \
    --cc=rosen.xu@intel.com \
    --cc=roy.fan.zhang@intel.com \
    --cc=rsanford@akamai.com \
    --cc=ruifeng.wang@arm.com \
    --cc=sachin.saxena@nxp.com \
    --cc=sameh.gobriel@intel.com \
    --cc=shahafs@mellanox.com \
    --cc=shepard.siegel@atomicrules.com \
    --cc=shshaikh@marvell.com \
    --cc=skori@marvell.com \
    --cc=skoteshwar@marvell.com \
    --cc=somnath.kotur@broadcom.com \
    --cc=srinivasan@marvell.com \
    --cc=ssahu@marvell.com \
    --cc=steven.webster@windriver.com \
    --cc=sthemmin@microsoft.com \
    --cc=sthotton@marvell.com \
    --cc=tdu@semihalf.com \
    --cc=thomas@monjalon.net \
    --cc=tianfei.zhang@intel.com \
    --cc=tiwei.bie@intel.com \
    --cc=tomasz.kantecki@intel.com \
    --cc=vattunuru@marvell.com \
    --cc=viacheslavo@mellanox.com \
    --cc=viktorin@rehivetech.com \
    --cc=vladimir.medvedkin@intel.com \
    --cc=wenzhuo.lu@intel.com \
    --cc=xavier.huwei@huawei.com \
    --cc=xiao.w.wang@intel.com \
    --cc=xiaolong.ye@intel.com \
    --cc=xiaoyun.li@intel.com \
    --cc=xuanziyang2@huawei.com \
    --cc=yipeng1.wang@intel.com \
    --cc=yisen.zhuang@huawei.com \
    --cc=yongwang@vmware.com \
    --cc=zhihong.wang@intel.com \
    --cc=zhouguoyang@huawei.com \
    --cc=zr@semihalf.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

DPDK-dev Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/dpdk-dev/0 dpdk-dev/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 dpdk-dev dpdk-dev/ https://lore.kernel.org/dpdk-dev \
		dev@dpdk.org
	public-inbox-index dpdk-dev

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.dpdk.dev


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git