DPDK-dev Archive on lore.kernel.org
 help / color / Atom feed
From: Jerin Jacob <jerinjacobk@gmail.com>
To: "Mattias Rönnblom" <mattias.ronnblom@ericsson.com>
Cc: David Marchand <david.marchand@redhat.com>,
	 Bruce Richardson <bruce.richardson@intel.com>,
	Jerin Jacob Kollanukkaran <jerinj@marvell.com>,
	 "dev@dpdk.org" <dev@dpdk.org>,
	Thomas Monjalon <thomas@monjalon.net>,
	 Ferruh Yigit <ferruh.yigit@intel.com>,
	Andrew Rybchenko <arybchenko@solarflare.com>,
	 Ajit Khaparde <ajit.khaparde@broadcom.com>,
	Qi Zhang <qi.z.zhang@intel.com>,
	 Xiaolong Ye <xiaolong.ye@intel.com>,
	Raslan Darawsheh <rasland@mellanox.com>,
	 Maxime Coquelin <maxime.coquelin@redhat.com>,
	Tiwei Bie <tiwei.bie@intel.com>,
	 Akhil Goyal <akhil.goyal@nxp.com>,
	Luca Boccassi <bluca@debian.org>,
	 Kevin Traynor <ktraynor@redhat.com>,
	"maintainers@dpdk.org" <maintainers@dpdk.org>,
	 John McNamara <john.mcnamara@intel.com>,
	Marko Kovacevic <marko.kovacevic@intel.com>,
	 Ray Kinsella <mdr@ashroe.eu>, Aaron Conole <aconole@redhat.com>,
	 Michael Santana <maicolgabriel@hotmail.com>,
	Harry van Haaren <harry.van.haaren@intel.com>,
	 Cristian Dumitrescu <cristian.dumitrescu@intel.com>,
	Phil Yang <phil.yang@arm.com>,  Joyce Kong <joyce.kong@arm.com>,
	Jan Viktorin <viktorin@rehivetech.com>,
	 Gavin Hu <gavin.hu@arm.com>,
	David Christensen <drc@linux.vnet.ibm.com>,
	 Konstantin Ananyev <konstantin.ananyev@intel.com>,
	Anatoly Burakov <anatoly.burakov@intel.com>,
	 Harini Ramakrishnan <harini.ramakrishnan@microsoft.com>,
	Omar Cardona <ocardona@microsoft.com>,
	 Anand Rawat <anand.rawat@intel.com>,
	Olivier Matz <olivier.matz@6wind.com>,
	 Gage Eads <gage.eads@intel.com>,
	Adrien Mazarguil <adrien.mazarguil@6wind.com>,
	Nicolas Chautru <nicolas.chautru@intel.com>,
	Declan Doherty <declan.doherty@intel.com>,
	 Fiona Trahe <fiona.trahe@intel.com>,
	Ashish Gupta <ashishg@marvell.com>,
	 Erik Gabriel Carrillo <erik.g.carrillo@intel.com>,
	Abhinandan Gujjar <abhinandan.gujjar@intel.com>,
	 Hemant Agrawal <hemant.agrawal@nxp.com>,
	"Artem V. Andreev" <artem.andreev@oktetlabs.ru>,
	 Nithin Kumar Dabilpuram <ndabilpuram@marvell.com>,
	Vamsi Krishna Attunuru <vattunuru@marvell.com>,
	 Rosen Xu <rosen.xu@intel.com>,
	Sachin Saxena <sachin.saxena@nxp.com>,
	 Stephen Hemminger <sthemmin@microsoft.com>,
	Chas Williams <chas3@att.com>,
	 "John W. Linville" <linville@tuxdriver.com>,
	Prasun Kapoor <pkapoor@marvell.com>,
	 Marcin Wojtas <mw@semihalf.com>,
	Michal Krawczyk <mk@semihalf.com>,
	Guy Tzalik <gtzalik@amazon.com>,
	 Evgeny Schemeilin <evgenys@amazon.com>,
	Igor Chauskin <igorch@amazon.com>,
	Ravi Kumar <ravi1.kumar@amd.com>,
	 Igor Russkikh <igor.russkikh@aquantia.com>,
	Pavel Belous <pavel.belous@aquantia.com>,
	 Shepard Siegel <shepard.siegel@atomicrules.com>,
	Ed Czeck <ed.czeck@atomicrules.com>,
	 John Miller <john.miller@atomicrules.com>,
	Somnath Kotur <somnath.kotur@broadcom.com>,
	 Maciej Czekaj <mczekaj@marvell.com>,
	Shijith Thotton <sthotton@marvell.com>,
	 Srisivasubramanian Srinivasan <srinivasan@marvell.com>,
	Rahul Lakkireddy <rahul.lakkireddy@chelsio.com>,
	 John Daley <johndale@cisco.com>,
	Hyong Youb Kim <hyonkim@cisco.com>,
	 "Wei Hu (Xavier" <xavier.huwei@huawei.com>,
	"Min Hu (Connor" <humin29@huawei.com>,
	 Yisen Zhuang <yisen.zhuang@huawei.com>,
	Ziyang Xuan <xuanziyang2@huawei.com>,
	 Xiaoyun Wang <cloud.wangxiaoyun@huawei.com>,
	Guoyang Zhou <zhouguoyang@huawei.com>,
	 Beilei Xing <beilei.xing@intel.com>,
	Xiao Wang <xiao.w.wang@intel.com>,
	 Jingjing Wu <jingjing.wu@intel.com>,
	Wenzhuo Lu <wenzhuo.lu@intel.com>,
	 Qiming Yang <qiming.yang@intel.com>,
	Tomasz Duszynski <tdu@semihalf.com>,
	 Liron Himi <lironh@marvell.com>, Zyta Szpak <zr@semihalf.com>,
	 Kiran Kumar Kokkilagadda <kirankumark@marvell.com>,
	Matan Azrad <matan@mellanox.com>,
	Shahaf Shuler <shahafs@mellanox.com>,
	Viacheslav Ovsiienko <viacheslavo@mellanox.com>,
	 "K. Y. Srinivasan" <kys@microsoft.com>,
	Haiyang Zhang <haiyangz@microsoft.com>,
	Jan Remes <remes@netcope.com>,
	Heinrich Kuhn <heinrich.kuhn@netronome.com>,
	 Jan Gutter <jan.gutter@netronome.com>,
	Gagandeep Singh <g.singh@nxp.com>,
	 Rasesh Mody <rmody@marvell.com>,
	Shahed Shaikh <shshaikh@marvell.com>,
	 Yong Wang <yongwang@vmware.com>,
	Zhihong Wang <zhihong.wang@intel.com>,
	 Steven Webster <steven.webster@windriver.com>,
	Matt Peters <matt.peters@windriver.com>,
	 Keith Wiles <keith.wiles@intel.com>,
	Tetsuya Mukawa <mtetsuyah@gmail.com>,
	 Jasvinder Singh <jasvinder.singh@intel.com>,
	Jakub Grajciar <jgrajcia@cisco.com>,
	 Ruifeng Wang <ruifeng.wang@arm.com>,
	Anoob Joseph <anoobj@marvell.com>,
	 Fan Zhang <roy.fan.zhang@intel.com>,
	Pablo de Lara <pablo.de.lara.guarch@intel.com>,
	 John Griffin <john.griffin@intel.com>,
	Deepak Kumar Jain <deepak.k.jain@intel.com>,
	 Michael Shamis <michaelsh@marvell.com>,
	Nagadheeraj Rottela <rnagadheeraj@marvell.com>,
	 Srikanth Jampala <jsrikanth@marvell.com>,
	Ankur Dwivedi <adwivedi@marvell.com>,
	Jay Zhou <jianjay.zhou@huawei.com>, Lee Daly <lee.daly@intel.com>,
	 Sunila Sahu <ssahu@marvell.com>,
	Nipun Gupta <nipun.gupta@nxp.com>,
	Liang Ma <liang.j.ma@intel.com>,
	 Peter Mccarthy <peter.mccarthy@intel.com>,
	Tianfei zhang <tianfei.zhang@intel.com>,
	 Satha Koteswara Rao Kottidi <skoteshwar@marvell.com>,
	Xiaoyun Li <xiaoyun.li@intel.com>,
	 Bernard Iremonger <bernard.iremonger@intel.com>,
	 Vladimir Medvedkin <vladimir.medvedkin@intel.com>,
	David Hunt <david.hunt@intel.com>,
	 Reshma Pattan <reshma.pattan@intel.com>,
	Byron Marohn <byron.marohn@intel.com>,
	Sameh Gobriel <sameh.gobriel@intel.com>,
	Yipeng Wang <yipeng1.wang@intel.com>,
	 Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>,
	Robert Sanford <rsanford@akamai.com>,
	 Kevin Laatz <kevin.laatz@intel.com>,
	Maryam Tahhan <maryam.tahhan@intel.com>,
	 Ori Kam <orika@mellanox.com>,
	Radu Nicolau <radu.nicolau@intel.com>,
	 Tomasz Kantecki <tomasz.kantecki@intel.com>,
	Sunil Kumar Kori <skori@marvell.com>,
	 Pavan Nikhilesh Bhagavatula <pbhagavatula@marvell.com>,
	 Kirill Rybalchenko <kirill.rybalchenko@intel.com>,
	"Kadam, Pallavi" <pallavi.kadam@intel.com>,
	dave@barachs.net
Subject: Re: [dpdk-dev] [RFC] DPDK Trace support
Date: Sat, 15 Feb 2020 15:51:53 +0530
Message-ID: <CALBAE1NPp2RhBO+--37cX+k7MxstoknNSDzdKC8sRmbfrmfRyA@mail.gmail.com> (raw)
In-Reply-To: <CALBAE1P-D--BNV5yMNeGa0AWTrwN=GHQRhMnbvteG0vtOnBB4g@mail.gmail.com>

On Fri, Jan 17, 2020 at 4:24 PM Jerin Jacob <jerinjacobk@gmail.com> wrote:
>
> On Fri, Jan 17, 2020 at 4:00 PM Mattias Rönnblom
> <mattias.ronnblom@ericsson.com> wrote:
> >
>
> > > LTTng kernel tracing only needs kmod support.
> > > For the userspace tracing at minium following libraries are required.
> > >
> > > a) LTTng-UST
> > > b) LTTng-tools
> > > c) liburcu
> > > d) libpopt-dev
> >
> > This "DPDK CTF trace emitter" would make DPDK interoperate with, but
> > without any build-time dependencies to, LTTng. Correct?
>
> Yes. Native CTF trace emitter without LTTng dependency.
>
> >
> > Do you have any idea of what the performance benefits one would receive
> > from having something DPDK native, compared to just depending on LTTng UST?
>
> I calibrated LTTng cost and pushed the test code to github[1].
>
> I just started working on the DPDK native CTF emitter.
> I am sure it will be less than LTTng as we can leverage hugepage, exploit
> dpdk worker thread usage to avoid atomics and use per core variables,
> avoid a lot function pointers in fast-path etc
> I can share the exact overhead after the PoC.

I did the almost feature-complete PoC. The code shared in Github[1]
The documentation and code cleanup etc is still pending.

[1]
https://github.com/jerinjacobk/dpdk-trace.git

trace overhead data on x86:[2]
# 236 cyles with LTTng(>100ns)
# 18 cycles(7ns) with Native DPDK CTF emitter.

trace overhead data on arm64:
#  312  cycles to  1100 cycles with LTTng based on the class of arm64 CPU.
#  11 cycles to 13 cycles with Native DPDK CTF emitter based on the
class of arm64 CPU.

18 cycles(on x86) vs 11 cycles(on arm64) is due to rdtsc() overhead in
x86. It seems  rdtsc takes around 15cycles in x86.

# The Native DPDK CTF trace support does not have any dependency on
third-party library.
The generated output file is compatible with LTTng as both are using
CTF trace format.

The performance gain comes from:
1) exploit dpdk worker thread usage model to avoid atomics and use per
core variables
2) use hugepage,
3) avoid a lot function pointers in fast-path etc
4) avoid unaligned store for arm64 etc

Features:
- APIs and Features are similar to rte_log dynamic framework
API(expect log prints on stdout vs it dumps on trace file)
- No specific limit on the events. A string-based event like rte_log
for pattern matching
- Dynamic enable/disable support.
- Instructmention overhead is ~1 cycle. i.e cost of adding the code
wth out using trace feature.
- Timestamp support for all the events using DPDK rte_rtdsc
- No dependency on another library. Cleanroom native implementation of CTF.

Functional test case:
a) echo "trace_autotest" | sudo ./build/app/test/dpdk-test  -c 0x3
--trace-level=8

The above command emits the following trace events
<code>
        uint8_t i;

        rte_trace_lib_eal_generic_void();
        rte_trace_lib_eal_generic_u64(0x10000000000000);
        rte_trace_lib_eal_generic_u32(0x10000000);
        rte_trace_lib_eal_generic_u16(0xffee);
        rte_trace_lib_eal_generic_u8(0xc);
        rte_trace_lib_eal_generic_i64(-1234);
        rte_trace_lib_eal_generic_i32(-1234567);
        rte_trace_lib_eal_generic_i16(12);
        rte_trace_lib_eal_generic_i8(-3);
        rte_trace_lib_eal_generic_string("my string");
        rte_trace_lib_eal_generic_function(__func__);

        for (i = 0; i < 128; i++)
                rte_trace_lib_eal_generic_u8(i);
</code>

Install babeltrace package in Linux and point the generated trace file
to babel trace. By default trace file created under
<user>/dpdk-traces/time_stamp/

example:
# babeltrace /root/dpdk-traces/rte-2020-02-15-PM-02-56-51 | more

[13:27:36.138468807] (+?.?????????) lib.eal.generic.void: { cpu_id =
0, name = "dpdk-test" }, { }
[13:27:36.138468851] (+0.000000044) lib.eal.generic.u64: { cpu_id = 0,
name = "dpdk-test" }, { in = 4503599627370496 }
[13:27:36.138468860] (+0.000000009) lib.eal.generic.u32: { cpu_id = 0,
name = "dpdk-test" }, { in = 268435456 }
[13:27:36.138468934] (+0.000000074) lib.eal.generic.u16: { cpu_id = 0,
name = "dpdk-test" }, { in = 65518 }
[13:27:36.138468949] (+0.000000015) lib.eal.generic.u8: { cpu_id = 0,
name = "dpdk-test" }, { in = 12 }
[13:27:36.138468956] (+0.000000007) lib.eal.generic.i64: { cpu_id = 0,
name = "dpdk-test" }, { in = -1234 }
[13:27:36.138468963] (+0.000000007) lib.eal.generic.i32: { cpu_id = 0,
name = "dpdk-test" }, { in = -1234567 }
[13:27:36.138469024] (+0.000000061) lib.eal.generic.i16: { cpu_id = 0,
name = "dpdk-test" }, { in = 12 }
[13:27:36.138469044] (+0.000000020) lib.eal.generic.i8: { cpu_id = 0,
name = "dpdk-test" }, { in = -3 }
[13:27:36.138469051] (+0.000000007) lib.eal.generic.string: { cpu_id =
0, name = "dpdk-test" }, { str = "my string" }
[13:27:36.138469203] (+0.000000152) lib.eal.generic.func: { cpu_id =
0, name = "dpdk-test" }, { func = "test_trace_points" }
[13:27:36.138469239] (+0.000000036) lib.eal.generic.u8: { cpu_id = 0,
name = "dpdk-test" }, { in = 0 }
[13:27:36.138469246] (+0.000000007) lib.eal.generic.u8: { cpu_id = 0,
name = "dpdk-test" }, { in = 1 }
[13:27:36.138469252] (+0.000000006) lib.eal.generic.u8: { cpu_id = 0,
name = "dpdk-test" }, { in = 2 }
[13:27:36.138469262] (+0.000000010) lib.eal.generic.u8: { cpu_id = 0,
name = "dpdk-test" }, { in = 3 }
[13:27:36.138469269] (+0.000000007) lib.eal.generic.u8: { cpu_id = 0,
name = "dpdk-test" }, { in = 4 }
[13:27:36.138469276] (+0.000000007) lib.eal.generic.u8: { cpu_id = 0,
name = "dpdk-test" }, { in = 5 }

# There is a  GUI based trace viewer available in Windows, Linux and Mac.
It is called as tracecompass.(https://www.eclipse.org/tracecompass/)

The example screenshot and Histogram of above DPDK trace using Tracecompass.

https://github.com/jerinjacobk/share/blob/master/dpdk_trace.JPG


[2] Added performance test case to find the Trace overhead.
Command to test trace overhead. It is the overhead of writing
zero-argument trace.

echo "trace_perf" | sudo ./build/app/test/dpdk-test  -c 0x3 --trace-level=8
EAL: Detected 56 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Trace dir: /root/dpdk-traces/rte-2020-02-15-PM-03-37-33
RTE>>trace_perf
Timer running at 2600.00MHz
ZERO_ARG: cycles=17.901031 ns=6.885012
Test OK

[3] The above test is ported to LTTng for finding the LTTng trace
overhead. It available at
https://github.com/jerinjacobk/lttng-overhead
https://github.com/jerinjacobk/lttng-overhead/blob/master/README

File walkthrough:

lib/librte_eal/common/include/rte_trace.h - Public API for Trace
provider and Trace control
lib/librte_eal/common/eal_common_trace.c - main trace implemention
lib/librte_eal/common/eal_common_trace_ctf.c - CTF metadata spec implementation
lib/librte_eal/common/eal_common_trace_utils.c - command line utils
and filesystem operations.
lib/librte_eal/common/eal_common_trace_points.c -  trace points for EAL library
lib/librte_eal/common/include/rte_trace_eal.h - EAL tracepoint public API.
lib/librte_eal/common/eal_trace.h - Private trace header file.


> I think, based on the performance we can decide one or another?

The above performance data shows a much higher improvement with LTTng.

Let me know if anyone have any comments on this and or any suggestion.
If there are no comments, I will submit the v1 with after code clean
up before the 20.05 proposal deadline.

  reply index

Thread overview: 24+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-01-13 10:40 Jerin Jacob Kollanukkaran
2020-01-13 11:00 ` Ray Kinsella
2020-01-13 12:04   ` [dpdk-dev] [EXT] " Jerin Jacob Kollanukkaran
2020-01-18 15:14   ` [dpdk-dev] " dave
2020-01-20 16:51     ` Stephen Hemminger
2020-01-13 13:05 ` Bruce Richardson
2020-01-13 14:46   ` Jerin Jacob
2020-01-13 14:58     ` Bruce Richardson
2020-01-13 15:13       ` Jerin Jacob
2020-01-13 16:12         ` Bruce Richardson
2020-01-17  4:41           ` Jerin Jacob
2020-01-17  8:04             ` David Marchand
2020-01-17  9:52               ` Jerin Jacob
2020-01-17 10:30                 ` Mattias Rönnblom
2020-01-17 10:54                   ` Jerin Jacob
2020-02-15 10:21                     ` Jerin Jacob [this message]
2020-02-17  9:35                       ` Mattias Rönnblom
2020-02-17 10:23                         ` Jerin Jacob
2020-01-17 10:43                 ` David Marchand
2020-01-17 11:08                   ` Jerin Jacob
2020-01-27 16:12 ` Aaron Conole
2020-01-27 17:23   ` Jerin Jacob
2020-01-20  4:48 Jerin Jacob Kollanukkaran
2020-01-20 12:08 ` Ray Kinsella

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CALBAE1NPp2RhBO+--37cX+k7MxstoknNSDzdKC8sRmbfrmfRyA@mail.gmail.com \
    --to=jerinjacobk@gmail.com \
    --cc=abhinandan.gujjar@intel.com \
    --cc=aconole@redhat.com \
    --cc=adrien.mazarguil@6wind.com \
    --cc=adwivedi@marvell.com \
    --cc=ajit.khaparde@broadcom.com \
    --cc=akhil.goyal@nxp.com \
    --cc=anand.rawat@intel.com \
    --cc=anatoly.burakov@intel.com \
    --cc=anoobj@marvell.com \
    --cc=artem.andreev@oktetlabs.ru \
    --cc=arybchenko@solarflare.com \
    --cc=ashishg@marvell.com \
    --cc=beilei.xing@intel.com \
    --cc=bernard.iremonger@intel.com \
    --cc=bluca@debian.org \
    --cc=bruce.richardson@intel.com \
    --cc=byron.marohn@intel.com \
    --cc=chas3@att.com \
    --cc=cloud.wangxiaoyun@huawei.com \
    --cc=cristian.dumitrescu@intel.com \
    --cc=dave@barachs.net \
    --cc=david.hunt@intel.com \
    --cc=david.marchand@redhat.com \
    --cc=declan.doherty@intel.com \
    --cc=deepak.k.jain@intel.com \
    --cc=dev@dpdk.org \
    --cc=drc@linux.vnet.ibm.com \
    --cc=ed.czeck@atomicrules.com \
    --cc=erik.g.carrillo@intel.com \
    --cc=evgenys@amazon.com \
    --cc=ferruh.yigit@intel.com \
    --cc=fiona.trahe@intel.com \
    --cc=g.singh@nxp.com \
    --cc=gage.eads@intel.com \
    --cc=gavin.hu@arm.com \
    --cc=gtzalik@amazon.com \
    --cc=haiyangz@microsoft.com \
    --cc=harini.ramakrishnan@microsoft.com \
    --cc=harry.van.haaren@intel.com \
    --cc=heinrich.kuhn@netronome.com \
    --cc=hemant.agrawal@nxp.com \
    --cc=honnappa.nagarahalli@arm.com \
    --cc=humin29@huawei.com \
    --cc=hyonkim@cisco.com \
    --cc=igor.russkikh@aquantia.com \
    --cc=igorch@amazon.com \
    --cc=jan.gutter@netronome.com \
    --cc=jasvinder.singh@intel.com \
    --cc=jerinj@marvell.com \
    --cc=jgrajcia@cisco.com \
    --cc=jianjay.zhou@huawei.com \
    --cc=jingjing.wu@intel.com \
    --cc=john.griffin@intel.com \
    --cc=john.mcnamara@intel.com \
    --cc=john.miller@atomicrules.com \
    --cc=johndale@cisco.com \
    --cc=joyce.kong@arm.com \
    --cc=jsrikanth@marvell.com \
    --cc=keith.wiles@intel.com \
    --cc=kevin.laatz@intel.com \
    --cc=kirankumark@marvell.com \
    --cc=kirill.rybalchenko@intel.com \
    --cc=konstantin.ananyev@intel.com \
    --cc=ktraynor@redhat.com \
    --cc=kys@microsoft.com \
    --cc=lee.daly@intel.com \
    --cc=liang.j.ma@intel.com \
    --cc=linville@tuxdriver.com \
    --cc=lironh@marvell.com \
    --cc=maicolgabriel@hotmail.com \
    --cc=maintainers@dpdk.org \
    --cc=marko.kovacevic@intel.com \
    --cc=maryam.tahhan@intel.com \
    --cc=matan@mellanox.com \
    --cc=matt.peters@windriver.com \
    --cc=mattias.ronnblom@ericsson.com \
    --cc=maxime.coquelin@redhat.com \
    --cc=mczekaj@marvell.com \
    --cc=mdr@ashroe.eu \
    --cc=michaelsh@marvell.com \
    --cc=mk@semihalf.com \
    --cc=mtetsuyah@gmail.com \
    --cc=mw@semihalf.com \
    --cc=ndabilpuram@marvell.com \
    --cc=nicolas.chautru@intel.com \
    --cc=nipun.gupta@nxp.com \
    --cc=ocardona@microsoft.com \
    --cc=olivier.matz@6wind.com \
    --cc=orika@mellanox.com \
    --cc=pablo.de.lara.guarch@intel.com \
    --cc=pallavi.kadam@intel.com \
    --cc=pavel.belous@aquantia.com \
    --cc=pbhagavatula@marvell.com \
    --cc=peter.mccarthy@intel.com \
    --cc=phil.yang@arm.com \
    --cc=pkapoor@marvell.com \
    --cc=qi.z.zhang@intel.com \
    --cc=qiming.yang@intel.com \
    --cc=radu.nicolau@intel.com \
    --cc=rahul.lakkireddy@chelsio.com \
    --cc=rasland@mellanox.com \
    --cc=ravi1.kumar@amd.com \
    --cc=remes@netcope.com \
    --cc=reshma.pattan@intel.com \
    --cc=rmody@marvell.com \
    --cc=rnagadheeraj@marvell.com \
    --cc=rosen.xu@intel.com \
    --cc=roy.fan.zhang@intel.com \
    --cc=rsanford@akamai.com \
    --cc=ruifeng.wang@arm.com \
    --cc=sachin.saxena@nxp.com \
    --cc=sameh.gobriel@intel.com \
    --cc=shahafs@mellanox.com \
    --cc=shepard.siegel@atomicrules.com \
    --cc=shshaikh@marvell.com \
    --cc=skori@marvell.com \
    --cc=skoteshwar@marvell.com \
    --cc=somnath.kotur@broadcom.com \
    --cc=srinivasan@marvell.com \
    --cc=ssahu@marvell.com \
    --cc=steven.webster@windriver.com \
    --cc=sthemmin@microsoft.com \
    --cc=sthotton@marvell.com \
    --cc=tdu@semihalf.com \
    --cc=thomas@monjalon.net \
    --cc=tianfei.zhang@intel.com \
    --cc=tiwei.bie@intel.com \
    --cc=tomasz.kantecki@intel.com \
    --cc=vattunuru@marvell.com \
    --cc=viacheslavo@mellanox.com \
    --cc=viktorin@rehivetech.com \
    --cc=vladimir.medvedkin@intel.com \
    --cc=wenzhuo.lu@intel.com \
    --cc=xavier.huwei@huawei.com \
    --cc=xiao.w.wang@intel.com \
    --cc=xiaolong.ye@intel.com \
    --cc=xiaoyun.li@intel.com \
    --cc=xuanziyang2@huawei.com \
    --cc=yipeng1.wang@intel.com \
    --cc=yisen.zhuang@huawei.com \
    --cc=yongwang@vmware.com \
    --cc=zhihong.wang@intel.com \
    --cc=zhouguoyang@huawei.com \
    --cc=zr@semihalf.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

DPDK-dev Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/dpdk-dev/0 dpdk-dev/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 dpdk-dev dpdk-dev/ https://lore.kernel.org/dpdk-dev \
		dev@dpdk.org
	public-inbox-index dpdk-dev

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.dpdk.dev


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git