From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.2 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS, USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06558C33CAD for ; Mon, 13 Jan 2020 13:06:18 +0000 (UTC) Received: from dpdk.org (dpdk.org [92.243.14.124]) by mail.kernel.org (Postfix) with ESMTP id 7222F207FF for ; Mon, 13 Jan 2020 13:06:17 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7222F207FF Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=dev-bounces@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id BB1571D174; Mon, 13 Jan 2020 14:06:16 +0100 (CET) Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by dpdk.org (Postfix) with ESMTP id C87E11D151; Mon, 13 Jan 2020 14:06:14 +0100 (CET) X-Amp-Result: UNKNOWN X-Amp-Original-Verdict: FILE UNKNOWN X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga101.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 13 Jan 2020 05:06:13 -0800 X-IronPort-AV: E=Sophos;i="5.69,429,1571727600"; d="scan'208";a="217400772" Received: from bricha3-mobl.ger.corp.intel.com ([10.237.221.26]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-SHA; 13 Jan 2020 05:05:46 -0800 Date: Mon, 13 Jan 2020 13:05:43 +0000 From: Bruce Richardson To: Jerin Jacob Kollanukkaran Cc: "dev@dpdk.org" , Thomas Monjalon , David Marchand , Ferruh Yigit , Andrew Rybchenko , Ajit Khaparde , Qi Zhang , Xiaolong Ye , Raslan Darawsheh , Maxime Coquelin , Tiwei Bie , Akhil Goyal , Luca Boccassi , Kevin Traynor , "maintainers@dpdk.org" , John McNamara , Marko Kovacevic , Ray Kinsella , Aaron Conole , Michael Santana , Harry van Haaren , Cristian Dumitrescu , Phil Yang , Joyce Kong , Mattias =?iso-8859-1?Q?R=F6nnblom?= , Jan Viktorin , Gavin Hu , David Christensen , Konstantin Ananyev , Anatoly Burakov , Harini Ramakrishnan , Omar Cardona , Anand Rawat , Ranjit Menon , Olivier Matz , Gage Eads , Adrien Mazarguil , Nicolas Chautru , Declan Doherty , Fiona Trahe , Ashish Gupta , Erik Gabriel Carrillo , Abhinandan Gujjar , Shreyansh Jain , Hemant Agrawal , "Artem V. Andreev" , Nithin Kumar Dabilpuram , Vamsi Krishna Attunuru , Rosen Xu , Sachin Saxena , Stephen Hemminger , Chas Williams , "John W. Linville" , Prasun Kapoor , Marcin Wojtas , Michal Krawczyk , Guy Tzalik , Evgeny Schemeilin , Igor Chauskin , Ravi Kumar , Igor Russkikh , Pavel Belous , Shepard Siegel , Ed Czeck , John Miller , Somnath Kotur , Maciej Czekaj , Shijith Thotton , Srisivasubramanian Srinivasan , Rahul Lakkireddy , John Daley , Hyong Youb Kim , "Wei Hu (Xavier" , "Min Hu (Connor" , Yisen Zhuang , Ziyang Xuan , Xiaoyun Wang , Guoyang Zhou , Beilei Xing , Xiao Wang , Jingjing Wu , Wenzhuo Lu , Qiming Yang , Tomasz Duszynski , Liron Himi , Zyta Szpak , Kiran Kumar Kokkilagadda , Matan Azrad , Shahaf Shuler , Viacheslav Ovsiienko , "K. Y. Srinivasan" , Haiyang Zhang , Jan Remes , Heinrich Kuhn , Jan Gutter , Gagandeep Singh , Rasesh Mody , Shahed Shaikh , Yong Wang , Zhihong Wang , Steven Webster , Matt Peters , Keith Wiles , Tetsuya Mukawa , Gaetan Rivet , Jasvinder Singh , Jakub Grajciar , Ruifeng Wang , Anoob Joseph , Fan Zhang , Pablo de Lara , John Griffin , Deepak Kumar Jain , Michael Shamis , Nagadheeraj Rottela , Srikanth Jampala , Ankur Dwivedi , Jay Zhou , Lee Daly , Sunila Sahu , Nipun Gupta , Liang Ma , Peter Mccarthy , Tianfei zhang , Satha Koteswara Rao Kottidi , Xiaoyun Li , Bernard Iremonger , Vladimir Medvedkin , David Hunt , Reshma Pattan , Byron Marohn , Sameh Gobriel , Yipeng Wang , Honnappa Nagarahalli , Robert Sanford , Kevin Laatz , Maryam Tahhan , Ori Kam , Radu Nicolau , Tomasz Kantecki , Sunil Kumar Kori , Pavan Nikhilesh Bhagavatula , Kirill Rybalchenko , "Kadam, Pallavi" Message-ID: <20200113130543.GC1645@bricha3-MOBL.ger.corp.intel.com> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.12.1 (2019-06-15) Subject: Re: [dpdk-dev] [RFC] DPDK Trace support X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Mon, Jan 13, 2020 at 10:40:13AM +0000, Jerin Jacob Kollanukkaran wrote: > Hi All, > > I would like to add tracing support for DPDK. > I am planning to add this support in v20.05 release. > > This RFC attempts to get feedback from the community on > > a) Tracing Use cases. > b) Tracing Requirements. > b) Implementation choices. > c) Trace format. > > Use-cases > --------- > - Most of the cases, The DPDK provider will not have access to the DPDK customer applications. > To debug/analyze the slow path and fast path DPDK API usage from the field, > we need to have integrated trace support in DPDK. > > - Need a low overhead Fast path multi-core PMD driver debugging/analysis > infrastructure in DPDK to fix the functional and performance issue(s) of PMD. > > - Post trace analysis tools can provide various status across the system such > as cpu_idle() using the timestamp added in the trace. > > > Requirements: > ------------- > - Support for Linux, FreeBSD and Windows OS > - Open trace format > - Multi-platform Open source trace viewer > - Absolute low overhead trace API for DPDK fast path tracing/debugging. > - Dynamic enable/disable of trace events > > > To enable trace support in DPDK, following items need to work out: > > a) Add the DPDK trace points in the DPDK source code. > > - This includes updating DPDK functions such as, > rte_eth_dev_configure(), rte_eth_dev_start(), rte_eth_dev_rx_burst() to emit the trace. > > b) Choosing suitable serialization-format > > - Common Trace Format, CTF, is an open format and language to describe trace formats. > This enables tool reuse, of which line-textual (babeltrace) and > graphical (TraceCompass) variants already exist. > > CTF should look familiar to C programmers but adds stronger typing. > See CTF - A Flexible, High-performance Binary Trace Format. > > https://diamon.org/ctf/ > > c) Writing the on-target serialization code, > > See the section below.(Lttng CTF trace emitter vs DPDK specific CTF trace emitter) > > d) Deciding on and writing the I/O transport mechanics, > > For performance reasons, it should be backed by a huge-page and write to file IO. > > e) Writing the PC-side deserializer/parser, > > Both the babletrace(CLI tool) and Trace Compass(GUI tool) support CTF. > See: > https://lttng.org/viewers/ > > f) Writing tools for filtering and presentation. > > See item (e) > > > Lttng CTF trace emitter vs DPDK specific CTF trace emitter > ---------------------------------------------------------- > > I have written a performance evaluation application to measure the overhead > of Lttng CTF emitter(The fastpath infrastructure used by https://lttng.org/ library to emit the trace) > > https://github.com/jerinjacobk/lttng-overhead > https://github.com/jerinjacobk/lttng-overhead/blob/master/README > > I could improve the performance by 30% by adding the "DPDK" > based plugin for get_clock() and get_cpu(), > Here are the performance numbers after adding the plugin on > x86 and various arm64 board that I have access to, > > On high-end x86, it comes around 236 cycles/~100ns @ 2.4GHz (See the last line in the log(ZERO_ARG)) > On arm64, it varies from 312 cycles to 1100 cycles(based on the class of CPU). > In short, Based on the "IPC capabilities", The cost would be around 100ns to 400ns > for single void trace(a trace without any argument) > > > [lttng-overhead-x86] $ sudo ./calibrate/build/app/calibrate -c 0xc0 > make[1]: Entering directory '/export/lttng-overhead-x86/calibrate' > make[1]: Leaving directory '/export/lttng-overhead-x86/calibrate' > EAL: Detected 56 lcore(s) > EAL: Detected 2 NUMA nodes > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket > EAL: Selected IOVA mode 'PA' > EAL: Probing VFIO support... > EAL: PCI device 0000:01:00.0 on NUMA socket 0 > EAL: probe driver: 8086:1521 net_e1000_igb > EAL: PCI device 0000:01:00.1 on NUMA socket 0 > EAL: probe driver: 8086:1521 net_e1000_igb > CPU Timer freq is 2600.000000MHz > NOP: cycles=0.194834 ns=0.074936 > GET_CLOCK: cycles=47.854658 ns=18.405638 > GET_CPU: cycles=30.995892 ns=11.921497 > ZERO_ARG: cycles=236.945113 ns=91.132736 > > > We will have only 16.75ns to process 59.2 mpps(40Gbps), So IMO, Lttng CTF emitter > may not fit the DPDK fast path purpose due to the cost associated with generic Lttng features. > > One option could be to have, native CTF emitter in EAL/DPDK to emit the > trace in a hugepage. I think it would be a handful of cycles if we limit the features > to the requirements above: > > The upside of using Lttng CTF emitter: > a) No need to write a new CTF trace emitter(the item (c)) > > The downside of Lttng CTF emitter(the item (c)) > a) performance issue(See above) > b) Lack of Windows OS support. It looks like, it has basic FreeBSD support. > c) dpdk library dependency to lttng for trace. > > So, Probably it good to have native CTF emitter in DPDK and reuse all > open-source trace viewer(babeltrace and TraceCompass) and format(CTF) infrastructure. > I think, it would be best of both world. > > Any thoughts on this subject? Based on the community feedback, I can work on the patch for v20.05. Forgive my ignorance of LTTng, but is there the concept of enabling/disabling the trace points? If so, the overhead you refer to, that is presumably with the trace enabled? /Bruce