From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.2 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS, USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 400B7C33CAD for ; Mon, 13 Jan 2020 11:00:58 +0000 (UTC) Received: from dpdk.org (dpdk.org [92.243.14.124]) by mail.kernel.org (Postfix) with ESMTP id 93766206DA for ; Mon, 13 Jan 2020 11:00:57 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 93766206DA Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ashroe.eu Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=dev-bounces@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id C5F471D542; Mon, 13 Jan 2020 12:00:56 +0100 (CET) Received: from q2relay28.mxroute.com (q2relay28.mxroute.com [160.202.107.28]) by dpdk.org (Postfix) with ESMTP id 1E6291D51B for ; Mon, 13 Jan 2020 12:00:54 +0100 (CET) Received: from filter003.mxroute.com [168.235.111.26] (Authenticated sender: mN4UYu2MZsgR) by q2relay28.mxroute.com (ZoneMTA) with ESMTPSA id 16f9e9043ac00025a7.002 for (version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES128-GCM-SHA256); Mon, 13 Jan 2020 11:00:52 +0000 X-Zone-Loop: ee1d64e7a8058e95fe8e68e8d4faf90a73e871ce6b0f X-Originating-IP: [168.235.111.26] Received: from galaxy.mxroute.com (unknown [23.92.70.113]) by filter003.mxroute.com (Postfix) with ESMTPS id 608606002B; Mon, 13 Jan 2020 11:00:50 +0000 (UTC) Received: from irdmzpr02-ext.ir.intel.com ([192.198.151.37]) by galaxy.mxroute.com with esmtpsa (TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128) (Exim 4.91) (envelope-from ) id 1iqxAn-0002rR-Kr; Mon, 13 Jan 2020 05:42:39 -0500 To: Jerin Jacob Kollanukkaran , dpdk-dev , "dave@barachs.net" References: From: Ray Kinsella Autocrypt: addr=mdr@ashroe.eu; keydata= mQINBFv8B3wBEAC+5ImcgbIvadt3axrTnt7Sxch3FsmWTTomXfB8YiuHT8KL8L/bFRQSL1f6 ASCHu3M89EjYazlY+vJUWLr0BhK5t/YI7bQzrOuYrl9K94vlLwzD19s/zB/g5YGGR5plJr0s JtJsFGEvF9LL3e+FKMRXveQxBB8A51nAHfwG0WSyx53d61DYz7lp4/Y4RagxaJoHp9lakn8j HV2N6rrnF+qt5ukj5SbbKWSzGg5HQF2t0QQ5tzWhCAKTfcPlnP0GymTBfNMGOReWivi3Qqzr S51Xo7hoGujUgNAM41sxpxmhx8xSwcQ5WzmxgAhJ/StNV9cb3HWIoE5StCwQ4uXOLplZNGnS uxNdegvKB95NHZjRVRChg/uMTGpg9PqYbTIFoPXjuk27sxZLRJRrueg4tLbb3HM39CJwSB++ YICcqf2N+GVD48STfcIlpp12/HI+EcDSThzfWFhaHDC0hyirHxJyHXjnZ8bUexI/5zATn/ux TpMbc/vicJxeN+qfaVqPkCbkS71cHKuPluM3jE8aNCIBNQY1/j87k5ELzg3qaesLo2n1krBH bKvFfAmQuUuJT84/IqfdVtrSCTabvDuNBDpYBV0dGbTwaRfE7i+LiJJclUr8lOvHUpJ4Y6a5 0cxEPxm498G12Z3NoY/mP5soItPIPtLR0rA0fage44zSPwp6cQARAQABtBxSYXkgS2luc2Vs bGEgPG1kckBhc2hyb2UuZXU+iQJUBBMBCAA+FiEEcDUDlKDJaDuJlfZfdJdaH/sCCpsFAlv8 B3wCGyMFCQlmAYAFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQdJdaH/sCCptdtRAAl0oE msa+djBVYLIsax+0f8acidtWg2l9f7kc2hEjp9h9aZCpPchQvhhemtew/nKavik3RSnLTAyn B3C/0GNlmvI1l5PFROOgPZwz4xhJKGN7jOsRrbkJa23a8ly5UXwF3Vqnlny7D3z+7cu1qq/f VRK8qFyWkAb+xgqeZ/hTcbJUWtW+l5Zb+68WGEp8hB7TuJLEWb4+VKgHTpQ4vElYj8H3Z94a 04s2PJMbLIZSgmKDASnyrKY0CzTpPXx5rSJ1q+B1FCsfepHLqt3vKSALa3ld6bJ8fSJtDUJ7 JLiU8dFZrywgDIVme01jPbjJtUScW6jONLvhI8Z2sheR71UoKqGomMHNQpZ03ViVWBEALzEt TcjWgJFn8yAmxqM4nBnZ+hE3LbMo34KCHJD4eg18ojDt3s9VrDLa+V9fNxUHPSib9FD9UX/1 +nGfU/ZABmiTuUDM7WZdXri7HaMpzDRJUKI6b+/uunF8xH/h/MHW16VuMzgI5dkOKKv1LejD dT5mA4R+2zBS+GsM0oa2hUeX9E5WwjaDzXtVDg6kYq8YvEd+m0z3M4e6diFeLS77/sAOgaYL 92UcoKD+Beym/fVuC6/55a0e12ksTmgk5/ZoEdoNQLlVgd2INtvnO+0k5BJcn66ZjKn3GbEC VqFbrnv1GnA58nEInRCTzR1k26h9nmS5Ag0EW/wHfAEQAMth1vHr3fOZkVOPfod3M6DkQir5 xJvUW5EHgYUjYCPIa2qzgIVVuLDqZgSCCinyooG5dUJONVHj3nCbITCpJp4eB3PI84RPfDcC hf/V34N/Gx5mTeoymSZDBmXT8YtvV/uJvn+LvHLO4ZJdvq5ZxmDyxfXFmkm3/lLw0+rrNdK5 pt6OnVlCqEU9tcDBezjUwDtOahyV20XqxtUttN4kQWbDRkhT+HrA9WN9l2HX91yEYC+zmF1S OhBqRoTPLrR6g4sCWgFywqztpvZWhyIicJipnjac7qL/wRS+wrWfsYy6qWLIV80beN7yoa6v ccnuy4pu2uiuhk9/edtlmFE4dNdoRf7843CV9k1yRASTlmPkU59n0TJbw+okTa9fbbQgbIb1 pWsAuicRHyLUIUz4f6kPgdgty2FgTKuPuIzJd1s8s6p2aC1qo+Obm2gnBTduB+/n1Jw+vKpt 07d+CKEKu4CWwvZZ8ktJJLeofi4hMupTYiq+oMzqH+V1k6QgNm0Da489gXllU+3EFC6W1qKj tkvQzg2rYoWeYD1Qn8iXcO4Fpk6wzylclvatBMddVlQ6qrYeTmSbCsk+m2KVrz5vIyja0o5Y yfeN29s9emXnikmNfv/dA5fpi8XCANNnz3zOfA93DOB9DBf0TQ2/OrSPGjB3op7RCfoPBZ7u AjJ9dM7VABEBAAGJAjwEGAEIACYWIQRwNQOUoMloO4mV9l90l1of+wIKmwUCW/wHfAIbDAUJ CWYBgAAKCRB0l1of+wIKm3KlD/9w/LOG5rtgtCUWPl4B3pZvGpNym6XdK8cop9saOnE85zWf u+sKWCrxNgYkYP7aZrYMPwqDvilxhbTsIJl5HhPgpTO1b0i+c0n1Tij3EElj5UCg3q8mEc17 c+5jRrY3oz77g7E3oPftAjaq1ybbXjY4K32o3JHFR6I8wX3m9wJZJe1+Y+UVrrjY65gZFxcA thNVnWKErarVQGjeNgHV4N1uF3pIx3kT1N4GSnxhoz4Bki91kvkbBhUgYfNflGURfZT3wIKK +d50jd7kqRouXUCzTdzmDh7jnYrcEFM4nvyaYu0JjSS5R672d9SK5LVIfWmoUGzqD4AVmUW8 pcv461+PXchuS8+zpltR9zajl72Q3ymlT4BTAQOlCWkD0snBoKNUB5d2EXPNV13nA0qlm4U2 GpROfJMQXjV6fyYRvttKYfM5xYKgRgtP0z5lTAbsjg9WFKq0Fndh7kUlmHjuAIwKIV4Tzo75 QO2zC0/NTaTjmrtiXhP+vkC4pcrOGNsbHuaqvsc/ZZ0siXyYsqbctj/sCd8ka2r94u+c7o4l BGaAm+FtwAfEAkXHu4y5Phuv2IRR+x1wTey1U1RaEPgN8xq0LQ1OitX4t2mQwjdPihZQBCnZ wzOrkbzlJMNrMKJpEgulmxAHmYJKgvZHXZXtLJSejFjR0GdHJcL5rwVOMWB8cg== Message-ID: Date: Mon, 13 Jan 2020 11:00:25 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.3.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-AuthUser: mdr@ashroe.eu Subject: Re: [dpdk-dev] [RFC] DPDK Trace support X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi Jerin, Any idea why lttng performance is so poor? I would have naturally gone there to benefit from the existing toolchain. Have you looked at the FD.io logging/tracing infrastructure for inspiration? https://wiki.fd.io/view/VPP/elog Ray K On 13/01/2020 10:40, Jerin Jacob Kollanukkaran wrote: > Hi All, > > I would like to add tracing support for DPDK. > I am planning to add this support in v20.05 release. > > This RFC attempts to get feedback from the community on > > a) Tracing Use cases. > b) Tracing Requirements. > b) Implementation choices. > c) Trace format. > > Use-cases > --------- > - Most of the cases, The DPDK provider will not have access to the DPDK customer applications. > To debug/analyze the slow path and fast path DPDK API usage from the field, > we need to have integrated trace support in DPDK. > > - Need a low overhead Fast path multi-core PMD driver debugging/analysis > infrastructure in DPDK to fix the functional and performance issue(s) of PMD. > > - Post trace analysis tools can provide various status across the system such > as cpu_idle() using the timestamp added in the trace. > > > Requirements: > ------------- > - Support for Linux, FreeBSD and Windows OS > - Open trace format > - Multi-platform Open source trace viewer > - Absolute low overhead trace API for DPDK fast path tracing/debugging. > - Dynamic enable/disable of trace events > > > To enable trace support in DPDK, following items need to work out: > > a) Add the DPDK trace points in the DPDK source code. > > - This includes updating DPDK functions such as, > rte_eth_dev_configure(), rte_eth_dev_start(), rte_eth_dev_rx_burst() to emit the trace. > > b) Choosing suitable serialization-format > > - Common Trace Format, CTF, is an open format and language to describe trace formats. > This enables tool reuse, of which line-textual (babeltrace) and > graphical (TraceCompass) variants already exist. > > CTF should look familiar to C programmers but adds stronger typing. > See CTF - A Flexible, High-performance Binary Trace Format. > > https://diamon.org/ctf/ > > c) Writing the on-target serialization code, > > See the section below.(Lttng CTF trace emitter vs DPDK specific CTF trace emitter) > > d) Deciding on and writing the I/O transport mechanics, > > For performance reasons, it should be backed by a huge-page and write to file IO. > > e) Writing the PC-side deserializer/parser, > > Both the babletrace(CLI tool) and Trace Compass(GUI tool) support CTF. > See: > https://lttng.org/viewers/ > > f) Writing tools for filtering and presentation. > > See item (e) > > > Lttng CTF trace emitter vs DPDK specific CTF trace emitter > ---------------------------------------------------------- > > I have written a performance evaluation application to measure the overhead > of Lttng CTF emitter(The fastpath infrastructure used by https://lttng.org/ library to emit the trace) > > https://github.com/jerinjacobk/lttng-overhead > https://github.com/jerinjacobk/lttng-overhead/blob/master/README > > I could improve the performance by 30% by adding the "DPDK" > based plugin for get_clock() and get_cpu(), > Here are the performance numbers after adding the plugin on > x86 and various arm64 board that I have access to, > > On high-end x86, it comes around 236 cycles/~100ns @ 2.4GHz (See the last line in the log(ZERO_ARG)) > On arm64, it varies from 312 cycles to 1100 cycles(based on the class of CPU). > In short, Based on the "IPC capabilities", The cost would be around 100ns to 400ns > for single void trace(a trace without any argument) > > > [lttng-overhead-x86] $ sudo ./calibrate/build/app/calibrate -c 0xc0 > make[1]: Entering directory '/export/lttng-overhead-x86/calibrate' > make[1]: Leaving directory '/export/lttng-overhead-x86/calibrate' > EAL: Detected 56 lcore(s) > EAL: Detected 2 NUMA nodes > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket > EAL: Selected IOVA mode 'PA' > EAL: Probing VFIO support... > EAL: PCI device 0000:01:00.0 on NUMA socket 0 > EAL: probe driver: 8086:1521 net_e1000_igb > EAL: PCI device 0000:01:00.1 on NUMA socket 0 > EAL: probe driver: 8086:1521 net_e1000_igb > CPU Timer freq is 2600.000000MHz > NOP: cycles=0.194834 ns=0.074936 > GET_CLOCK: cycles=47.854658 ns=18.405638 > GET_CPU: cycles=30.995892 ns=11.921497 > ZERO_ARG: cycles=236.945113 ns=91.132736 > > > We will have only 16.75ns to process 59.2 mpps(40Gbps), So IMO, Lttng CTF emitter > may not fit the DPDK fast path purpose due to the cost associated with generic Lttng features. > > One option could be to have, native CTF emitter in EAL/DPDK to emit the > trace in a hugepage. I think it would be a handful of cycles if we limit the features > to the requirements above: > > The upside of using Lttng CTF emitter: > a) No need to write a new CTF trace emitter(the item (c)) > > The downside of Lttng CTF emitter(the item (c)) > a) performance issue(See above) > b) Lack of Windows OS support. It looks like, it has basic FreeBSD support. > c) dpdk library dependency to lttng for trace. > > So, Probably it good to have native CTF emitter in DPDK and reuse all > open-source trace viewer(babeltrace and TraceCompass) and format(CTF) infrastructure. > I think, it would be best of both world. > > Any thoughts on this subject? Based on the community feedback, I can work on the patch for v20.05. >