From mboxrd@z Thu Jan 1 00:00:00 1970 From: Adrien Mazarguil Subject: Re: Possible bug in mlx5_tx_burst_mpw? Date: Fri, 16 Sep 2016 09:14:03 +0200 Message-ID: <20160916071403.GW17252@6wind.com> References: <20160914143016.GT17252@6wind.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: "dev@dpdk.org" To: Luke Gorrie Return-path: Received: from mail-wm0-f42.google.com (mail-wm0-f42.google.com [74.125.82.42]) by dpdk.org (Postfix) with ESMTP id C76572C2A for ; Fri, 16 Sep 2016 09:14:09 +0200 (CEST) Received: by mail-wm0-f42.google.com with SMTP id k186so24021247wmd.0 for ; Fri, 16 Sep 2016 00:14:09 -0700 (PDT) Content-Disposition: inline In-Reply-To: List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Wed, Sep 14, 2016 at 09:33:18PM +0200, Luke Gorrie wrote: > Hi Adrien, > > On 14 September 2016 at 16:30, Adrien Mazarguil > wrote: > > > Your interpretation is correct (this is intentional and not a bug). > > > > Thanks very much for clarifying. > > This is interesting to me because I am also working on a ConnectX-4 (Lx) > driver based on the newly released driver interface specification [1] and I > am wondering how interested I should be in this MPW feature that is > currently not documented. Seems like this document only describes established features whose interface won't be subject to firmware evolutions, I think MPW is not one of them. AFAIK currently MPW cannot be used with LSO which we intend to support soon. Our implementation is a stripped down version of the code found in libmlx5. I guess you could ask Mellanox directly if you need more information. > In the event successive packets share a few properties (length, number of > > segments, offload flags), these can be factored out as an optimization to > > lower the amount of traffic on the PCI bus. This feature is currently > > supported by the ConnectX-4 Lx family of adapters. > > > > I have a concern here that I hope you will forgive me for voicing. > > This optimization seems to run the risk of inflating scores on > constant-packet-size IXIA-style benchmarks like [2] and making them less > useful for predicting real-world performance. That seems like a negative to > me as an application developer. I wonder if I am overlooking some practical > benefits that motivate implementing this in silicon and in the driver and > enabling it by default? Your concern is understandable, no offense taken. You are obviously right about benchmarks with constant packets, whose results can be improved by MPW. Performance-wise, with the right traffic patterns MPW allows ConnectX-4 Lx adapters to outperform their non-Lx counterparts (e.g. comparing 40G EN Lx PCIe 8x vs. 40G EN PCIe 8x) when measuring traffic rate (Mpps), not throughput. Disabling MPW yields comparable results, which is why it is considered to be an optimization. Since processing MPW consumes a few additional CPU cycles, it can be disabled at runtime with the txq_mpw_en switch (documented in mlx5.rst). Now about the real-world scenario, we are not talking about needing millions of identical packets to notice an improvement. MPW is effective from 2 to at most 5 consecutive packets that share some meta-data (length, number of segments and offload flags), all within the same burst. Just to be clear, neither their destination nor their payload need to be the same, it would have been useless otherwise. Sending a few packets at once with such similar properties is common occurrence in the real world, think about forwarding TCP traffic that has been shaped to a constant size by LSO or MTU. Like many optimizations, this one targets a specific yet common use-case. If you would rather get a constant rate out of any traffic pattern for predictable latency, DPDK which is burst-oriented is probably not what your application needs if used as-is. > [1] > http://www.mellanox.com/related-docs/user_manuals/Ethernet_Adapters_Programming_Manual.pdf > [2] > https://www.mellanox.com/blog/2016/06/performance-beyond-numbers-stephen-curry-style-server-io/ -- Adrien Mazarguil 6WIND