From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D85C3C282CE for ; Tue, 4 Jun 2019 09:16:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id B8C0E22CF8 for ; Tue, 4 Jun 2019 09:16:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727157AbfFDJQB (ORCPT ); Tue, 4 Jun 2019 05:16:01 -0400 Received: from mx1.redhat.com ([209.132.183.28]:36020 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726948AbfFDJQB (ORCPT ); Tue, 4 Jun 2019 05:16:01 -0400 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id D27393091D73; Tue, 4 Jun 2019 09:15:59 +0000 (UTC) Received: from carbon (ovpn-200-32.brq.redhat.com [10.40.200.32]) by smtp.corp.redhat.com (Postfix) with ESMTP id 26750105B202; Tue, 4 Jun 2019 09:15:55 +0000 (UTC) Date: Tue, 4 Jun 2019 11:15:54 +0200 From: Jesper Dangaard Brouer To: Tom Barbette Cc: Saeed Mahameed , "toke@redhat.com" , "xdp-newbies@vger.kernel.org" , Leon Romanovsky , Tariq Toukan , brouer@redhat.com, =?UTF-8?B?QmrDtnJuIFQ=?= =?UTF-8?B?w7ZwZWw=?= , "Karlsson, Magnus" , Jakub Kicinski , "netdev@vger.kernel.org" Subject: Re: Bad XDP performance with mlx5 Message-ID: <20190604111554.749ddd87@carbon> In-Reply-To: <9f116335-0fad-079b-4070-89f24af4ab55@kth.se> References: <20190529191602.71eb6c87@carbon> <0836bd30-828a-9126-5d99-1d35b931e3ab@kth.se> <20190530094053.364b1147@carbon> <2218141a-7026-1cb8-c594-37e38eef7b15@kth.se> <20190531181817.34039c9f@carbon> <19ca7cd9a878b2ecc593cd2838b8ae0412463593.camel@mellanox.com> <9f116335-0fad-079b-4070-89f24af4ab55@kth.se> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.42]); Tue, 04 Jun 2019 09:16:01 +0000 (UTC) Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org On Tue, 4 Jun 2019 09:28:22 +0200 Tom Barbette wrote: > Thanks Jesper for looking into this! > > I don't think I will be of much help further on this matter. My take > out would be: as a first-time user looking into XDP after watching a > dozen of XDP talks, I would have expected XDP default settings to be > identical to SKB, so I don't have to watch out for a set of > per-driver parameter checklist to avoid increasing my CPU consumption > by 15% when inserting "a super efficient and light BPF program". But > I understand it's not that easy... The gap should not be this large, but as I demonstrated it was primarily because you hit an unfortunate interaction with TCP and how the mlx5 driver does page-caching (p.s. we are working on removing this driver local recycle-cache). When loading an XDP/eBPF-prog then the driver change the underlying RX memory model, which waste memory to gain packets-per-sec speed, but TCP sees this memory waste and gives us a penalty. It is important to understand, that XDP is not optimized for TCP. XDP is designed and optimized for L2-L3 handling of packets (TCP is L4). Before XDP these L2-L3 use-cases were "slow", because the kernel netstack assumes a L4/socket use-case (full SKB), when less was really needed. This is actually another good example of why XDP programs per RX-queue, will be useful (notice: which is not implemented upstream, yet...). -- Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer