From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.1 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DF407C2F441 for ; Mon, 21 Jan 2019 16:34:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id A30E920870 for ; Mon, 21 Jan 2019 16:34:31 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="up0qSvka" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730588AbfAUQea (ORCPT ); Mon, 21 Jan 2019 11:34:30 -0500 Received: from mail-pf1-f196.google.com ([209.85.210.196]:39066 "EHLO mail-pf1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729485AbfAUQea (ORCPT ); Mon, 21 Jan 2019 11:34:30 -0500 Received: by mail-pf1-f196.google.com with SMTP id r136so10361126pfc.6 for ; Mon, 21 Jan 2019 08:34:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=ZjcFvHbd6iRIroDPAuVeiqXQw0ipZPZGgmNWF9u8XnU=; b=up0qSvkaVwOVLQHXKK1oEpZJ/OvUV8GR9+9/sa0hZkvVXrin7cjgVmpXhLGpTkeMfM /TjSh727zRpwHx71/jqtDGmbWk8QD8ygi/zLBoRzTH5r5PXZIeoMGz0XHtEkmIyxgPCu bOlrmJy+ZLJkeHyQ/o7GUHBFaTOaS3vIJeYu7P7I/vT3fss6WAQhlOzb5dv3J3gM1wUu rUxr3IU5JSagi8y6/DYkh+kcjm5eVYcO8KzE2qUjMLdj+cdhBhTAvqVQg0fy+EDHzZ48 lZcF0RXBL2Ul85HGpFAP+qAJh7T0i4IN70g+GiNqU2bBV7qyop2YIapefQxTBAbIEmU1 WUBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=ZjcFvHbd6iRIroDPAuVeiqXQw0ipZPZGgmNWF9u8XnU=; b=uN3I6MkBzvUJQZtxJFPAcL6YH9+GfeW/4RJHWqbcdPvXooc123O9OwklRZjj+8zKf+ 0T8FcNl5TehsOmtU/h2mbwVghFYDcBEMhguj9lYpm9bi7raxMIMm40FawZpWswyDIdLi WzKq/VdjXwpGh04gQL6axgYdOp8l+OoApVYyFNG3PsdZVRRRT3eXRVGqZGp1ZtbOiySf Zo0O5MvyAXfcGZzgggOqmc2agrHyoHVf/flEirtS1H7UBNUgixM6J2xDKbHFMDGAzudq 5XEr9x9R4+w6VJAQQ8UKxNSrCk08ZvJdJcDlyKfhO8yE/YHQtY08sMxqbTqtuCM2LpdO V4FQ== X-Gm-Message-State: AJcUuke8HAjykscXX+0od4aluIq7Y4tKoScPhPl5rXH3Dza/vDaxVzhx e8Rj81xOyI7QJSlfAM494KE= X-Google-Smtp-Source: ALg8bN5cv37P89UnfREgOtCmuZ27gsozgfFCvhC3z/mxQvUu5RxluFfkX6OGQMVmKqlO2acjxRGHlw== X-Received: by 2002:a62:37c3:: with SMTP id e186mr30924484pfa.251.1548088468640; Mon, 21 Jan 2019 08:34:28 -0800 (PST) Received: from btopel-mobl.isw.intel.com ([192.55.54.42]) by smtp.gmail.com with ESMTPSA id e9sm16225989pff.5.2019.01.21.08.34.26 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 21 Jan 2019 08:34:28 -0800 (PST) From: bjorn.topel@gmail.com To: intel-wired-lan@lists.osuosl.org Cc: =?UTF-8?q?Bj=C3=B6rn=20T=C3=B6pel?= , brouer@redhat.com, magnus.karlsson@intel.com, magnus.karlsson@gmail.com, netdev@vger.kernel.org Subject: [PATCH] i40e: replace switch-statement with if-clause Date: Mon, 21 Jan 2019 17:33:56 +0100 Message-Id: <20190121163356.31332-1-bjorn.topel@gmail.com> X-Mailer: git-send-email 2.19.1 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Björn Töpel GCC will generate jump tables for switch-statements with more than 5 case statements. An entry into the jump table is an indirect call, which means that for CONFIG_RETPOLINE builds, this is rather expensive. This commit replaces the switch-statement that acts on the XDP program result with an if-clause. The if-clause was also refactored into a common function that can be used by AF_XDP zero-copy and non-zero-copy code. Performance prior this patch: $ sudo ./xdp_rxq_info --dev enp134s0f0 --action XDP_DROP Running XDP on dev:enp134s0f0 (ifindex:7) action:XDP_DROP options:no_touch XDP stats CPU pps issue-pps XDP-RX CPU 20 18983018 0 XDP-RX CPU total 18983018 RXQ stats RXQ:CPU pps issue-pps rx_queue_index 20:20 18983012 0 rx_queue_index 20:sum 18983012 $ sudo ./xdpsock -i enp134s0f0 -q 20 -n 2 -z -r sock0@enp134s0f0:20 rxdrop pps pkts 2.00 rx 14,641,496 144,751,092 tx 0 0 And after: $ sudo ./xdp_rxq_info --dev enp134s0f0 --action XDP_DROP Running XDP on dev:enp134s0f0 (ifindex:7) action:XDP_DROP options:no_touch XDP stats CPU pps issue-pps XDP-RX CPU 20 24000986 0 XDP-RX CPU total 24000986 RXQ stats RXQ:CPU pps issue-pps rx_queue_index 20:20 24000985 0 rx_queue_index 20:sum 24000985 +26% $ sudo ./xdpsock -i enp134s0f0 -q 20 -n 2 -z -r sock0@enp134s0f0:20 rxdrop pps pkts 2.00 rx 17,623,578 163,503,263 tx 0 0 +20% Signed-off-by: Björn Töpel --- drivers/net/ethernet/intel/i40e/i40e_txrx.c | 31 ++++--------------- .../ethernet/intel/i40e/i40e_txrx_common.h | 27 ++++++++++++++++ drivers/net/ethernet/intel/i40e/i40e_xsk.c | 24 ++------------ 3 files changed, 35 insertions(+), 47 deletions(-) diff --git a/drivers/net/ethernet/intel/i40e/i40e_txrx.c b/drivers/net/ethernet/intel/i40e/i40e_txrx.c index a7e14e98889f..b339b7ee6380 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_txrx.c +++ b/drivers/net/ethernet/intel/i40e/i40e_txrx.c @@ -2,7 +2,6 @@ /* Copyright(c) 2013 - 2018 Intel Corporation. */ #include -#include #include #include "i40e.h" #include "i40e_trace.h" @@ -2195,41 +2194,23 @@ int i40e_xmit_xdp_tx_ring(struct xdp_buff *xdp, struct i40e_ring *xdp_ring) static struct sk_buff *i40e_run_xdp(struct i40e_ring *rx_ring, struct xdp_buff *xdp) { - int err, result = I40E_XDP_PASS; - struct i40e_ring *xdp_ring; struct bpf_prog *xdp_prog; + int result; u32 act; rcu_read_lock(); xdp_prog = READ_ONCE(rx_ring->xdp_prog); - if (!xdp_prog) + if (!xdp_prog) { + result = I40E_XDP_PASS; goto xdp_out; + } prefetchw(xdp->data_hard_start); /* xdp_frame write */ act = bpf_prog_run_xdp(xdp_prog, xdp); - switch (act) { - case XDP_PASS: - break; - case XDP_TX: - xdp_ring = rx_ring->vsi->xdp_rings[rx_ring->queue_index]; - result = i40e_xmit_xdp_tx_ring(xdp, xdp_ring); - break; - case XDP_REDIRECT: - err = xdp_do_redirect(rx_ring->netdev, xdp, xdp_prog); - result = !err ? I40E_XDP_REDIR : I40E_XDP_CONSUMED; - break; - default: - bpf_warn_invalid_xdp_action(act); - /* fall through */ - case XDP_ABORTED: - trace_xdp_exception(rx_ring->netdev, xdp_prog, act); - /* fall through -- handle aborts by dropping packet */ - case XDP_DROP: - result = I40E_XDP_CONSUMED; - break; - } + i40e_xdp_do_action(act, &result, rx_ring, xdp, xdp_prog); + xdp_out: rcu_read_unlock(); return ERR_PTR(-result); diff --git a/drivers/net/ethernet/intel/i40e/i40e_txrx_common.h b/drivers/net/ethernet/intel/i40e/i40e_txrx_common.h index 8af0e99c6c0d..8cc4d8365f9e 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_txrx_common.h +++ b/drivers/net/ethernet/intel/i40e/i40e_txrx_common.h @@ -4,6 +4,8 @@ #ifndef I40E_TXRX_COMMON_ #define I40E_TXRX_COMMON_ +#include + void i40e_fd_handle_status(struct i40e_ring *rx_ring, union i40e_rx_desc *rx_desc, u8 prog_id); int i40e_xmit_xdp_tx_ring(struct xdp_buff *xdp, struct i40e_ring *xdp_ring); @@ -88,4 +90,29 @@ void i40e_xsk_clean_rx_ring(struct i40e_ring *rx_ring); void i40e_xsk_clean_tx_ring(struct i40e_ring *tx_ring); bool i40e_xsk_any_rx_ring_enabled(struct i40e_vsi *vsi); +static inline void i40e_xdp_do_action(u32 act, int *result, + struct i40e_ring *rx_ring, + struct xdp_buff *xdp, + struct bpf_prog *xdp_prog) +{ + struct i40e_ring *xdp_ring; + int err; + + if (act == XDP_TX) { + xdp_ring = rx_ring->vsi->xdp_rings[rx_ring->queue_index]; + *result = i40e_xmit_xdp_tx_ring(xdp, xdp_ring); + } else if (act == XDP_REDIRECT) { + err = xdp_do_redirect(rx_ring->netdev, xdp, xdp_prog); + *result = !err ? I40E_XDP_REDIR : I40E_XDP_CONSUMED; + } else if (act == XDP_PASS) { + *result = I40E_XDP_PASS; + } else if (act == XDP_DROP) { + *result = I40E_XDP_CONSUMED; + } else { + if (act != XDP_ABORTED) + bpf_warn_invalid_xdp_action(act); + trace_xdp_exception(rx_ring->netdev, xdp_prog, act); + *result = I40E_XDP_CONSUMED; + } +} #endif /* I40E_TXRX_COMMON_ */ diff --git a/drivers/net/ethernet/intel/i40e/i40e_xsk.c b/drivers/net/ethernet/intel/i40e/i40e_xsk.c index 870cf654e436..1ed56475ec78 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_xsk.c +++ b/drivers/net/ethernet/intel/i40e/i40e_xsk.c @@ -282,9 +282,8 @@ int i40e_xsk_umem_setup(struct i40e_vsi *vsi, struct xdp_umem *umem, **/ static int i40e_run_xdp_zc(struct i40e_ring *rx_ring, struct xdp_buff *xdp) { - int err, result = I40E_XDP_PASS; - struct i40e_ring *xdp_ring; struct bpf_prog *xdp_prog; + int result; u32 act; rcu_read_lock(); @@ -294,26 +293,7 @@ static int i40e_run_xdp_zc(struct i40e_ring *rx_ring, struct xdp_buff *xdp) xdp_prog = READ_ONCE(rx_ring->xdp_prog); act = bpf_prog_run_xdp(xdp_prog, xdp); xdp->handle += xdp->data - xdp->data_hard_start; - switch (act) { - case XDP_PASS: - break; - case XDP_TX: - xdp_ring = rx_ring->vsi->xdp_rings[rx_ring->queue_index]; - result = i40e_xmit_xdp_tx_ring(xdp, xdp_ring); - break; - case XDP_REDIRECT: - err = xdp_do_redirect(rx_ring->netdev, xdp, xdp_prog); - result = !err ? I40E_XDP_REDIR : I40E_XDP_CONSUMED; - break; - default: - bpf_warn_invalid_xdp_action(act); - case XDP_ABORTED: - trace_xdp_exception(rx_ring->netdev, xdp_prog, act); - /* fallthrough -- handle aborts by dropping packet */ - case XDP_DROP: - result = I40E_XDP_CONSUMED; - break; - } + i40e_xdp_do_action(act, &result, rx_ring, xdp, xdp_prog); rcu_read_unlock(); return result; } -- 2.19.1 From mboxrd@z Thu Jan 1 00:00:00 1970 From: bjorn.topel@gmail.com Date: Mon, 21 Jan 2019 17:33:56 +0100 Subject: [Intel-wired-lan] [PATCH] i40e: replace switch-statement with if-clause Message-ID: <20190121163356.31332-1-bjorn.topel@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: intel-wired-lan@osuosl.org List-ID: From: Bj?rn T?pel GCC will generate jump tables for switch-statements with more than 5 case statements. An entry into the jump table is an indirect call, which means that for CONFIG_RETPOLINE builds, this is rather expensive. This commit replaces the switch-statement that acts on the XDP program result with an if-clause. The if-clause was also refactored into a common function that can be used by AF_XDP zero-copy and non-zero-copy code. Performance prior this patch: $ sudo ./xdp_rxq_info --dev enp134s0f0 --action XDP_DROP Running XDP on dev:enp134s0f0 (ifindex:7) action:XDP_DROP options:no_touch XDP stats CPU pps issue-pps XDP-RX CPU 20 18983018 0 XDP-RX CPU total 18983018 RXQ stats RXQ:CPU pps issue-pps rx_queue_index 20:20 18983012 0 rx_queue_index 20:sum 18983012 $ sudo ./xdpsock -i enp134s0f0 -q 20 -n 2 -z -r sock0 at enp134s0f0:20 rxdrop pps pkts 2.00 rx 14,641,496 144,751,092 tx 0 0 And after: $ sudo ./xdp_rxq_info --dev enp134s0f0 --action XDP_DROP Running XDP on dev:enp134s0f0 (ifindex:7) action:XDP_DROP options:no_touch XDP stats CPU pps issue-pps XDP-RX CPU 20 24000986 0 XDP-RX CPU total 24000986 RXQ stats RXQ:CPU pps issue-pps rx_queue_index 20:20 24000985 0 rx_queue_index 20:sum 24000985 +26% $ sudo ./xdpsock -i enp134s0f0 -q 20 -n 2 -z -r sock0 at enp134s0f0:20 rxdrop pps pkts 2.00 rx 17,623,578 163,503,263 tx 0 0 +20% Signed-off-by: Bj?rn T?pel --- drivers/net/ethernet/intel/i40e/i40e_txrx.c | 31 ++++--------------- .../ethernet/intel/i40e/i40e_txrx_common.h | 27 ++++++++++++++++ drivers/net/ethernet/intel/i40e/i40e_xsk.c | 24 ++------------ 3 files changed, 35 insertions(+), 47 deletions(-) diff --git a/drivers/net/ethernet/intel/i40e/i40e_txrx.c b/drivers/net/ethernet/intel/i40e/i40e_txrx.c index a7e14e98889f..b339b7ee6380 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_txrx.c +++ b/drivers/net/ethernet/intel/i40e/i40e_txrx.c @@ -2,7 +2,6 @@ /* Copyright(c) 2013 - 2018 Intel Corporation. */ #include -#include #include #include "i40e.h" #include "i40e_trace.h" @@ -2195,41 +2194,23 @@ int i40e_xmit_xdp_tx_ring(struct xdp_buff *xdp, struct i40e_ring *xdp_ring) static struct sk_buff *i40e_run_xdp(struct i40e_ring *rx_ring, struct xdp_buff *xdp) { - int err, result = I40E_XDP_PASS; - struct i40e_ring *xdp_ring; struct bpf_prog *xdp_prog; + int result; u32 act; rcu_read_lock(); xdp_prog = READ_ONCE(rx_ring->xdp_prog); - if (!xdp_prog) + if (!xdp_prog) { + result = I40E_XDP_PASS; goto xdp_out; + } prefetchw(xdp->data_hard_start); /* xdp_frame write */ act = bpf_prog_run_xdp(xdp_prog, xdp); - switch (act) { - case XDP_PASS: - break; - case XDP_TX: - xdp_ring = rx_ring->vsi->xdp_rings[rx_ring->queue_index]; - result = i40e_xmit_xdp_tx_ring(xdp, xdp_ring); - break; - case XDP_REDIRECT: - err = xdp_do_redirect(rx_ring->netdev, xdp, xdp_prog); - result = !err ? I40E_XDP_REDIR : I40E_XDP_CONSUMED; - break; - default: - bpf_warn_invalid_xdp_action(act); - /* fall through */ - case XDP_ABORTED: - trace_xdp_exception(rx_ring->netdev, xdp_prog, act); - /* fall through -- handle aborts by dropping packet */ - case XDP_DROP: - result = I40E_XDP_CONSUMED; - break; - } + i40e_xdp_do_action(act, &result, rx_ring, xdp, xdp_prog); + xdp_out: rcu_read_unlock(); return ERR_PTR(-result); diff --git a/drivers/net/ethernet/intel/i40e/i40e_txrx_common.h b/drivers/net/ethernet/intel/i40e/i40e_txrx_common.h index 8af0e99c6c0d..8cc4d8365f9e 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_txrx_common.h +++ b/drivers/net/ethernet/intel/i40e/i40e_txrx_common.h @@ -4,6 +4,8 @@ #ifndef I40E_TXRX_COMMON_ #define I40E_TXRX_COMMON_ +#include + void i40e_fd_handle_status(struct i40e_ring *rx_ring, union i40e_rx_desc *rx_desc, u8 prog_id); int i40e_xmit_xdp_tx_ring(struct xdp_buff *xdp, struct i40e_ring *xdp_ring); @@ -88,4 +90,29 @@ void i40e_xsk_clean_rx_ring(struct i40e_ring *rx_ring); void i40e_xsk_clean_tx_ring(struct i40e_ring *tx_ring); bool i40e_xsk_any_rx_ring_enabled(struct i40e_vsi *vsi); +static inline void i40e_xdp_do_action(u32 act, int *result, + struct i40e_ring *rx_ring, + struct xdp_buff *xdp, + struct bpf_prog *xdp_prog) +{ + struct i40e_ring *xdp_ring; + int err; + + if (act == XDP_TX) { + xdp_ring = rx_ring->vsi->xdp_rings[rx_ring->queue_index]; + *result = i40e_xmit_xdp_tx_ring(xdp, xdp_ring); + } else if (act == XDP_REDIRECT) { + err = xdp_do_redirect(rx_ring->netdev, xdp, xdp_prog); + *result = !err ? I40E_XDP_REDIR : I40E_XDP_CONSUMED; + } else if (act == XDP_PASS) { + *result = I40E_XDP_PASS; + } else if (act == XDP_DROP) { + *result = I40E_XDP_CONSUMED; + } else { + if (act != XDP_ABORTED) + bpf_warn_invalid_xdp_action(act); + trace_xdp_exception(rx_ring->netdev, xdp_prog, act); + *result = I40E_XDP_CONSUMED; + } +} #endif /* I40E_TXRX_COMMON_ */ diff --git a/drivers/net/ethernet/intel/i40e/i40e_xsk.c b/drivers/net/ethernet/intel/i40e/i40e_xsk.c index 870cf654e436..1ed56475ec78 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_xsk.c +++ b/drivers/net/ethernet/intel/i40e/i40e_xsk.c @@ -282,9 +282,8 @@ int i40e_xsk_umem_setup(struct i40e_vsi *vsi, struct xdp_umem *umem, **/ static int i40e_run_xdp_zc(struct i40e_ring *rx_ring, struct xdp_buff *xdp) { - int err, result = I40E_XDP_PASS; - struct i40e_ring *xdp_ring; struct bpf_prog *xdp_prog; + int result; u32 act; rcu_read_lock(); @@ -294,26 +293,7 @@ static int i40e_run_xdp_zc(struct i40e_ring *rx_ring, struct xdp_buff *xdp) xdp_prog = READ_ONCE(rx_ring->xdp_prog); act = bpf_prog_run_xdp(xdp_prog, xdp); xdp->handle += xdp->data - xdp->data_hard_start; - switch (act) { - case XDP_PASS: - break; - case XDP_TX: - xdp_ring = rx_ring->vsi->xdp_rings[rx_ring->queue_index]; - result = i40e_xmit_xdp_tx_ring(xdp, xdp_ring); - break; - case XDP_REDIRECT: - err = xdp_do_redirect(rx_ring->netdev, xdp, xdp_prog); - result = !err ? I40E_XDP_REDIR : I40E_XDP_CONSUMED; - break; - default: - bpf_warn_invalid_xdp_action(act); - case XDP_ABORTED: - trace_xdp_exception(rx_ring->netdev, xdp_prog, act); - /* fallthrough -- handle aborts by dropping packet */ - case XDP_DROP: - result = I40E_XDP_CONSUMED; - break; - } + i40e_xdp_do_action(act, &result, rx_ring, xdp, xdp_prog); rcu_read_unlock(); return result; } -- 2.19.1