* [PATCH net-next v3 0/2] net/mlx5: use indirect call wrappers @ 2019-06-12 10:18 Paolo Abeni 2019-06-12 10:18 ` [PATCH net-next v3 1/2] net/mlx5e: use indirect calls wrapper for skb allocation Paolo Abeni ` (2 more replies) 0 siblings, 3 replies; 6+ messages in thread From: Paolo Abeni @ 2019-06-12 10:18 UTC (permalink / raw) To: netdev; +Cc: David S. Miller, Saeed Mahameed, Leon Romanovsky The mlx5_core driver uses several indirect calls in fast-path, some of them are invoked on each ingress packet, even for the XDP-only traffic. This series leverage the indirect call wrappers infrastructure the avoid the expansive RETPOLINE overhead for 2 indirect calls in fast-path. Each call is addressed on a different patch, plus we need to introduce a couple of additional helpers to cope with the higher number of possible direct-call alternatives. v2 -> v3: - do not add more INDIRECT_CALL_* macros - use only the direct calls always available regardless of the mlx5 build options in the last patch v1 -> v2: - update the direct call list and use a macro to define it, as per Saeed suggestion. An intermediated additional macro is needed to allow arg list expansion - patch 2/3 is unchanged, as the generated code looks better this way than with possible alternative (dropping BP hits) Paolo Abeni (2): net/mlx5e: use indirect calls wrapper for skb allocation net/mlx5e: use indirect calls wrapper for the rx packet handler .../net/ethernet/mellanox/mlx5/core/en_rx.c | 25 ++++++++++++++----- 1 file changed, 19 insertions(+), 6 deletions(-) -- 2.20.1 ^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH net-next v3 1/2] net/mlx5e: use indirect calls wrapper for skb allocation 2019-06-12 10:18 [PATCH net-next v3 0/2] net/mlx5: use indirect call wrappers Paolo Abeni @ 2019-06-12 10:18 ` Paolo Abeni 2019-06-14 18:41 ` Saeed Mahameed 2019-06-12 10:18 ` [PATCH net-next v3 2/2] net/mlx5e: use indirect calls wrapper for the rx packet handler Paolo Abeni 2019-06-14 22:35 ` [PATCH net-next v3 0/2] net/mlx5: use indirect call wrappers David Miller 2 siblings, 1 reply; 6+ messages in thread From: Paolo Abeni @ 2019-06-12 10:18 UTC (permalink / raw) To: netdev; +Cc: David S. Miller, Saeed Mahameed, Leon Romanovsky We can avoid an indirect call per packet wrapping the skb creation with the appropriate helper. Signed-off-by: Paolo Abeni <pabeni@redhat.com> --- .../net/ethernet/mellanox/mlx5/core/en_rx.c | 22 ++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c index 13133e7f088e..0fe5f13d07cc 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c @@ -34,6 +34,7 @@ #include <linux/ip.h> #include <linux/ipv6.h> #include <linux/tcp.h> +#include <linux/indirect_call_wrapper.h> #include <net/ip6_checksum.h> #include <net/page_pool.h> #include <net/inet_ecn.h> @@ -1092,7 +1093,10 @@ void mlx5e_handle_rx_cqe(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe) wi = get_frag(rq, ci); cqe_bcnt = be32_to_cpu(cqe->byte_cnt); - skb = rq->wqe.skb_from_cqe(rq, cqe, wi, cqe_bcnt); + skb = INDIRECT_CALL_2(rq->wqe.skb_from_cqe, + mlx5e_skb_from_cqe_linear, + mlx5e_skb_from_cqe_nonlinear, + rq, cqe, wi, cqe_bcnt); if (!skb) { /* probably for XDP */ if (__test_and_clear_bit(MLX5E_RQ_FLAG_XDP_XMIT, rq->flags)) { @@ -1279,8 +1283,10 @@ void mlx5e_handle_rx_cqe_mpwrq(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe) cqe_bcnt = mpwrq_get_cqe_byte_cnt(cqe); - skb = rq->mpwqe.skb_from_cqe_mpwrq(rq, wi, cqe_bcnt, head_offset, - page_idx); + skb = INDIRECT_CALL_2(rq->mpwqe.skb_from_cqe_mpwrq, + mlx5e_skb_from_cqe_mpwrq_linear, + mlx5e_skb_from_cqe_mpwrq_nonlinear, + rq, wi, cqe_bcnt, head_offset, page_idx); if (!skb) goto mpwrq_cqe_out; @@ -1437,7 +1443,10 @@ void mlx5i_handle_rx_cqe(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe) wi = get_frag(rq, ci); cqe_bcnt = be32_to_cpu(cqe->byte_cnt); - skb = rq->wqe.skb_from_cqe(rq, cqe, wi, cqe_bcnt); + skb = INDIRECT_CALL_2(rq->wqe.skb_from_cqe, + mlx5e_skb_from_cqe_linear, + mlx5e_skb_from_cqe_nonlinear, + rq, cqe, wi, cqe_bcnt); if (!skb) goto wq_free_wqe; @@ -1469,7 +1478,10 @@ void mlx5e_ipsec_handle_rx_cqe(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe) wi = get_frag(rq, ci); cqe_bcnt = be32_to_cpu(cqe->byte_cnt); - skb = rq->wqe.skb_from_cqe(rq, cqe, wi, cqe_bcnt); + skb = INDIRECT_CALL_2(rq->wqe.skb_from_cqe, + mlx5e_skb_from_cqe_linear, + mlx5e_skb_from_cqe_nonlinear, + rq, cqe, wi, cqe_bcnt); if (unlikely(!skb)) { /* a DROP, save the page-reuse checks */ mlx5e_free_rx_wqe(rq, wi, true); -- 2.20.1 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH net-next v3 1/2] net/mlx5e: use indirect calls wrapper for skb allocation 2019-06-12 10:18 ` [PATCH net-next v3 1/2] net/mlx5e: use indirect calls wrapper for skb allocation Paolo Abeni @ 2019-06-14 18:41 ` Saeed Mahameed 0 siblings, 0 replies; 6+ messages in thread From: Saeed Mahameed @ 2019-06-14 18:41 UTC (permalink / raw) To: netdev, pabeni; +Cc: davem, leon On Wed, 2019-06-12 at 12:18 +0200, Paolo Abeni wrote: > We can avoid an indirect call per packet wrapping the skb creation > with the appropriate helper. > > Signed-off-by: Paolo Abeni <pabeni@redhat.com> Acked-by: Saeed Mahameed <saeedm@mellanox.com> ^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH net-next v3 2/2] net/mlx5e: use indirect calls wrapper for the rx packet handler 2019-06-12 10:18 [PATCH net-next v3 0/2] net/mlx5: use indirect call wrappers Paolo Abeni 2019-06-12 10:18 ` [PATCH net-next v3 1/2] net/mlx5e: use indirect calls wrapper for skb allocation Paolo Abeni @ 2019-06-12 10:18 ` Paolo Abeni 2019-06-14 18:42 ` Saeed Mahameed 2019-06-14 22:35 ` [PATCH net-next v3 0/2] net/mlx5: use indirect call wrappers David Miller 2 siblings, 1 reply; 6+ messages in thread From: Paolo Abeni @ 2019-06-12 10:18 UTC (permalink / raw) To: netdev; +Cc: David S. Miller, Saeed Mahameed, Leon Romanovsky We can avoid another indirect call per packet wrapping the rx handler call with the proper helper. To ensure that even the last listed direct call experience measurable gain, despite the additional conditionals we must traverse before reaching it, I tested reversing the order of the listed options, with performance differences below noise level. Together with the previous indirect call patch, this gives ~6% performance improvement in raw UDP tput. v2 -> v3: - use only the direct calls always available regardless of the mlx5 build options - drop the direct call list macro, to keep the code as simple as possible for future rework v1 -> v2: - update the direct call list and use a macro to define it, as per Saeed suggestion. An intermediated additional macro is needed to allow arg list expansion Signed-off-by: Paolo Abeni <pabeni@redhat.com> --- drivers/net/ethernet/mellanox/mlx5/core/en_rx.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c index 0fe5f13d07cc..935bf62eddc1 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c @@ -1333,7 +1333,8 @@ int mlx5e_poll_rx_cq(struct mlx5e_cq *cq, int budget) mlx5_cqwq_pop(cqwq); - rq->handle_rx_cqe(rq, cqe); + INDIRECT_CALL_2(rq->handle_rx_cqe, mlx5e_handle_rx_cqe_mpwrq, + mlx5e_handle_rx_cqe, rq, cqe); } while ((++work_done < budget) && (cqe = mlx5_cqwq_get_cqe(cqwq))); out: -- 2.20.1 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH net-next v3 2/2] net/mlx5e: use indirect calls wrapper for the rx packet handler 2019-06-12 10:18 ` [PATCH net-next v3 2/2] net/mlx5e: use indirect calls wrapper for the rx packet handler Paolo Abeni @ 2019-06-14 18:42 ` Saeed Mahameed 0 siblings, 0 replies; 6+ messages in thread From: Saeed Mahameed @ 2019-06-14 18:42 UTC (permalink / raw) To: netdev, pabeni; +Cc: davem, leon On Wed, 2019-06-12 at 12:18 +0200, Paolo Abeni wrote: > We can avoid another indirect call per packet wrapping the rx > handler call with the proper helper. > > To ensure that even the last listed direct call experience > measurable gain, despite the additional conditionals we must > traverse before reaching it, I tested reversing the order of the > listed options, with performance differences below noise level. > > Together with the previous indirect call patch, this gives > ~6% performance improvement in raw UDP tput. > > v2 -> v3: > - use only the direct calls always available regardless of > the mlx5 build options > - drop the direct call list macro, to keep the code as simple > as possible for future rework > > v1 -> v2: > - update the direct call list and use a macro to define it, > as per Saeed suggestion. An intermediated additional > macro is needed to allow arg list expansion > > Signed-off-by: Paolo Abeni <pabeni@redhat.com> > Acked-by: Saeed Mahameed <saeedm@mellanox.com> ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH net-next v3 0/2] net/mlx5: use indirect call wrappers 2019-06-12 10:18 [PATCH net-next v3 0/2] net/mlx5: use indirect call wrappers Paolo Abeni 2019-06-12 10:18 ` [PATCH net-next v3 1/2] net/mlx5e: use indirect calls wrapper for skb allocation Paolo Abeni 2019-06-12 10:18 ` [PATCH net-next v3 2/2] net/mlx5e: use indirect calls wrapper for the rx packet handler Paolo Abeni @ 2019-06-14 22:35 ` David Miller 2 siblings, 0 replies; 6+ messages in thread From: David Miller @ 2019-06-14 22:35 UTC (permalink / raw) To: pabeni; +Cc: netdev, saeedm, leon From: Paolo Abeni <pabeni@redhat.com> Date: Wed, 12 Jun 2019 12:18:34 +0200 > The mlx5_core driver uses several indirect calls in fast-path, some of them > are invoked on each ingress packet, even for the XDP-only traffic. > > This series leverage the indirect call wrappers infrastructure the avoid > the expansive RETPOLINE overhead for 2 indirect calls in fast-path. > > Each call is addressed on a different patch, plus we need to introduce a couple > of additional helpers to cope with the higher number of possible direct-call > alternatives. > > v2 -> v3: > - do not add more INDIRECT_CALL_* macros > - use only the direct calls always available regardless of > the mlx5 build options in the last patch > > v1 -> v2: > - update the direct call list and use a macro to define it, > as per Saeed suggestion. An intermediated additional > macro is needed to allow arg list expansion > - patch 2/3 is unchanged, as the generated code looks better this way than > with possible alternative (dropping BP hits) Applied, thanks. ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2019-06-14 22:35 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-06-12 10:18 [PATCH net-next v3 0/2] net/mlx5: use indirect call wrappers Paolo Abeni 2019-06-12 10:18 ` [PATCH net-next v3 1/2] net/mlx5e: use indirect calls wrapper for skb allocation Paolo Abeni 2019-06-14 18:41 ` Saeed Mahameed 2019-06-12 10:18 ` [PATCH net-next v3 2/2] net/mlx5e: use indirect calls wrapper for the rx packet handler Paolo Abeni 2019-06-14 18:42 ` Saeed Mahameed 2019-06-14 22:35 ` [PATCH net-next v3 0/2] net/mlx5: use indirect call wrappers David Miller
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).