From: Jesper Dangaard Brouer <brouer@redhat.com>
To: Tariq Toukan <tariqt@mellanox.com>
Cc: sameehj@amazon.com, "Saeed Mahameed" <saeedm@mellanox.com>,
netdev@vger.kernel.org, bpf@vger.kernel.org, zorik@amazon.com,
akiyano@amazon.com, gtzalik@amazon.com,
"Toke Høiland-Jørgensen" <toke@redhat.com>,
"Daniel Borkmann" <borkmann@iogearbox.net>,
"Alexei Starovoitov" <alexei.starovoitov@gmail.com>,
"John Fastabend" <john.fastabend@gmail.com>,
"Alexander Duyck" <alexander.duyck@gmail.com>,
"Jeff Kirsher" <jeffrey.t.kirsher@intel.com>,
"David Ahern" <dsahern@gmail.com>,
"Willem de Bruijn" <willemdebruijn.kernel@gmail.com>,
"Ilias Apalodimas" <ilias.apalodimas@linaro.org>,
"Lorenzo Bianconi" <lorenzo@kernel.org>,
steffen.klassert@secunet.com, brouer@redhat.com
Subject: Re: [PATCH net-next v2 28/33] mlx5: rx queue setup time determine frame_sz for XDP
Date: Fri, 1 May 2020 15:01:58 +0200 [thread overview]
Message-ID: <20200501150158.05e647f7@carbon> (raw)
In-Reply-To: <a5be329e-39e3-fdfc-500d-383953546d40@mellanox.com>
On Thu, 30 Apr 2020 20:07:43 +0300
Tariq Toukan <tariqt@mellanox.com> wrote:
> On 4/30/2020 2:22 PM, Jesper Dangaard Brouer wrote:
> > The mlx5 driver have multiple memory models, which are also changed
> > according to whether a XDP bpf_prog is attached.
> >
> > The 'rx_striding_rq' setting is adjusted via ethtool priv-flags e.g.:
> > # ethtool --set-priv-flags mlx5p2 rx_striding_rq off
> >
> > On the general case with 4K page_size and regular MTU packet, then
> > the frame_sz is 2048 and 4096 when XDP is enabled, in both modes.
> >
> > The info on the given frame size is stored differently depending on the
> > RQ-mode and encoded in a union in struct mlx5e_rq union wqe/mpwqe.
> > In rx striding mode rq->mpwqe.log_stride_sz is either 11 or 12, which
> > corresponds to 2048 or 4096 (MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ).
> > In non-striding mode (MLX5_WQ_TYPE_CYCLIC) the frag_stride is stored
> > in rq->wqe.info.arr[0].frag_stride, for the first fragment, which is
> > what the XDP case cares about.
> >
> > To reduce effect on fast-path, this patch determine the frame_sz at
> > setup time, to avoid determining the memory model runtime. Variable
> > is named first_frame_sz to make it clear that this is only the frame
> > size of the first fragment.
> >
> > This mlx5 driver does a DMA-sync on XDP_TX action, but grow is safe
> > as it have done a DMA-map on the entire PAGE_SIZE. The driver also
> > already does a XDP length check against sq->hw_mtu on the possible
> > XDP xmit paths mlx5e_xmit_xdp_frame() + mlx5e_xmit_xdp_frame_mpwqe().
> >
> > V2: Fix that frag_size need to be recalc before creating SKB.
> >
> > Cc: Tariq Toukan <tariqt@mellanox.com>
> > Cc: Saeed Mahameed <saeedm@mellanox.com>
> > Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
> > ---
> > drivers/net/ethernet/mellanox/mlx5/core/en.h | 1 +
> > drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c | 1 +
> > drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 6 ++++++
> > drivers/net/ethernet/mellanox/mlx5/core/en_rx.c | 2 ++
> > 4 files changed, 10 insertions(+)
> >
[... cut ...]
> > diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
> > index e2beb89c1832..04671ed977a5 100644
> > --- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
> > +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
> > @@ -1084,6 +1084,7 @@ mlx5e_skb_from_cqe_linear(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe,
> > if (consumed)
> > return NULL; /* page/packet was consumed by XDP */
> >
> > + frag_size = MLX5_SKB_FRAG_SZ(rx_headroom + cqe_bcnt);
>
> This is a re-calculation of frag_size, using the exact same command used
> earlier in this function, but with a newer value of rx_headroom.
> This wasn't part of the previous patchset. I understand the need.
Yes, kernel will crash without this change.
> However, this code repetition looks weird and non-optimal to me. I think
> we can come up with something better.
>
> > skb = mlx5e_build_linear_skb(rq, va, frag_size, rx_headroom, cqe_bcnt);
> > if (unlikely(!skb))
> > return NULL;
> > @@ -1385,6 +1386,7 @@ mlx5e_skb_from_cqe_mpwrq_linear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *wi,
> > return NULL; /* page/packet was consumed by XDP */
> > }
> >
> > + frag_size = MLX5_SKB_FRAG_SZ(rx_headroom + cqe_bcnt32);
>
> Same here.
>
> > skb = mlx5e_build_linear_skb(rq, va, frag_size, rx_headroom, cqe_bcnt32);
> > if (unlikely(!skb))
> > return NULL;
> >
> >
>
> My suggetion is:
> Pass &frag_size to mlx5e_xdp_handle(), and update it within it, just
> next to the update of rx_headroom.
> All the needed information is there: the new rx_headroom, and cqe_bcnt.
First of all, passing yet-another argument to mlx5e_xdp_handle(), also
looks weird, and is on the brink of becoming a performance issue, as on
x86_64 you can pass max 6 arguments in registers before they get pushed
on the stack. Adding this would be the 7th argument.
Second the MLX5_SKB_FRAG_SZ() calculation is also weird, because it
does not provide any tailroom in the packet, I guess it is for
supporting another memory mode, as in case XDP is activated there are
plenty of tailroom.
I though about increasing the frag_size, but I choose not to, because
then this patch would change the driver behavior beyond adding frame_sz
for XDP.
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
bool mlx5e_xdp_handle(struct mlx5e_rq *rq, struct mlx5e_dma_info *di,
void *va, u16 *rx_headroom, u32 *len, bool xsk)
{
[...]
}
next prev parent reply other threads:[~2020-05-01 13:02 UTC|newest]
Thread overview: 47+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <158824557985.2172139.4173570969543904434.stgit@firesoul>
2020-04-30 11:20 ` [PATCH net-next v2 01/33] xdp: add frame size to xdp_buff Jesper Dangaard Brouer
2020-04-30 11:20 ` [PATCH net-next v2 02/33] bnxt: add XDP frame size to driver Jesper Dangaard Brouer
2020-04-30 11:20 ` [PATCH net-next v2 03/33] sfc: add XDP frame size Jesper Dangaard Brouer
2020-04-30 11:20 ` [PATCH net-next v2 04/33] mvneta: add XDP frame size to driver Jesper Dangaard Brouer
2020-04-30 11:20 ` [PATCH net-next v2 05/33] net: netsec: Add support for XDP frame size Jesper Dangaard Brouer
2020-04-30 11:20 ` [PATCH net-next v2 06/33] net: XDP-generic determining " Jesper Dangaard Brouer
2020-04-30 11:20 ` [PATCH net-next v2 07/33] xdp: xdp_frame add member frame_sz and handle in convert_to_xdp_frame Jesper Dangaard Brouer
2020-04-30 11:21 ` [PATCH net-next v2 08/33] xdp: cpumap redirect use frame_sz and increase skb_tailroom Jesper Dangaard Brouer
2020-04-30 11:21 ` [PATCH net-next v2 09/33] veth: adjust hard_start offset on redirect XDP frames Jesper Dangaard Brouer
2020-04-30 11:21 ` [PATCH net-next v2 10/33] veth: xdp using frame_sz in veth driver Jesper Dangaard Brouer
2020-04-30 11:21 ` [PATCH net-next v2 11/33] dpaa2-eth: add XDP frame size Jesper Dangaard Brouer
2020-04-30 11:21 ` [PATCH net-next v2 12/33] hv_netvsc: add XDP frame size to driver Jesper Dangaard Brouer
2020-04-30 14:20 ` Haiyang Zhang
2020-05-01 14:47 ` Jesper Dangaard Brouer
2020-04-30 11:21 ` [PATCH net-next v2 13/33] qlogic/qede: " Jesper Dangaard Brouer
2020-04-30 11:21 ` [PATCH net-next v2 14/33] net: ethernet: ti: add XDP frame size to driver cpsw Jesper Dangaard Brouer
2020-04-30 11:21 ` [PATCH net-next v2 15/33] ena: add XDP frame size to amazon NIC driver Jesper Dangaard Brouer
2020-04-30 11:21 ` [PATCH net-next v2 16/33] mlx4: add XDP frame size and adjust max XDP MTU Jesper Dangaard Brouer
2020-04-30 11:21 ` [PATCH net-next v2 17/33] net: thunderx: add XDP frame size Jesper Dangaard Brouer
2020-04-30 11:21 ` [PATCH net-next v2 18/33] nfp: add XDP frame size to netronome driver Jesper Dangaard Brouer
2020-04-30 11:21 ` [PATCH net-next v2 19/33] tun: add XDP frame size Jesper Dangaard Brouer
2020-05-06 20:32 ` Michael S. Tsirkin
2020-04-30 11:22 ` [PATCH net-next v2 20/33] vhost_net: also populate " Jesper Dangaard Brouer
2020-05-06 6:41 ` Jason Wang
2020-05-06 6:49 ` Jason Wang
2020-05-06 20:33 ` Michael S. Tsirkin
2020-04-30 11:22 ` [PATCH net-next v2 21/33] virtio_net: add XDP frame size in two code paths Jesper Dangaard Brouer
2020-05-06 20:34 ` Michael S. Tsirkin
2020-05-08 2:05 ` Jason Wang
2020-05-08 7:21 ` Jesper Dangaard Brouer
2020-04-30 11:22 ` [PATCH net-next v2 22/33] ixgbe: fix XDP redirect on archs with PAGE_SIZE above 4K Jesper Dangaard Brouer
2020-04-30 11:22 ` [PATCH net-next v2 23/33] ixgbe: add XDP frame size to driver Jesper Dangaard Brouer
2020-04-30 11:22 ` [PATCH net-next v2 24/33] ixgbevf: add XDP frame size to VF driver Jesper Dangaard Brouer
2020-04-30 11:22 ` [PATCH net-next v2 25/33] i40e: add XDP frame size to driver Jesper Dangaard Brouer
2020-04-30 11:22 ` [PATCH net-next v2 26/33] ice: " Jesper Dangaard Brouer
2020-04-30 11:22 ` [PATCH net-next v2 27/33] xdp: for Intel AF_XDP drivers add XDP frame_sz Jesper Dangaard Brouer
2020-04-30 11:22 ` [PATCH net-next v2 28/33] mlx5: rx queue setup time determine frame_sz for XDP Jesper Dangaard Brouer
2020-04-30 17:07 ` Tariq Toukan
2020-04-30 17:12 ` Tariq Toukan
2020-05-01 12:32 ` Jesper Dangaard Brouer
2020-05-08 10:49 ` Jesper Dangaard Brouer
2020-05-01 13:01 ` Jesper Dangaard Brouer [this message]
2020-04-30 11:22 ` [PATCH net-next v2 29/33] xdp: allow bpf_xdp_adjust_tail() to grow packet size Jesper Dangaard Brouer
2020-04-30 11:22 ` [PATCH net-next v2 30/33] xdp: clear grow memory in bpf_xdp_adjust_tail() Jesper Dangaard Brouer
2020-04-30 11:22 ` [PATCH net-next v2 31/33] bpf: add xdp.frame_sz in bpf_prog_test_run_xdp() Jesper Dangaard Brouer
2020-04-30 11:23 ` [PATCH net-next v2 32/33] selftests/bpf: adjust BPF selftest for xdp_adjust_tail Jesper Dangaard Brouer
2020-04-30 11:23 ` [PATCH net-next v2 33/33] selftests/bpf: xdp_adjust_tail add grow tail tests Jesper Dangaard Brouer
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200501150158.05e647f7@carbon \
--to=brouer@redhat.com \
--cc=akiyano@amazon.com \
--cc=alexander.duyck@gmail.com \
--cc=alexei.starovoitov@gmail.com \
--cc=borkmann@iogearbox.net \
--cc=bpf@vger.kernel.org \
--cc=dsahern@gmail.com \
--cc=gtzalik@amazon.com \
--cc=ilias.apalodimas@linaro.org \
--cc=jeffrey.t.kirsher@intel.com \
--cc=john.fastabend@gmail.com \
--cc=lorenzo@kernel.org \
--cc=netdev@vger.kernel.org \
--cc=saeedm@mellanox.com \
--cc=sameehj@amazon.com \
--cc=steffen.klassert@secunet.com \
--cc=tariqt@mellanox.com \
--cc=toke@redhat.com \
--cc=willemdebruijn.kernel@gmail.com \
--cc=zorik@amazon.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).