Re: [xdp-hints] Re: [PATCH bpf-next v7 15/17] net/mlx5e: Introduce wrapper for xdp_buff

From: Tariq Toukan <ttoukan.linux@gmail.com>
To: "Toke Høiland-Jørgensen" <toke@redhat.com>,
	"Stanislav Fomichev" <sdf@google.com>
Cc: bpf@vger.kernel.org, ast@kernel.org, daniel@iogearbox.net,
	andrii@kernel.org, martin.lau@linux.dev, song@kernel.org,
	yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org,
	haoluo@google.com, jolsa@kernel.org,
	Saeed Mahameed <saeedm@nvidia.com>,
	David Ahern <dsahern@gmail.com>, Jakub Kicinski <kuba@kernel.org>,
	Willem de Bruijn <willemb@google.com>,
	Jesper Dangaard Brouer <brouer@redhat.com>,
	Anatoly Burakov <anatoly.burakov@intel.com>,
	Alexander Lobakin <alexandr.lobakin@intel.com>,
	Magnus Karlsson <magnus.karlsson@gmail.com>,
	Maryam Tahhan <mtahhan@redhat.com>,
	xdp-hints@xdp-project.net, netdev@vger.kernel.org
Subject: Re: [xdp-hints] Re: [PATCH bpf-next v7 15/17] net/mlx5e: Introduce wrapper for xdp_buff
Date: Sun, 15 Jan 2023 08:59:15 +0200	[thread overview]
Message-ID: <493fd525-10b3-c136-8458-a1560ed2cdcb@gmail.com> (raw)
In-Reply-To: <87358ef7e8.fsf@toke.dk>

On 13/01/2023 23:31, Toke Høiland-Jørgensen wrote:
> Tariq Toukan <ttoukan.linux@gmail.com> writes:
> 
>> On 12/01/2023 23:55, Toke Høiland-Jørgensen wrote:
>>> Toke Høiland-Jørgensen <toke@redhat.com> writes:
>>>
>>>> Stanislav Fomichev <sdf@google.com> writes:
>>>>
>>>>> On Thu, Jan 12, 2023 at 12:07 AM Tariq Toukan <ttoukan.linux@gmail.com> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 12/01/2023 2:32, Stanislav Fomichev wrote:
>>>>>>> From: Toke Høiland-Jørgensen <toke@redhat.com>
>>>>>>>
>>>>>>> Preparation for implementing HW metadata kfuncs. No functional change.
>>>>>>>
>>>>>>> Cc: Tariq Toukan <tariqt@nvidia.com>
>>>>>>> Cc: Saeed Mahameed <saeedm@nvidia.com>
>>>>>>> Cc: John Fastabend <john.fastabend@gmail.com>
>>>>>>> Cc: David Ahern <dsahern@gmail.com>
>>>>>>> Cc: Martin KaFai Lau <martin.lau@linux.dev>
>>>>>>> Cc: Jakub Kicinski <kuba@kernel.org>
>>>>>>> Cc: Willem de Bruijn <willemb@google.com>
>>>>>>> Cc: Jesper Dangaard Brouer <brouer@redhat.com>
>>>>>>> Cc: Anatoly Burakov <anatoly.burakov@intel.com>
>>>>>>> Cc: Alexander Lobakin <alexandr.lobakin@intel.com>
>>>>>>> Cc: Magnus Karlsson <magnus.karlsson@gmail.com>
>>>>>>> Cc: Maryam Tahhan <mtahhan@redhat.com>
>>>>>>> Cc: xdp-hints@xdp-project.net
>>>>>>> Cc: netdev@vger.kernel.org
>>>>>>> Signed-off-by: Toke Høiland-Jørgensen <toke@redhat.com>
>>>>>>> Signed-off-by: Stanislav Fomichev <sdf@google.com>
>>>>>>> ---
>>>>>>>     drivers/net/ethernet/mellanox/mlx5/core/en.h  |  1 +
>>>>>>>     .../net/ethernet/mellanox/mlx5/core/en/xdp.c  |  3 +-
>>>>>>>     .../net/ethernet/mellanox/mlx5/core/en/xdp.h  |  6 +-
>>>>>>>     .../ethernet/mellanox/mlx5/core/en/xsk/rx.c   | 25 ++++----
>>>>>>>     .../net/ethernet/mellanox/mlx5/core/en_rx.c   | 58 +++++++++----------
>>>>>>>     5 files changed, 50 insertions(+), 43 deletions(-)
>>>>>>>
>>>>>>> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h
>>>>>>> index 2d77fb8a8a01..af663978d1b4 100644
>>>>>>> --- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
>>>>>>> +++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h
>>>>>>> @@ -469,6 +469,7 @@ struct mlx5e_txqsq {
>>>>>>>     union mlx5e_alloc_unit {
>>>>>>>         struct page *page;
>>>>>>>         struct xdp_buff *xsk;
>>>>>>> +     struct mlx5e_xdp_buff *mxbuf;
>>>>>>
>>>>>> In XSK files below you mix usage of both alloc_units[page_idx].mxbuf and
>>>>>> alloc_units[page_idx].xsk, while both fields share the memory of a union.
>>>>>>
>>>>>> As struct mlx5e_xdp_buff wraps struct xdp_buff, I think that you just
>>>>>> need to change the existing xsk field type from struct xdp_buff *xsk
>>>>>> into struct mlx5e_xdp_buff *xsk and align the usage.
>>>>>
>>>>> Hmmm, good point. I'm actually not sure how it works currently.
>>>>> mlx5e_alloc_unit.mxbuf doesn't seem to be initialized anywhere? Toke,
>>>>> am I missing something?
>>>>
>>>> It's initialised piecemeal in different places; but yeah, we're mixing
>>>> things a bit...
>>>>
>>>>> I'm thinking about something like this:
>>>>>
>>>>> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h
>>>>> b/drivers/net/ethernet/mellanox/mlx5/core/en.h
>>>>> index af663978d1b4..2d77fb8a8a01 100644
>>>>> --- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
>>>>> +++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h
>>>>> @@ -469,7 +469,6 @@ struct mlx5e_txqsq {
>>>>>    union mlx5e_alloc_unit {
>>>>>           struct page *page;
>>>>>           struct xdp_buff *xsk;
>>>>> -       struct mlx5e_xdp_buff *mxbuf;
>>>>>    };
>>>>
>>>> Hmm, for consistency with the non-XSK path we should rather go the other
>>>> direction and lose the xsk member, moving everything to mxbuf? Let me
>>>> give that a shot...
>>>
>>> Something like the below?
>>>
>>> -Toke
>>>
>>> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h
>>> index 6de02d8aeab8..cb9cdb6421c5 100644
>>> --- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
>>> +++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h
>>> @@ -468,7 +468,6 @@ struct mlx5e_txqsq {
>>>    
>>>    union mlx5e_alloc_unit {
>>>    	struct page *page;
>>> -	struct xdp_buff *xsk;
>>>    	struct mlx5e_xdp_buff *mxbuf;
>>>    };
>>>    
>>> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h
>>> index cb568c62aba0..95694a25ec31 100644
>>> --- a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h
>>> +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h
>>> @@ -33,6 +33,7 @@
>>>    #define __MLX5_EN_XDP_H__
>>>    
>>>    #include <linux/indirect_call_wrapper.h>
>>> +#include <net/xdp_sock_drv.h>
>>>    
>>>    #include "en.h"
>>>    #include "en/txrx.h"
>>> @@ -112,6 +113,21 @@ static inline void mlx5e_xmit_xdp_doorbell(struct mlx5e_xdpsq *sq)
>>>    	}
>>>    }
>>>    
>>> +static inline struct mlx5e_xdp_buff *mlx5e_xsk_buff_alloc(struct xsk_buff_pool *pool)
>>> +{
>>> +	return (struct mlx5e_xdp_buff *)xsk_buff_alloc(pool);
>>
>> What about the space needed for the rq / cqe fields? xsk_buff_alloc
>> won't allocate it.
> 
> It will! See patch 14 in the series that adds a 'cb' field to
> xdp_buff_xsk, meaning there's actually space after the xdp_buff struct
> being allocated by the xsk_buff_alloc API. The XSK_CHECK_PRIV_TYPE macro
> call is there to ensure the cb field is big enough for the struct we're
> casting to in the driver.
> 

Oh okay, got it.

>>> +}
>>> +
>>> +static inline void mlx5e_xsk_buff_free(struct mlx5e_xdp_buff *mxbuf)
>>> +{
>>> +	xsk_buff_free(&mxbuf->xdp);
>>> +}
>>> +
>>> +static inline dma_addr_t mlx5e_xsk_buff_xdp_get_frame_dma(struct mlx5e_xdp_buff *mxbuf)
>>> +{
>>> +	return xsk_buff_xdp_get_frame_dma(&mxbuf->xdp);
>>> +}
>>> +
>>>    /* Enable inline WQEs to shift some load from a congested HCA (HW) to
>>>     * a less congested cpu (SW).
>>>     */
>>> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c
>>> index 8bf3029abd3c..1f166dbb7f22 100644
>>> --- a/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c
>>> +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c
>>> @@ -3,7 +3,6 @@
>>>    
>>>    #include "rx.h"
>>>    #include "en/xdp.h"
>>> -#include <net/xdp_sock_drv.h>
>>>    #include <linux/filter.h>
>>>    
>>>    /* RX data path */
>>> @@ -21,7 +20,7 @@ int mlx5e_xsk_alloc_rx_mpwqe(struct mlx5e_rq *rq, u16 ix)
>>>    	if (unlikely(!xsk_buff_can_alloc(rq->xsk_pool, rq->mpwqe.pages_per_wqe)))
>>>    		goto err;
>>>    
>>> -	BUILD_BUG_ON(sizeof(wi->alloc_units[0]) != sizeof(wi->alloc_units[0].xsk));
>>> +	BUILD_BUG_ON(sizeof(wi->alloc_units[0]) != sizeof(wi->alloc_units[0].mxbuf));
>>>    	XSK_CHECK_PRIV_TYPE(struct mlx5e_xdp_buff);
>>>    	batch = xsk_buff_alloc_batch(rq->xsk_pool, (struct xdp_buff **)wi->alloc_units,
>>>    				     rq->mpwqe.pages_per_wqe);
>>
>> This batching API gets broken as well...
>> xsk_buff_alloc_batch fills an array of struct xdp_buff pointers, it
>> cannot correctly act on the array of struct mlx5e_xdp_buff, as it
>> contains additional fields.
> 
> See above for why this does, in fact, work. I agree it's not totally
> obvious, and in any case there's going to be a point where the cast
> happens where type safety will break, which is what I was alluding to in
> my reply to Stanislav.
> 
> I guess we could try to rework the API in xdp_sock_drv.h to make this
> more obvious instead of using the casting driver-specific wrappers I
> suggested here. Or we could go with Stanislav's suggestion of keeping
> allocation etc using xdp_buff and only casting to mlx5e_xdp_buff in the
> function where it's used; then we can keep the casting localised to that
> function, and put a comment there explaining why it works?
> 

Stanislav's proposal LGTM.
Let's keep the casting localised, and make sure there's a comment there.

>> Maybe letting mlx5e_xdp_buff point to its struct xdp_buff (instead of
>> wrapping it) will solve the problems here, then we'll loop over the
>> xdp_buff * array and copy the pointers into the struct mlx5e_xdp_buff *
>> array.
>> Need to give it deeper thoughts...
>>
>> struct mlx5e_xdp_buff {
>> 	struct xdp_buff *xdp;
>> 	struct mlx5_cqe64 *cqe;
>> 	struct mlx5e_rq *rq;
>> };
> 
> This was actually my original proposal; we discussed this back on v2 of
> this patch series. People generally felt that the 'cb' field approach
> (originally suggested by Jakub) was better.

I agree.

> See the discussion starting
> from here:
> 
> https://lore.kernel.org/r/20221123111431.7b54668e@kernel.org
> 
> -Toke
> 