Re: mlx5 error when the skb linear space is empty

From: Saeed Mahameed <saeed@kernel.org>
To: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
Cc: Magnus Karlsson <magnus.karlsson@gmail.com>,
	Leon Romanovsky <leonro@nvidia.com>,
	netdev@vger.kernel.org
Subject: Re: mlx5 error when the skb linear space is empty
Date: Tue, 05 Jan 2021 12:51:02 -0800	[thread overview]
Message-ID: <741209d2a42d46ebdb8249caaef7531f5ad8fa76.camel@kernel.org> (raw)
In-Reply-To: <1609757998.875103-1-xuanzhuo@linux.alibaba.com>

On Mon, 2021-01-04 at 18:59 +0800, Xuan Zhuo wrote:
> hi
> 
> In the process of developing xdp socket, we tried to directly use
> page to
> construct skb directly, to avoid data copy. And the MAC information
> is also in
> the page, which caused the linear space of skb to be empty. In this
> case, I
> encountered a problem :
> 
> mlx5_core 0000:3b:00.1 eth1: Error cqe on cqn 0x817, ci 0x8, qn
> 0x1dbb, opcode 0xd, syndrome 0x1, vendor syndrome 0x68
> 00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 00000030: 00 00 00 00 60 10 68 01 0a 00 1d bb 00 0f 9f d2
> WQE DUMP: WQ size 1024 WQ cur size 0, WQE index 0xf, len: 64
> 00000000: 00 00 0f 0a 00 1d bb 03 00 00 00 08 00 00 00 00
> 00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 00000020: 00 00 00 2b 00 08 00 00 00 00 00 05 9e e3 08 00
> 00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> mlx5_core 0000:3b:00.1 eth1: ERR CQE on SQ: 0x1dbb
> 
> 
> And when I try to copy only the mac address into the linear space of
> skb, the
> other parts are still placed in the page. When constructing skb in
> this way, I
> found that although the data can be sent successfully, the sending
> performance
> is relatively poor!!
> 

Hi,

This is an expected behavior of ConnectX4-LX, ConnectX4-LX requires the
driver to copy at least the L2 headers into the linear part, in some
DCB/DSCP configuration it will require L3 headers.
to check what the current configuration, you can check from the driver
code:
mlx5e_calc_min_inline() // Calculates the minimum required headers to
copy to linear part per packet 

and sq->min_inline_mode; stores the minimum required by the FW.

This "must copy" requirement doesn't exist for ConnectX5 and above .. 

> I would like to ask, is there any way to solve this problem?
> 
> dev info:
>     driver: mlx5_core
>     version: 5.10.0+
>     firmware-version: 14.21.2328 (MT_2470112034)
>     expansion-rom-version:
>     bus-info: 0000:3b:00.0
>     supports-statistics: yes
>     supports-test: yes
>     supports-eeprom-access: no
>     supports-register-dump: no
>     supports-priv-flags: yes
> 
> 
> 
> 