* [PATCH v24 00/20] nvme-tcp receive offloads
@ 2024-04-04 12:36 Aurelien Aptel
2024-04-04 12:36 ` [PATCH v24 01/20] net: Introduce direct data placement tcp offload Aurelien Aptel
` (20 more replies)
0 siblings, 21 replies; 44+ messages in thread
From: Aurelien Aptel @ 2024-04-04 12:36 UTC (permalink / raw)
To: linux-nvme, netdev, sagi, hch, kbusch, axboe, chaitanyak, davem, kuba
Cc: Aurelien Aptel, aurelien.aptel, smalin, malin1024, ogerlitz,
yorayz, borisp, galshalom, mgurtovoy, edumazet
Hi,
The next iteration of our nvme-tcp receive offload series.
Previous submission (v23):
https://lore.kernel.org/netdev/20240228125733.299384-1-aaptel@nvidia.com/
The changes are also available through git:
Repo: https://github.com/aaptel/linux.git branch nvme-rx-offload-v24
Web: https://github.com/aaptel/linux/tree/nvme-rx-offload-v24
The NVMe-TCP offload was presented in netdev 0x16 (video available):
- https://netdevconf.info/0x16/session.html?NVMeTCP-Offload-%E2%80%93-Implementation-and-Performance-Gains
- https://youtu.be/W74TR-SNgi4
From: Aurelien Aptel <aaptel@nvidia.com>
From: Shai Malin <smalin@nvidia.com>
From: Boris Pismenny <borisp@nvidia.com>
From: Or Gerlitz <ogerlitz@nvidia.com>
From: Yoray Zack <yorayz@nvidia.com>
From: Max Gurtovoy <mgurtovoy@nvidia.com>
=========================================
This series adds support for NVMe-TCP receive offloads. The method here
does not mandate the offload of the network stack to the device.
Instead, these work together with TCP to offload:
1. copy from SKB to the block layer buffers.
2. CRC calculation and verification for received PDU.
The series implements these as a generic offload infrastructure for storage
protocols, which calls TCP Direct Data Placement and TCP Offload CRC
respectively. We use this infrastructure to implement NVMe-TCP offload for
copy and CRC.
Future implementations can reuse the same infrastructure for other protocols
such as iSCSI.
Note:
These offloads are similar in nature to the packet-based NIC TLS offloads,
which are already upstream (see net/tls/tls_device.c).
You can read more about TLS offload here:
https://www.kernel.org/doc/html/latest/networking/tls-offload.html
Queue Level
===========
The offload for IO queues is initialized after the handshake of the
NVMe-TCP protocol is finished by calling `nvme_tcp_offload_socket`
with the tcp socket of the nvme_tcp_queue:
This operation sets all relevant hardware contexts in
hardware. If it fails, then the IO queue proceeds as usual with no offload.
If it succeeds then `nvme_tcp_setup_ddp` and `nvme_tcp_teardown_ddp` may be
called to perform copy offload, and crc offload will be used.
This initialization does not change the normal operation of NVMe-TCP in any
way besides adding the option to call the above mentioned NDO operations.
For the admin queue, NVMe-TCP does not initialize the offload.
Instead, NVMe-TCP calls the driver to configure limits for the controller,
such as max_hw_sectors and max_segments, these must be limited to accommodate
potential HW resource limits, and to improve performance.
If some error occurs, and the IO queue must be closed or reconnected, then
offload is teardown and initialized again. Additionally, we handle netdev
down events via the existing error recovery flow.
IO Level
========
The NVMe-TCP layer calls the NIC driver to map block layer buffers to CID
using `nvme_tcp_setup_ddp` before sending the read request. When the response
is received, then the NIC HW will write the PDU payload directly into the
designated buffer, and build an SKB such that it points into the destination
buffer. This SKB represents the entire packet received on the wire, but it
points to the block layer buffers. Once NVMe-TCP attempts to copy data from
this SKB to the block layer buffer it can skip the copy by checking in the
copying function: if (src == dst) -> skip copy
Finally, when the PDU has been processed to completion, the NVMe-TCP layer
releases the NIC HW context by calling `nvme_tcp_teardown_ddp` which
asynchronously unmaps the buffers from NIC HW.
The NIC must release its mapping between command IDs and the target buffers.
This mapping is released when NVMe-TCP calls the NIC
driver (`nvme_tcp_offload_socket`).
As completing IOs is performance critical, we introduce asynchronous
completions for NVMe-TCP, i.e. NVMe-TCP calls the NIC, which will later
call NVMe-TCP to complete the IO (`nvme_tcp_ddp_teardown_done`).
On the IO level, and in order to use the offload only when a clear
performance improvement is expected, the offload is used only for IOs
which are bigger than io_threshold.
SKB
===
The DDP (zero-copy) and CRC offloads require two additional bits in the SKB.
The ddp bit is useful to prevent condensing of SKBs which are targeted
for zero-copy. The crc bit is useful to prevent GRO coalescing SKBs with
different offload values. This bit is similar in concept to the
"decrypted" bit.
After offload is initialized, we use the SKB's crc bit to indicate that:
"there was no problem with the verification of all CRC fields in this packet's
payload". The bit is set to zero if there was an error, or if HW skipped
offload for some reason. If *any* SKB in a PDU has (crc != 1), then the
calling driver must compute the CRC, and check it. We perform this check, and
accompanying software fallback at the end of the processing of a received PDU.
Resynchronization flow
======================
The resynchronization flow is performed to reset the hardware tracking of
NVMe-TCP PDUs within the TCP stream. The flow consists of a request from
the hardware proxied by the driver, regarding a possible location of a
PDU header. Followed by a response from the NVMe-TCP driver.
This flow is rare, and it should happen only after packet loss or
reordering events that involve NVMe-TCP PDU headers.
CID Mapping
===========
ConnectX-7 assumes linear CID (0...N-1 for queue of size N) where the Linux NVMe
driver uses part of the 16 bit CCID for generation counter.
To address that, we use the existing quirk in the NVMe layer when the HW
driver advertises that they don't support the full 16 bit CCID range.
Enablement on ConnectX-7
========================
By default, NVMeTCP offload is disabled in the mlx driver and in the nvme-tcp host.
In order to enable it:
# Disable CQE compression (specific for ConnectX)
ethtool --set-priv-flags <device> rx_cqe_compress off
# Enable the ULP-DDP
./tools/net/ynl/cli.py \
--spec Documentation/netlink/specs/ulp_ddp.yaml --do caps-set \
--json '{"ifindex": <device index>, "wanted": 3, "wanted_mask": 3}'
# Enable ULP offload in nvme-tcp
modprobe nvme-tcp ddp_offload=1
Following the device ULP-DDP enablement, all the IO queues/sockets which are
running on the device are offloaded.
Performance
===========
With this implementation, using the ConnectX-7 NIC, we were able to
demonstrate the following CPU utilization improvement:
Without data digest:
For 64K queued read IOs – up to 32% improvement in the BW/IOPS (111 Gbps vs. 84 Gbps).
For 512K queued read IOs – up to 55% improvement in the BW/IOPS (148 Gbps vs. 98 Gbps).
With data digest:
For 64K queued read IOs – up to 107% improvement in the BW/IOPS (111 Gbps vs. 53 Gbps).
For 512K queued read IOs – up to 138% improvement in the BW/IOPS (146 Gbps vs. 61 Gbps).
With small IOs we are not expecting that the offload will show a performance gain.
The test configuration:
- fio command: qd=128, jobs=8.
- Server: Intel(R) Xeon(R) Platinum 8380 CPU @ 2.30GHz, 160 cores.
Patches
=======
Patch 1: Introduce the infrastructure for all ULP DDP and ULP DDP CRC offloads.
Patch 2: Add netlink family to manage ULP DDP capabilities & stats.
Patch 3: The iov_iter change to skip copy if (src == dst).
Patch 4: Export the get_netdev_for_sock function from TLS to generic location.
Patch 5: NVMe-TCP changes to call NIC driver on queue init/teardown and resync.
Patch 6: NVMe-TCP changes to call NIC driver on IO operation
setup/teardown, and support async completions.
Patch 7: NVMe-TCP changes to support CRC offload on receive
Also, this patch moves CRC calculation to the end of PDU
in case offload requires software fallback.
Patch 8: NVMe-TCP handling of netdev events: stop the offload if netdev is
going down.
Patch 9: Documentation of ULP DDP offloads.
The rest of the series is the mlx5 implementation of the offload.
Testing
=======
This series was tested on ConnectX-7 HW using various configurations
of IO sizes, queue depths, MTUs, and with both the SPDK and kernel NVMe-TCP
targets.
Compatibility
=============
* The offload works with bare-metal or SRIOV.
* The HW can support up to 64K connections per device (assuming no
other HW accelerations are used). In this series, we will introduce
the support for up to 4k connections, and we have plans to increase it.
* In the current HW implementation, the combination of NVMeTCP offload
with TLS is not supported. In the future, if it will be implemented,
the impact on the NVMe/TCP layer will be minimal.
* The NVMeTCP offload ConnectX 7 HW can support tunneling, but we
don't see the need for this feature yet.
* NVMe poll queues are not in the scope of this series.
Future Work
===========
* NVMeTCP transmit offload.
* NVMeTCP offloads incremental features.
Changes since v23:
=================
- ulp_ddp.h: remove queue_id (Sagi).
- nvme-tcp: remove nvme_status, always set req->{result,status} (Sagi).
- nvme-tcp: rename label to ddgst_valid (Sagi).
- mlx5: remove newline from error messages (Jakub).
Changes since v22:
=================
- protect ->set_caps() with rtnl_lock().
- refactor of netdev GOING_DOWN event handler (Sagi).
- fix DDGST recalc for IOs under offload threshold.
- rebase against new mlx5 driver changes.
Changes since v21:
=================
- add netdevice_tracker to get_netdev_for_sock() (Jakub).
- remove redundant q->data_digest check (Max).
Changes since v20:
=================
- get caps&limits from nvme and remove query_limits() (Sagi).
- rename queue->ddp_status to queue->nvme_status and move ouf of ifdef (Sagi).
- call setup_ddp() during request setup (Sagi).
- remove null check in ddgst_recalc() (Sagi).
- remove local var in offload_socket() (Sagi).
- remove ifindex and hdr from netlink context data (Jiri).
- clean netlink notify handling and use nla_get_uint() (Jiri).
- normalize doc in ulp_ddp netlink spec (Jiri).
Changes since v19:
=================
- rebase against net-next.
- fix ulp_ddp_is_cap_active() error reported by the kernel test bot.
Changes since v18:
=================
- rebase against net-next.
- integrate with nvme-tcp tls.
- add const in parameter for skb_is_no_condense() and skb_is_ulp_crc().
- update documentation.
Changes since v17:
=================
- move capabilities from netdev to driver and add get_caps() op (Jiri).
- set stats by name explicitly, remove dump ops (Jiri).
- rename struct, functions, YAML attributes, reuse caps enum (Jiri).
- use uint instead of u64 in YAML spec (Jakub).
Changes since v16:
=================
- rebase against net-next
- minor whitespace changes
- updated CC list
Changes since v15:
=================
- add API func to get netdev & limits together (Sagi).
- add nvme_tcp_stop_admin_queue()
- hide config.io_cpu in the interface (Sagi).
- rename skb->ulp_ddp to skb->no_condense (David).
Changes since v14:
=================
- Added dumpit op for ULP_DDP_CMD_{GET,STATS} (Jakub).
- Remove redundant "-ddp-" fom stat names.
- Fix checkpatch/sparse warnings.
Changes since v13:
=================
- Replace ethtool interface with a new netlink family (Jakub).
- Simplify and squash mlx5e refactoring changes.
Changes since v12:
=================
- Rebase on top of NVMe-TCP kTLS v10 patches.
- Add ULP DDP wrappers for common code and ref accounting (Sagi).
- Fold modparam and tls patches into control-path patch (Sagi).
- Take one netdev ref for the admin queue (Sagi).
- Simplify start_queue() logic (Sagi).
- Rename
* modparam ulp_offload modparam -> ddp_offload (Sagi).
* queue->offload_xxx to queue->ddp_xxx (Sagi).
* queue->resync_req -> resync_tcp_seq (Sagi).
- Use SECTOR_SHIFT (Sagi).
- Use nvme_cid(rq) (Sagi).
- Use sock->sk->sk_incoming_cpu instead of queue->io_cpu (Sagi).
- Move limits results to ctrl struct.
- Add missing ifdefs.
- Fix docs and reverse xmas tree (Simon).
Changes since v11:
=================
- Rebase on top of NVMe-TCP kTLS offload.
- Add tls support bit in struct ulp_ddp_limits.
- Simplify logic in NVMe-TCP queue init.
- Use new page pool in mlx5 driver.
Changes since v10:
=================
- Pass extack to drivers for better error reporting in the .set_caps
callback (Jakub).
- netlink: use new callbacks, existing macros, padding, fix size
add notifications, update specs (Jakub).
Changes since v9:
=================
- Add missing crc checks in tcp_try_coalesce() (Paolo).
- Add missing ifdef guard for socket ops (Paolo).
- Remove verbose netlink format for statistics (Jakub).
- Use regular attributes for statistics (Jakub).
- Expose and document individual stats to uAPI (Jakub).
- Move ethtool ops for caps&stats to netdev_ops->ulp_ddp_ops (Jakub).
Changes since v8:
=================
- Make stats stringset global instead of per-device (Jakub).
- Remove per-queue stats (Jakub).
- Rename ETH_SS_ULP_DDP stringset to ETH_SS_ULP_DDP_CAPS.
- Update & fix kdoc comments.
- Use 80 columns limit for nvme code.
Changes since v7:
=================
- Remove ULP DDP netdev->feature bit (Jakub).
- Expose ULP DDP capabilities to userspace via ethtool netlink messages (Jakub).
- Move ULP DDP stats to dedicated stats group (Jakub).
- Add ethtool_ops operations for setting capabilities and getting stats (Jakub).
- Move ulp_ddp_netdev_ops into net_device_ops (Jakub).
- Use union for protocol-specific struct instances (Jakub).
- Fold netdev_sk_get_lowest_dev() into get_netdev_for_sock (Christoph).
- Rename memcpy skip patch to something more obvious (Christoph).
- Move capabilities from ulp_ddp.h to ulp_ddp_caps.h.
- Add "Compatibility" section on the cover letter (Sagi).
Changes since v6:
=================
- Moved IS_ULP_{DDP,CRC} macros to skb_is_ulp_{ddp,crc} inline functions (Jakub).
- Fix copyright notice (Leon).
- Added missing ifdef to allow build with MLX5_EN_TLS disabled.
- Fix space alignment, indent and long lines (max 99 columns).
- Add missing field documentation in ulp_ddp.h.
Changes since v5:
=================
- Limit the series to RX offloads.
- Added two separated skb indications to avoid wrong flushing of GRO
when aggerating offloaded packets.
- Use accessor functions for skb->ddp and skb->crc (Eric D) bits.
- Add kernel-doc for get_netdev_for_sock (Christoph).
- Remove ddp_iter* routines and only modify _copy_to_iter (Al Viro, Christoph).
- Remove consume skb (Sagi).
- Add a knob in the ddp limits struct for the HW driver to advertise
if they need the nvme-tcp driver to apply the generation counter
quirk. Use this knob for the mlx5 CX7 offload.
- bugfix: use u8 flags instead of bool in mlx5e_nvmeotcp_queue->dgst.
- bugfix: use sg_dma_len(sgl) instead of sgl->length.
- bugfix: remove sgl leak in nvme_tcp_setup_ddp().
- bugfix: remove sgl leak when only using DDGST_RX offload.
- Add error check for dma_map_sg().
- Reduce #ifdef by using dummy macros/functions.
- Remove redundant netdev null check in nvme_tcp_pdu_last_send().
- Rename ULP_DDP_RESYNC_{REQ -> PENDING}.
- Add per-ulp limits struct (Sagi).
- Add ULP DDP capabilities querying (Sagi).
- Simplify RX DDGST logic (Sagi).
- Document resync flow better.
- Add ulp_offload param to nvme-tcp module to enable ULP offload (Sagi).
- Add a revert commit to reintroduce nvme_tcp_queue->queue_size.
Changes since v4:
=================
- Add transmit offload patches.
- Use one feature bit for both receive and transmit offload.
Changes since v3:
=================
- Use DDP_TCP ifdefs in iov_iter and skb iterators to minimize impact
when compiled out (Christoph).
- Simplify netdev references and reduce the use of
get_netdev_for_sock (Sagi).
- Avoid "static" in it's own line, move it one line down (Christoph)
- Pass (queue, skb, *offset) and retrieve the pdu_seq in
nvme_tcp_resync_response (Sagi).
- Add missing assignment of offloading_netdev to null in offload_limits
error case (Sagi).
- Set req->offloaded = false once -- the lifetime rules are:
set to false on cmd_setup / set to true when ddp setup succeeds (Sagi).
- Replace pr_info_ratelimited with dev_info_ratelimited (Sagi).
- Add nvme_tcp_complete_request and invoke it from two similar call
sites (Sagi).
- Introduce nvme_tcp_req_map_sg earlier in the series (Sagi).
- Add nvme_tcp_consume_skb and put into it a hunk from
nvme_tcp_recv_data to handle copy with and without offload.
Changes since v2:
=================
- Use skb->ddp_crc for copy offload to avoid skb_condense.
- Default mellanox driver support to no (experimental feature).
- In iov_iter use non-ddp functions for kvec and iovec.
- Remove typecasting in NVMe-TCP.
Changes since v1:
=================
- Rework iov_iter copy skip if src==dst to be less intrusive (David Ahern).
- Add tcp-ddp documentation (David Ahern).
- Refactor mellanox driver patches into more patches (Saeed Mahameed).
- Avoid pointer casting (David Ahern).
- Rename NVMe-TCP offload flags (Shai Malin).
- Update cover-letter according to the above.
Changes since RFC v1:
=====================
- Split mlx5 driver patches to several commits.
- Fix NVMe-TCP handling of recovery flows. In particular, move queue offload.
init/teardown to the start/stop functions.
Signed-off-by: Or Gerlitz <ogerlitz@nvidia.com>
Aurelien Aptel (3):
netlink: add new family to manage ULP_DDP enablement and stats
net/tls,core: export get_netdev_for_sock
net/mlx5e: NVMEoTCP, statistics
Ben Ben-Ishay (8):
iov_iter: skip copy if src == dst for direct data placement
net/mlx5: Add NVMEoTCP caps, HW bits, 128B CQE and enumerations
net/mlx5e: NVMEoTCP, offload initialization
net/mlx5e: NVMEoTCP, use KLM UMRs for buffer registration
net/mlx5e: NVMEoTCP, queue init/teardown
net/mlx5e: NVMEoTCP, ddp setup and resync
net/mlx5e: NVMEoTCP, async ddp invalidation
net/mlx5e: NVMEoTCP, data-path for DDP+DDGST offload
Boris Pismenny (4):
net: Introduce direct data placement tcp offload
nvme-tcp: Add DDP offload control path
nvme-tcp: Add DDP data-path
net/mlx5e: TCP flow steering for nvme-tcp acceleration
Or Gerlitz (3):
nvme-tcp: Deal with netdevice DOWN events
net/mlx5e: Rename from tls to transport static params
net/mlx5e: Refactor ico sq polling to get budget
Yoray Zack (2):
nvme-tcp: RX DDGST offload
Documentation: add ULP DDP offload documentation
Documentation/netlink/specs/ulp_ddp.yaml | 172 +++
Documentation/networking/index.rst | 1 +
Documentation/networking/ulp-ddp-offload.rst | 372 ++++++
.../net/ethernet/mellanox/mlx5/core/Kconfig | 11 +
.../net/ethernet/mellanox/mlx5/core/Makefile | 3 +
drivers/net/ethernet/mellanox/mlx5/core/en.h | 9 +
.../net/ethernet/mellanox/mlx5/core/en/fs.h | 4 +-
.../ethernet/mellanox/mlx5/core/en/params.c | 12 +-
.../ethernet/mellanox/mlx5/core/en/params.h | 3 +
.../mellanox/mlx5/core/en/reporter_rx.c | 4 +-
.../ethernet/mellanox/mlx5/core/en/rx_res.c | 28 +
.../ethernet/mellanox/mlx5/core/en/rx_res.h | 4 +
.../net/ethernet/mellanox/mlx5/core/en/tir.c | 15 +
.../net/ethernet/mellanox/mlx5/core/en/tir.h | 2 +
.../net/ethernet/mellanox/mlx5/core/en/txrx.h | 34 +-
.../mlx5/core/en_accel/common_utils.h | 32 +
.../mellanox/mlx5/core/en_accel/en_accel.h | 3 +
.../mellanox/mlx5/core/en_accel/fs_tcp.c | 12 +-
.../mellanox/mlx5/core/en_accel/fs_tcp.h | 2 +-
.../mellanox/mlx5/core/en_accel/ktls.c | 2 +-
.../mellanox/mlx5/core/en_accel/ktls_rx.c | 6 +-
.../mellanox/mlx5/core/en_accel/ktls_tx.c | 8 +-
.../mellanox/mlx5/core/en_accel/ktls_txrx.c | 36 +-
.../mellanox/mlx5/core/en_accel/ktls_utils.h | 17 +-
.../mellanox/mlx5/core/en_accel/nvmeotcp.c | 1118 +++++++++++++++++
.../mellanox/mlx5/core/en_accel/nvmeotcp.h | 141 +++
.../mlx5/core/en_accel/nvmeotcp_rxtx.c | 354 ++++++
.../mlx5/core/en_accel/nvmeotcp_rxtx.h | 37 +
.../mlx5/core/en_accel/nvmeotcp_stats.c | 66 +
.../mlx5/core/en_accel/nvmeotcp_utils.h | 66 +
.../ethernet/mellanox/mlx5/core/en_ethtool.c | 6 +
.../net/ethernet/mellanox/mlx5/core/en_fs.c | 4 +-
.../net/ethernet/mellanox/mlx5/core/en_main.c | 29 +-
.../net/ethernet/mellanox/mlx5/core/en_rx.c | 71 +-
.../ethernet/mellanox/mlx5/core/en_stats.h | 8 +
.../net/ethernet/mellanox/mlx5/core/en_txrx.c | 4 +-
drivers/net/ethernet/mellanox/mlx5/core/fw.c | 6 +
.../net/ethernet/mellanox/mlx5/core/main.c | 1 +
drivers/nvme/host/tcp.c | 536 +++++++-
include/linux/mlx5/device.h | 59 +-
include/linux/mlx5/mlx5_ifc.h | 83 +-
include/linux/mlx5/qp.h | 1 +
include/linux/netdevice.h | 10 +-
include/linux/skbuff.h | 25 +-
include/net/inet_connection_sock.h | 6 +
include/net/ulp_ddp.h | 321 +++++
include/uapi/linux/ulp_ddp.h | 61 +
lib/iov_iter.c | 9 +-
net/Kconfig | 20 +
net/core/Makefile | 1 +
net/core/dev.c | 30 +-
net/core/skbuff.c | 3 +-
net/core/ulp_ddp.c | 52 +
net/core/ulp_ddp_gen_nl.c | 75 ++
net/core/ulp_ddp_gen_nl.h | 30 +
net/core/ulp_ddp_nl.c | 344 +++++
net/ipv4/tcp_input.c | 13 +-
net/ipv4/tcp_ipv4.c | 3 +
net/ipv4/tcp_offload.c | 3 +
net/tls/tls_device.c | 31 +-
60 files changed, 4258 insertions(+), 161 deletions(-)
create mode 100644 Documentation/netlink/specs/ulp_ddp.yaml
create mode 100644 Documentation/networking/ulp-ddp-offload.rst
create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/common_utils.h
create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c
create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h
create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.c
create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.h
create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_stats.c
create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_utils.h
create mode 100644 include/net/ulp_ddp.h
create mode 100644 include/uapi/linux/ulp_ddp.h
create mode 100644 net/core/ulp_ddp.c
create mode 100644 net/core/ulp_ddp_gen_nl.c
create mode 100644 net/core/ulp_ddp_gen_nl.h
create mode 100644 net/core/ulp_ddp_nl.c
--
2.34.1
^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v24 01/20] net: Introduce direct data placement tcp offload
2024-04-04 12:36 [PATCH v24 00/20] nvme-tcp receive offloads Aurelien Aptel
@ 2024-04-04 12:36 ` Aurelien Aptel
2024-04-21 11:47 ` Sagi Grimberg
2024-04-04 12:36 ` [PATCH v24 02/20] netlink: add new family to manage ULP_DDP enablement and stats Aurelien Aptel
` (19 subsequent siblings)
20 siblings, 1 reply; 44+ messages in thread
From: Aurelien Aptel @ 2024-04-04 12:36 UTC (permalink / raw)
To: linux-nvme, netdev, sagi, hch, kbusch, axboe, chaitanyak, davem, kuba
Cc: Boris Pismenny, aaptel, aurelien.aptel, smalin, malin1024,
ogerlitz, yorayz, galshalom, mgurtovoy, edumazet, pabeni,
dsahern, ast, jacob.e.keller
From: Boris Pismenny <borisp@nvidia.com>
This commit introduces direct data placement (DDP) offload for TCP.
The motivation is saving compute resources/cycles that are spent
to copy data from SKBs to the block layer buffers and CRC
calculation/verification for received PDUs (Protocol Data Units).
The DDP capability is accompanied by new net_device operations that
configure hardware contexts.
There is a context per socket, and a context per DDP operation.
Additionally, a resynchronization routine is used to assist
hardware handle TCP OOO, and continue the offload. Furthermore,
we let the offloading driver advertise what is the max hw
sectors/segments.
The interface includes the following net-device ddp operations:
1. sk_add - add offload for the queue represented by socket+config pair
2. sk_del - remove the offload for the socket/queue
3. ddp_setup - request copy offload for buffers associated with an IO
4. ddp_teardown - release offload resources for that IO
5. limits - query NIC driver for quirks and limitations (e.g.
max number of scatter gather entries per IO)
6. set_caps - request ULP DDP capabilities enablement
7. get_caps - request current ULP DDP capabilities
8. get_stats - query NIC driver for ULP DDP stats
Using this interface, the NIC hardware will scatter TCP payload
directly to the BIO pages according to the command_id.
To maintain the correctness of the network stack, the driver is
expected to construct SKBs that point to the BIO pages.
The SKB passed to the network stack from the driver represents
data as it is on the wire, while it is pointing directly to data
in destination buffers.
As a result, data from page frags should not be copied out to
the linear part. To avoid needless copies, such as when using
skb_condense, we mark the skb->no_condense bit.
In addition, the skb->ulp_crc will be used by the upper layers to
determine if CRC re-calculation is required. The two separated skb
indications are needed to avoid false positives GRO flushing events.
Follow-up patches will use this interface for DDP in NVMe-TCP.
Capability bits stored in net_device allow drivers to report which
ULP DDP capabilities a device supports. Control over these
capabilities will be exposed to userspace in later patches.
Signed-off-by: Boris Pismenny <borisp@nvidia.com>
Signed-off-by: Ben Ben-Ishay <benishay@nvidia.com>
Signed-off-by: Or Gerlitz <ogerlitz@nvidia.com>
Signed-off-by: Yoray Zack <yorayz@nvidia.com>
Signed-off-by: Shai Malin <smalin@nvidia.com>
Signed-off-by: Aurelien Aptel <aaptel@nvidia.com>
---
include/linux/netdevice.h | 5 +
include/linux/skbuff.h | 25 ++-
include/net/inet_connection_sock.h | 6 +
include/net/ulp_ddp.h | 320 +++++++++++++++++++++++++++++
net/Kconfig | 20 ++
net/core/Makefile | 1 +
net/core/skbuff.c | 3 +-
net/core/ulp_ddp.c | 52 +++++
net/ipv4/tcp_input.c | 13 +-
net/ipv4/tcp_ipv4.c | 3 +
net/ipv4/tcp_offload.c | 3 +
11 files changed, 448 insertions(+), 3 deletions(-)
create mode 100644 include/net/ulp_ddp.h
create mode 100644 net/core/ulp_ddp.c
diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index 0c198620ac93..2fc33d9470e5 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -1351,6 +1351,8 @@ struct netdev_net_notifier {
* struct kernel_hwtstamp_config *kernel_config,
* struct netlink_ext_ack *extack);
* Change the hardware timestamping parameters for NIC device.
+ * struct ulp_ddp_dev_ops *ulp_ddp_ops;
+ * ULP DDP operations (see include/net/ulp_ddp.h)
*/
struct net_device_ops {
int (*ndo_init)(struct net_device *dev);
@@ -1596,6 +1598,9 @@ struct net_device_ops {
int (*ndo_hwtstamp_set)(struct net_device *dev,
struct kernel_hwtstamp_config *kernel_config,
struct netlink_ext_ack *extack);
+#if IS_ENABLED(CONFIG_ULP_DDP)
+ const struct ulp_ddp_dev_ops *ulp_ddp_ops;
+#endif
};
/**
diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
index 03ea36a82cdd..36e6ebf5e694 100644
--- a/include/linux/skbuff.h
+++ b/include/linux/skbuff.h
@@ -823,6 +823,8 @@ typedef unsigned char *sk_buff_data_t;
* delivery_time in mono clock base (i.e. EDT). Otherwise, the
* skb->tstamp has the (rcv) timestamp at ingress and
* delivery_time at egress.
+ * @no_condense: When set, don't condense fragments (DDP offloaded)
+ * @ulp_crc: CRC offloaded
* @napi_id: id of the NAPI struct this skb came from
* @sender_cpu: (aka @napi_id) source CPU in XPS
* @alloc_cpu: CPU which did the skb allocation.
@@ -999,7 +1001,10 @@ struct sk_buff {
#if IS_ENABLED(CONFIG_IP_SCTP)
__u8 csum_not_inet:1;
#endif
-
+#ifdef CONFIG_ULP_DDP
+ __u8 no_condense:1;
+ __u8 ulp_crc:1;
+#endif
#if defined(CONFIG_NET_SCHED) || defined(CONFIG_NET_XGRESS)
__u16 tc_index; /* traffic control index */
#endif
@@ -5148,5 +5153,23 @@ static inline void skb_mark_for_recycle(struct sk_buff *skb)
ssize_t skb_splice_from_iter(struct sk_buff *skb, struct iov_iter *iter,
ssize_t maxsize, gfp_t gfp);
+static inline bool skb_is_no_condense(const struct sk_buff *skb)
+{
+#ifdef CONFIG_ULP_DDP
+ return skb->no_condense;
+#else
+ return 0;
+#endif
+}
+
+static inline bool skb_is_ulp_crc(const struct sk_buff *skb)
+{
+#ifdef CONFIG_ULP_DDP
+ return skb->ulp_crc;
+#else
+ return 0;
+#endif
+}
+
#endif /* __KERNEL__ */
#endif /* _LINUX_SKBUFF_H */
diff --git a/include/net/inet_connection_sock.h b/include/net/inet_connection_sock.h
index 20e7b0c0b3d1..c68e3c4bf74c 100644
--- a/include/net/inet_connection_sock.h
+++ b/include/net/inet_connection_sock.h
@@ -67,6 +67,8 @@ struct inet_connection_sock_af_ops {
* @icsk_ulp_ops Pluggable ULP control hook
* @icsk_ulp_data ULP private data
* @icsk_clean_acked Clean acked data hook
+ * @icsk_ulp_ddp_ops Pluggable ULP direct data placement control hook
+ * @icsk_ulp_ddp_data ULP direct data placement private data
* @icsk_ca_state: Congestion control state
* @icsk_retransmits: Number of unrecovered [RTO] timeouts
* @icsk_pending: Scheduled timer event
@@ -97,6 +99,10 @@ struct inet_connection_sock {
const struct tcp_ulp_ops *icsk_ulp_ops;
void __rcu *icsk_ulp_data;
void (*icsk_clean_acked)(struct sock *sk, u32 acked_seq);
+#ifdef CONFIG_ULP_DDP
+ const struct ulp_ddp_ulp_ops *icsk_ulp_ddp_ops;
+ void __rcu *icsk_ulp_ddp_data;
+#endif
unsigned int (*icsk_sync_mss)(struct sock *sk, u32 pmtu);
__u8 icsk_ca_state:5,
icsk_ca_initialized:1,
diff --git a/include/net/ulp_ddp.h b/include/net/ulp_ddp.h
new file mode 100644
index 000000000000..dc5405f37bba
--- /dev/null
+++ b/include/net/ulp_ddp.h
@@ -0,0 +1,320 @@
+/* SPDX-License-Identifier: GPL-2.0
+ *
+ * ulp_ddp.h
+ * Author: Boris Pismenny <borisp@nvidia.com>
+ * Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+ */
+#ifndef _ULP_DDP_H
+#define _ULP_DDP_H
+
+#include <linux/netdevice.h>
+#include <net/inet_connection_sock.h>
+#include <net/sock.h>
+
+enum ulp_ddp_type {
+ ULP_DDP_NVME = 1,
+};
+
+/**
+ * struct nvme_tcp_ddp_limits - nvme tcp driver limitations
+ *
+ * @full_ccid_range: true if the driver supports the full CID range
+ */
+struct nvme_tcp_ddp_limits {
+ bool full_ccid_range;
+};
+
+/**
+ * struct ulp_ddp_limits - Generic ulp ddp limits: tcp ddp
+ * protocol limits.
+ * Add new instances of ulp_ddp_limits in the union below (nvme-tcp, etc.).
+ *
+ * @type: type of this limits struct
+ * @max_ddp_sgl_len: maximum sgl size supported (zero means no limit)
+ * @io_threshold: minimum payload size required to offload
+ * @tls: support for ULP over TLS
+ * @nvmeotcp: NVMe-TCP specific limits
+ */
+struct ulp_ddp_limits {
+ enum ulp_ddp_type type;
+ int max_ddp_sgl_len;
+ int io_threshold;
+ bool tls:1;
+ union {
+ struct nvme_tcp_ddp_limits nvmeotcp;
+ };
+};
+
+/**
+ * struct nvme_tcp_ddp_config - nvme tcp ddp configuration for an IO queue
+ *
+ * @pfv: pdu version (e.g., NVME_TCP_PFV_1_0)
+ * @cpda: controller pdu data alignment (dwords, 0's based)
+ * @dgst: digest types enabled (header or data, see
+ * enum nvme_tcp_digest_option).
+ * The netdev will offload crc if it is supported.
+ * @queue_size: number of nvme-tcp IO queue elements
+ */
+struct nvme_tcp_ddp_config {
+ u16 pfv;
+ u8 cpda;
+ u8 dgst;
+ int queue_size;
+};
+
+/**
+ * struct ulp_ddp_config - Generic ulp ddp configuration
+ * Add new instances of ulp_ddp_config in the union below (nvme-tcp, etc.).
+ *
+ * @type: type of this config struct
+ * @nvmeotcp: NVMe-TCP specific config
+ * @io_cpu: cpu core running the IO thread for this socket
+ */
+struct ulp_ddp_config {
+ enum ulp_ddp_type type;
+ int io_cpu;
+ union {
+ struct nvme_tcp_ddp_config nvmeotcp;
+ };
+};
+
+/**
+ * struct ulp_ddp_io - ulp ddp configuration for an IO request.
+ *
+ * @command_id: identifier on the wire associated with these buffers
+ * @nents: number of entries in the sg_table
+ * @sg_table: describing the buffers for this IO request
+ * @first_sgl: first SGL in sg_table
+ */
+struct ulp_ddp_io {
+ u32 command_id;
+ int nents;
+ struct sg_table sg_table;
+ struct scatterlist first_sgl[SG_CHUNK_SIZE];
+};
+
+/**
+ * struct ulp_ddp_stats - ULP DDP offload statistics
+ * @rx_nvmeotcp_sk_add: number of sockets successfully prepared for offloading.
+ * @rx_nvmeotcp_sk_add_fail: number of sockets that failed to be prepared
+ * for offloading.
+ * @rx_nvmeotcp_sk_del: number of sockets where offloading has been removed.
+ * @rx_nvmeotcp_ddp_setup: number of NVMeTCP PDU successfully prepared for
+ * Direct Data Placement.
+ * @rx_nvmeotcp_ddp_setup_fail: number of PDUs that failed DDP preparation.
+ * @rx_nvmeotcp_ddp_teardown: number of PDUs done with DDP.
+ * @rx_nvmeotcp_drop: number of PDUs dropped.
+ * @rx_nvmeotcp_resync: number of resync.
+ * @rx_nvmeotcp_packets: number of offloaded PDUs.
+ * @rx_nvmeotcp_bytes: number of offloaded bytes.
+ */
+struct ulp_ddp_stats {
+ u64 rx_nvmeotcp_sk_add;
+ u64 rx_nvmeotcp_sk_add_fail;
+ u64 rx_nvmeotcp_sk_del;
+ u64 rx_nvmeotcp_ddp_setup;
+ u64 rx_nvmeotcp_ddp_setup_fail;
+ u64 rx_nvmeotcp_ddp_teardown;
+ u64 rx_nvmeotcp_drop;
+ u64 rx_nvmeotcp_resync;
+ u64 rx_nvmeotcp_packets;
+ u64 rx_nvmeotcp_bytes;
+
+ /*
+ * add new stats at the end and keep in sync with
+ * Documentation/netlink/specs/ulp_ddp.yaml
+ */
+};
+
+#define ULP_DDP_CAP_COUNT 1
+
+struct ulp_ddp_dev_caps {
+ DECLARE_BITMAP(active, ULP_DDP_CAP_COUNT);
+ DECLARE_BITMAP(hw, ULP_DDP_CAP_COUNT);
+};
+
+struct netlink_ext_ack;
+
+/**
+ * struct ulp_ddp_dev_ops - operations used by an upper layer protocol
+ * to configure ddp offload
+ *
+ * @limits: query ulp driver limitations and quirks.
+ * @sk_add: add offload for the queue represented by socket+config
+ * pair. this function is used to configure either copy, crc
+ * or both offloads.
+ * @sk_del: remove offload from the socket, and release any device
+ * related resources.
+ * @setup: request copy offload for buffers associated with a
+ * command_id in ulp_ddp_io.
+ * @teardown: release offload resources association between buffers
+ * and command_id in ulp_ddp_io.
+ * @resync: respond to the driver's resync_request. Called only if
+ * resync is successful.
+ * @set_caps: set device ULP DDP capabilities.
+ * returns a negative error code or zero.
+ * @get_caps: get device ULP DDP capabilities.
+ * @get_stats: query ULP DDP statistics.
+ */
+struct ulp_ddp_dev_ops {
+ int (*limits)(struct net_device *netdev,
+ struct ulp_ddp_limits *limits);
+ int (*sk_add)(struct net_device *netdev,
+ struct sock *sk,
+ struct ulp_ddp_config *config);
+ void (*sk_del)(struct net_device *netdev,
+ struct sock *sk);
+ int (*setup)(struct net_device *netdev,
+ struct sock *sk,
+ struct ulp_ddp_io *io);
+ void (*teardown)(struct net_device *netdev,
+ struct sock *sk,
+ struct ulp_ddp_io *io,
+ void *ddp_ctx);
+ void (*resync)(struct net_device *netdev,
+ struct sock *sk, u32 seq);
+ int (*set_caps)(struct net_device *dev, unsigned long *bits,
+ struct netlink_ext_ack *extack);
+ void (*get_caps)(struct net_device *dev,
+ struct ulp_ddp_dev_caps *caps);
+ int (*get_stats)(struct net_device *dev,
+ struct ulp_ddp_stats *stats);
+};
+
+#define ULP_DDP_RESYNC_PENDING BIT(0)
+
+/**
+ * struct ulp_ddp_ulp_ops - Interface to register upper layer
+ * Direct Data Placement (DDP) TCP offload.
+ * @resync_request: NIC requests ulp to indicate if @seq is the start
+ * of a message.
+ * @ddp_teardown_done: NIC driver informs the ulp that teardown is done,
+ * used for async completions.
+ */
+struct ulp_ddp_ulp_ops {
+ bool (*resync_request)(struct sock *sk, u32 seq, u32 flags);
+ void (*ddp_teardown_done)(void *ddp_ctx);
+};
+
+/**
+ * struct ulp_ddp_ctx - Generic ulp ddp context
+ *
+ * @type: type of this context struct
+ * @buf: protocol-specific context struct
+ */
+struct ulp_ddp_ctx {
+ enum ulp_ddp_type type;
+ unsigned char buf[];
+};
+
+static inline struct ulp_ddp_ctx *ulp_ddp_get_ctx(struct sock *sk)
+{
+#ifdef CONFIG_ULP_DDP
+ struct inet_connection_sock *icsk = inet_csk(sk);
+
+ return (__force struct ulp_ddp_ctx *)icsk->icsk_ulp_ddp_data;
+#else
+ return NULL;
+#endif
+}
+
+static inline void ulp_ddp_set_ctx(struct sock *sk, void *ctx)
+{
+#ifdef CONFIG_ULP_DDP
+ struct inet_connection_sock *icsk = inet_csk(sk);
+
+ rcu_assign_pointer(icsk->icsk_ulp_ddp_data, ctx);
+#endif
+}
+
+static inline int ulp_ddp_setup(struct net_device *netdev,
+ struct sock *sk,
+ struct ulp_ddp_io *io)
+{
+#ifdef CONFIG_ULP_DDP
+ return netdev->netdev_ops->ulp_ddp_ops->setup(netdev, sk, io);
+#else
+ return -EOPNOTSUPP;
+#endif
+}
+
+static inline void ulp_ddp_teardown(struct net_device *netdev,
+ struct sock *sk,
+ struct ulp_ddp_io *io,
+ void *ddp_ctx)
+{
+#ifdef CONFIG_ULP_DDP
+ netdev->netdev_ops->ulp_ddp_ops->teardown(netdev, sk, io, ddp_ctx);
+#endif
+}
+
+static inline void ulp_ddp_resync(struct net_device *netdev,
+ struct sock *sk,
+ u32 seq)
+{
+#ifdef CONFIG_ULP_DDP
+ netdev->netdev_ops->ulp_ddp_ops->resync(netdev, sk, seq);
+#endif
+}
+
+static inline int ulp_ddp_get_limits(struct net_device *netdev,
+ struct ulp_ddp_limits *limits,
+ enum ulp_ddp_type type)
+{
+#ifdef CONFIG_ULP_DDP
+ limits->type = type;
+ return netdev->netdev_ops->ulp_ddp_ops->limits(netdev, limits);
+#else
+ return -EOPNOTSUPP;
+#endif
+}
+
+static inline bool ulp_ddp_cap_turned_on(unsigned long *old,
+ unsigned long *new,
+ int bit_nr)
+{
+ return !test_bit(bit_nr, old) && test_bit(bit_nr, new);
+}
+
+static inline bool ulp_ddp_cap_turned_off(unsigned long *old,
+ unsigned long *new,
+ int bit_nr)
+{
+ return test_bit(bit_nr, old) && !test_bit(bit_nr, new);
+}
+
+#ifdef CONFIG_ULP_DDP
+
+int ulp_ddp_sk_add(struct net_device *netdev,
+ struct sock *sk,
+ struct ulp_ddp_config *config,
+ const struct ulp_ddp_ulp_ops *ops);
+
+void ulp_ddp_sk_del(struct net_device *netdev,
+ struct sock *sk);
+
+bool ulp_ddp_is_cap_active(struct net_device *netdev, int cap_bit_nr);
+
+#else
+
+static inline int ulp_ddp_sk_add(struct net_device *netdev,
+ struct sock *sk,
+ struct ulp_ddp_config *config,
+ const struct ulp_ddp_ulp_ops *ops)
+{
+ return -EOPNOTSUPP;
+}
+
+static inline void ulp_ddp_sk_del(struct net_device *netdev,
+ struct sock *sk)
+{}
+
+static inline bool ulp_ddp_is_cap_active(struct net_device *netdev,
+ int cap_bit_nr)
+{
+ return false;
+}
+
+#endif
+
+#endif /* _ULP_DDP_H */
diff --git a/net/Kconfig b/net/Kconfig
index 3e57ccf0da27..fb1a854523c1 100644
--- a/net/Kconfig
+++ b/net/Kconfig
@@ -516,4 +516,24 @@ config NET_TEST
If unsure, say N.
+config ULP_DDP
+ bool "ULP direct data placement offload"
+ help
+ This feature provides a generic infrastructure for Direct
+ Data Placement (DDP) offload for Upper Layer Protocols (ULP,
+ such as NVMe-TCP).
+
+ If the ULP and NIC driver supports it, the ULP code can
+ request the NIC to place ULP response data directly
+ into application memory, avoiding a costly copy.
+
+ This infrastructure also allows for offloading the ULP data
+ integrity checks (e.g. data digest) that would otherwise
+ require another costly pass on the data we managed to avoid
+ copying.
+
+ For more information, see
+ <file:Documentation/networking/ulp-ddp-offload.rst>.
+
+
endif # if NET
diff --git a/net/core/Makefile b/net/core/Makefile
index 21d6fbc7e884..b7ea11bd9c0b 100644
--- a/net/core/Makefile
+++ b/net/core/Makefile
@@ -19,6 +19,7 @@ obj-$(CONFIG_NETDEV_ADDR_LIST_TEST) += dev_addr_lists_test.o
obj-y += net-sysfs.o
obj-y += hotdata.o
+obj-$(CONFIG_ULP_DDP) += ulp_ddp.o
obj-$(CONFIG_PAGE_POOL) += page_pool.o page_pool_user.o
obj-$(CONFIG_PROC_FS) += net-procfs.o
obj-$(CONFIG_NET_PKTGEN) += pktgen.o
diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index 2a5ce6667bbb..8ab25e344d43 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -77,6 +77,7 @@
#include <net/mctp.h>
#include <net/page_pool/helpers.h>
#include <net/dropreason.h>
+#include <net/ulp_ddp.h>
#include <linux/uaccess.h>
#include <trace/events/skb.h>
@@ -6752,7 +6753,7 @@ void skb_condense(struct sk_buff *skb)
{
if (skb->data_len) {
if (skb->data_len > skb->end - skb->tail ||
- skb_cloned(skb))
+ skb_cloned(skb) || skb_is_no_condense(skb))
return;
/* Nice, we can free page frag(s) right now */
diff --git a/net/core/ulp_ddp.c b/net/core/ulp_ddp.c
new file mode 100644
index 000000000000..658e67620c0f
--- /dev/null
+++ b/net/core/ulp_ddp.c
@@ -0,0 +1,52 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ *
+ * ulp_ddp.c
+ * Author: Aurelien Aptel <aaptel@nvidia.com>
+ * Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+ */
+
+#include <net/ulp_ddp.h>
+
+int ulp_ddp_sk_add(struct net_device *netdev,
+ struct sock *sk,
+ struct ulp_ddp_config *config,
+ const struct ulp_ddp_ulp_ops *ops)
+{
+ int ret;
+
+ /* put in ulp_ddp_sk_del() */
+ dev_hold(netdev);
+
+ config->io_cpu = sk->sk_incoming_cpu;
+ ret = netdev->netdev_ops->ulp_ddp_ops->sk_add(netdev, sk, config);
+ if (ret) {
+ dev_put(netdev);
+ return ret;
+ }
+
+ inet_csk(sk)->icsk_ulp_ddp_ops = ops;
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(ulp_ddp_sk_add);
+
+void ulp_ddp_sk_del(struct net_device *netdev,
+ struct sock *sk)
+{
+ netdev->netdev_ops->ulp_ddp_ops->sk_del(netdev, sk);
+ inet_csk(sk)->icsk_ulp_ddp_ops = NULL;
+ dev_put(netdev);
+}
+EXPORT_SYMBOL_GPL(ulp_ddp_sk_del);
+
+bool ulp_ddp_is_cap_active(struct net_device *netdev, int cap_bit_nr)
+{
+ struct ulp_ddp_dev_caps caps;
+
+ if (!netdev->netdev_ops->ulp_ddp_ops->get_caps)
+ return false;
+ netdev->netdev_ops->ulp_ddp_ops->get_caps(netdev, &caps);
+ return test_bit(cap_bit_nr, caps.active);
+}
+EXPORT_SYMBOL_GPL(ulp_ddp_is_cap_active);
diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
index 1b6cd3840012..52563f4f35a7 100644
--- a/net/ipv4/tcp_input.c
+++ b/net/ipv4/tcp_input.c
@@ -4807,7 +4807,10 @@ static bool tcp_try_coalesce(struct sock *sk,
if (from->decrypted != to->decrypted)
return false;
#endif
-
+#ifdef CONFIG_ULP_DDP
+ if (skb_is_ulp_crc(from) != skb_is_ulp_crc(to))
+ return false;
+#endif
if (!skb_try_coalesce(to, from, fragstolen, &delta))
return false;
@@ -5377,6 +5380,10 @@ tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root,
memcpy(nskb->cb, skb->cb, sizeof(skb->cb));
#ifdef CONFIG_TLS_DEVICE
nskb->decrypted = skb->decrypted;
+#endif
+#ifdef CONFIG_ULP_DDP
+ nskb->no_condense = skb->no_condense;
+ nskb->ulp_crc = skb->ulp_crc;
#endif
TCP_SKB_CB(nskb)->seq = TCP_SKB_CB(nskb)->end_seq = start;
if (list)
@@ -5410,6 +5417,10 @@ tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root,
#ifdef CONFIG_TLS_DEVICE
if (skb->decrypted != nskb->decrypted)
goto end;
+#endif
+#ifdef CONFIG_ULP_DDP
+ if (skb_is_ulp_crc(skb) != skb_is_ulp_crc(nskb))
+ goto end;
#endif
}
}
diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c
index 1e0a9762f92e..3dc7607660b4 100644
--- a/net/ipv4/tcp_ipv4.c
+++ b/net/ipv4/tcp_ipv4.c
@@ -2049,6 +2049,9 @@ bool tcp_add_backlog(struct sock *sk, struct sk_buff *skb,
tail->decrypted != skb->decrypted ||
#endif
!mptcp_skb_can_collapse(tail, skb) ||
+#ifdef CONFIG_ULP_DDP
+ skb_is_ulp_crc(tail) != skb_is_ulp_crc(skb) ||
+#endif
thtail->doff != th->doff ||
memcmp(thtail + 1, th + 1, hdrlen - sizeof(*th)))
goto no_coalesce;
diff --git a/net/ipv4/tcp_offload.c b/net/ipv4/tcp_offload.c
index ebe4722bb020..a7a607d4812f 100644
--- a/net/ipv4/tcp_offload.c
+++ b/net/ipv4/tcp_offload.c
@@ -268,6 +268,9 @@ struct sk_buff *tcp_gro_receive(struct list_head *head, struct sk_buff *skb)
#ifdef CONFIG_TLS_DEVICE
flush |= p->decrypted ^ skb->decrypted;
#endif
+#ifdef CONFIG_ULP_DDP
+ flush |= skb_is_ulp_crc(p) ^ skb_is_ulp_crc(skb);
+#endif
if (flush || skb_gro_receive(p, skb)) {
mss = 1;
--
2.34.1
^ permalink raw reply related [flat|nested] 44+ messages in thread
* [PATCH v24 02/20] netlink: add new family to manage ULP_DDP enablement and stats
2024-04-04 12:36 [PATCH v24 00/20] nvme-tcp receive offloads Aurelien Aptel
2024-04-04 12:36 ` [PATCH v24 01/20] net: Introduce direct data placement tcp offload Aurelien Aptel
@ 2024-04-04 12:36 ` Aurelien Aptel
2024-04-04 12:37 ` [PATCH v24 03/20] iov_iter: skip copy if src == dst for direct data placement Aurelien Aptel
` (18 subsequent siblings)
20 siblings, 0 replies; 44+ messages in thread
From: Aurelien Aptel @ 2024-04-04 12:36 UTC (permalink / raw)
To: linux-nvme, netdev, sagi, hch, kbusch, axboe, chaitanyak, davem, kuba
Cc: Aurelien Aptel, aurelien.aptel, smalin, malin1024, ogerlitz,
yorayz, borisp, galshalom, mgurtovoy, edumazet, pabeni
Add a new netlink family to get/set ULP DDP capabilities on a network
device and to retrieve statistics.
The messages use the genetlink infrastructure and are specified in a
YAML file which was used to generate some of the files in this commit:
./tools/net/ynl/ynl-gen-c.py --mode kernel \
--spec ./Documentation/netlink/specs/ulp_ddp.yaml --header \
-o net/core/ulp_ddp_gen_nl.h
./tools/net/ynl/ynl-gen-c.py --mode kernel \
--spec ./Documentation/netlink/specs/ulp_ddp.yaml --source \
-o net/core/ulp_ddp_gen_nl.c
./tools/net/ynl/ynl-gen-c.py --mode uapi \
--spec ./Documentation/netlink/specs/ulp_ddp.yaml --header \
> include/uapi/linux/ulp_ddp.h
Signed-off-by: Shai Malin <smalin@nvidia.com>
Signed-off-by: Aurelien Aptel <aaptel@nvidia.com>
Reviewed-by: Jiri Pirko <jiri@nvidia.com>
---
Documentation/netlink/specs/ulp_ddp.yaml | 172 ++++++++++++
include/net/ulp_ddp.h | 3 +-
include/uapi/linux/ulp_ddp.h | 61 ++++
net/core/Makefile | 2 +-
net/core/ulp_ddp_gen_nl.c | 75 +++++
net/core/ulp_ddp_gen_nl.h | 30 ++
net/core/ulp_ddp_nl.c | 344 +++++++++++++++++++++++
7 files changed, 685 insertions(+), 2 deletions(-)
create mode 100644 Documentation/netlink/specs/ulp_ddp.yaml
create mode 100644 include/uapi/linux/ulp_ddp.h
create mode 100644 net/core/ulp_ddp_gen_nl.c
create mode 100644 net/core/ulp_ddp_gen_nl.h
create mode 100644 net/core/ulp_ddp_nl.c
diff --git a/Documentation/netlink/specs/ulp_ddp.yaml b/Documentation/netlink/specs/ulp_ddp.yaml
new file mode 100644
index 000000000000..27a0b905ec28
--- /dev/null
+++ b/Documentation/netlink/specs/ulp_ddp.yaml
@@ -0,0 +1,172 @@
+# SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause)
+#
+# Author: Aurelien Aptel <aaptel@nvidia.com>
+#
+# Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+#
+
+name: ulp_ddp
+
+protocol: genetlink
+
+doc: Netlink protocol to manage ULP DPP on network devices.
+
+definitions:
+ -
+ type: enum
+ name: cap
+ render-max: true
+ entries:
+ - nvme-tcp
+ - nvme-tcp-ddgst-rx
+
+attribute-sets:
+ -
+ name: stats
+ attributes:
+ -
+ name: ifindex
+ doc: Interface index of the net device.
+ type: u32
+ -
+ name: rx-nvme-tcp-sk-add
+ doc: Sockets successfully configured for NVMeTCP offloading.
+ type: uint
+ -
+ name: rx-nvme-tcp-sk-add-fail
+ doc: Sockets failed to be configured for NVMeTCP offloading.
+ type: uint
+ -
+ name: rx-nvme-tcp-sk-del
+ doc: Sockets with NVMeTCP offloading configuration removed.
+ type: uint
+ -
+ name: rx-nvme-tcp-setup
+ doc: NVMe-TCP IOs successfully configured for Rx Direct Data Placement.
+ type: uint
+ -
+ name: rx-nvme-tcp-setup-fail
+ doc: NVMe-TCP IOs failed to be configured for Rx Direct Data Placement.
+ type: uint
+ -
+ name: rx-nvme-tcp-teardown
+ doc: NVMe-TCP IOs with Rx Direct Data Placement configuration removed.
+ type: uint
+ -
+ name: rx-nvme-tcp-drop
+ doc: Packets failed the NVMeTCP offload validation.
+ type: uint
+ -
+ name: rx-nvme-tcp-resync
+ doc: >
+ NVMe-TCP resync operations were processed due to Rx TCP packets
+ re-ordering.
+ type: uint
+ -
+ name: rx-nvme-tcp-packets
+ doc: TCP packets successfully processed by the NVMeTCP offload.
+ type: uint
+ -
+ name: rx-nvme-tcp-bytes
+ doc: Bytes were successfully processed by the NVMeTCP offload.
+ type: uint
+ -
+ name: caps
+ attributes:
+ -
+ name: ifindex
+ doc: Interface index of the net device.
+ type: u32
+ -
+ name: hw
+ doc: Bitmask of the capabilities supported by the device.
+ type: uint
+ enum: cap
+ enum-as-flags: true
+ -
+ name: active
+ doc: Bitmask of the capabilities currently enabled on the device.
+ type: uint
+ enum: cap
+ enum-as-flags: true
+ -
+ name: wanted
+ doc: >
+ New active bit values of the capabilities we want to set on the
+ device.
+ type: uint
+ enum: cap
+ enum-as-flags: true
+ -
+ name: wanted_mask
+ doc: Bitmask of the meaningful bits in the wanted field.
+ type: uint
+ enum: cap
+ enum-as-flags: true
+
+operations:
+ list:
+ -
+ name: caps-get
+ doc: Get ULP DDP capabilities.
+ attribute-set: caps
+ do:
+ request:
+ attributes:
+ - ifindex
+ reply:
+ attributes:
+ - ifindex
+ - hw
+ - active
+ pre: ulp_ddp_get_netdev
+ post: ulp_ddp_put_netdev
+ -
+ name: stats-get
+ doc: Get ULP DDP stats.
+ attribute-set: stats
+ do:
+ request:
+ attributes:
+ - ifindex
+ reply:
+ attributes:
+ - ifindex
+ - rx-nvme-tcp-sk-add
+ - rx-nvme-tcp-sk-add-fail
+ - rx-nvme-tcp-sk-del
+ - rx-nvme-tcp-setup
+ - rx-nvme-tcp-setup-fail
+ - rx-nvme-tcp-teardown
+ - rx-nvme-tcp-drop
+ - rx-nvme-tcp-resync
+ - rx-nvme-tcp-packets
+ - rx-nvme-tcp-bytes
+ pre: ulp_ddp_get_netdev
+ post: ulp_ddp_put_netdev
+ -
+ name: caps-set
+ doc: Set ULP DDP capabilities.
+ attribute-set: caps
+ do:
+ request:
+ attributes:
+ - ifindex
+ - wanted
+ - wanted_mask
+ reply:
+ attributes:
+ - ifindex
+ - hw
+ - active
+ pre: ulp_ddp_get_netdev
+ post: ulp_ddp_put_netdev
+ -
+ name: caps-set-ntf
+ doc: Notification for change in ULP DDP capabilities.
+ notify: caps-get
+
+mcast-groups:
+ list:
+ -
+ name: mgmt
diff --git a/include/net/ulp_ddp.h b/include/net/ulp_ddp.h
index dc5405f37bba..dc5f760f7dea 100644
--- a/include/net/ulp_ddp.h
+++ b/include/net/ulp_ddp.h
@@ -10,6 +10,7 @@
#include <linux/netdevice.h>
#include <net/inet_connection_sock.h>
#include <net/sock.h>
+#include <uapi/linux/ulp_ddp.h>
enum ulp_ddp_type {
ULP_DDP_NVME = 1,
@@ -126,7 +127,7 @@ struct ulp_ddp_stats {
*/
};
-#define ULP_DDP_CAP_COUNT 1
+#define ULP_DDP_CAP_COUNT (ULP_DDP_CAP_MAX + 1)
struct ulp_ddp_dev_caps {
DECLARE_BITMAP(active, ULP_DDP_CAP_COUNT);
diff --git a/include/uapi/linux/ulp_ddp.h b/include/uapi/linux/ulp_ddp.h
new file mode 100644
index 000000000000..dbf6399d3aef
--- /dev/null
+++ b/include/uapi/linux/ulp_ddp.h
@@ -0,0 +1,61 @@
+/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) */
+/* Do not edit directly, auto-generated from: */
+/* Documentation/netlink/specs/ulp_ddp.yaml */
+/* YNL-GEN uapi header */
+
+#ifndef _UAPI_LINUX_ULP_DDP_H
+#define _UAPI_LINUX_ULP_DDP_H
+
+#define ULP_DDP_FAMILY_NAME "ulp_ddp"
+#define ULP_DDP_FAMILY_VERSION 1
+
+enum ulp_ddp_cap {
+ ULP_DDP_CAP_NVME_TCP,
+ ULP_DDP_CAP_NVME_TCP_DDGST_RX,
+
+ /* private: */
+ __ULP_DDP_CAP_MAX,
+ ULP_DDP_CAP_MAX = (__ULP_DDP_CAP_MAX - 1)
+};
+
+enum {
+ ULP_DDP_A_STATS_IFINDEX = 1,
+ ULP_DDP_A_STATS_RX_NVME_TCP_SK_ADD,
+ ULP_DDP_A_STATS_RX_NVME_TCP_SK_ADD_FAIL,
+ ULP_DDP_A_STATS_RX_NVME_TCP_SK_DEL,
+ ULP_DDP_A_STATS_RX_NVME_TCP_SETUP,
+ ULP_DDP_A_STATS_RX_NVME_TCP_SETUP_FAIL,
+ ULP_DDP_A_STATS_RX_NVME_TCP_TEARDOWN,
+ ULP_DDP_A_STATS_RX_NVME_TCP_DROP,
+ ULP_DDP_A_STATS_RX_NVME_TCP_RESYNC,
+ ULP_DDP_A_STATS_RX_NVME_TCP_PACKETS,
+ ULP_DDP_A_STATS_RX_NVME_TCP_BYTES,
+
+ __ULP_DDP_A_STATS_MAX,
+ ULP_DDP_A_STATS_MAX = (__ULP_DDP_A_STATS_MAX - 1)
+};
+
+enum {
+ ULP_DDP_A_CAPS_IFINDEX = 1,
+ ULP_DDP_A_CAPS_HW,
+ ULP_DDP_A_CAPS_ACTIVE,
+ ULP_DDP_A_CAPS_WANTED,
+ ULP_DDP_A_CAPS_WANTED_MASK,
+
+ __ULP_DDP_A_CAPS_MAX,
+ ULP_DDP_A_CAPS_MAX = (__ULP_DDP_A_CAPS_MAX - 1)
+};
+
+enum {
+ ULP_DDP_CMD_CAPS_GET = 1,
+ ULP_DDP_CMD_STATS_GET,
+ ULP_DDP_CMD_CAPS_SET,
+ ULP_DDP_CMD_CAPS_SET_NTF,
+
+ __ULP_DDP_CMD_MAX,
+ ULP_DDP_CMD_MAX = (__ULP_DDP_CMD_MAX - 1)
+};
+
+#define ULP_DDP_MCGRP_MGMT "mgmt"
+
+#endif /* _UAPI_LINUX_ULP_DDP_H */
diff --git a/net/core/Makefile b/net/core/Makefile
index b7ea11bd9c0b..7c0cbd1cc582 100644
--- a/net/core/Makefile
+++ b/net/core/Makefile
@@ -19,7 +19,7 @@ obj-$(CONFIG_NETDEV_ADDR_LIST_TEST) += dev_addr_lists_test.o
obj-y += net-sysfs.o
obj-y += hotdata.o
-obj-$(CONFIG_ULP_DDP) += ulp_ddp.o
+obj-$(CONFIG_ULP_DDP) += ulp_ddp.o ulp_ddp_nl.o ulp_ddp_gen_nl.o
obj-$(CONFIG_PAGE_POOL) += page_pool.o page_pool_user.o
obj-$(CONFIG_PROC_FS) += net-procfs.o
obj-$(CONFIG_NET_PKTGEN) += pktgen.o
diff --git a/net/core/ulp_ddp_gen_nl.c b/net/core/ulp_ddp_gen_nl.c
new file mode 100644
index 000000000000..5675193ad8ca
--- /dev/null
+++ b/net/core/ulp_ddp_gen_nl.c
@@ -0,0 +1,75 @@
+// SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause)
+/* Do not edit directly, auto-generated from: */
+/* Documentation/netlink/specs/ulp_ddp.yaml */
+/* YNL-GEN kernel source */
+
+#include <net/netlink.h>
+#include <net/genetlink.h>
+
+#include "ulp_ddp_gen_nl.h"
+
+#include <uapi/linux/ulp_ddp.h>
+
+/* ULP_DDP_CMD_CAPS_GET - do */
+static const struct nla_policy ulp_ddp_caps_get_nl_policy[ULP_DDP_A_CAPS_IFINDEX + 1] = {
+ [ULP_DDP_A_CAPS_IFINDEX] = { .type = NLA_U32, },
+};
+
+/* ULP_DDP_CMD_STATS_GET - do */
+static const struct nla_policy ulp_ddp_stats_get_nl_policy[ULP_DDP_A_STATS_IFINDEX + 1] = {
+ [ULP_DDP_A_STATS_IFINDEX] = { .type = NLA_U32, },
+};
+
+/* ULP_DDP_CMD_CAPS_SET - do */
+static const struct nla_policy ulp_ddp_caps_set_nl_policy[ULP_DDP_A_CAPS_WANTED_MASK + 1] = {
+ [ULP_DDP_A_CAPS_IFINDEX] = { .type = NLA_U32, },
+ [ULP_DDP_A_CAPS_WANTED] = NLA_POLICY_MASK(NLA_UINT, 0x3),
+ [ULP_DDP_A_CAPS_WANTED_MASK] = NLA_POLICY_MASK(NLA_UINT, 0x3),
+};
+
+/* Ops table for ulp_ddp */
+static const struct genl_split_ops ulp_ddp_nl_ops[] = {
+ {
+ .cmd = ULP_DDP_CMD_CAPS_GET,
+ .pre_doit = ulp_ddp_get_netdev,
+ .doit = ulp_ddp_nl_caps_get_doit,
+ .post_doit = ulp_ddp_put_netdev,
+ .policy = ulp_ddp_caps_get_nl_policy,
+ .maxattr = ULP_DDP_A_CAPS_IFINDEX,
+ .flags = GENL_CMD_CAP_DO,
+ },
+ {
+ .cmd = ULP_DDP_CMD_STATS_GET,
+ .pre_doit = ulp_ddp_get_netdev,
+ .doit = ulp_ddp_nl_stats_get_doit,
+ .post_doit = ulp_ddp_put_netdev,
+ .policy = ulp_ddp_stats_get_nl_policy,
+ .maxattr = ULP_DDP_A_STATS_IFINDEX,
+ .flags = GENL_CMD_CAP_DO,
+ },
+ {
+ .cmd = ULP_DDP_CMD_CAPS_SET,
+ .pre_doit = ulp_ddp_get_netdev,
+ .doit = ulp_ddp_nl_caps_set_doit,
+ .post_doit = ulp_ddp_put_netdev,
+ .policy = ulp_ddp_caps_set_nl_policy,
+ .maxattr = ULP_DDP_A_CAPS_WANTED_MASK,
+ .flags = GENL_CMD_CAP_DO,
+ },
+};
+
+static const struct genl_multicast_group ulp_ddp_nl_mcgrps[] = {
+ [ULP_DDP_NLGRP_MGMT] = { "mgmt", },
+};
+
+struct genl_family ulp_ddp_nl_family __ro_after_init = {
+ .name = ULP_DDP_FAMILY_NAME,
+ .version = ULP_DDP_FAMILY_VERSION,
+ .netnsok = true,
+ .parallel_ops = true,
+ .module = THIS_MODULE,
+ .split_ops = ulp_ddp_nl_ops,
+ .n_split_ops = ARRAY_SIZE(ulp_ddp_nl_ops),
+ .mcgrps = ulp_ddp_nl_mcgrps,
+ .n_mcgrps = ARRAY_SIZE(ulp_ddp_nl_mcgrps),
+};
diff --git a/net/core/ulp_ddp_gen_nl.h b/net/core/ulp_ddp_gen_nl.h
new file mode 100644
index 000000000000..368433cfa867
--- /dev/null
+++ b/net/core/ulp_ddp_gen_nl.h
@@ -0,0 +1,30 @@
+/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) */
+/* Do not edit directly, auto-generated from: */
+/* Documentation/netlink/specs/ulp_ddp.yaml */
+/* YNL-GEN kernel header */
+
+#ifndef _LINUX_ULP_DDP_GEN_H
+#define _LINUX_ULP_DDP_GEN_H
+
+#include <net/netlink.h>
+#include <net/genetlink.h>
+
+#include <uapi/linux/ulp_ddp.h>
+
+int ulp_ddp_get_netdev(const struct genl_split_ops *ops, struct sk_buff *skb,
+ struct genl_info *info);
+void
+ulp_ddp_put_netdev(const struct genl_split_ops *ops, struct sk_buff *skb,
+ struct genl_info *info);
+
+int ulp_ddp_nl_caps_get_doit(struct sk_buff *skb, struct genl_info *info);
+int ulp_ddp_nl_stats_get_doit(struct sk_buff *skb, struct genl_info *info);
+int ulp_ddp_nl_caps_set_doit(struct sk_buff *skb, struct genl_info *info);
+
+enum {
+ ULP_DDP_NLGRP_MGMT,
+};
+
+extern struct genl_family ulp_ddp_nl_family;
+
+#endif /* _LINUX_ULP_DDP_GEN_H */
diff --git a/net/core/ulp_ddp_nl.c b/net/core/ulp_ddp_nl.c
new file mode 100644
index 000000000000..04feefce23c2
--- /dev/null
+++ b/net/core/ulp_ddp_nl.c
@@ -0,0 +1,344 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * ulp_ddp_nl.c
+ * Author: Aurelien Aptel <aaptel@nvidia.com>
+ * Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+ */
+#include <net/ulp_ddp.h>
+#include "ulp_ddp_gen_nl.h"
+
+#define ULP_DDP_STATS_CNT (sizeof(struct ulp_ddp_stats) / sizeof(u64))
+
+struct ulp_ddp_reply_context {
+ struct net_device *dev;
+ netdevice_tracker tracker;
+ struct ulp_ddp_dev_caps caps;
+ struct ulp_ddp_stats stats;
+};
+
+static size_t ulp_ddp_reply_size(int cmd)
+{
+ size_t len = 0;
+
+ BUILD_BUG_ON(ULP_DDP_CAP_COUNT > 64);
+
+ /* ifindex */
+ len += nla_total_size(sizeof(u32));
+
+ switch (cmd) {
+ case ULP_DDP_CMD_CAPS_GET:
+ case ULP_DDP_CMD_CAPS_SET:
+ case ULP_DDP_CMD_CAPS_SET_NTF:
+ /* hw */
+ len += nla_total_size_64bit(sizeof(u64));
+
+ /* active */
+ len += nla_total_size_64bit(sizeof(u64));
+ break;
+ case ULP_DDP_CMD_STATS_GET:
+ /* stats */
+ len += nla_total_size_64bit(sizeof(u64)) * ULP_DDP_STATS_CNT;
+ break;
+ }
+
+ return len;
+}
+
+/* pre_doit */
+int ulp_ddp_get_netdev(const struct genl_split_ops *ops,
+ struct sk_buff *skb, struct genl_info *info)
+{
+ struct ulp_ddp_reply_context *ctx;
+ u32 ifindex;
+
+ if (GENL_REQ_ATTR_CHECK(info, ULP_DDP_A_CAPS_IFINDEX))
+ return -EINVAL;
+
+ ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
+ if (!ctx)
+ return -ENOMEM;
+
+ ifindex = nla_get_u32(info->attrs[ULP_DDP_A_CAPS_IFINDEX]);
+ ctx->dev = netdev_get_by_index(genl_info_net(info),
+ ifindex,
+ &ctx->tracker,
+ GFP_KERNEL);
+ if (!ctx->dev) {
+ kfree(ctx);
+ NL_SET_ERR_MSG_ATTR(info->extack,
+ info->attrs[ULP_DDP_A_CAPS_IFINDEX],
+ "Network interface does not exist");
+ return -ENODEV;
+ }
+
+ if (!ctx->dev->netdev_ops->ulp_ddp_ops) {
+ netdev_put(ctx->dev, &ctx->tracker);
+ kfree(ctx);
+ NL_SET_ERR_MSG_ATTR(info->extack,
+ info->attrs[ULP_DDP_A_CAPS_IFINDEX],
+ "Network interface does not support ULP DDP");
+ return -EOPNOTSUPP;
+ }
+
+ info->user_ptr[0] = ctx;
+ return 0;
+}
+
+/* post_doit */
+void ulp_ddp_put_netdev(const struct genl_split_ops *ops, struct sk_buff *skb,
+ struct genl_info *info)
+{
+ struct ulp_ddp_reply_context *ctx = info->user_ptr[0];
+
+ netdev_put(ctx->dev, &ctx->tracker);
+ kfree(ctx);
+}
+
+static int ulp_ddp_prepare_context(struct ulp_ddp_reply_context *ctx, int cmd)
+{
+ const struct ulp_ddp_dev_ops *ops = ctx->dev->netdev_ops->ulp_ddp_ops;
+
+ switch (cmd) {
+ case ULP_DDP_CMD_CAPS_GET:
+ case ULP_DDP_CMD_CAPS_SET:
+ case ULP_DDP_CMD_CAPS_SET_NTF:
+ ops->get_caps(ctx->dev, &ctx->caps);
+ break;
+ case ULP_DDP_CMD_STATS_GET:
+ ops->get_stats(ctx->dev, &ctx->stats);
+ break;
+ }
+
+ return 0;
+}
+
+static int ulp_ddp_write_reply(struct sk_buff *rsp,
+ struct ulp_ddp_reply_context *ctx,
+ int cmd,
+ const struct genl_info *info)
+{
+ void *hdr;
+
+ hdr = genlmsg_iput(rsp, info);
+ if (!hdr)
+ return -EMSGSIZE;
+
+ switch (cmd) {
+ case ULP_DDP_CMD_CAPS_GET:
+ case ULP_DDP_CMD_CAPS_SET:
+ case ULP_DDP_CMD_CAPS_SET_NTF:
+ if (nla_put_u32(rsp, ULP_DDP_A_CAPS_IFINDEX,
+ ctx->dev->ifindex) ||
+ nla_put_uint(rsp, ULP_DDP_A_CAPS_HW, ctx->caps.hw[0]) ||
+ nla_put_uint(rsp, ULP_DDP_A_CAPS_ACTIVE,
+ ctx->caps.active[0]))
+ goto err_cancel_msg;
+ break;
+ case ULP_DDP_CMD_STATS_GET:
+ if (nla_put_u32(rsp, ULP_DDP_A_STATS_IFINDEX,
+ ctx->dev->ifindex) ||
+ nla_put_uint(rsp,
+ ULP_DDP_A_STATS_RX_NVME_TCP_SK_ADD,
+ ctx->stats.rx_nvmeotcp_sk_add) ||
+ nla_put_uint(rsp,
+ ULP_DDP_A_STATS_RX_NVME_TCP_SK_ADD_FAIL,
+ ctx->stats.rx_nvmeotcp_sk_add_fail) ||
+ nla_put_uint(rsp,
+ ULP_DDP_A_STATS_RX_NVME_TCP_SK_DEL,
+ ctx->stats.rx_nvmeotcp_sk_del) ||
+ nla_put_uint(rsp,
+ ULP_DDP_A_STATS_RX_NVME_TCP_SETUP,
+ ctx->stats.rx_nvmeotcp_ddp_setup) ||
+ nla_put_uint(rsp,
+ ULP_DDP_A_STATS_RX_NVME_TCP_SETUP_FAIL,
+ ctx->stats.rx_nvmeotcp_ddp_setup_fail) ||
+ nla_put_uint(rsp,
+ ULP_DDP_A_STATS_RX_NVME_TCP_TEARDOWN,
+ ctx->stats.rx_nvmeotcp_ddp_teardown) ||
+ nla_put_uint(rsp,
+ ULP_DDP_A_STATS_RX_NVME_TCP_DROP,
+ ctx->stats.rx_nvmeotcp_drop) ||
+ nla_put_uint(rsp,
+ ULP_DDP_A_STATS_RX_NVME_TCP_RESYNC,
+ ctx->stats.rx_nvmeotcp_resync) ||
+ nla_put_uint(rsp,
+ ULP_DDP_A_STATS_RX_NVME_TCP_PACKETS,
+ ctx->stats.rx_nvmeotcp_packets) ||
+ nla_put_uint(rsp,
+ ULP_DDP_A_STATS_RX_NVME_TCP_BYTES,
+ ctx->stats.rx_nvmeotcp_bytes))
+ goto err_cancel_msg;
+ }
+ genlmsg_end(rsp, hdr);
+
+ return 0;
+
+err_cancel_msg:
+ genlmsg_cancel(rsp, hdr);
+
+ return -EMSGSIZE;
+}
+
+int ulp_ddp_nl_caps_get_doit(struct sk_buff *req, struct genl_info *info)
+{
+ struct ulp_ddp_reply_context *ctx = info->user_ptr[0];
+ struct sk_buff *rsp;
+ int ret = 0;
+
+ ret = ulp_ddp_prepare_context(ctx, ULP_DDP_CMD_CAPS_GET);
+ if (ret)
+ return ret;
+
+ rsp = genlmsg_new(ulp_ddp_reply_size(ULP_DDP_CMD_CAPS_GET), GFP_KERNEL);
+ if (!rsp)
+ return -EMSGSIZE;
+
+ ret = ulp_ddp_write_reply(rsp, ctx, ULP_DDP_CMD_CAPS_GET, info);
+ if (ret)
+ goto err_rsp;
+
+ return genlmsg_reply(rsp, info);
+
+err_rsp:
+ nlmsg_free(rsp);
+ return ret;
+}
+
+static void ulp_ddp_nl_notify_dev(struct ulp_ddp_reply_context *ctx)
+{
+ struct genl_info info;
+ struct sk_buff *ntf;
+ int ret;
+
+ if (!genl_has_listeners(&ulp_ddp_nl_family, dev_net(ctx->dev),
+ ULP_DDP_NLGRP_MGMT))
+ return;
+
+ genl_info_init_ntf(&info, &ulp_ddp_nl_family, ULP_DDP_CMD_CAPS_SET_NTF);
+ ntf = genlmsg_new(ulp_ddp_reply_size(ULP_DDP_CMD_CAPS_SET_NTF), GFP_KERNEL);
+ if (!ntf)
+ return;
+
+ ret = ulp_ddp_write_reply(ntf, ctx, ULP_DDP_CMD_CAPS_SET_NTF, &info);
+ if (ret) {
+ nlmsg_free(ntf);
+ return;
+ }
+
+ genlmsg_multicast_netns(&ulp_ddp_nl_family, dev_net(ctx->dev), ntf,
+ 0, ULP_DDP_NLGRP_MGMT, GFP_KERNEL);
+}
+
+static int ulp_ddp_apply_bits(struct ulp_ddp_reply_context *ctx,
+ unsigned long *req_wanted,
+ unsigned long *req_mask,
+ struct genl_info *info,
+ bool *notify)
+{
+ DECLARE_BITMAP(old_active, ULP_DDP_CAP_COUNT);
+ DECLARE_BITMAP(new_active, ULP_DDP_CAP_COUNT);
+ const struct ulp_ddp_dev_ops *ops;
+ struct ulp_ddp_dev_caps caps;
+ int ret;
+
+ ops = ctx->dev->netdev_ops->ulp_ddp_ops;
+ ops->get_caps(ctx->dev, &caps);
+
+ /* new_active = (old_active & ~req_mask) | (wanted & req_mask)
+ * new_active &= caps_hw
+ */
+ bitmap_copy(old_active, caps.active, ULP_DDP_CAP_COUNT);
+ bitmap_and(req_wanted, req_wanted, req_mask, ULP_DDP_CAP_COUNT);
+ bitmap_andnot(new_active, old_active, req_mask, ULP_DDP_CAP_COUNT);
+ bitmap_or(new_active, new_active, req_wanted, ULP_DDP_CAP_COUNT);
+ bitmap_and(new_active, new_active, caps.hw, ULP_DDP_CAP_COUNT);
+ if (!bitmap_equal(old_active, new_active, ULP_DDP_CAP_COUNT)) {
+ ret = ops->set_caps(ctx->dev, new_active, info->extack);
+ if (ret)
+ return ret;
+ ops->get_caps(ctx->dev, &caps);
+ bitmap_copy(new_active, caps.active, ULP_DDP_CAP_COUNT);
+ }
+
+ /* notify if capabilities were changed */
+ *notify = !bitmap_equal(old_active, new_active, ULP_DDP_CAP_COUNT);
+
+ return 0;
+}
+
+int ulp_ddp_nl_caps_set_doit(struct sk_buff *skb, struct genl_info *info)
+{
+ struct ulp_ddp_reply_context *ctx = info->user_ptr[0];
+ unsigned long wanted, wanted_mask;
+ struct sk_buff *rsp;
+ bool notify = false;
+ int ret;
+
+ if (GENL_REQ_ATTR_CHECK(info, ULP_DDP_A_CAPS_WANTED) ||
+ GENL_REQ_ATTR_CHECK(info, ULP_DDP_A_CAPS_WANTED_MASK))
+ return -EINVAL;
+
+ rsp = genlmsg_new(ulp_ddp_reply_size(ULP_DDP_CMD_CAPS_SET), GFP_KERNEL);
+ if (!rsp)
+ return -EMSGSIZE;
+
+ wanted = nla_get_uint(info->attrs[ULP_DDP_A_CAPS_WANTED]);
+ wanted_mask = nla_get_uint(info->attrs[ULP_DDP_A_CAPS_WANTED_MASK]);
+
+ rtnl_lock();
+ ret = ulp_ddp_apply_bits(ctx, &wanted, &wanted_mask, info, ¬ify);
+ rtnl_unlock();
+ if (ret)
+ goto err_rsp;
+
+ ret = ulp_ddp_prepare_context(ctx, ULP_DDP_CMD_CAPS_SET);
+ if (ret)
+ goto err_rsp;
+
+ ret = ulp_ddp_write_reply(rsp, ctx, ULP_DDP_CMD_CAPS_SET, info);
+ if (ret)
+ goto err_rsp;
+
+ ret = genlmsg_reply(rsp, info);
+ if (notify)
+ ulp_ddp_nl_notify_dev(ctx);
+
+ return ret;
+
+err_rsp:
+ nlmsg_free(rsp);
+
+ return ret;
+}
+
+int ulp_ddp_nl_stats_get_doit(struct sk_buff *skb, struct genl_info *info)
+{
+ struct ulp_ddp_reply_context *ctx = info->user_ptr[0];
+ struct sk_buff *rsp;
+ int ret = 0;
+
+ ret = ulp_ddp_prepare_context(ctx, ULP_DDP_CMD_STATS_GET);
+ if (ret)
+ return ret;
+
+ rsp = genlmsg_new(ulp_ddp_reply_size(ULP_DDP_CMD_STATS_GET), GFP_KERNEL);
+ if (!rsp)
+ return -EMSGSIZE;
+
+ ret = ulp_ddp_write_reply(rsp, ctx, ULP_DDP_CMD_STATS_GET, info);
+ if (ret)
+ goto err_rsp;
+
+ return genlmsg_reply(rsp, info);
+
+err_rsp:
+ nlmsg_free(rsp);
+ return ret;
+}
+
+static int __init ulp_ddp_init(void)
+{
+ return genl_register_family(&ulp_ddp_nl_family);
+}
+
+subsys_initcall(ulp_ddp_init);
--
2.34.1
^ permalink raw reply related [flat|nested] 44+ messages in thread
* [PATCH v24 03/20] iov_iter: skip copy if src == dst for direct data placement
2024-04-04 12:36 [PATCH v24 00/20] nvme-tcp receive offloads Aurelien Aptel
2024-04-04 12:36 ` [PATCH v24 01/20] net: Introduce direct data placement tcp offload Aurelien Aptel
2024-04-04 12:36 ` [PATCH v24 02/20] netlink: add new family to manage ULP_DDP enablement and stats Aurelien Aptel
@ 2024-04-04 12:37 ` Aurelien Aptel
2024-04-15 14:28 ` Max Gurtovoy
2024-04-16 20:30 ` David Laight
2024-04-04 12:37 ` [PATCH v24 04/20] net/tls,core: export get_netdev_for_sock Aurelien Aptel
` (17 subsequent siblings)
20 siblings, 2 replies; 44+ messages in thread
From: Aurelien Aptel @ 2024-04-04 12:37 UTC (permalink / raw)
To: linux-nvme, netdev, sagi, hch, kbusch, axboe, chaitanyak, davem, kuba
Cc: aaptel, aurelien.aptel, smalin, malin1024, ogerlitz, yorayz,
borisp, galshalom, mgurtovoy, viro
From: Ben Ben-Ishay <benishay@nvidia.com>
When using direct data placement (DDP) the NIC could write the payload
directly into the destination buffer and constructs SKBs such that
they point to this data. To skip copies when SKB data already resides
in the destination buffer we check if (src == dst), and skip the copy
when it's true.
Signed-off-by: Ben Ben-Ishay <benishay@nvidia.com>
Signed-off-by: Boris Pismenny <borisp@nvidia.com>
Signed-off-by: Or Gerlitz <ogerlitz@nvidia.com>
Signed-off-by: Yoray Zack <yorayz@nvidia.com>
Signed-off-by: Shai Malin <smalin@nvidia.com>
Signed-off-by: Aurelien Aptel <aaptel@nvidia.com>
Reviewed-by: Chaitanya Kulkarni <kch@nvidia.com>
---
lib/iov_iter.c | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/lib/iov_iter.c b/lib/iov_iter.c
index 4a6a9f419bd7..a85125485174 100644
--- a/lib/iov_iter.c
+++ b/lib/iov_iter.c
@@ -62,7 +62,14 @@ static __always_inline
size_t memcpy_to_iter(void *iter_to, size_t progress,
size_t len, void *from, void *priv2)
{
- memcpy(iter_to, from + progress, len);
+ /*
+ * When using direct data placement (DDP) the hardware writes
+ * data directly to the destination buffer, and constructs
+ * IOVs such that they point to this data.
+ * Thus, when the src == dst we skip the memcpy.
+ */
+ if (iter_to != from + progress)
+ memcpy(iter_to, from + progress, len);
return 0;
}
--
2.34.1
^ permalink raw reply related [flat|nested] 44+ messages in thread
* [PATCH v24 04/20] net/tls,core: export get_netdev_for_sock
2024-04-04 12:36 [PATCH v24 00/20] nvme-tcp receive offloads Aurelien Aptel
` (2 preceding siblings ...)
2024-04-04 12:37 ` [PATCH v24 03/20] iov_iter: skip copy if src == dst for direct data placement Aurelien Aptel
@ 2024-04-04 12:37 ` Aurelien Aptel
2024-04-21 11:45 ` Sagi Grimberg
2024-04-04 12:37 ` [PATCH v24 05/20] nvme-tcp: Add DDP offload control path Aurelien Aptel
` (16 subsequent siblings)
20 siblings, 1 reply; 44+ messages in thread
From: Aurelien Aptel @ 2024-04-04 12:37 UTC (permalink / raw)
To: linux-nvme, netdev, sagi, hch, kbusch, axboe, chaitanyak, davem, kuba
Cc: Aurelien Aptel, aurelien.aptel, smalin, malin1024, ogerlitz,
yorayz, borisp, galshalom, mgurtovoy, edumazet, pabeni,
john.fastabend, daniel
* remove netdev_sk_get_lowest_dev() from net/core
* move get_netdev_for_sock() from net/tls to net/core
* update exising users in net/tls/tls_device.c
get_netdev_for_sock() is a utility that is used to obtain
the net_device structure from a connected socket.
Later patches will use this for nvme-tcp DDP and DDP DDGST offloads.
Suggested-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Ben Ben-Ishay <benishay@nvidia.com>
Signed-off-by: Shai Malin <smalin@nvidia.com>
Signed-off-by: Aurelien Aptel <aaptel@nvidia.com>
---
include/linux/netdevice.h | 5 +++--
net/core/dev.c | 30 +++++++++++++++++-------------
net/tls/tls_device.c | 31 +++++++++----------------------
3 files changed, 29 insertions(+), 37 deletions(-)
diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index 2fc33d9470e5..cd490b6f003c 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -3129,8 +3129,9 @@ void init_dummy_netdev(struct net_device *dev);
struct net_device *netdev_get_xmit_slave(struct net_device *dev,
struct sk_buff *skb,
bool all_slaves);
-struct net_device *netdev_sk_get_lowest_dev(struct net_device *dev,
- struct sock *sk);
+struct net_device *get_netdev_for_sock(struct sock *sk,
+ netdevice_tracker *tracker,
+ gfp_t gfp);
struct net_device *dev_get_by_index(struct net *net, int ifindex);
struct net_device *__dev_get_by_index(struct net *net, int ifindex);
struct net_device *netdev_get_by_index(struct net *net, int ifindex,
diff --git a/net/core/dev.c b/net/core/dev.c
index 818699dea9d7..cc01c76f65c6 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -8376,27 +8376,31 @@ static struct net_device *netdev_sk_get_lower_dev(struct net_device *dev,
}
/**
- * netdev_sk_get_lowest_dev - Get the lowest device in chain given device and socket
- * @dev: device
+ * get_netdev_for_sock - Get the lowest device in socket
* @sk: the socket
+ * @tracker: tracking object for the acquired reference
+ * @gfp: allocation flags for the tracker
*
- * %NULL is returned if no lower device is found.
+ * Assumes that the socket is already connected.
+ * Returns the lower device or %NULL if no lower device is found.
*/
-
-struct net_device *netdev_sk_get_lowest_dev(struct net_device *dev,
- struct sock *sk)
+struct net_device *get_netdev_for_sock(struct sock *sk,
+ netdevice_tracker *tracker,
+ gfp_t gfp)
{
- struct net_device *lower;
+ struct dst_entry *dst = sk_dst_get(sk);
+ struct net_device *dev, *lower;
- lower = netdev_sk_get_lower_dev(dev, sk);
- while (lower) {
+ if (unlikely(!dst))
+ return NULL;
+ dev = dst->dev;
+ while ((lower = netdev_sk_get_lower_dev(dev, sk)))
dev = lower;
- lower = netdev_sk_get_lower_dev(dev, sk);
- }
-
+ netdev_hold(dev, tracker, gfp);
+ dst_release(dst);
return dev;
}
-EXPORT_SYMBOL(netdev_sk_get_lowest_dev);
+EXPORT_SYMBOL_GPL(get_netdev_for_sock);
static void netdev_adjacent_add_links(struct net_device *dev)
{
diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c
index bf8ed36b1ad6..5868daf36ae2 100644
--- a/net/tls/tls_device.c
+++ b/net/tls/tls_device.c
@@ -119,22 +119,6 @@ static void tls_device_queue_ctx_destruction(struct tls_context *ctx)
tls_device_free_ctx(ctx);
}
-/* We assume that the socket is already connected */
-static struct net_device *get_netdev_for_sock(struct sock *sk)
-{
- struct dst_entry *dst = sk_dst_get(sk);
- struct net_device *netdev = NULL;
-
- if (likely(dst)) {
- netdev = netdev_sk_get_lowest_dev(dst->dev, sk);
- dev_hold(netdev);
- }
-
- dst_release(dst);
-
- return netdev;
-}
-
static void destroy_record(struct tls_record_info *record)
{
int i;
@@ -1063,6 +1047,7 @@ int tls_set_device_offload(struct sock *sk)
struct tls_offload_context_tx *offload_ctx;
const struct tls_cipher_desc *cipher_desc;
struct tls_crypto_info *crypto_info;
+ netdevice_tracker netdev_tracker;
struct tls_prot_info *prot;
struct net_device *netdev;
struct tls_context *ctx;
@@ -1076,7 +1061,7 @@ int tls_set_device_offload(struct sock *sk)
if (ctx->priv_ctx_tx)
return -EEXIST;
- netdev = get_netdev_for_sock(sk);
+ netdev = get_netdev_for_sock(sk, &netdev_tracker, GFP_KERNEL);
if (!netdev) {
pr_err_ratelimited("%s: netdev not found\n", __func__);
return -EINVAL;
@@ -1172,7 +1157,7 @@ int tls_set_device_offload(struct sock *sk)
* by the netdev's xmit function.
*/
smp_store_release(&sk->sk_validate_xmit_skb, tls_validate_xmit_skb);
- dev_put(netdev);
+ netdev_put(netdev, &netdev_tracker);
return 0;
@@ -1186,7 +1171,7 @@ int tls_set_device_offload(struct sock *sk)
free_marker_record:
kfree(start_marker_record);
release_netdev:
- dev_put(netdev);
+ netdev_put(netdev, &netdev_tracker);
return rc;
}
@@ -1194,13 +1179,15 @@ int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx)
{
struct tls12_crypto_info_aes_gcm_128 *info;
struct tls_offload_context_rx *context;
+ netdevice_tracker netdev_tracker;
struct net_device *netdev;
+
int rc = 0;
if (ctx->crypto_recv.info.version != TLS_1_2_VERSION)
return -EOPNOTSUPP;
- netdev = get_netdev_for_sock(sk);
+ netdev = get_netdev_for_sock(sk, &netdev_tracker, GFP_KERNEL);
if (!netdev) {
pr_err_ratelimited("%s: netdev not found\n", __func__);
return -EINVAL;
@@ -1249,7 +1236,7 @@ int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx)
tls_device_attach(ctx, sk, netdev);
up_read(&device_offload_lock);
- dev_put(netdev);
+ netdev_put(netdev, &netdev_tracker);
return 0;
@@ -1262,7 +1249,7 @@ int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx)
release_lock:
up_read(&device_offload_lock);
release_netdev:
- dev_put(netdev);
+ netdev_put(netdev, &netdev_tracker);
return rc;
}
--
2.34.1
^ permalink raw reply related [flat|nested] 44+ messages in thread
* [PATCH v24 05/20] nvme-tcp: Add DDP offload control path
2024-04-04 12:36 [PATCH v24 00/20] nvme-tcp receive offloads Aurelien Aptel
` (3 preceding siblings ...)
2024-04-04 12:37 ` [PATCH v24 04/20] net/tls,core: export get_netdev_for_sock Aurelien Aptel
@ 2024-04-04 12:37 ` Aurelien Aptel
2024-04-07 22:08 ` Sagi Grimberg
2024-04-04 12:37 ` [PATCH v24 06/20] nvme-tcp: Add DDP data-path Aurelien Aptel
` (15 subsequent siblings)
20 siblings, 1 reply; 44+ messages in thread
From: Aurelien Aptel @ 2024-04-04 12:37 UTC (permalink / raw)
To: linux-nvme, netdev, sagi, hch, kbusch, axboe, chaitanyak, davem, kuba
Cc: Boris Pismenny, aaptel, aurelien.aptel, smalin, malin1024,
ogerlitz, yorayz, galshalom, mgurtovoy, brauner
From: Boris Pismenny <borisp@nvidia.com>
This commit introduces direct data placement offload to NVME
TCP. There is a context per queue, which is established after the
handshake using the sk_add/del NDOs.
Additionally, a resynchronization routine is used to assist
hardware recovery from TCP OOO, and continue the offload.
Resynchronization operates as follows:
1. TCP OOO causes the NIC HW to stop the offload
2. NIC HW identifies a PDU header at some TCP sequence number,
and asks NVMe-TCP to confirm it.
This request is delivered from the NIC driver to NVMe-TCP by first
finding the socket for the packet that triggered the request, and
then finding the nvme_tcp_queue that is used by this routine.
Finally, the request is recorded in the nvme_tcp_queue.
3. When NVMe-TCP observes the requested TCP sequence, it will compare
it with the PDU header TCP sequence, and report the result to the
NIC driver (resync), which will update the HW, and resume offload
when all is successful.
Some HW implementation such as ConnectX-7 assume linear CCID (0...N-1
for queue of size N) where the linux nvme driver uses part of the 16
bit CCID for generation counter. To address that, we use the existing
quirk in the nvme layer when the HW driver advertises if the device is
not supports the full 16 bit CCID range.
Furthermore, we let the offloading driver advertise what is the max hw
sectors/segments via ulp_ddp_limits.
A follow-up patch introduces the data-path changes required for this
offload.
Socket operations need a netdev reference. This reference is
dropped on NETDEV_GOING_DOWN events to allow the device to go down in
a follow-up patch.
Signed-off-by: Boris Pismenny <borisp@nvidia.com>
Signed-off-by: Ben Ben-Ishay <benishay@nvidia.com>
Signed-off-by: Or Gerlitz <ogerlitz@nvidia.com>
Signed-off-by: Yoray Zack <yorayz@nvidia.com>
Signed-off-by: Shai Malin <smalin@nvidia.com>
Signed-off-by: Aurelien Aptel <aaptel@nvidia.com>
Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
Reviewed-by: Chaitanya Kulkarni <kch@nvidia.com>
Reviewed-by: Max Gurtovoy <mgurtovoy@nvidia.com>
---
drivers/nvme/host/tcp.c | 264 ++++++++++++++++++++++++++++++++++++++--
1 file changed, 251 insertions(+), 13 deletions(-)
diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index fdbcdcedcee9..529eee4b8f98 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -21,6 +21,10 @@
#include <net/busy_poll.h>
#include <trace/events/sock.h>
+#ifdef CONFIG_ULP_DDP
+#include <net/ulp_ddp.h>
+#endif
+
#include "nvme.h"
#include "fabrics.h"
@@ -54,6 +58,16 @@ MODULE_PARM_DESC(tls_handshake_timeout,
"nvme TLS handshake timeout in seconds (default 10)");
#endif
+#ifdef CONFIG_ULP_DDP
+/* NVMeTCP direct data placement and data digest offload will not
+ * happen if this parameter false (default), regardless of what the
+ * underlying netdev capabilities are.
+ */
+static bool ddp_offload;
+module_param(ddp_offload, bool, 0644);
+MODULE_PARM_DESC(ddp_offload, "Enable or disable NVMeTCP direct data placement support");
+#endif
+
#ifdef CONFIG_DEBUG_LOCK_ALLOC
/* lockdep can detect a circular dependency of the form
* sk_lock -> mmap_lock (page fault) -> fs locks -> sk_lock
@@ -127,6 +141,7 @@ enum nvme_tcp_queue_flags {
NVME_TCP_Q_ALLOCATED = 0,
NVME_TCP_Q_LIVE = 1,
NVME_TCP_Q_POLLING = 2,
+ NVME_TCP_Q_OFF_DDP = 3,
};
enum nvme_tcp_recv_state {
@@ -154,6 +169,18 @@ struct nvme_tcp_queue {
size_t ddgst_remaining;
unsigned int nr_cqe;
+#ifdef CONFIG_ULP_DDP
+ /*
+ * resync_tcp_seq is a speculative PDU header tcp seq number (with
+ * an additional flag in the lower 32 bits) that the HW send to
+ * the SW, for the SW to verify.
+ * - The 32 high bits store the seq number
+ * - The 32 low bits are used as a flag to know if a request
+ * is pending (ULP_DDP_RESYNC_PENDING).
+ */
+ atomic64_t resync_tcp_seq;
+#endif
+
/* send state */
struct nvme_tcp_request *request;
@@ -194,6 +221,13 @@ struct nvme_tcp_ctrl {
struct delayed_work connect_work;
struct nvme_tcp_request async_req;
u32 io_queues[HCTX_MAX_TYPES];
+
+ struct net_device *ddp_netdev;
+ netdevice_tracker netdev_tracker;
+ u32 ddp_threshold;
+#ifdef CONFIG_ULP_DDP
+ struct ulp_ddp_limits ddp_limits;
+#endif
};
static LIST_HEAD(nvme_tcp_ctrl_list);
@@ -305,6 +339,170 @@ static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req,
return nvme_tcp_pdu_data_left(req) <= len;
}
+#ifdef CONFIG_ULP_DDP
+
+static struct net_device *
+nvme_tcp_get_ddp_netdev_with_limits(struct nvme_tcp_ctrl *ctrl)
+{
+ struct net_device *netdev;
+ int ret;
+
+ if (!ddp_offload)
+ return NULL;
+
+ /* netdev ref is put in nvme_tcp_stop_admin_queue() */
+ netdev = get_netdev_for_sock(ctrl->queues[0].sock->sk, &ctrl->netdev_tracker, GFP_KERNEL);
+ if (!netdev) {
+ dev_dbg(ctrl->ctrl.device, "netdev not found\n");
+ return NULL;
+ }
+
+ if (!ulp_ddp_is_cap_active(netdev, ULP_DDP_CAP_NVME_TCP))
+ goto err;
+
+ ret = ulp_ddp_get_limits(netdev, &ctrl->ddp_limits, ULP_DDP_NVME);
+ if (ret)
+ goto err;
+
+ if (ctrl->ctrl.opts->tls && !ctrl->ddp_limits.tls)
+ goto err;
+
+ return netdev;
+err:
+ netdev_put(netdev, &ctrl->netdev_tracker);
+ return NULL;
+}
+
+static bool nvme_tcp_resync_request(struct sock *sk, u32 seq, u32 flags);
+static const struct ulp_ddp_ulp_ops nvme_tcp_ddp_ulp_ops = {
+ .resync_request = nvme_tcp_resync_request,
+};
+
+static int nvme_tcp_offload_socket(struct nvme_tcp_queue *queue)
+{
+ struct ulp_ddp_config config = {.type = ULP_DDP_NVME};
+ int ret;
+
+ config.nvmeotcp.pfv = NVME_TCP_PFV_1_0;
+ config.nvmeotcp.cpda = 0;
+ config.nvmeotcp.dgst =
+ queue->hdr_digest ? NVME_TCP_HDR_DIGEST_ENABLE : 0;
+ config.nvmeotcp.dgst |=
+ queue->data_digest ? NVME_TCP_DATA_DIGEST_ENABLE : 0;
+ config.nvmeotcp.queue_size = queue->ctrl->ctrl.sqsize + 1;
+
+ ret = ulp_ddp_sk_add(queue->ctrl->ddp_netdev,
+ queue->sock->sk,
+ &config,
+ &nvme_tcp_ddp_ulp_ops);
+ if (ret)
+ return ret;
+
+ set_bit(NVME_TCP_Q_OFF_DDP, &queue->flags);
+
+ return 0;
+}
+
+static void nvme_tcp_unoffload_socket(struct nvme_tcp_queue *queue)
+{
+ clear_bit(NVME_TCP_Q_OFF_DDP, &queue->flags);
+ ulp_ddp_sk_del(queue->ctrl->ddp_netdev, queue->sock->sk);
+}
+
+static void nvme_tcp_ddp_apply_limits(struct nvme_tcp_ctrl *ctrl)
+{
+ ctrl->ctrl.max_segments = ctrl->ddp_limits.max_ddp_sgl_len;
+ ctrl->ctrl.max_hw_sectors =
+ ctrl->ddp_limits.max_ddp_sgl_len << (NVME_CTRL_PAGE_SHIFT - SECTOR_SHIFT);
+ ctrl->ddp_threshold = ctrl->ddp_limits.io_threshold;
+
+ /* offloading HW doesn't support full ccid range, apply the quirk */
+ ctrl->ctrl.quirks |=
+ ctrl->ddp_limits.nvmeotcp.full_ccid_range ? 0 : NVME_QUIRK_SKIP_CID_GEN;
+}
+
+/* In presence of packet drops or network packet reordering, the device may lose
+ * synchronization between the TCP stream and the L5P framing, and require a
+ * resync with the kernel's TCP stack.
+ *
+ * - NIC HW identifies a PDU header at some TCP sequence number,
+ * and asks NVMe-TCP to confirm it.
+ * - When NVMe-TCP observes the requested TCP sequence, it will compare
+ * it with the PDU header TCP sequence, and report the result to the
+ * NIC driver
+ */
+static void nvme_tcp_resync_response(struct nvme_tcp_queue *queue,
+ struct sk_buff *skb, unsigned int offset)
+{
+ u64 pdu_seq = TCP_SKB_CB(skb)->seq + offset - queue->pdu_offset;
+ struct net_device *netdev = queue->ctrl->ddp_netdev;
+ u64 pdu_val = (pdu_seq << 32) | ULP_DDP_RESYNC_PENDING;
+ u64 resync_val;
+ u32 resync_seq;
+
+ resync_val = atomic64_read(&queue->resync_tcp_seq);
+ /* Lower 32 bit flags. Check validity of the request */
+ if ((resync_val & ULP_DDP_RESYNC_PENDING) == 0)
+ return;
+
+ /*
+ * Obtain and check requested sequence number: is this PDU header
+ * before the request?
+ */
+ resync_seq = resync_val >> 32;
+ if (before(pdu_seq, resync_seq))
+ return;
+
+ /*
+ * The atomic operation guarantees that we don't miss any NIC driver
+ * resync requests submitted after the above checks.
+ */
+ if (atomic64_cmpxchg(&queue->resync_tcp_seq, pdu_val,
+ pdu_val & ~ULP_DDP_RESYNC_PENDING) !=
+ atomic64_read(&queue->resync_tcp_seq))
+ ulp_ddp_resync(netdev, queue->sock->sk, pdu_seq);
+}
+
+static bool nvme_tcp_resync_request(struct sock *sk, u32 seq, u32 flags)
+{
+ struct nvme_tcp_queue *queue = sk->sk_user_data;
+
+ /*
+ * "seq" (TCP seq number) is what the HW assumes is the
+ * beginning of a PDU. The nvme-tcp layer needs to store the
+ * number along with the "flags" (ULP_DDP_RESYNC_PENDING) to
+ * indicate that a request is pending.
+ */
+ atomic64_set(&queue->resync_tcp_seq, (((uint64_t)seq << 32) | flags));
+
+ return true;
+}
+
+#else
+
+static struct net_device *
+nvme_tcp_get_ddp_netdev_with_limits(struct nvme_tcp_ctrl *ctrl)
+{
+ return NULL;
+}
+
+static void nvme_tcp_ddp_apply_limits(struct nvme_tcp_ctrl *ctrl)
+{}
+
+static int nvme_tcp_offload_socket(struct nvme_tcp_queue *queue)
+{
+ return 0;
+}
+
+static void nvme_tcp_unoffload_socket(struct nvme_tcp_queue *queue)
+{}
+
+static void nvme_tcp_resync_response(struct nvme_tcp_queue *queue,
+ struct sk_buff *skb, unsigned int offset)
+{}
+
+#endif
+
static void nvme_tcp_init_iter(struct nvme_tcp_request *req,
unsigned int dir)
{
@@ -747,6 +945,9 @@ static int nvme_tcp_recv_pdu(struct nvme_tcp_queue *queue, struct sk_buff *skb,
size_t rcv_len = min_t(size_t, *len, queue->pdu_remaining);
int ret;
+ if (test_bit(NVME_TCP_Q_OFF_DDP, &queue->flags))
+ nvme_tcp_resync_response(queue, skb, *offset);
+
ret = skb_copy_bits(skb, *offset,
&pdu[queue->pdu_offset], rcv_len);
if (unlikely(ret))
@@ -1810,6 +2011,8 @@ static void __nvme_tcp_stop_queue(struct nvme_tcp_queue *queue)
kernel_sock_shutdown(queue->sock, SHUT_RDWR);
nvme_tcp_restore_sock_ops(queue);
cancel_work_sync(&queue->io_work);
+ if (test_bit(NVME_TCP_Q_OFF_DDP, &queue->flags))
+ nvme_tcp_unoffload_socket(queue);
}
static void nvme_tcp_stop_queue(struct nvme_ctrl *nctrl, int qid)
@@ -1826,6 +2029,20 @@ static void nvme_tcp_stop_queue(struct nvme_ctrl *nctrl, int qid)
mutex_unlock(&queue->queue_lock);
}
+static void nvme_tcp_stop_admin_queue(struct nvme_ctrl *nctrl)
+{
+ struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(nctrl);
+
+ nvme_tcp_stop_queue(nctrl, 0);
+
+ /*
+ * We are called twice by nvme_tcp_teardown_admin_queue()
+ * Set ddp_netdev to NULL to avoid putting it twice
+ */
+ netdev_put(ctrl->ddp_netdev, &ctrl->netdev_tracker);
+ ctrl->ddp_netdev = NULL;
+}
+
static void nvme_tcp_setup_sock_ops(struct nvme_tcp_queue *queue)
{
write_lock_bh(&queue->sock->sk->sk_callback_lock);
@@ -1852,19 +2069,37 @@ static int nvme_tcp_start_queue(struct nvme_ctrl *nctrl, int idx)
nvme_tcp_init_recv_ctx(queue);
nvme_tcp_setup_sock_ops(queue);
- if (idx)
+ if (idx) {
ret = nvmf_connect_io_queue(nctrl, idx);
- else
+ if (ret)
+ goto err;
+
+ if (ctrl->ddp_netdev) {
+ ret = nvme_tcp_offload_socket(queue);
+ if (ret) {
+ dev_info(nctrl->device,
+ "failed to setup offload on queue %d ret=%d\n",
+ idx, ret);
+ }
+ }
+ } else {
ret = nvmf_connect_admin_queue(nctrl);
+ if (ret)
+ goto err;
+
+ ctrl->ddp_netdev = nvme_tcp_get_ddp_netdev_with_limits(ctrl);
+ if (ctrl->ddp_netdev)
+ nvme_tcp_ddp_apply_limits(ctrl);
- if (!ret) {
- set_bit(NVME_TCP_Q_LIVE, &queue->flags);
- } else {
- if (test_bit(NVME_TCP_Q_ALLOCATED, &queue->flags))
- __nvme_tcp_stop_queue(queue);
- dev_err(nctrl->device,
- "failed to connect queue: %d ret=%d\n", idx, ret);
}
+
+ set_bit(NVME_TCP_Q_LIVE, &queue->flags);
+ return 0;
+err:
+ if (test_bit(NVME_TCP_Q_ALLOCATED, &queue->flags))
+ __nvme_tcp_stop_queue(queue);
+ dev_err(nctrl->device,
+ "failed to connect queue: %d ret=%d\n", idx, ret);
return ret;
}
@@ -2075,7 +2310,7 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)
static void nvme_tcp_destroy_admin_queue(struct nvme_ctrl *ctrl, bool remove)
{
- nvme_tcp_stop_queue(ctrl, 0);
+ nvme_tcp_stop_admin_queue(ctrl);
if (remove)
nvme_remove_admin_tag_set(ctrl);
nvme_tcp_free_admin_queue(ctrl);
@@ -2118,7 +2353,7 @@ static int nvme_tcp_configure_admin_queue(struct nvme_ctrl *ctrl, bool new)
nvme_quiesce_admin_queue(ctrl);
blk_sync_queue(ctrl->admin_q);
out_stop_queue:
- nvme_tcp_stop_queue(ctrl, 0);
+ nvme_tcp_stop_admin_queue(ctrl);
nvme_cancel_admin_tagset(ctrl);
out_cleanup_tagset:
if (new)
@@ -2133,7 +2368,7 @@ static void nvme_tcp_teardown_admin_queue(struct nvme_ctrl *ctrl,
{
nvme_quiesce_admin_queue(ctrl);
blk_sync_queue(ctrl->admin_q);
- nvme_tcp_stop_queue(ctrl, 0);
+ nvme_tcp_stop_admin_queue(ctrl);
nvme_cancel_admin_tagset(ctrl);
if (remove)
nvme_unquiesce_admin_queue(ctrl);
@@ -2425,7 +2660,10 @@ static void nvme_tcp_complete_timed_out(struct request *rq)
struct nvme_tcp_request *req = blk_mq_rq_to_pdu(rq);
struct nvme_ctrl *ctrl = &req->queue->ctrl->ctrl;
- nvme_tcp_stop_queue(ctrl, nvme_tcp_queue_id(req->queue));
+ if (nvme_tcp_admin_queue(req->queue))
+ nvme_tcp_stop_admin_queue(ctrl);
+ else
+ nvme_tcp_stop_queue(ctrl, nvme_tcp_queue_id(req->queue));
nvmf_complete_timed_out_request(rq);
}
--
2.34.1
^ permalink raw reply related [flat|nested] 44+ messages in thread
* [PATCH v24 06/20] nvme-tcp: Add DDP data-path
2024-04-04 12:36 [PATCH v24 00/20] nvme-tcp receive offloads Aurelien Aptel
` (4 preceding siblings ...)
2024-04-04 12:37 ` [PATCH v24 05/20] nvme-tcp: Add DDP offload control path Aurelien Aptel
@ 2024-04-04 12:37 ` Aurelien Aptel
2024-04-07 22:08 ` Sagi Grimberg
2024-04-04 12:37 ` [PATCH v24 07/20] nvme-tcp: RX DDGST offload Aurelien Aptel
` (14 subsequent siblings)
20 siblings, 1 reply; 44+ messages in thread
From: Aurelien Aptel @ 2024-04-04 12:37 UTC (permalink / raw)
To: linux-nvme, netdev, sagi, hch, kbusch, axboe, chaitanyak, davem, kuba
Cc: Boris Pismenny, aaptel, aurelien.aptel, smalin, malin1024,
ogerlitz, yorayz, galshalom, mgurtovoy
From: Boris Pismenny <borisp@nvidia.com>
Introduce the NVMe-TCP DDP data-path offload.
Using this interface, the NIC hardware will scatter TCP payload directly
to the BIO pages according to the command_id in the PDU.
To maintain the correctness of the network stack, the driver is expected
to construct SKBs that point to the BIO pages.
The data-path interface contains two routines: setup/teardown.
The setup provides the mapping from command_id to the request buffers,
while the teardown removes this mapping.
For efficiency, we introduce an asynchronous nvme completion, which is
split between NVMe-TCP and the NIC driver as follows:
NVMe-TCP performs the specific completion, while NIC driver performs the
generic mq_blk completion.
Signed-off-by: Boris Pismenny <borisp@nvidia.com>
Signed-off-by: Ben Ben-Ishay <benishay@nvidia.com>
Signed-off-by: Or Gerlitz <ogerlitz@nvidia.com>
Signed-off-by: Yoray Zack <yorayz@nvidia.com>
Signed-off-by: Shai Malin <smalin@nvidia.com>
Signed-off-by: Aurelien Aptel <aaptel@nvidia.com>
Reviewed-by: Chaitanya Kulkarni <kch@nvidia.com>
Reviewed-by: Max Gurtovoy <mgurtovoy@nvidia.com>
---
drivers/nvme/host/tcp.c | 136 ++++++++++++++++++++++++++++++++++++++--
1 file changed, 131 insertions(+), 5 deletions(-)
diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index 529eee4b8f98..345c7885c476 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -128,6 +128,9 @@ struct nvme_tcp_request {
struct llist_node lentry;
__le32 ddgst;
+ /* ddp async completion */
+ union nvme_result result;
+
struct bio *curr_bio;
struct iov_iter iter;
@@ -135,6 +138,11 @@ struct nvme_tcp_request {
size_t offset;
size_t data_sent;
enum nvme_tcp_send_state state;
+
+#ifdef CONFIG_ULP_DDP
+ bool offloaded;
+ struct ulp_ddp_io ddp;
+#endif
};
enum nvme_tcp_queue_flags {
@@ -341,6 +349,25 @@ static inline size_t nvme_tcp_pdu_last_send(struct nvme_tcp_request *req,
#ifdef CONFIG_ULP_DDP
+static bool nvme_tcp_is_ddp_offloaded(const struct nvme_tcp_request *req)
+{
+ return req->offloaded;
+}
+
+static int nvme_tcp_ddp_alloc_sgl(struct nvme_tcp_request *req, struct request *rq)
+{
+ int ret;
+
+ req->ddp.sg_table.sgl = req->ddp.first_sgl;
+ ret = sg_alloc_table_chained(&req->ddp.sg_table,
+ blk_rq_nr_phys_segments(rq),
+ req->ddp.sg_table.sgl, SG_CHUNK_SIZE);
+ if (ret)
+ return -ENOMEM;
+ req->ddp.nents = blk_rq_map_sg(rq->q, rq, req->ddp.sg_table.sgl);
+ return 0;
+}
+
static struct net_device *
nvme_tcp_get_ddp_netdev_with_limits(struct nvme_tcp_ctrl *ctrl)
{
@@ -374,10 +401,68 @@ nvme_tcp_get_ddp_netdev_with_limits(struct nvme_tcp_ctrl *ctrl)
}
static bool nvme_tcp_resync_request(struct sock *sk, u32 seq, u32 flags);
+static void nvme_tcp_ddp_teardown_done(void *ddp_ctx);
static const struct ulp_ddp_ulp_ops nvme_tcp_ddp_ulp_ops = {
.resync_request = nvme_tcp_resync_request,
+ .ddp_teardown_done = nvme_tcp_ddp_teardown_done,
};
+static void nvme_tcp_teardown_ddp(struct nvme_tcp_queue *queue,
+ struct request *rq)
+{
+ struct net_device *netdev = queue->ctrl->ddp_netdev;
+ struct nvme_tcp_request *req = blk_mq_rq_to_pdu(rq);
+
+ ulp_ddp_teardown(netdev, queue->sock->sk, &req->ddp, rq);
+ sg_free_table_chained(&req->ddp.sg_table, SG_CHUNK_SIZE);
+}
+
+static void nvme_tcp_ddp_teardown_done(void *ddp_ctx)
+{
+ struct request *rq = ddp_ctx;
+ struct nvme_tcp_request *req = blk_mq_rq_to_pdu(rq);
+
+ if (!nvme_try_complete_req(rq, req->status, req->result))
+ nvme_complete_rq(rq);
+}
+
+static void nvme_tcp_setup_ddp(struct nvme_tcp_queue *queue,
+ struct request *rq)
+{
+ struct net_device *netdev = queue->ctrl->ddp_netdev;
+ struct nvme_tcp_request *req = blk_mq_rq_to_pdu(rq);
+ int ret;
+
+ if (rq_data_dir(rq) != READ ||
+ queue->ctrl->ddp_threshold > blk_rq_payload_bytes(rq))
+ return;
+
+ /*
+ * DDP offload is best-effort, errors are ignored.
+ */
+
+ req->ddp.command_id = nvme_cid(rq);
+ ret = nvme_tcp_ddp_alloc_sgl(req, rq);
+ if (ret)
+ goto err;
+
+ ret = ulp_ddp_setup(netdev, queue->sock->sk, &req->ddp);
+ if (ret) {
+ sg_free_table_chained(&req->ddp.sg_table, SG_CHUNK_SIZE);
+ goto err;
+ }
+
+ /* if successful, sg table is freed in nvme_tcp_teardown_ddp() */
+ req->offloaded = true;
+
+ return;
+err:
+ WARN_ONCE(ret, "ddp setup failed (queue 0x%x, cid 0x%x, ret=%d)",
+ nvme_tcp_queue_id(queue),
+ nvme_cid(rq),
+ ret);
+}
+
static int nvme_tcp_offload_socket(struct nvme_tcp_queue *queue)
{
struct ulp_ddp_config config = {.type = ULP_DDP_NVME};
@@ -480,6 +565,11 @@ static bool nvme_tcp_resync_request(struct sock *sk, u32 seq, u32 flags)
#else
+static bool nvme_tcp_is_ddp_offloaded(const struct nvme_tcp_request *req)
+{
+ return false;
+}
+
static struct net_device *
nvme_tcp_get_ddp_netdev_with_limits(struct nvme_tcp_ctrl *ctrl)
{
@@ -497,6 +587,14 @@ static int nvme_tcp_offload_socket(struct nvme_tcp_queue *queue)
static void nvme_tcp_unoffload_socket(struct nvme_tcp_queue *queue)
{}
+static void nvme_tcp_setup_ddp(struct nvme_tcp_queue *queue,
+ struct request *rq)
+{}
+
+static void nvme_tcp_teardown_ddp(struct nvme_tcp_queue *queue,
+ struct request *rq)
+{}
+
static void nvme_tcp_resync_response(struct nvme_tcp_queue *queue,
struct sk_buff *skb, unsigned int offset)
{}
@@ -772,6 +870,26 @@ static void nvme_tcp_error_recovery(struct nvme_ctrl *ctrl)
queue_work(nvme_reset_wq, &to_tcp_ctrl(ctrl)->err_work);
}
+static void nvme_tcp_complete_request(struct request *rq,
+ __le16 status,
+ union nvme_result result,
+ __u16 command_id)
+{
+ struct nvme_tcp_request *req = blk_mq_rq_to_pdu(rq);
+
+ req->status = status;
+ req->result = result;
+
+ if (nvme_tcp_is_ddp_offloaded(req)) {
+ /* complete when teardown is confirmed to be done */
+ nvme_tcp_teardown_ddp(req->queue, rq);
+ return;
+ }
+
+ if (!nvme_try_complete_req(rq, status, result))
+ nvme_complete_rq(rq);
+}
+
static int nvme_tcp_process_nvme_cqe(struct nvme_tcp_queue *queue,
struct nvme_completion *cqe)
{
@@ -791,10 +909,9 @@ static int nvme_tcp_process_nvme_cqe(struct nvme_tcp_queue *queue,
if (req->status == cpu_to_le16(NVME_SC_SUCCESS))
req->status = cqe->status;
- if (!nvme_try_complete_req(rq, req->status, cqe->result))
- nvme_complete_rq(rq);
+ nvme_tcp_complete_request(rq, req->status, cqe->result,
+ cqe->command_id);
queue->nr_cqe++;
-
return 0;
}
@@ -992,10 +1109,13 @@ static int nvme_tcp_recv_pdu(struct nvme_tcp_queue *queue, struct sk_buff *skb,
static inline void nvme_tcp_end_request(struct request *rq, u16 status)
{
+ struct nvme_tcp_request *req = blk_mq_rq_to_pdu(rq);
+ struct nvme_tcp_queue *queue = req->queue;
+ struct nvme_tcp_data_pdu *pdu = (void *)queue->pdu;
union nvme_result res = {};
- if (!nvme_try_complete_req(rq, cpu_to_le16(status << 1), res))
- nvme_complete_rq(rq);
+ nvme_tcp_complete_request(rq, cpu_to_le16(status << 1), res,
+ pdu->command_id);
}
static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb,
@@ -2739,6 +2859,9 @@ static blk_status_t nvme_tcp_setup_cmd_pdu(struct nvme_ns *ns,
if (ret)
return ret;
+#ifdef CONFIG_ULP_DDP
+ req->offloaded = false;
+#endif
req->state = NVME_TCP_SEND_CMD_PDU;
req->status = cpu_to_le16(NVME_SC_SUCCESS);
req->offset = 0;
@@ -2777,6 +2900,9 @@ static blk_status_t nvme_tcp_setup_cmd_pdu(struct nvme_ns *ns,
return ret;
}
+ if (test_bit(NVME_TCP_Q_OFF_DDP, &queue->flags))
+ nvme_tcp_setup_ddp(queue, rq);
+
return 0;
}
--
2.34.1
^ permalink raw reply related [flat|nested] 44+ messages in thread
* [PATCH v24 07/20] nvme-tcp: RX DDGST offload
2024-04-04 12:36 [PATCH v24 00/20] nvme-tcp receive offloads Aurelien Aptel
` (5 preceding siblings ...)
2024-04-04 12:37 ` [PATCH v24 06/20] nvme-tcp: Add DDP data-path Aurelien Aptel
@ 2024-04-04 12:37 ` Aurelien Aptel
2024-04-04 12:37 ` [PATCH v24 08/20] nvme-tcp: Deal with netdevice DOWN events Aurelien Aptel
` (13 subsequent siblings)
20 siblings, 0 replies; 44+ messages in thread
From: Aurelien Aptel @ 2024-04-04 12:37 UTC (permalink / raw)
To: linux-nvme, netdev, sagi, hch, kbusch, axboe, chaitanyak, davem, kuba
Cc: Yoray Zack, aaptel, aurelien.aptel, smalin, malin1024, ogerlitz,
borisp, galshalom, mgurtovoy
From: Yoray Zack <yorayz@nvidia.com>
Enable rx side of DDGST offload when supported.
At the end of the capsule, check if all the skb bits are on, and if not
recalculate the DDGST in SW and check it.
Signed-off-by: Yoray Zack <yorayz@nvidia.com>
Signed-off-by: Boris Pismenny <borisp@nvidia.com>
Signed-off-by: Ben Ben-Ishay <benishay@nvidia.com>
Signed-off-by: Or Gerlitz <ogerlitz@nvidia.com>
Signed-off-by: Shai Malin <smalin@nvidia.com>
Signed-off-by: Aurelien Aptel <aaptel@nvidia.com>
Reviewed-by: Chaitanya Kulkarni <kch@nvidia.com>
Reviewed-by: Max Gurtovoy <mgurtovoy@nvidia.com>
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
---
drivers/nvme/host/tcp.c | 96 ++++++++++++++++++++++++++++++++++++++---
1 file changed, 91 insertions(+), 5 deletions(-)
diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index 345c7885c476..c9b307137569 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -150,6 +150,7 @@ enum nvme_tcp_queue_flags {
NVME_TCP_Q_LIVE = 1,
NVME_TCP_Q_POLLING = 2,
NVME_TCP_Q_OFF_DDP = 3,
+ NVME_TCP_Q_OFF_DDGST_RX = 4,
};
enum nvme_tcp_recv_state {
@@ -187,6 +188,7 @@ struct nvme_tcp_queue {
* is pending (ULP_DDP_RESYNC_PENDING).
*/
atomic64_t resync_tcp_seq;
+ bool ddp_ddgst_valid;
#endif
/* send state */
@@ -400,6 +402,46 @@ nvme_tcp_get_ddp_netdev_with_limits(struct nvme_tcp_ctrl *ctrl)
return NULL;
}
+static inline bool nvme_tcp_ddp_ddgst_ok(struct nvme_tcp_queue *queue)
+{
+ return queue->ddp_ddgst_valid;
+}
+
+static inline void nvme_tcp_ddp_ddgst_update(struct nvme_tcp_queue *queue,
+ struct sk_buff *skb)
+{
+ if (queue->ddp_ddgst_valid)
+ queue->ddp_ddgst_valid = skb_is_ulp_crc(skb);
+}
+
+static void nvme_tcp_ddp_ddgst_recalc(struct ahash_request *hash,
+ struct request *rq,
+ __le32 *ddgst)
+{
+ struct nvme_tcp_request *req;
+
+ req = blk_mq_rq_to_pdu(rq);
+ if (!req->offloaded) {
+ /* if we have DDGST_RX offload but DDP was skipped
+ * because it's under the min IO threshold then the
+ * request won't have an SGL, so we need to make it
+ * here
+ */
+ if (nvme_tcp_ddp_alloc_sgl(req, rq))
+ return;
+ }
+
+ ahash_request_set_crypt(hash, req->ddp.sg_table.sgl, (u8 *)ddgst,
+ req->data_len);
+ crypto_ahash_digest(hash);
+ if (!req->offloaded) {
+ /* without DDP, ddp_teardown() won't be called, so
+ * free the table here
+ */
+ sg_free_table_chained(&req->ddp.sg_table, SG_CHUNK_SIZE);
+ }
+}
+
static bool nvme_tcp_resync_request(struct sock *sk, u32 seq, u32 flags);
static void nvme_tcp_ddp_teardown_done(void *ddp_ctx);
static const struct ulp_ddp_ulp_ops nvme_tcp_ddp_ulp_ops = {
@@ -484,6 +526,10 @@ static int nvme_tcp_offload_socket(struct nvme_tcp_queue *queue)
return ret;
set_bit(NVME_TCP_Q_OFF_DDP, &queue->flags);
+ if (queue->data_digest &&
+ ulp_ddp_is_cap_active(queue->ctrl->ddp_netdev,
+ ULP_DDP_CAP_NVME_TCP_DDGST_RX))
+ set_bit(NVME_TCP_Q_OFF_DDGST_RX, &queue->flags);
return 0;
}
@@ -491,6 +537,7 @@ static int nvme_tcp_offload_socket(struct nvme_tcp_queue *queue)
static void nvme_tcp_unoffload_socket(struct nvme_tcp_queue *queue)
{
clear_bit(NVME_TCP_Q_OFF_DDP, &queue->flags);
+ clear_bit(NVME_TCP_Q_OFF_DDGST_RX, &queue->flags);
ulp_ddp_sk_del(queue->ctrl->ddp_netdev, queue->sock->sk);
}
@@ -599,6 +646,20 @@ static void nvme_tcp_resync_response(struct nvme_tcp_queue *queue,
struct sk_buff *skb, unsigned int offset)
{}
+static inline bool nvme_tcp_ddp_ddgst_ok(struct nvme_tcp_queue *queue)
+{
+ return false;
+}
+
+static inline void nvme_tcp_ddp_ddgst_update(struct nvme_tcp_queue *queue,
+ struct sk_buff *skb)
+{}
+
+static void nvme_tcp_ddp_ddgst_recalc(struct ahash_request *hash,
+ struct request *rq,
+ __le32 *ddgst)
+{}
+
#endif
static void nvme_tcp_init_iter(struct nvme_tcp_request *req,
@@ -859,6 +920,9 @@ static void nvme_tcp_init_recv_ctx(struct nvme_tcp_queue *queue)
queue->pdu_offset = 0;
queue->data_remaining = -1;
queue->ddgst_remaining = 0;
+#ifdef CONFIG_ULP_DDP
+ queue->ddp_ddgst_valid = true;
+#endif
}
static void nvme_tcp_error_recovery(struct nvme_ctrl *ctrl)
@@ -1126,6 +1190,9 @@ static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb,
nvme_cid_to_rq(nvme_tcp_tagset(queue), pdu->command_id);
struct nvme_tcp_request *req = blk_mq_rq_to_pdu(rq);
+ if (test_bit(NVME_TCP_Q_OFF_DDGST_RX, &queue->flags))
+ nvme_tcp_ddp_ddgst_update(queue, skb);
+
while (true) {
int recv_len, ret;
@@ -1154,7 +1221,8 @@ static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb,
recv_len = min_t(size_t, recv_len,
iov_iter_count(&req->iter));
- if (queue->data_digest)
+ if (queue->data_digest &&
+ !test_bit(NVME_TCP_Q_OFF_DDGST_RX, &queue->flags))
ret = skb_copy_and_hash_datagram_iter(skb, *offset,
&req->iter, recv_len, queue->rcv_hash);
else
@@ -1196,8 +1264,11 @@ static int nvme_tcp_recv_ddgst(struct nvme_tcp_queue *queue,
char *ddgst = (char *)&queue->recv_ddgst;
size_t recv_len = min_t(size_t, *len, queue->ddgst_remaining);
off_t off = NVME_TCP_DIGEST_LENGTH - queue->ddgst_remaining;
+ struct request *rq;
int ret;
+ if (test_bit(NVME_TCP_Q_OFF_DDGST_RX, &queue->flags))
+ nvme_tcp_ddp_ddgst_update(queue, skb);
ret = skb_copy_bits(skb, *offset, &ddgst[off], recv_len);
if (unlikely(ret))
return ret;
@@ -1208,9 +1279,25 @@ static int nvme_tcp_recv_ddgst(struct nvme_tcp_queue *queue,
if (queue->ddgst_remaining)
return 0;
+ rq = nvme_cid_to_rq(nvme_tcp_tagset(queue),
+ pdu->command_id);
+
+ if (test_bit(NVME_TCP_Q_OFF_DDGST_RX, &queue->flags)) {
+ /*
+ * If HW successfully offloaded the digest
+ * verification, we can skip it
+ */
+ if (nvme_tcp_ddp_ddgst_ok(queue))
+ goto ddgst_valid;
+ /*
+ * Otherwise we have to recalculate and verify the
+ * digest with the software-fallback
+ */
+ nvme_tcp_ddp_ddgst_recalc(queue->rcv_hash, rq,
+ &queue->exp_ddgst);
+ }
+
if (queue->recv_ddgst != queue->exp_ddgst) {
- struct request *rq = nvme_cid_to_rq(nvme_tcp_tagset(queue),
- pdu->command_id);
struct nvme_tcp_request *req = blk_mq_rq_to_pdu(rq);
req->status = cpu_to_le16(NVME_SC_DATA_XFER_ERROR);
@@ -1221,9 +1308,8 @@ static int nvme_tcp_recv_ddgst(struct nvme_tcp_queue *queue,
le32_to_cpu(queue->exp_ddgst));
}
+ddgst_valid:
if (pdu->hdr.flags & NVME_TCP_F_DATA_SUCCESS) {
- struct request *rq = nvme_cid_to_rq(nvme_tcp_tagset(queue),
- pdu->command_id);
struct nvme_tcp_request *req = blk_mq_rq_to_pdu(rq);
nvme_tcp_end_request(rq, le16_to_cpu(req->status));
--
2.34.1
^ permalink raw reply related [flat|nested] 44+ messages in thread
* [PATCH v24 08/20] nvme-tcp: Deal with netdevice DOWN events
2024-04-04 12:36 [PATCH v24 00/20] nvme-tcp receive offloads Aurelien Aptel
` (6 preceding siblings ...)
2024-04-04 12:37 ` [PATCH v24 07/20] nvme-tcp: RX DDGST offload Aurelien Aptel
@ 2024-04-04 12:37 ` Aurelien Aptel
2024-04-04 12:37 ` [PATCH v24 09/20] Documentation: add ULP DDP offload documentation Aurelien Aptel
` (12 subsequent siblings)
20 siblings, 0 replies; 44+ messages in thread
From: Aurelien Aptel @ 2024-04-04 12:37 UTC (permalink / raw)
To: linux-nvme, netdev, sagi, hch, kbusch, axboe, chaitanyak, davem, kuba
Cc: Or Gerlitz, aaptel, aurelien.aptel, smalin, malin1024, yorayz,
borisp, galshalom, mgurtovoy
From: Or Gerlitz <ogerlitz@nvidia.com>
For ddp setup/teardown and resync, the offloading logic
uses HW resources at the NIC driver such as SQ and CQ.
These resources are destroyed when the netdevice does down
and hence we must stop using them before the NIC driver
destroys them.
Use netdevice notifier for that matter -- offloaded connections
are stopped before the stack continues to call the NIC driver
close ndo.
We use the existing recovery flow which has the advantage
of resuming the offload once the connection is re-set.
This also buys us proper handling for the UNREGISTER event
b/c our offloading starts in the UP state, and down is always
there between up to unregister.
Signed-off-by: Or Gerlitz <ogerlitz@nvidia.com>
Signed-off-by: Boris Pismenny <borisp@nvidia.com>
Signed-off-by: Ben Ben-Ishay <benishay@nvidia.com>
Signed-off-by: Yoray Zack <yorayz@nvidia.com>
Signed-off-by: Shai Malin <smalin@nvidia.com>
Signed-off-by: Aurelien Aptel <aaptel@nvidia.com>
Reviewed-by: Chaitanya Kulkarni <kch@nvidia.com>
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
---
drivers/nvme/host/tcp.c | 40 ++++++++++++++++++++++++++++++++++++++++
1 file changed, 40 insertions(+)
diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index c9b307137569..723c292dea07 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -242,6 +242,7 @@ struct nvme_tcp_ctrl {
static LIST_HEAD(nvme_tcp_ctrl_list);
static DEFINE_MUTEX(nvme_tcp_ctrl_mutex);
+static struct notifier_block nvme_tcp_netdevice_nb;
static struct workqueue_struct *nvme_tcp_wq;
static const struct blk_mq_ops nvme_tcp_mq_ops;
static const struct blk_mq_ops nvme_tcp_admin_mq_ops;
@@ -3231,6 +3232,32 @@ static struct nvme_ctrl *nvme_tcp_create_ctrl(struct device *dev,
return ERR_PTR(ret);
}
+static int nvme_tcp_netdev_event(struct notifier_block *this,
+ unsigned long event, void *ptr)
+{
+#ifdef CONFIG_ULP_DDP
+ struct net_device *ndev = netdev_notifier_info_to_dev(ptr);
+ struct nvme_tcp_ctrl *ctrl;
+
+ if (event != NETDEV_GOING_DOWN)
+ return NOTIFY_DONE;
+
+ mutex_lock(&nvme_tcp_ctrl_mutex);
+ list_for_each_entry(ctrl, &nvme_tcp_ctrl_list, list) {
+ if (ndev == ctrl->ddp_netdev)
+ nvme_tcp_error_recovery(&ctrl->ctrl);
+ }
+ mutex_unlock(&nvme_tcp_ctrl_mutex);
+ flush_workqueue(nvme_reset_wq);
+ /*
+ * The associated controllers teardown has completed,
+ * ddp contexts were also torn down so we should be
+ * safe to continue...
+ */
+#endif
+ return NOTIFY_DONE;
+}
+
static struct nvmf_transport_ops nvme_tcp_transport = {
.name = "tcp",
.module = THIS_MODULE,
@@ -3247,6 +3274,7 @@ static struct nvmf_transport_ops nvme_tcp_transport = {
static int __init nvme_tcp_init_module(void)
{
unsigned int wq_flags = WQ_MEM_RECLAIM | WQ_HIGHPRI | WQ_SYSFS;
+ int ret;
BUILD_BUG_ON(sizeof(struct nvme_tcp_hdr) != 8);
BUILD_BUG_ON(sizeof(struct nvme_tcp_cmd_pdu) != 72);
@@ -3264,8 +3292,19 @@ static int __init nvme_tcp_init_module(void)
if (!nvme_tcp_wq)
return -ENOMEM;
+ nvme_tcp_netdevice_nb.notifier_call = nvme_tcp_netdev_event;
+ ret = register_netdevice_notifier(&nvme_tcp_netdevice_nb);
+ if (ret) {
+ pr_err("failed to register netdev notifier\n");
+ goto out_free_workqueue;
+ }
+
nvmf_register_transport(&nvme_tcp_transport);
return 0;
+
+out_free_workqueue:
+ destroy_workqueue(nvme_tcp_wq);
+ return ret;
}
static void __exit nvme_tcp_cleanup_module(void)
@@ -3273,6 +3312,7 @@ static void __exit nvme_tcp_cleanup_module(void)
struct nvme_tcp_ctrl *ctrl;
nvmf_unregister_transport(&nvme_tcp_transport);
+ unregister_netdevice_notifier(&nvme_tcp_netdevice_nb);
mutex_lock(&nvme_tcp_ctrl_mutex);
list_for_each_entry(ctrl, &nvme_tcp_ctrl_list, list)
--
2.34.1
^ permalink raw reply related [flat|nested] 44+ messages in thread
* [PATCH v24 09/20] Documentation: add ULP DDP offload documentation
2024-04-04 12:36 [PATCH v24 00/20] nvme-tcp receive offloads Aurelien Aptel
` (7 preceding siblings ...)
2024-04-04 12:37 ` [PATCH v24 08/20] nvme-tcp: Deal with netdevice DOWN events Aurelien Aptel
@ 2024-04-04 12:37 ` Aurelien Aptel
2024-04-09 8:49 ` Bagas Sanjaya
2024-04-04 12:37 ` [PATCH v24 10/20] net/mlx5e: Rename from tls to transport static params Aurelien Aptel
` (11 subsequent siblings)
20 siblings, 1 reply; 44+ messages in thread
From: Aurelien Aptel @ 2024-04-04 12:37 UTC (permalink / raw)
To: linux-nvme, netdev, sagi, hch, kbusch, axboe, chaitanyak, davem, kuba
Cc: Yoray Zack, aaptel, aurelien.aptel, smalin, malin1024, ogerlitz,
borisp, galshalom, mgurtovoy, linux-doc, edumazet, pabeni,
corbet
From: Yoray Zack <yorayz@nvidia.com>
Document the new ULP DDP API and add it under "networking".
Use NVMe-TCP implementation as an example.
Signed-off-by: Boris Pismenny <borisp@nvidia.com>
Signed-off-by: Ben Ben-Ishay <benishay@nvidia.com>
Signed-off-by: Or Gerlitz <ogerlitz@nvidia.com>
Signed-off-by: Yoray Zack <yorayz@nvidia.com>
Signed-off-by: Shai Malin <smalin@nvidia.com>
Signed-off-by: Aurelien Aptel <aaptel@nvidia.com>
---
Documentation/networking/index.rst | 1 +
Documentation/networking/ulp-ddp-offload.rst | 372 +++++++++++++++++++
2 files changed, 373 insertions(+)
create mode 100644 Documentation/networking/ulp-ddp-offload.rst
diff --git a/Documentation/networking/index.rst b/Documentation/networking/index.rst
index 473d72c36d61..77d69aa68f4c 100644
--- a/Documentation/networking/index.rst
+++ b/Documentation/networking/index.rst
@@ -111,6 +111,7 @@ Contents:
tc-queue-filters
tcp_ao
tcp-thin
+ ulp-ddp-offload
team
timestamping
tipc
diff --git a/Documentation/networking/ulp-ddp-offload.rst b/Documentation/networking/ulp-ddp-offload.rst
new file mode 100644
index 000000000000..4133e5094ff5
--- /dev/null
+++ b/Documentation/networking/ulp-ddp-offload.rst
@@ -0,0 +1,372 @@
+.. SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+
+=================================
+ULP direct data placement offload
+=================================
+
+Overview
+========
+
+The Linux kernel ULP direct data placement (DDP) offload infrastructure
+provides tagged request-response protocols, such as NVMe-TCP, the ability to
+place response data directly in pre-registered buffers according to header
+tags. DDP is particularly useful for data-intensive pipelined protocols whose
+responses may be reordered.
+
+For example, in NVMe-TCP numerous read requests are sent together and each
+request is tagged using the PDU header CID field. Receiving servers process
+requests as fast as possible and sometimes responses for smaller requests
+bypasses responses to larger requests, e.g., 4KB reads bypass 1GB reads.
+Thereafter, clients correlate responses to requests using PDU header CID tags.
+The processing of each response requires copying data from SKBs to read
+request destination buffers; The offload avoids this copy. The offload is
+oblivious to destination buffers which can reside either in userspace
+(O_DIRECT) or in kernel pagecache.
+
+Request TCP byte-stream:
+
+.. parsed-literal::
+
+ +---------------+-------+---------------+-------+---------------+-------+
+ | PDU hdr CID=1 | Req 1 | PDU hdr CID=2 | Req 2 | PDU hdr CID=3 | Req 3 |
+ +---------------+-------+---------------+-------+---------------+-------+
+
+Response TCP byte-stream:
+
+.. parsed-literal::
+
+ +---------------+--------+---------------+--------+---------------+--------+
+ | PDU hdr CID=2 | Resp 2 | PDU hdr CID=3 | Resp 3 | PDU hdr CID=1 | Resp 1 |
+ +---------------+--------+---------------+--------+---------------+--------+
+
+The driver builds SKB page fragments that point to destination buffers.
+Consequently, SKBs represent the original data on the wire, which enables
+*transparent* inter-operation with the network stack. To avoid copies between
+SKBs and destination buffers, the layer-5 protocol (L5P) will check
+``if (src == dst)`` for SKB page fragments, success indicates that data is
+already placed there by NIC hardware and copy should be skipped.
+
+In addition, L5P might have DDGST which ensures data integrity over
+the network. If not offloaded, ULP DDP might not be efficient as L5P
+will need to go over the data and calculate it by itself, cancelling
+out the benefits of the DDP copy skip. ULP DDP has support for Rx/Tx
+DDGST offload. On the received side the NIC will verify DDGST for
+received PDUs and update SKB->ulp_ddp and SKB->ulp_crc bits. If all the SKBs
+making up a L5P PDU have crc on, L5P will skip on calculating and
+verifying the DDGST for the corresponding PDU. On the Tx side, the NIC
+will be responsible for calculating and filling the DDGST fields in
+the sent PDUs.
+
+Offloading does require NIC hardware to track L5P protocol framing, similarly
+to RX TLS offload (see Documentation/networking/tls-offload.rst). NIC hardware
+will parse PDU headers, extract fields such as operation type, length, tag
+identifier, etc. and only offload segments that correspond to tags registered
+with the NIC, see the :ref:`buf_reg` section.
+
+Device configuration
+====================
+
+During driver initialization the driver sets the ULP DDP operations
+for the :c:type:`struct net_device <net_device>` via
+`netdev->netdev_ops->ulp_ddp_ops`.
+
+The :c:member:`get_caps` operation returns the ULP DDP capabilities
+enabled and/or supported by the device to the caller. The current list
+of capabilities is represented as a bitset:
+
+.. code-block:: c
+
+ enum ulp_ddp_cap {
+ ULP_DDP_CAP_NVME_TCP,
+ ULP_DDP_CAP_NVME_TCP_DDGST,
+ };
+
+The enablement of capabilities can be controlled via the
+:c:member:`set_caps` operation. This operation is exposed to userspace
+via netlink. See Documentation/netlink/specs/ulp_ddp.yaml for more
+details.
+
+Later, after the L5P completes its handshake, the L5P queries the
+driver for its runtime limitations via the :c:member:`limits` operation:
+
+.. code-block:: c
+
+ int (*limits)(struct net_device *netdev,
+ struct ulp_ddp_limits *lim);
+
+
+All L5P share a common set of limits and parameters (:c:type:`struct ulp_ddp_limits <ulp_ddp_limits>`):
+
+.. code-block:: c
+
+ /**
+ * struct ulp_ddp_limits - Generic ulp ddp limits: tcp ddp
+ * protocol limits.
+ * Add new instances of ulp_ddp_limits in the union below (nvme-tcp, etc.).
+ *
+ * @type: type of this limits struct
+ * @max_ddp_sgl_len: maximum sgl size supported (zero means no limit)
+ * @io_threshold: minimum payload size required to offload
+ * @tls: support for ULP over TLS
+ * @nvmeotcp: NVMe-TCP specific limits
+ */
+ struct ulp_ddp_limits {
+ enum ulp_ddp_type type;
+ int max_ddp_sgl_len;
+ int io_threshold;
+ bool tls:1;
+ union {
+ /* ... protocol-specific limits ... */
+ struct nvme_tcp_ddp_limits nvmeotcp;
+ };
+ };
+
+But each L5P can also add protocol-specific limits e.g.:
+
+.. code-block:: c
+
+ /**
+ * struct nvme_tcp_ddp_limits - nvme tcp driver limitations
+ *
+ * @full_ccid_range: true if the driver supports the full CID range
+ */
+ struct nvme_tcp_ddp_limits {
+ bool full_ccid_range;
+ };
+
+Once the L5P has made sure the device is supported the offload
+operations are installed on the socket.
+
+If offload installation fails, then the connection is handled by software as if
+offload was not attempted.
+
+To request offload for a socket `sk`, the L5P calls :c:member:`sk_add`:
+
+.. code-block:: c
+
+ int (*sk_add)(struct net_device *netdev,
+ struct sock *sk,
+ struct ulp_ddp_config *config);
+
+The function return 0 for success. In case of failure, L5P software should
+fallback to normal non-offloaded operations. The `config` parameter indicates
+the L5P type and any metadata relevant for that protocol. For example, in
+NVMe-TCP the following config is used:
+
+.. code-block:: c
+
+ /**
+ * struct nvme_tcp_ddp_config - nvme tcp ddp configuration for an IO queue
+ *
+ * @pfv: pdu version (e.g., NVME_TCP_PFV_1_0)
+ * @cpda: controller pdu data alignment (dwords, 0's based)
+ * @dgst: digest types enabled.
+ * The netdev will offload crc if L5P data digest is supported.
+ * @queue_size: number of nvme-tcp IO queue elements
+ */
+ struct nvme_tcp_ddp_config {
+ u16 pfv;
+ u8 cpda;
+ u8 dgst;
+ int queue_size;
+ };
+
+When offload is not needed anymore, e.g. when the socket is being released, the L5P
+calls :c:member:`sk_del` to release device contexts:
+
+.. code-block:: c
+
+ void (*sk_del)(struct net_device *netdev,
+ struct sock *sk);
+
+Normal operation
+================
+
+At the very least, the device maintains the following state for each connection:
+
+ * 5-tuple
+ * expected TCP sequence number
+ * mapping between tags and corresponding buffers
+ * current offset within PDU, PDU length, current PDU tag
+
+NICs should not assume any correlation between PDUs and TCP packets.
+If TCP packets arrive in-order, offload will place PDU payloads
+directly inside corresponding registered buffers. NIC offload should
+not delay packets. If offload is not possible, than the packet is
+passed as-is to software. To perform offload on incoming packets
+without buffering packets in the NIC, the NIC stores some inter-packet
+state, such as partial PDU headers.
+
+RX data-path
+------------
+
+After the device validates TCP checksums, it can perform DDP offload. The
+packet is steered to the DDP offload context according to the 5-tuple.
+Thereafter, the expected TCP sequence number is checked against the packet
+TCP sequence number. If there is a match, offload is performed: the PDU payload
+is DMA written to the corresponding destination buffer according to the PDU header
+tag. The data should be DMAed only once, and the NIC receive ring will only
+store the remaining TCP and PDU headers.
+
+We remark that a single TCP packet may have numerous PDUs embedded inside. NICs
+can choose to offload one or more of these PDUs according to various
+trade-offs. Possibly, offloading such small PDUs is of little value, and it is
+better to leave it to software.
+
+Upon receiving a DDP offloaded packet, the driver reconstructs the original SKB
+using page frags, while pointing to the destination buffers whenever possible.
+This method enables seamless integration with the network stack, which can
+inspect and modify packet fields transparently to the offload.
+
+.. _buf_reg:
+
+Destination buffer registration
+-------------------------------
+
+To register the mapping between tags and destination buffers for a socket
+`sk`, the L5P calls :c:member:`setup` of :c:type:`struct ulp_ddp_dev_ops
+<ulp_ddp_dev_ops>`:
+
+.. code-block:: c
+
+ int (*setup)(struct net_device *netdev,
+ struct sock *sk,
+ struct ulp_ddp_io *io);
+
+
+The `io` provides the buffer via scatter-gather list (`sg_table`) and
+corresponding tag (`command_id`):
+
+.. code-block:: c
+
+ /**
+ * struct ulp_ddp_io - tcp ddp configuration for an IO request.
+ *
+ * @command_id: identifier on the wire associated with these buffers
+ * @nents: number of entries in the sg_table
+ * @sg_table: describing the buffers for this IO request
+ * @first_sgl: first SGL in sg_table
+ */
+ struct ulp_ddp_io {
+ u32 command_id;
+ int nents;
+ struct sg_table sg_table;
+ struct scatterlist first_sgl[SG_CHUNK_SIZE];
+ };
+
+After the buffers have been consumed by the L5P, to release the NIC mapping of
+buffers the L5P calls :c:member:`teardown` of :c:type:`struct
+ulp_ddp_dev_ops <ulp_ddp_dev_ops>`:
+
+.. code-block:: c
+
+ void (*teardown)(struct net_device *netdev,
+ struct sock *sk,
+ struct ulp_ddp_io *io,
+ void *ddp_ctx);
+
+`teardown` receives the same `io` context and an additional opaque
+`ddp_ctx` that is used for asynchronous teardown, see the :ref:`async_release`
+section.
+
+.. _async_release:
+
+Asynchronous teardown
+---------------------
+
+To teardown the association between tags and buffers and allow tag reuse NIC HW
+is called by the NIC driver during `teardown`. This operation may be
+performed either synchronously or asynchronously. In asynchronous teardown,
+`teardown` returns immediately without unmapping NIC HW buffers. Later,
+when the unmapping completes by NIC HW, the NIC driver will call up to L5P
+using :c:member:`ddp_teardown_done` of :c:type:`struct ulp_ddp_ulp_ops <ulp_ddp_ulp_ops>`:
+
+.. code-block:: c
+
+ void (*ddp_teardown_done)(void *ddp_ctx);
+
+The `ddp_ctx` parameter passed in `ddp_teardown_done` is the same on provided
+in `teardown` and it is used to carry some context about the buffers
+and tags that are released.
+
+Resync handling
+===============
+
+RX
+--
+In presence of packet drops or network packet reordering, the device may lose
+synchronization between the TCP stream and the L5P framing, and require a
+resync with the kernel's TCP stack. When the device is out of sync, no offload
+takes place, and packets are passed as-is to software. Resync is very similar
+to TLS offload (see documentation at Documentation/networking/tls-offload.rst)
+
+If only packets with L5P data are lost or reordered, then resynchronization may
+be avoided by NIC HW that keeps tracking PDU headers. If, however, PDU headers
+are reordered, then resynchronization is necessary.
+
+To resynchronize hardware during traffic, we use a handshake between hardware
+and software. The NIC HW searches for a sequence of bytes that identifies L5P
+headers (i.e., magic pattern). For example, in NVMe-TCP, the PDU operation
+type can be used for this purpose. Using the PDU header length field, the NIC
+HW will continue to find and match magic patterns in subsequent PDU headers. If
+the pattern is missing in an expected position, then searching for the pattern
+starts anew.
+
+The NIC will not resume offload when the magic pattern is first identified.
+Instead, it will request L5P software to confirm that indeed this is a PDU
+header. To request confirmation the NIC driver calls up to L5P using
+:c:member:`resync_request` of :c:type:`struct ulp_ddp_ulp_ops <ulp_ddp_ulp_ops>`:
+
+.. code-block:: c
+
+ bool (*resync_request)(struct sock *sk, u32 seq, u32 flags);
+
+The `seq` parameter contains the TCP sequence of the last byte in the PDU header.
+The `flags` parameter contains a flag (`ULP_DDP_RESYNC_PENDING`) indicating whether
+a request is pending or not.
+L5P software will respond to this request after observing the packet containing
+TCP sequence `seq` in-order. If the PDU header is indeed there, then L5P
+software calls the NIC driver using the :c:member:`resync` function of
+the :c:type:`struct ulp_ddp_dev_ops <ulp_ddp_ops>` inside the :c:type:`struct
+net_device <net_device>` while passing the same `seq` to confirm it is a PDU
+header.
+
+.. code-block:: c
+
+ void (*resync)(struct net_device *netdev,
+ struct sock *sk, u32 seq);
+
+Statistics
+==========
+
+Per L5P protocol, the NIC driver must report statistics for the above
+netdevice operations and packets processed by offload.
+These statistics are per-device and can be retrieved from userspace
+via netlink (see Documentation/netlink/specs/ulp_ddp.yaml).
+
+For example, NVMe-TCP offload reports:
+
+ * ``rx_nvme_tcp_sk_add`` - number of NVMe-TCP Rx offload contexts created.
+ * ``rx_nvme_tcp_sk_add_fail`` - number of NVMe-TCP Rx offload context creation
+ failures.
+ * ``rx_nvme_tcp_sk_del`` - number of NVMe-TCP Rx offload contexts destroyed.
+ * ``rx_nvme_tcp_setup`` - number of DDP buffers mapped.
+ * ``rx_nvme_tcp_setup_fail`` - number of DDP buffers mapping that failed.
+ * ``rx_nvme_tcp_teardown`` - number of DDP buffers unmapped.
+ * ``rx_nvme_tcp_drop`` - number of packets dropped in the driver due to fatal
+ errors.
+ * ``rx_nvme_tcp_resync`` - number of packets with resync requests.
+ * ``rx_nvme_tcp_packets`` - number of packets that used offload.
+ * ``rx_nvme_tcp_bytes`` - number of bytes placed in DDP buffers.
+
+NIC requirements
+================
+
+NIC hardware should meet the following requirements to provide this offload:
+
+ * Offload must never buffer TCP packets.
+ * Offload must never modify TCP packet headers.
+ * Offload must never reorder TCP packets within a flow.
+ * Offload must never drop TCP packets.
+ * Offload must not depend on any TCP fields beyond the
+ 5-tuple and TCP sequence number.
--
2.34.1
^ permalink raw reply related [flat|nested] 44+ messages in thread
* [PATCH v24 10/20] net/mlx5e: Rename from tls to transport static params
2024-04-04 12:36 [PATCH v24 00/20] nvme-tcp receive offloads Aurelien Aptel
` (8 preceding siblings ...)
2024-04-04 12:37 ` [PATCH v24 09/20] Documentation: add ULP DDP offload documentation Aurelien Aptel
@ 2024-04-04 12:37 ` Aurelien Aptel
2024-04-04 12:37 ` [PATCH v24 11/20] net/mlx5e: Refactor ico sq polling to get budget Aurelien Aptel
` (10 subsequent siblings)
20 siblings, 0 replies; 44+ messages in thread
From: Aurelien Aptel @ 2024-04-04 12:37 UTC (permalink / raw)
To: linux-nvme, netdev, sagi, hch, kbusch, axboe, chaitanyak, davem, kuba
Cc: Or Gerlitz, aaptel, aurelien.aptel, smalin, malin1024, yorayz,
borisp, galshalom, mgurtovoy
From: Or Gerlitz <ogerlitz@nvidia.com>
The static params structure is used in TLS but also in other
transports we're offloading like nvmeotcp:
- Rename the relevant structures/fields
- Create common file for appropriate transports
- Apply changes in the TLS code
No functional change here.
Signed-off-by: Or Gerlitz <ogerlitz@nvidia.com>
Signed-off-by: Ben Ben-Ishay <benishay@nvidia.com>
Signed-off-by: Aurelien Aptel <aaptel@nvidia.com>
Reviewed-by: Tariq Toukan <tariqt@nvidia.com>
---
.../mlx5/core/en_accel/common_utils.h | 32 +++++++++++++++++
.../mellanox/mlx5/core/en_accel/ktls.c | 2 +-
.../mellanox/mlx5/core/en_accel/ktls_rx.c | 6 ++--
.../mellanox/mlx5/core/en_accel/ktls_tx.c | 8 ++---
.../mellanox/mlx5/core/en_accel/ktls_txrx.c | 36 ++++++++-----------
.../mellanox/mlx5/core/en_accel/ktls_utils.h | 17 ++-------
include/linux/mlx5/device.h | 8 ++---
include/linux/mlx5/mlx5_ifc.h | 8 +++--
8 files changed, 67 insertions(+), 50 deletions(-)
create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/common_utils.h
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/common_utils.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/common_utils.h
new file mode 100644
index 000000000000..efdf48125848
--- /dev/null
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/common_utils.h
@@ -0,0 +1,32 @@
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/* Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. */
+#ifndef __MLX5E_COMMON_UTILS_H__
+#define __MLX5E_COMMON_UTILS_H__
+
+#include "en.h"
+
+struct mlx5e_set_transport_static_params_wqe {
+ struct mlx5_wqe_ctrl_seg ctrl;
+ struct mlx5_wqe_umr_ctrl_seg uctrl;
+ struct mlx5_mkey_seg mkc;
+ struct mlx5_wqe_transport_static_params_seg params;
+};
+
+/* macros for transport_static_params handling */
+#define MLX5E_TRANSPORT_SET_STATIC_PARAMS_WQEBBS \
+ (DIV_ROUND_UP(sizeof(struct mlx5e_set_transport_static_params_wqe), MLX5_SEND_WQE_BB))
+
+#define MLX5E_TRANSPORT_FETCH_SET_STATIC_PARAMS_WQE(sq, pi) \
+ ((struct mlx5e_set_transport_static_params_wqe *)\
+ mlx5e_fetch_wqe(&(sq)->wq, pi, sizeof(struct mlx5e_set_transport_static_params_wqe)))
+
+#define MLX5E_TRANSPORT_STATIC_PARAMS_WQE_SZ \
+ (sizeof(struct mlx5e_set_transport_static_params_wqe))
+
+#define MLX5E_TRANSPORT_STATIC_PARAMS_DS_CNT \
+ (DIV_ROUND_UP(MLX5E_TRANSPORT_STATIC_PARAMS_WQE_SZ, MLX5_SEND_WQE_DS))
+
+#define MLX5E_TRANSPORT_STATIC_PARAMS_OCTWORD_SIZE \
+ (MLX5_ST_SZ_BYTES(transport_static_params) / MLX5_SEND_WQE_DS)
+
+#endif /* __MLX5E_COMMON_UTILS_H__ */
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c
index e3e57c849436..ab7468bddf42 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c
@@ -100,7 +100,7 @@ bool mlx5e_is_ktls_rx(struct mlx5_core_dev *mdev)
return false;
/* Check the possibility to post the required ICOSQ WQEs. */
- if (WARN_ON_ONCE(max_sq_wqebbs < MLX5E_TLS_SET_STATIC_PARAMS_WQEBBS))
+ if (WARN_ON_ONCE(max_sq_wqebbs < MLX5E_TRANSPORT_SET_STATIC_PARAMS_WQEBBS))
return false;
if (WARN_ON_ONCE(max_sq_wqebbs < MLX5E_TLS_SET_PROGRESS_PARAMS_WQEBBS))
return false;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c
index 65ccb33edafb..3c501466634c 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c
@@ -136,16 +136,16 @@ static struct mlx5_wqe_ctrl_seg *
post_static_params(struct mlx5e_icosq *sq,
struct mlx5e_ktls_offload_context_rx *priv_rx)
{
- struct mlx5e_set_tls_static_params_wqe *wqe;
+ struct mlx5e_set_transport_static_params_wqe *wqe;
struct mlx5e_icosq_wqe_info wi;
u16 pi, num_wqebbs;
- num_wqebbs = MLX5E_TLS_SET_STATIC_PARAMS_WQEBBS;
+ num_wqebbs = MLX5E_TRANSPORT_SET_STATIC_PARAMS_WQEBBS;
if (unlikely(!mlx5e_icosq_can_post_wqe(sq, num_wqebbs)))
return ERR_PTR(-ENOSPC);
pi = mlx5e_icosq_get_next_pi(sq, num_wqebbs);
- wqe = MLX5E_TLS_FETCH_SET_STATIC_PARAMS_WQE(sq, pi);
+ wqe = MLX5E_TRANSPORT_FETCH_SET_STATIC_PARAMS_WQE(sq, pi);
mlx5e_ktls_build_static_params(wqe, sq->pc, sq->sqn, &priv_rx->crypto_info,
mlx5e_tir_get_tirn(&priv_rx->tir),
mlx5_crypto_dek_get_id(priv_rx->dek),
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c
index d61be26a4df1..0691995470e2 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c
@@ -33,7 +33,7 @@ u16 mlx5e_ktls_get_stop_room(struct mlx5_core_dev *mdev, struct mlx5e_params *pa
num_dumps = mlx5e_ktls_dumps_num_wqes(params, MAX_SKB_FRAGS, TLS_MAX_PAYLOAD_SIZE);
- stop_room += mlx5e_stop_room_for_wqe(mdev, MLX5E_TLS_SET_STATIC_PARAMS_WQEBBS);
+ stop_room += mlx5e_stop_room_for_wqe(mdev, MLX5E_TRANSPORT_SET_STATIC_PARAMS_WQEBBS);
stop_room += mlx5e_stop_room_for_wqe(mdev, MLX5E_TLS_SET_PROGRESS_PARAMS_WQEBBS);
stop_room += num_dumps * mlx5e_stop_room_for_wqe(mdev, MLX5E_KTLS_DUMP_WQEBBS);
stop_room += 1; /* fence nop */
@@ -550,12 +550,12 @@ post_static_params(struct mlx5e_txqsq *sq,
struct mlx5e_ktls_offload_context_tx *priv_tx,
bool fence)
{
- struct mlx5e_set_tls_static_params_wqe *wqe;
+ struct mlx5e_set_transport_static_params_wqe *wqe;
u16 pi, num_wqebbs;
- num_wqebbs = MLX5E_TLS_SET_STATIC_PARAMS_WQEBBS;
+ num_wqebbs = MLX5E_TRANSPORT_SET_STATIC_PARAMS_WQEBBS;
pi = mlx5e_txqsq_get_next_pi(sq, num_wqebbs);
- wqe = MLX5E_TLS_FETCH_SET_STATIC_PARAMS_WQE(sq, pi);
+ wqe = MLX5E_TRANSPORT_FETCH_SET_STATIC_PARAMS_WQE(sq, pi);
mlx5e_ktls_build_static_params(wqe, sq->pc, sq->sqn, &priv_tx->crypto_info,
priv_tx->tisn,
mlx5_crypto_dek_get_id(priv_tx->dek),
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c
index 570a912dd6fa..8abea6fe6cd9 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c
@@ -8,10 +8,6 @@ enum {
MLX5E_STATIC_PARAMS_CONTEXT_TLS_1_2 = 0x2,
};
-enum {
- MLX5E_ENCRYPTION_STANDARD_TLS = 0x1,
-};
-
#define EXTRACT_INFO_FIELDS do { \
salt = info->salt; \
rec_seq = info->rec_seq; \
@@ -20,7 +16,7 @@ enum {
} while (0)
static void
-fill_static_params(struct mlx5_wqe_tls_static_params_seg *params,
+fill_static_params(struct mlx5_wqe_transport_static_params_seg *params,
union mlx5e_crypto_info *crypto_info,
u32 key_id, u32 resync_tcp_sn)
{
@@ -53,25 +49,25 @@ fill_static_params(struct mlx5_wqe_tls_static_params_seg *params,
return;
}
- gcm_iv = MLX5_ADDR_OF(tls_static_params, ctx, gcm_iv);
- initial_rn = MLX5_ADDR_OF(tls_static_params, ctx, initial_record_number);
+ gcm_iv = MLX5_ADDR_OF(transport_static_params, ctx, gcm_iv);
+ initial_rn = MLX5_ADDR_OF(transport_static_params, ctx, initial_record_number);
memcpy(gcm_iv, salt, salt_sz);
memcpy(initial_rn, rec_seq, rec_seq_sz);
tls_version = MLX5E_STATIC_PARAMS_CONTEXT_TLS_1_2;
- MLX5_SET(tls_static_params, ctx, tls_version, tls_version);
- MLX5_SET(tls_static_params, ctx, const_1, 1);
- MLX5_SET(tls_static_params, ctx, const_2, 2);
- MLX5_SET(tls_static_params, ctx, encryption_standard,
- MLX5E_ENCRYPTION_STANDARD_TLS);
- MLX5_SET(tls_static_params, ctx, resync_tcp_sn, resync_tcp_sn);
- MLX5_SET(tls_static_params, ctx, dek_index, key_id);
+ MLX5_SET(transport_static_params, ctx, tls_version, tls_version);
+ MLX5_SET(transport_static_params, ctx, const_1, 1);
+ MLX5_SET(transport_static_params, ctx, const_2, 2);
+ MLX5_SET(transport_static_params, ctx, acc_type,
+ MLX5_TRANSPORT_STATIC_PARAMS_ACC_TYPE_TLS);
+ MLX5_SET(transport_static_params, ctx, resync_tcp_sn, resync_tcp_sn);
+ MLX5_SET(transport_static_params, ctx, dek_index, key_id);
}
void
-mlx5e_ktls_build_static_params(struct mlx5e_set_tls_static_params_wqe *wqe,
+mlx5e_ktls_build_static_params(struct mlx5e_set_transport_static_params_wqe *wqe,
u16 pc, u32 sqn,
union mlx5e_crypto_info *crypto_info,
u32 tis_tir_num, u32 key_id, u32 resync_tcp_sn,
@@ -80,19 +76,17 @@ mlx5e_ktls_build_static_params(struct mlx5e_set_tls_static_params_wqe *wqe,
struct mlx5_wqe_umr_ctrl_seg *ucseg = &wqe->uctrl;
struct mlx5_wqe_ctrl_seg *cseg = &wqe->ctrl;
u8 opmod = direction == TLS_OFFLOAD_CTX_DIR_TX ?
- MLX5_OPC_MOD_TLS_TIS_STATIC_PARAMS :
- MLX5_OPC_MOD_TLS_TIR_STATIC_PARAMS;
-
-#define STATIC_PARAMS_DS_CNT DIV_ROUND_UP(sizeof(*wqe), MLX5_SEND_WQE_DS)
+ MLX5_OPC_MOD_TRANSPORT_TIS_STATIC_PARAMS :
+ MLX5_OPC_MOD_TRANSPORT_TIR_STATIC_PARAMS;
cseg->opmod_idx_opcode = cpu_to_be32((pc << 8) | MLX5_OPCODE_UMR | (opmod << 24));
cseg->qpn_ds = cpu_to_be32((sqn << MLX5_WQE_CTRL_QPN_SHIFT) |
- STATIC_PARAMS_DS_CNT);
+ MLX5E_TRANSPORT_STATIC_PARAMS_DS_CNT);
cseg->fm_ce_se = fence ? MLX5_FENCE_MODE_INITIATOR_SMALL : 0;
cseg->tis_tir_num = cpu_to_be32(tis_tir_num << 8);
ucseg->flags = MLX5_UMR_INLINE;
- ucseg->bsf_octowords = cpu_to_be16(MLX5_ST_SZ_BYTES(tls_static_params) / 16);
+ ucseg->bsf_octowords = cpu_to_be16(MLX5E_TRANSPORT_STATIC_PARAMS_OCTWORD_SIZE);
fill_static_params(&wqe->params, crypto_info, key_id, resync_tcp_sn);
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_utils.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_utils.h
index 3d79cd379890..5e2d186778aa 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_utils.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_utils.h
@@ -6,6 +6,7 @@
#include <net/tls.h>
#include "en.h"
+#include "en_accel/common_utils.h"
enum {
MLX5E_TLS_PROGRESS_PARAMS_AUTH_STATE_NO_OFFLOAD = 0,
@@ -33,13 +34,6 @@ union mlx5e_crypto_info {
struct tls12_crypto_info_aes_gcm_256 crypto_info_256;
};
-struct mlx5e_set_tls_static_params_wqe {
- struct mlx5_wqe_ctrl_seg ctrl;
- struct mlx5_wqe_umr_ctrl_seg uctrl;
- struct mlx5_mkey_seg mkc;
- struct mlx5_wqe_tls_static_params_seg params;
-};
-
struct mlx5e_set_tls_progress_params_wqe {
struct mlx5_wqe_ctrl_seg ctrl;
struct mlx5_wqe_tls_progress_params_seg params;
@@ -50,19 +44,12 @@ struct mlx5e_get_tls_progress_params_wqe {
struct mlx5_seg_get_psv psv;
};
-#define MLX5E_TLS_SET_STATIC_PARAMS_WQEBBS \
- (DIV_ROUND_UP(sizeof(struct mlx5e_set_tls_static_params_wqe), MLX5_SEND_WQE_BB))
-
#define MLX5E_TLS_SET_PROGRESS_PARAMS_WQEBBS \
(DIV_ROUND_UP(sizeof(struct mlx5e_set_tls_progress_params_wqe), MLX5_SEND_WQE_BB))
#define MLX5E_KTLS_GET_PROGRESS_WQEBBS \
(DIV_ROUND_UP(sizeof(struct mlx5e_get_tls_progress_params_wqe), MLX5_SEND_WQE_BB))
-#define MLX5E_TLS_FETCH_SET_STATIC_PARAMS_WQE(sq, pi) \
- ((struct mlx5e_set_tls_static_params_wqe *)\
- mlx5e_fetch_wqe(&(sq)->wq, pi, sizeof(struct mlx5e_set_tls_static_params_wqe)))
-
#define MLX5E_TLS_FETCH_SET_PROGRESS_PARAMS_WQE(sq, pi) \
((struct mlx5e_set_tls_progress_params_wqe *)\
mlx5e_fetch_wqe(&(sq)->wq, pi, sizeof(struct mlx5e_set_tls_progress_params_wqe)))
@@ -76,7 +63,7 @@ struct mlx5e_get_tls_progress_params_wqe {
mlx5e_fetch_wqe(&(sq)->wq, pi, sizeof(struct mlx5e_dump_wqe)))
void
-mlx5e_ktls_build_static_params(struct mlx5e_set_tls_static_params_wqe *wqe,
+mlx5e_ktls_build_static_params(struct mlx5e_set_transport_static_params_wqe *wqe,
u16 pc, u32 sqn,
union mlx5e_crypto_info *crypto_info,
u32 tis_tir_num, u32 key_id, u32 resync_tcp_sn,
diff --git a/include/linux/mlx5/device.h b/include/linux/mlx5/device.h
index 01275c6e8468..66783d63ab97 100644
--- a/include/linux/mlx5/device.h
+++ b/include/linux/mlx5/device.h
@@ -454,8 +454,8 @@ enum {
};
enum {
- MLX5_OPC_MOD_TLS_TIS_STATIC_PARAMS = 0x1,
- MLX5_OPC_MOD_TLS_TIR_STATIC_PARAMS = 0x2,
+ MLX5_OPC_MOD_TRANSPORT_TIS_STATIC_PARAMS = 0x1,
+ MLX5_OPC_MOD_TRANSPORT_TIR_STATIC_PARAMS = 0x2,
};
enum {
@@ -463,8 +463,8 @@ enum {
MLX5_OPC_MOD_TLS_TIR_PROGRESS_PARAMS = 0x2,
};
-struct mlx5_wqe_tls_static_params_seg {
- u8 ctx[MLX5_ST_SZ_BYTES(tls_static_params)];
+struct mlx5_wqe_transport_static_params_seg {
+ u8 ctx[MLX5_ST_SZ_BYTES(transport_static_params)];
};
struct mlx5_wqe_tls_progress_params_seg {
diff --git a/include/linux/mlx5/mlx5_ifc.h b/include/linux/mlx5/mlx5_ifc.h
index c940b329a475..f7784cee832e 100644
--- a/include/linux/mlx5/mlx5_ifc.h
+++ b/include/linux/mlx5/mlx5_ifc.h
@@ -12413,12 +12413,16 @@ enum {
MLX5_GENERAL_OBJECT_TYPE_ENCRYPTION_KEY_PURPOSE_MACSEC = 0x4,
};
-struct mlx5_ifc_tls_static_params_bits {
+enum {
+ MLX5_TRANSPORT_STATIC_PARAMS_ACC_TYPE_TLS = 0x1,
+};
+
+struct mlx5_ifc_transport_static_params_bits {
u8 const_2[0x2];
u8 tls_version[0x4];
u8 const_1[0x2];
u8 reserved_at_8[0x14];
- u8 encryption_standard[0x4];
+ u8 acc_type[0x4];
u8 reserved_at_20[0x20];
--
2.34.1
^ permalink raw reply related [flat|nested] 44+ messages in thread
* [PATCH v24 11/20] net/mlx5e: Refactor ico sq polling to get budget
2024-04-04 12:36 [PATCH v24 00/20] nvme-tcp receive offloads Aurelien Aptel
` (9 preceding siblings ...)
2024-04-04 12:37 ` [PATCH v24 10/20] net/mlx5e: Rename from tls to transport static params Aurelien Aptel
@ 2024-04-04 12:37 ` Aurelien Aptel
2024-04-04 12:37 ` [PATCH v24 12/20] net/mlx5: Add NVMEoTCP caps, HW bits, 128B CQE and enumerations Aurelien Aptel
` (9 subsequent siblings)
20 siblings, 0 replies; 44+ messages in thread
From: Aurelien Aptel @ 2024-04-04 12:37 UTC (permalink / raw)
To: linux-nvme, netdev, sagi, hch, kbusch, axboe, chaitanyak, davem, kuba
Cc: Or Gerlitz, aaptel, aurelien.aptel, smalin, malin1024, yorayz,
borisp, galshalom, mgurtovoy
From: Or Gerlitz <ogerlitz@nvidia.com>
The mlx5e driver uses ICO SQs for internal control operations which
are not visible to the network stack, such as UMR mapping for striding
RQ (MPWQ) and etc more cases.
The upcoming nvmeotcp offload uses ico sq for umr mapping as part of the
offload. As a pre-step for nvmeotcp ico sqs which have their own napi and
need to comply with budget, add the budget as parameter to the polling of
cqs related to ico sqs.
The polling already stops after a limit is reached, so just have the
caller to provide this limit as the budget.
Additionnaly, we move the mdev pointer directly on the icosq structure.
This provides better separation between channels to ICO SQs for use-cases
where they are not tightly coupled (such as the upcoming nvmeotcp code).
No functional change here.
Signed-off-by: Or Gerlitz <ogerlitz@nvidia.com>
Signed-off-by: Aurelien Aptel <aaptel@nvidia.com>
Reviewed-by: Tariq Toukan <tariqt@nvidia.com>
---
drivers/net/ethernet/mellanox/mlx5/core/en.h | 1 +
drivers/net/ethernet/mellanox/mlx5/core/en/reporter_rx.c | 4 ++--
drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h | 2 +-
drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 5 ++---
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c | 4 ++--
drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c | 4 ++--
6 files changed, 10 insertions(+), 10 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h
index 84db05fb9389..e3471c442c8b 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h
@@ -557,6 +557,7 @@ struct mlx5e_icosq {
/* control path */
struct mlx5_wq_ctrl wq_ctrl;
struct mlx5e_channel *channel;
+ struct mlx5_core_dev *mdev;
struct work_struct recover_work;
} ____cacheline_aligned_in_smp;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/reporter_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en/reporter_rx.c
index 25d751eba99b..b45497e6ab1a 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/reporter_rx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/reporter_rx.c
@@ -46,7 +46,7 @@ static int mlx5e_query_rq_state(struct mlx5_core_dev *dev, u32 rqn, u8 *state)
static int mlx5e_wait_for_icosq_flush(struct mlx5e_icosq *icosq)
{
- struct mlx5_core_dev *dev = icosq->channel->mdev;
+ struct mlx5_core_dev *dev = icosq->mdev;
unsigned long exp_time;
exp_time = jiffies + msecs_to_jiffies(mlx5_tout_ms(dev, FLUSH_ON_ERROR));
@@ -91,7 +91,7 @@ static int mlx5e_rx_reporter_err_icosq_cqe_recover(void *ctx)
rq = &icosq->channel->rq;
if (test_bit(MLX5E_RQ_STATE_ENABLED, &icosq->channel->xskrq.state))
xskrq = &icosq->channel->xskrq;
- mdev = icosq->channel->mdev;
+ mdev = icosq->mdev;
dev = icosq->channel->netdev;
err = mlx5_core_query_sq_state(mdev, icosq->sqn, &state);
if (err) {
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h b/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h
index 879d698b6119..cdd7fbf218ae 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h
@@ -62,7 +62,7 @@ void mlx5e_trigger_irq(struct mlx5e_icosq *sq);
void mlx5e_completion_event(struct mlx5_core_cq *mcq, struct mlx5_eqe *eqe);
void mlx5e_cq_error_event(struct mlx5_core_cq *mcq, enum mlx5_event event);
int mlx5e_napi_poll(struct napi_struct *napi, int budget);
-int mlx5e_poll_ico_cq(struct mlx5e_cq *cq);
+int mlx5e_poll_ico_cq(struct mlx5e_cq *cq, int budget);
/* RX */
INDIRECT_CALLABLE_DECLARE(bool mlx5e_post_rx_wqes(struct mlx5e_rq *rq));
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index 81e1c1e401f9..a467a011c599 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -1514,6 +1514,7 @@ static int mlx5e_alloc_icosq(struct mlx5e_channel *c,
int err;
sq->channel = c;
+ sq->mdev = mdev;
sq->uar_map = mdev->mlx5e_res.hw_objs.bfreg.map;
sq->reserved_room = param->stop_room;
@@ -1914,11 +1915,9 @@ void mlx5e_deactivate_icosq(struct mlx5e_icosq *icosq)
static void mlx5e_close_icosq(struct mlx5e_icosq *sq)
{
- struct mlx5e_channel *c = sq->channel;
-
if (sq->ktls_resync)
mlx5e_ktls_rx_resync_destroy_resp_list(sq->ktls_resync);
- mlx5e_destroy_sq(c->mdev, sq->sqn);
+ mlx5e_destroy_sq(sq->mdev, sq->sqn);
mlx5e_free_icosq_descs(sq);
mlx5e_free_icosq(sq);
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
index d601b5faaed5..780902f63823 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
@@ -993,7 +993,7 @@ static void mlx5e_handle_shampo_hd_umr(struct mlx5e_shampo_umr umr,
shampo->ci = (shampo->ci + umr.len) & (shampo->hd_per_wq - 1);
}
-int mlx5e_poll_ico_cq(struct mlx5e_cq *cq)
+int mlx5e_poll_ico_cq(struct mlx5e_cq *cq, int budget)
{
struct mlx5e_icosq *sq = container_of(cq, struct mlx5e_icosq, cq);
struct mlx5_cqe64 *cqe;
@@ -1068,7 +1068,7 @@ int mlx5e_poll_ico_cq(struct mlx5e_cq *cq)
wi->wqe_type);
}
} while (!last_wqe);
- } while ((++i < MLX5E_TX_CQ_POLL_BUDGET) && (cqe = mlx5_cqwq_get_cqe(&cq->wq)));
+ } while ((++i < budget) && (cqe = mlx5_cqwq_get_cqe(&cq->wq)));
sq->cc = sqcc;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
index a7d9b7cb4297..fd52311aada9 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
@@ -178,8 +178,8 @@ int mlx5e_napi_poll(struct napi_struct *napi, int budget)
busy |= work_done == budget;
- mlx5e_poll_ico_cq(&c->icosq.cq);
- if (mlx5e_poll_ico_cq(&c->async_icosq.cq))
+ mlx5e_poll_ico_cq(&c->icosq.cq, MLX5E_TX_CQ_POLL_BUDGET);
+ if (mlx5e_poll_ico_cq(&c->async_icosq.cq, MLX5E_TX_CQ_POLL_BUDGET))
/* Don't clear the flag if nothing was polled to prevent
* queueing more WQEs and overflowing the async ICOSQ.
*/
--
2.34.1
^ permalink raw reply related [flat|nested] 44+ messages in thread
* [PATCH v24 12/20] net/mlx5: Add NVMEoTCP caps, HW bits, 128B CQE and enumerations
2024-04-04 12:36 [PATCH v24 00/20] nvme-tcp receive offloads Aurelien Aptel
` (10 preceding siblings ...)
2024-04-04 12:37 ` [PATCH v24 11/20] net/mlx5e: Refactor ico sq polling to get budget Aurelien Aptel
@ 2024-04-04 12:37 ` Aurelien Aptel
2024-04-04 12:37 ` [PATCH v24 13/20] net/mlx5e: NVMEoTCP, offload initialization Aurelien Aptel
` (8 subsequent siblings)
20 siblings, 0 replies; 44+ messages in thread
From: Aurelien Aptel @ 2024-04-04 12:37 UTC (permalink / raw)
To: linux-nvme, netdev, sagi, hch, kbusch, axboe, chaitanyak, davem, kuba
Cc: aaptel, aurelien.aptel, smalin, malin1024, ogerlitz, yorayz,
borisp, galshalom, mgurtovoy
From: Ben Ben-Ishay <benishay@nvidia.com>
Add the necessary infrastructure for NVMEoTCP offload:
- Create mlx5_cqe128 structure for NVMEoTCP offload.
The new structure consist from the regular mlx5_cqe64 +
NVMEoTCP data information for offloaded packets.
- Add nvmetcp field to mlx5_cqe64, this field define the type
of the data that the additional NVMEoTCP part represents.
- Add nvmeotcp_zero_copy_en + nvmeotcp_crc_en bit
to the TIR, for identify NVMEoTCP offload flow
and tag_buffer_id that will be used by the
connected nvmeotcp_queues.
- Add new capability to HCA_CAP that represents the
NVMEoTCP offload ability.
Signed-off-by: Ben Ben-Ishay <benishay@nvidia.com>
Signed-off-by: Or Gerlitz <ogerlitz@nvidia.com>
Signed-off-by: Aurelien Aptel <aaptel@nvidia.com>
Reviewed-by: Tariq Toukan <tariqt@nvidia.com>
---
drivers/net/ethernet/mellanox/mlx5/core/fw.c | 6 ++
include/linux/mlx5/device.h | 51 ++++++++++++-
include/linux/mlx5/mlx5_ifc.h | 75 ++++++++++++++++++--
include/linux/mlx5/qp.h | 1 +
4 files changed, 128 insertions(+), 5 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fw.c b/drivers/net/ethernet/mellanox/mlx5/core/fw.c
index e7faf7e73ca4..b76aafa679f6 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/fw.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/fw.c
@@ -280,6 +280,12 @@ int mlx5_query_hca_caps(struct mlx5_core_dev *dev)
return err;
}
+ if (MLX5_CAP_GEN(dev, nvmeotcp)) {
+ err = mlx5_core_get_caps(dev, MLX5_CAP_DEV_NVMEOTCP);
+ if (err)
+ return err;
+ }
+
return 0;
}
diff --git a/include/linux/mlx5/device.h b/include/linux/mlx5/device.h
index 66783d63ab97..99cea3707bbc 100644
--- a/include/linux/mlx5/device.h
+++ b/include/linux/mlx5/device.h
@@ -264,6 +264,7 @@ enum {
enum {
MLX5_MKEY_MASK_LEN = 1ull << 0,
MLX5_MKEY_MASK_PAGE_SIZE = 1ull << 1,
+ MLX5_MKEY_MASK_XLT_OCT_SIZE = 1ull << 2,
MLX5_MKEY_MASK_START_ADDR = 1ull << 6,
MLX5_MKEY_MASK_PD = 1ull << 7,
MLX5_MKEY_MASK_EN_RINVAL = 1ull << 8,
@@ -798,7 +799,11 @@ struct mlx5_err_cqe {
struct mlx5_cqe64 {
u8 tls_outer_l3_tunneled;
- u8 rsvd0;
+ u8 rsvd16bit:4;
+ u8 nvmeotcp_zc:1;
+ u8 nvmeotcp_ddgst:1;
+ u8 nvmeotcp_resync:1;
+ u8 rsvd23bit:1;
__be16 wqe_id;
union {
struct {
@@ -847,6 +852,19 @@ struct mlx5_cqe64 {
u8 op_own;
};
+struct mlx5e_cqe128 {
+ __be16 cclen;
+ __be16 hlen;
+ union {
+ __be32 resync_tcp_sn;
+ __be32 ccoff;
+ };
+ __be16 ccid;
+ __be16 rsvd8;
+ u8 rsvd12[52];
+ struct mlx5_cqe64 cqe64;
+};
+
struct mlx5_mini_cqe8 {
union {
__be32 rx_hash_result;
@@ -882,6 +900,28 @@ enum {
#define MLX5_MINI_CQE_ARRAY_SIZE 8
+static inline bool cqe_is_nvmeotcp_resync(struct mlx5_cqe64 *cqe)
+{
+ return cqe->nvmeotcp_resync;
+}
+
+static inline bool cqe_is_nvmeotcp_crcvalid(struct mlx5_cqe64 *cqe)
+{
+ return cqe->nvmeotcp_ddgst;
+}
+
+static inline bool cqe_is_nvmeotcp_zc(struct mlx5_cqe64 *cqe)
+{
+ return cqe->nvmeotcp_zc;
+}
+
+/* check if cqe is zc or crc or resync */
+static inline bool cqe_is_nvmeotcp(struct mlx5_cqe64 *cqe)
+{
+ return cqe_is_nvmeotcp_zc(cqe) || cqe_is_nvmeotcp_crcvalid(cqe) ||
+ cqe_is_nvmeotcp_resync(cqe);
+}
+
static inline u8 mlx5_get_cqe_format(struct mlx5_cqe64 *cqe)
{
return (cqe->op_own >> 2) & 0x3;
@@ -1222,6 +1262,7 @@ enum mlx5_cap_type {
MLX5_CAP_VDPA_EMULATION = 0x13,
MLX5_CAP_DEV_EVENT = 0x14,
MLX5_CAP_IPSEC,
+ MLX5_CAP_DEV_NVMEOTCP = 0x19,
MLX5_CAP_CRYPTO = 0x1a,
MLX5_CAP_MACSEC = 0x1f,
MLX5_CAP_GENERAL_2 = 0x20,
@@ -1429,6 +1470,14 @@ enum mlx5_qcam_feature_groups {
#define MLX5_CAP_MACSEC(mdev, cap)\
MLX5_GET(macsec_cap, (mdev)->caps.hca[MLX5_CAP_MACSEC]->cur, cap)
+#define MLX5_CAP_DEV_NVMEOTCP(mdev, cap)\
+ MLX5_GET(nvmeotcp_cap, \
+ (mdev)->caps.hca[MLX5_CAP_DEV_NVMEOTCP]->cur, cap)
+
+#define MLX5_CAP64_DEV_NVMEOTCP(mdev, cap)\
+ MLX5_GET64(nvmeotcp_cap, \
+ (mdev)->caps.hca[MLX5_CAP_DEV_NVMEOTCP]->cur, cap)
+
enum {
MLX5_CMD_STAT_OK = 0x0,
MLX5_CMD_STAT_INT_ERR = 0x1,
diff --git a/include/linux/mlx5/mlx5_ifc.h b/include/linux/mlx5/mlx5_ifc.h
index f7784cee832e..fe8c8b9bd72e 100644
--- a/include/linux/mlx5/mlx5_ifc.h
+++ b/include/linux/mlx5/mlx5_ifc.h
@@ -1468,6 +1468,20 @@ enum {
MLX5_STEERING_FORMAT_CONNECTX_7 = 2,
};
+struct mlx5_ifc_nvmeotcp_cap_bits {
+ u8 zerocopy[0x1];
+ u8 crc_rx[0x1];
+ u8 crc_tx[0x1];
+ u8 reserved_at_3[0x15];
+ u8 version[0x8];
+
+ u8 reserved_at_20[0x13];
+ u8 log_max_nvmeotcp_tag_buffer_table[0x5];
+ u8 reserved_at_38[0x3];
+ u8 log_max_nvmeotcp_tag_buffer_size[0x5];
+ u8 reserved_at_40[0x7c0];
+};
+
struct mlx5_ifc_cmd_hca_cap_bits {
u8 reserved_at_0[0x10];
u8 shared_object_to_user_object_allowed[0x1];
@@ -1492,7 +1506,9 @@ struct mlx5_ifc_cmd_hca_cap_bits {
u8 event_cap[0x1];
u8 reserved_at_91[0x2];
u8 isolate_vl_tc_new[0x1];
- u8 reserved_at_94[0x4];
+ u8 reserved_at_94[0x2];
+ u8 nvmeotcp[0x1];
+ u8 reserved_at_97[0x1];
u8 prio_tag_required[0x1];
u8 reserved_at_99[0x2];
u8 log_max_qp[0x5];
@@ -3494,6 +3510,7 @@ union mlx5_ifc_hca_cap_union_bits {
struct mlx5_ifc_macsec_cap_bits macsec_cap;
struct mlx5_ifc_crypto_cap_bits crypto_cap;
struct mlx5_ifc_ipsec_cap_bits ipsec_cap;
+ struct mlx5_ifc_nvmeotcp_cap_bits nvmeotcp_cap;
u8 reserved_at_0[0x8000];
};
@@ -3746,7 +3763,9 @@ struct mlx5_ifc_tirc_bits {
u8 disp_type[0x4];
u8 tls_en[0x1];
- u8 reserved_at_25[0x1b];
+ u8 nvmeotcp_zero_copy_en[0x1];
+ u8 nvmeotcp_crc_en[0x1];
+ u8 reserved_at_27[0x19];
u8 reserved_at_40[0x40];
@@ -3777,7 +3796,8 @@ struct mlx5_ifc_tirc_bits {
struct mlx5_ifc_rx_hash_field_select_bits rx_hash_field_selector_inner;
- u8 reserved_at_2c0[0x4c0];
+ u8 nvmeotcp_tag_buffer_table_id[0x20];
+ u8 reserved_at_2e0[0x4a0];
};
enum {
@@ -12022,6 +12042,7 @@ enum {
MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_ENCRYPTION_KEY = BIT_ULL(0xc),
MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_IPSEC = BIT_ULL(0x13),
MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_SAMPLER = BIT_ULL(0x20),
+ MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_NVMEOTCP_TAG_BUFFER_TABLE = BIT_ULL(0x21),
MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_FLOW_METER_ASO = BIT_ULL(0x24),
};
@@ -12029,6 +12050,7 @@ enum {
MLX5_GENERAL_OBJECT_TYPES_ENCRYPTION_KEY = 0xc,
MLX5_GENERAL_OBJECT_TYPES_IPSEC = 0x13,
MLX5_GENERAL_OBJECT_TYPES_SAMPLER = 0x20,
+ MLX5_GENERAL_OBJECT_TYPES_NVMEOTCP_TAG_BUFFER_TABLE = 0x21,
MLX5_GENERAL_OBJECT_TYPES_FLOW_METER_ASO = 0x24,
MLX5_GENERAL_OBJECT_TYPES_MACSEC = 0x27,
MLX5_GENERAL_OBJECT_TYPES_INT_KEK = 0x47,
@@ -12402,6 +12424,20 @@ struct mlx5_ifc_query_sampler_obj_out_bits {
struct mlx5_ifc_sampler_obj_bits sampler_object;
};
+struct mlx5_ifc_nvmeotcp_tag_buf_table_obj_bits {
+ u8 modify_field_select[0x40];
+
+ u8 reserved_at_40[0x20];
+
+ u8 reserved_at_60[0x1b];
+ u8 log_tag_buffer_table_size[0x5];
+};
+
+struct mlx5_ifc_create_nvmeotcp_tag_buf_table_in_bits {
+ struct mlx5_ifc_general_obj_in_cmd_hdr_bits general_obj_in_cmd_hdr;
+ struct mlx5_ifc_nvmeotcp_tag_buf_table_obj_bits nvmeotcp_tag_buf_table_obj;
+};
+
enum {
MLX5_GENERAL_OBJECT_TYPE_ENCRYPTION_KEY_KEY_SIZE_128 = 0x0,
MLX5_GENERAL_OBJECT_TYPE_ENCRYPTION_KEY_KEY_SIZE_256 = 0x1,
@@ -12415,6 +12451,13 @@ enum {
enum {
MLX5_TRANSPORT_STATIC_PARAMS_ACC_TYPE_TLS = 0x1,
+ MLX5_TRANSPORT_STATIC_PARAMS_ACC_TYPE_NVMETCP = 0x2,
+ MLX5_TRANSPORT_STATIC_PARAMS_ACC_TYPE_NVMETCP_WITH_TLS = 0x3,
+};
+
+enum {
+ MLX5_TRANSPORT_STATIC_PARAMS_TI_INITIATOR = 0x0,
+ MLX5_TRANSPORT_STATIC_PARAMS_TI_TARGET = 0x1,
};
struct mlx5_ifc_transport_static_params_bits {
@@ -12437,7 +12480,20 @@ struct mlx5_ifc_transport_static_params_bits {
u8 reserved_at_100[0x8];
u8 dek_index[0x18];
- u8 reserved_at_120[0xe0];
+ u8 reserved_at_120[0x14];
+
+ u8 cccid_ttag[0x1];
+ u8 ti[0x1];
+ u8 zero_copy_en[0x1];
+ u8 ddgst_offload_en[0x1];
+ u8 hdgst_offload_en[0x1];
+ u8 ddgst_en[0x1];
+ u8 hddgst_en[0x1];
+ u8 pda[0x5];
+
+ u8 nvme_resync_tcp_sn[0x20];
+
+ u8 reserved_at_160[0xa0];
};
struct mlx5_ifc_tls_progress_params_bits {
@@ -12749,4 +12805,15 @@ struct mlx5_ifc_msees_reg_bits {
u8 reserved_at_80[0x180];
};
+struct mlx5_ifc_nvmeotcp_progress_params_bits {
+ u8 next_pdu_tcp_sn[0x20];
+
+ u8 hw_resync_tcp_sn[0x20];
+
+ u8 pdu_tracker_state[0x2];
+ u8 offloading_state[0x2];
+ u8 reserved_at_44[0xc];
+ u8 cccid_ttag[0x10];
+};
+
#endif /* MLX5_IFC_H */
diff --git a/include/linux/mlx5/qp.h b/include/linux/mlx5/qp.h
index f0e55bf3ec8b..c2805cf8d7fd 100644
--- a/include/linux/mlx5/qp.h
+++ b/include/linux/mlx5/qp.h
@@ -227,6 +227,7 @@ struct mlx5_wqe_ctrl_seg {
#define MLX5_WQE_CTRL_OPCODE_MASK 0xff
#define MLX5_WQE_CTRL_WQE_INDEX_MASK 0x00ffff00
#define MLX5_WQE_CTRL_WQE_INDEX_SHIFT 8
+#define MLX5_WQE_CTRL_TIR_TIS_INDEX_SHIFT 8
enum {
MLX5_ETH_WQE_L3_INNER_CSUM = 1 << 4,
--
2.34.1
^ permalink raw reply related [flat|nested] 44+ messages in thread
* [PATCH v24 13/20] net/mlx5e: NVMEoTCP, offload initialization
2024-04-04 12:36 [PATCH v24 00/20] nvme-tcp receive offloads Aurelien Aptel
` (11 preceding siblings ...)
2024-04-04 12:37 ` [PATCH v24 12/20] net/mlx5: Add NVMEoTCP caps, HW bits, 128B CQE and enumerations Aurelien Aptel
@ 2024-04-04 12:37 ` Aurelien Aptel
2024-04-04 12:37 ` [PATCH v24 14/20] net/mlx5e: TCP flow steering for nvme-tcp acceleration Aurelien Aptel
` (7 subsequent siblings)
20 siblings, 0 replies; 44+ messages in thread
From: Aurelien Aptel @ 2024-04-04 12:37 UTC (permalink / raw)
To: linux-nvme, netdev, sagi, hch, kbusch, axboe, chaitanyak, davem, kuba
Cc: aaptel, aurelien.aptel, smalin, malin1024, ogerlitz, yorayz,
borisp, galshalom, mgurtovoy
From: Ben Ben-Ishay <benishay@nvidia.com>
This commit introduces the driver structures and initialization blocks
for NVMEoTCP offload. The mlx5 nvmeotcp structures are:
- queue (mlx5e_nvmeotcp_queue) - pairs 1:1 with nvmeotcp driver queues and
deals with the offloading parts. The mlx5e queue is accessed in the ddp
ops: initialized on sk_add, used in ddp setup,teardown,resync and in the
fast path when dealing with packets, destroyed in the sk_del op.
- queue entry (nvmeotcp_queue_entry) - pairs 1:1 with offloaded IO from
that queue. Keeps pointers to the SG elements describing the buffers
used for the IO and the ddp context of it.
- queue handler (mlx5e_nvmeotcp_queue_handler) - we use icosq per NVME-TCP
queue for UMR mapping as part of the ddp offload. Those dedicated SQs are
unique in the sense that they are driven directly by the NVME-TCP layer
to submit and invalidate ddp requests.
Since the life-cycle of these icosqs is not tied to the channels, we
create dedicated napi contexts for polling them such that channels can be
re-created during offloading. The queue handler has pointer to the cq
associated with the queue's sq and napi context.
- main offload context (mlx5e_nvmeotcp) - has ida and hash table instances.
Each offloaded queue gets an ID from the ida instance and the <id, queue>
pairs are kept in the hash table. The id is programmed as flow tag to be
set by HW on the completion (cqe) of all packets related to this queue
(by 5-tuple steering). The fast path which deals with packets uses the
flow tag to access the hash table and retrieve the queue for the
processing.
We query nvmeotcp capabilities to see if the offload can be supported and
use 128B CQEs when this happens. By default, the offload is off but can
be enabled with `ethtool --ulp-ddp <device> nvme-tcp-ddp on`.
Signed-off-by: Ben Ben-Ishay <benishay@nvidia.com>
Signed-off-by: Boris Pismenny <borisp@nvidia.com>
Signed-off-by: Or Gerlitz <ogerlitz@nvidia.com>
Signed-off-by: Yoray Zack <yorayz@nvidia.com>
Signed-off-by: Shai Malin <smalin@nvidia.com>
Signed-off-by: Aurelien Aptel <aaptel@nvidia.com>
Reviewed-by: Tariq Toukan <tariqt@nvidia.com>
---
.../net/ethernet/mellanox/mlx5/core/Kconfig | 11 +
.../net/ethernet/mellanox/mlx5/core/Makefile | 2 +
drivers/net/ethernet/mellanox/mlx5/core/en.h | 4 +
.../net/ethernet/mellanox/mlx5/core/en/fs.h | 4 +-
.../ethernet/mellanox/mlx5/core/en/params.c | 12 +-
.../ethernet/mellanox/mlx5/core/en/params.h | 3 +
.../mellanox/mlx5/core/en_accel/en_accel.h | 3 +
.../mellanox/mlx5/core/en_accel/fs_tcp.h | 2 +-
.../mellanox/mlx5/core/en_accel/nvmeotcp.c | 217 ++++++++++++++++++
.../mellanox/mlx5/core/en_accel/nvmeotcp.h | 120 ++++++++++
.../ethernet/mellanox/mlx5/core/en_ethtool.c | 6 +
.../net/ethernet/mellanox/mlx5/core/en_fs.c | 4 +-
.../net/ethernet/mellanox/mlx5/core/en_main.c | 16 ++
.../net/ethernet/mellanox/mlx5/core/main.c | 1 +
14 files changed, 396 insertions(+), 9 deletions(-)
create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c
create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/Kconfig b/drivers/net/ethernet/mellanox/mlx5/core/Kconfig
index 685335832a93..5935c2cdefec 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/Kconfig
+++ b/drivers/net/ethernet/mellanox/mlx5/core/Kconfig
@@ -164,6 +164,17 @@ config MLX5_EN_TLS
help
Build support for TLS cryptography-offload acceleration in the NIC.
+config MLX5_EN_NVMEOTCP
+ bool "NVMEoTCP acceleration"
+ depends on ULP_DDP
+ depends on MLX5_CORE_EN
+ default y
+ help
+ Build support for NVMEoTCP acceleration in the NIC.
+ This includes Direct Data Placement and CRC offload.
+ Note: Support for hardware with this capability needs to be selected
+ for this option to become available.
+
config MLX5_SW_STEERING
bool "Mellanox Technologies software-managed steering"
depends on MLX5_CORE_EN && MLX5_ESWITCH
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/Makefile b/drivers/net/ethernet/mellanox/mlx5/core/Makefile
index 76dc5a9b9648..41cb8f831632 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/Makefile
+++ b/drivers/net/ethernet/mellanox/mlx5/core/Makefile
@@ -109,6 +109,8 @@ mlx5_core-$(CONFIG_MLX5_EN_TLS) += en_accel/ktls_stats.o \
en_accel/fs_tcp.o en_accel/ktls.o en_accel/ktls_txrx.o \
en_accel/ktls_tx.o en_accel/ktls_rx.o
+mlx5_core-$(CONFIG_MLX5_EN_NVMEOTCP) += en_accel/fs_tcp.o en_accel/nvmeotcp.o
+
mlx5_core-$(CONFIG_MLX5_SW_STEERING) += steering/dr_domain.o steering/dr_table.o \
steering/dr_matcher.o steering/dr_rule.o \
steering/dr_icm_pool.o steering/dr_buddy.o \
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h
index e3471c442c8b..6f2a1d419f64 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h
@@ -327,6 +327,7 @@ struct mlx5e_params {
int hard_mtu;
bool ptp_rx;
__be32 terminate_lkey_be;
+ bool nvmeotcp;
};
static inline u8 mlx5e_get_dcb_num_tc(struct mlx5e_params *params)
@@ -938,6 +939,9 @@ struct mlx5e_priv {
#endif
#ifdef CONFIG_MLX5_EN_TLS
struct mlx5e_tls *tls;
+#endif
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+ struct mlx5e_nvmeotcp *nvmeotcp;
#endif
struct devlink_health_reporter *tx_reporter;
struct devlink_health_reporter *rx_reporter;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/fs.h b/drivers/net/ethernet/mellanox/mlx5/core/en/fs.h
index 4d6225e0eec7..780e8b5ae8e0 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/fs.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/fs.h
@@ -77,7 +77,7 @@ enum {
MLX5E_INNER_TTC_FT_LEVEL,
MLX5E_FS_TT_UDP_FT_LEVEL = MLX5E_INNER_TTC_FT_LEVEL + 1,
MLX5E_FS_TT_ANY_FT_LEVEL = MLX5E_INNER_TTC_FT_LEVEL + 1,
-#ifdef CONFIG_MLX5_EN_TLS
+#if defined(CONFIG_MLX5_EN_TLS) || defined(CONFIG_MLX5_EN_NVMEOTCP)
MLX5E_ACCEL_FS_TCP_FT_LEVEL = MLX5E_INNER_TTC_FT_LEVEL + 1,
#endif
#ifdef CONFIG_MLX5_EN_ARFS
@@ -169,7 +169,7 @@ struct mlx5e_fs_any *mlx5e_fs_get_any(struct mlx5e_flow_steering *fs);
void mlx5e_fs_set_any(struct mlx5e_flow_steering *fs, struct mlx5e_fs_any *any);
struct mlx5e_fs_udp *mlx5e_fs_get_udp(struct mlx5e_flow_steering *fs);
void mlx5e_fs_set_udp(struct mlx5e_flow_steering *fs, struct mlx5e_fs_udp *udp);
-#ifdef CONFIG_MLX5_EN_TLS
+#if defined(CONFIG_MLX5_EN_TLS) || defined(CONFIG_MLX5_EN_NVMEOTCP)
struct mlx5e_accel_fs_tcp *mlx5e_fs_get_accel_tcp(struct mlx5e_flow_steering *fs);
void mlx5e_fs_set_accel_tcp(struct mlx5e_flow_steering *fs, struct mlx5e_accel_fs_tcp *accel_tcp);
#endif
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/params.c b/drivers/net/ethernet/mellanox/mlx5/core/en/params.c
index a3f31d9d527e..e694bfd872cc 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/params.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/params.c
@@ -889,7 +889,8 @@ static void mlx5e_build_common_cq_param(struct mlx5_core_dev *mdev,
void *cqc = param->cqc;
MLX5_SET(cqc, cqc, uar_page, mdev->priv.uar->index);
- if (MLX5_CAP_GEN(mdev, cqe_128_always) && cache_line_size() >= 128)
+ if (MLX5_CAP_GEN(mdev, cqe_128_always) &&
+ (cache_line_size() >= 128 || param->force_cqe128))
MLX5_SET(cqc, cqc, cqe_sz, CQE_STRIDE_128_PAD);
}
@@ -919,6 +920,9 @@ static void mlx5e_build_rx_cq_param(struct mlx5_core_dev *mdev,
void *cqc = param->cqc;
u8 log_cq_size;
+ /* nvme-tcp offload mandates 128 byte cqes */
+ param->force_cqe128 |= IS_ENABLED(CONFIG_MLX5_EN_NVMEOTCP) && params->nvmeotcp;
+
switch (params->rq_wq_type) {
case MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ:
hw_stridx = MLX5_CAP_GEN(mdev, mini_cqe_resp_stride_index);
@@ -1254,9 +1258,9 @@ static u8 mlx5e_build_async_icosq_log_wq_sz(struct mlx5_core_dev *mdev)
return MLX5E_PARAMS_MINIMUM_LOG_SQ_SIZE;
}
-static void mlx5e_build_icosq_param(struct mlx5_core_dev *mdev,
- u8 log_wq_size,
- struct mlx5e_sq_param *param)
+void mlx5e_build_icosq_param(struct mlx5_core_dev *mdev,
+ u8 log_wq_size,
+ struct mlx5e_sq_param *param)
{
void *sqc = param->sqc;
void *wq = MLX5_ADDR_OF(sqc, sqc, wq);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/params.h b/drivers/net/ethernet/mellanox/mlx5/core/en/params.h
index 9a781f18b57f..ce3c87e64edc 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/params.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/params.h
@@ -17,6 +17,7 @@ struct mlx5e_cq_param {
struct mlx5_wq_param wq;
u16 eq_ix;
u8 cq_period_mode;
+ bool force_cqe128;
};
struct mlx5e_rq_param {
@@ -145,6 +146,8 @@ void mlx5e_build_xdpsq_param(struct mlx5_core_dev *mdev,
struct mlx5e_params *params,
struct mlx5e_xsk_param *xsk,
struct mlx5e_sq_param *param);
+void mlx5e_build_icosq_param(struct mlx5_core_dev *mdev,
+ u8 log_wq_size, struct mlx5e_sq_param *param);
int mlx5e_build_channel_param(struct mlx5_core_dev *mdev,
struct mlx5e_params *params,
struct mlx5e_channel_param *cparam);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/en_accel.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/en_accel.h
index caa34b9c161e..070dabb03bd4 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/en_accel.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/en_accel.h
@@ -40,6 +40,7 @@
#include "en_accel/ktls.h"
#include "en_accel/ktls_txrx.h"
#include <en_accel/macsec.h>
+#include "en_accel/nvmeotcp.h"
#include "en.h"
#include "en/txrx.h"
@@ -202,11 +203,13 @@ static inline void mlx5e_accel_tx_finish(struct mlx5e_txqsq *sq,
static inline int mlx5e_accel_init_rx(struct mlx5e_priv *priv)
{
+ mlx5e_nvmeotcp_init_rx(priv);
return mlx5e_ktls_init_rx(priv);
}
static inline void mlx5e_accel_cleanup_rx(struct mlx5e_priv *priv)
{
+ mlx5e_nvmeotcp_cleanup_rx(priv);
mlx5e_ktls_cleanup_rx(priv);
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.h
index a032bff482a6..d907e352ffae 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.h
@@ -6,7 +6,7 @@
#include "en/fs.h"
-#ifdef CONFIG_MLX5_EN_TLS
+#if defined(CONFIG_MLX5_EN_TLS) || defined(CONFIG_MLX5_EN_NVMEOTCP)
int mlx5e_accel_fs_tcp_create(struct mlx5e_flow_steering *fs);
void mlx5e_accel_fs_tcp_destroy(struct mlx5e_flow_steering *fs);
struct mlx5_flow_handle *mlx5e_accel_fs_add_sk(struct mlx5e_flow_steering *fs,
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c
new file mode 100644
index 000000000000..9965757873f9
--- /dev/null
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c
@@ -0,0 +1,217 @@
+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
+// Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES.
+
+#include <linux/netdevice.h>
+#include <linux/idr.h>
+#include "en_accel/nvmeotcp.h"
+#include "en_accel/fs_tcp.h"
+#include "en/txrx.h"
+
+#define MAX_NUM_NVMEOTCP_QUEUES (4000)
+#define MIN_NUM_NVMEOTCP_QUEUES (1)
+
+static const struct rhashtable_params rhash_queues = {
+ .key_len = sizeof(int),
+ .key_offset = offsetof(struct mlx5e_nvmeotcp_queue, id),
+ .head_offset = offsetof(struct mlx5e_nvmeotcp_queue, hash),
+ .automatic_shrinking = true,
+ .min_size = MIN_NUM_NVMEOTCP_QUEUES,
+ .max_size = MAX_NUM_NVMEOTCP_QUEUES,
+};
+
+static int
+mlx5e_nvmeotcp_offload_limits(struct net_device *netdev,
+ struct ulp_ddp_limits *limits)
+{
+ return 0;
+}
+
+static int
+mlx5e_nvmeotcp_queue_init(struct net_device *netdev,
+ struct sock *sk,
+ struct ulp_ddp_config *tconfig)
+{
+ return 0;
+}
+
+static void
+mlx5e_nvmeotcp_queue_teardown(struct net_device *netdev,
+ struct sock *sk)
+{
+}
+
+static int
+mlx5e_nvmeotcp_ddp_setup(struct net_device *netdev,
+ struct sock *sk,
+ struct ulp_ddp_io *ddp)
+{
+ return 0;
+}
+
+static void
+mlx5e_nvmeotcp_ddp_teardown(struct net_device *netdev,
+ struct sock *sk,
+ struct ulp_ddp_io *ddp,
+ void *ddp_ctx)
+{
+}
+
+static void
+mlx5e_nvmeotcp_ddp_resync(struct net_device *netdev,
+ struct sock *sk, u32 seq)
+{
+}
+
+int set_ulp_ddp_nvme_tcp(struct net_device *netdev, bool enable)
+{
+ struct mlx5e_priv *priv = netdev_priv(netdev);
+ struct mlx5e_params new_params;
+ int err = 0;
+
+ /* There may be offloaded queues when an netlink callback to disable the feature is made.
+ * Hence, we can't destroy the tcp flow-table since it may be referenced by the offload
+ * related flows and we'll keep the 128B CQEs on the channel RQs. Also, since we don't
+ * deref/destroy the fs tcp table when the feature is disabled, we don't ref it again
+ * if the feature is enabled multiple times.
+ */
+ if (!enable || priv->nvmeotcp->enabled)
+ return 0;
+
+ err = mlx5e_accel_fs_tcp_create(priv->fs);
+ if (err)
+ return err;
+
+ new_params = priv->channels.params;
+ new_params.nvmeotcp = enable;
+ err = mlx5e_safe_switch_params(priv, &new_params, NULL, NULL, true);
+ if (err)
+ goto fs_tcp_destroy;
+
+ priv->nvmeotcp->enabled = true;
+ return 0;
+
+fs_tcp_destroy:
+ mlx5e_accel_fs_tcp_destroy(priv->fs);
+ return err;
+}
+
+static int mlx5e_ulp_ddp_set_caps(struct net_device *netdev, unsigned long *new_caps,
+ struct netlink_ext_ack *extack)
+{
+ struct mlx5e_priv *priv = netdev_priv(netdev);
+ DECLARE_BITMAP(old_caps, ULP_DDP_CAP_COUNT);
+ struct mlx5e_params *params;
+ int ret = 0;
+ int nvme = -1;
+
+ mutex_lock(&priv->state_lock);
+ params = &priv->channels.params;
+ bitmap_copy(old_caps, priv->nvmeotcp->ddp_caps.active, ULP_DDP_CAP_COUNT);
+
+ /* always handle nvme-tcp-ddp and nvme-tcp-ddgst-rx together (all or nothing) */
+
+ if (ulp_ddp_cap_turned_on(old_caps, new_caps, ULP_DDP_CAP_NVME_TCP) &&
+ ulp_ddp_cap_turned_on(old_caps, new_caps, ULP_DDP_CAP_NVME_TCP_DDGST_RX)) {
+ if (MLX5E_GET_PFLAG(params, MLX5E_PFLAG_RX_CQE_COMPRESS)) {
+ NL_SET_ERR_MSG_MOD(extack,
+ "NVMe-TCP offload not supported when CQE compress is active. Disable rx_cqe_compress ethtool private flag first");
+ goto out;
+ }
+
+ if (netdev->features & (NETIF_F_LRO | NETIF_F_GRO_HW)) {
+ NL_SET_ERR_MSG_MOD(extack,
+ "NVMe-TCP offload not supported when HW_GRO/LRO is active. Disable rx-gro-hw ethtool feature first");
+ goto out;
+ }
+ nvme = 1;
+ } else if (ulp_ddp_cap_turned_off(old_caps, new_caps, ULP_DDP_CAP_NVME_TCP) &&
+ ulp_ddp_cap_turned_off(old_caps, new_caps, ULP_DDP_CAP_NVME_TCP_DDGST_RX)) {
+ nvme = 0;
+ }
+
+ if (nvme >= 0) {
+ ret = set_ulp_ddp_nvme_tcp(netdev, nvme);
+ if (ret)
+ goto out;
+ change_bit(ULP_DDP_CAP_NVME_TCP, priv->nvmeotcp->ddp_caps.active);
+ change_bit(ULP_DDP_CAP_NVME_TCP_DDGST_RX, priv->nvmeotcp->ddp_caps.active);
+ }
+
+out:
+ mutex_unlock(&priv->state_lock);
+ return ret;
+}
+
+static void mlx5e_ulp_ddp_get_caps(struct net_device *dev,
+ struct ulp_ddp_dev_caps *caps)
+{
+ struct mlx5e_priv *priv = netdev_priv(dev);
+
+ mutex_lock(&priv->state_lock);
+ memcpy(caps, &priv->nvmeotcp->ddp_caps, sizeof(*caps));
+ mutex_unlock(&priv->state_lock);
+}
+
+const struct ulp_ddp_dev_ops mlx5e_nvmeotcp_ops = {
+ .limits = mlx5e_nvmeotcp_offload_limits,
+ .sk_add = mlx5e_nvmeotcp_queue_init,
+ .sk_del = mlx5e_nvmeotcp_queue_teardown,
+ .setup = mlx5e_nvmeotcp_ddp_setup,
+ .teardown = mlx5e_nvmeotcp_ddp_teardown,
+ .resync = mlx5e_nvmeotcp_ddp_resync,
+ .set_caps = mlx5e_ulp_ddp_set_caps,
+ .get_caps = mlx5e_ulp_ddp_get_caps,
+};
+
+void mlx5e_nvmeotcp_cleanup_rx(struct mlx5e_priv *priv)
+{
+ if (priv->nvmeotcp && priv->nvmeotcp->enabled)
+ mlx5e_accel_fs_tcp_destroy(priv->fs);
+}
+
+int mlx5e_nvmeotcp_init(struct mlx5e_priv *priv)
+{
+ struct mlx5e_nvmeotcp *nvmeotcp = NULL;
+ int ret = 0;
+
+ if (!(MLX5_CAP_GEN(priv->mdev, nvmeotcp) &&
+ MLX5_CAP_DEV_NVMEOTCP(priv->mdev, zerocopy) &&
+ MLX5_CAP_DEV_NVMEOTCP(priv->mdev, crc_rx) &&
+ MLX5_CAP_GEN(priv->mdev, cqe_128_always)))
+ return 0;
+
+ nvmeotcp = kzalloc(sizeof(*nvmeotcp), GFP_KERNEL);
+
+ if (!nvmeotcp)
+ return -ENOMEM;
+
+ ida_init(&nvmeotcp->queue_ids);
+ ret = rhashtable_init(&nvmeotcp->queue_hash, &rhash_queues);
+ if (ret)
+ goto err_ida;
+
+ /* report ULP DPP as supported, but don't enable it by default */
+ set_bit(ULP_DDP_CAP_NVME_TCP, nvmeotcp->ddp_caps.hw);
+ set_bit(ULP_DDP_CAP_NVME_TCP_DDGST_RX, nvmeotcp->ddp_caps.hw);
+ nvmeotcp->enabled = false;
+ priv->nvmeotcp = nvmeotcp;
+ return 0;
+
+err_ida:
+ ida_destroy(&nvmeotcp->queue_ids);
+ kfree(nvmeotcp);
+ return ret;
+}
+
+void mlx5e_nvmeotcp_cleanup(struct mlx5e_priv *priv)
+{
+ struct mlx5e_nvmeotcp *nvmeotcp = priv->nvmeotcp;
+
+ if (!nvmeotcp)
+ return;
+
+ rhashtable_destroy(&nvmeotcp->queue_hash);
+ ida_destroy(&nvmeotcp->queue_ids);
+ kfree(nvmeotcp);
+ priv->nvmeotcp = NULL;
+}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h
new file mode 100644
index 000000000000..29546992791f
--- /dev/null
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h
@@ -0,0 +1,120 @@
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/* Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. */
+#ifndef __MLX5E_NVMEOTCP_H__
+#define __MLX5E_NVMEOTCP_H__
+
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+
+#include <net/ulp_ddp.h>
+#include "en.h"
+#include "en/params.h"
+
+struct mlx5e_nvmeotcp_queue_entry {
+ struct mlx5e_nvmeotcp_queue *queue;
+ u32 sgl_length;
+ u32 klm_mkey;
+ struct scatterlist *sgl;
+ u32 ccid_gen;
+ u64 size;
+
+ /* for the ddp invalidate done callback */
+ void *ddp_ctx;
+ struct ulp_ddp_io *ddp;
+};
+
+struct mlx5e_nvmeotcp_queue_handler {
+ struct napi_struct napi;
+ struct mlx5e_cq *cq;
+};
+
+/**
+ * struct mlx5e_nvmeotcp_queue - mlx5 metadata for NVMEoTCP queue
+ * @ulp_ddp_ctx: Generic ulp ddp context
+ * @tir: Destination TIR created for NVMEoTCP offload
+ * @fh: Flow handle representing the 5-tuple steering for this flow
+ * @id: Flow tag ID used to identify this queue
+ * @size: NVMEoTCP queue depth
+ * @ccid_gen: Generation ID for the CCID, used to avoid conflicts in DDP
+ * @max_klms_per_wqe: Number of KLMs per DDP operation
+ * @hash: Hash table of queues mapped by @id
+ * @pda: Padding alignment
+ * @tag_buf_table_id: Tag buffer table for CCIDs
+ * @dgst: Digest supported (header and/or data)
+ * @sq: Send queue used for posting umrs
+ * @ref_count: Reference count for this structure
+ * @after_resync_cqe: Indicate if resync occurred
+ * @ccid_table: Table holding metadata for each CC (Command Capsule)
+ * @ccid: ID of the current CC
+ * @ccsglidx: Index within the scatter-gather list (SGL) of the current CC
+ * @ccoff: Offset within the current CC
+ * @ccoff_inner: Current offset within the @ccsglidx element
+ * @channel_ix: Channel IX for this nvmeotcp_queue
+ * @sk: The socket used by the NVMe-TCP queue
+ * @crc_rx: CRC Rx offload indication for this queue
+ * @priv: mlx5e netdev priv
+ * @static_params_done: Async completion structure for the initial umr mapping
+ * synchronization
+ * @sq_lock: Spin lock for the icosq
+ * @qh: Completion queue handler for processing umr completions
+ */
+struct mlx5e_nvmeotcp_queue {
+ struct ulp_ddp_ctx ulp_ddp_ctx;
+ struct mlx5e_tir tir;
+ struct mlx5_flow_handle *fh;
+ int id;
+ u32 size;
+ /* needed when the upper layer immediately reuses CCID + some packet loss happens */
+ u32 ccid_gen;
+ u32 max_klms_per_wqe;
+ struct rhash_head hash;
+ int pda;
+ u32 tag_buf_table_id;
+ u8 dgst;
+ struct mlx5e_icosq sq;
+
+ /* data-path section cache aligned */
+ refcount_t ref_count;
+ /* for MASK HW resync cqe */
+ bool after_resync_cqe;
+ struct mlx5e_nvmeotcp_queue_entry *ccid_table;
+ /* current ccid fields */
+ int ccid;
+ int ccsglidx;
+ off_t ccoff;
+ int ccoff_inner;
+
+ u32 channel_ix;
+ struct sock *sk;
+ u8 crc_rx:1;
+ /* for ddp invalidate flow */
+ struct mlx5e_priv *priv;
+ /* end of data-path section */
+
+ struct completion static_params_done;
+ /* spin lock for the ico sq, ULP can issue requests from multiple contexts */
+ spinlock_t sq_lock;
+ struct mlx5e_nvmeotcp_queue_handler qh;
+};
+
+struct mlx5e_nvmeotcp {
+ struct ida queue_ids;
+ struct rhashtable queue_hash;
+ struct ulp_ddp_dev_caps ddp_caps;
+ bool enabled;
+};
+
+int mlx5e_nvmeotcp_init(struct mlx5e_priv *priv);
+int set_ulp_ddp_nvme_tcp(struct net_device *netdev, bool enable);
+void mlx5e_nvmeotcp_cleanup(struct mlx5e_priv *priv);
+static inline void mlx5e_nvmeotcp_init_rx(struct mlx5e_priv *priv) {}
+void mlx5e_nvmeotcp_cleanup_rx(struct mlx5e_priv *priv);
+extern const struct ulp_ddp_dev_ops mlx5e_nvmeotcp_ops;
+#else
+
+static inline int mlx5e_nvmeotcp_init(struct mlx5e_priv *priv) { return 0; }
+static inline void mlx5e_nvmeotcp_cleanup(struct mlx5e_priv *priv) {}
+static inline int set_ulp_ddp_nvme_tcp(struct net_device *dev, bool en) { return -EOPNOTSUPP; }
+static inline void mlx5e_nvmeotcp_init_rx(struct mlx5e_priv *priv) {}
+static inline void mlx5e_nvmeotcp_cleanup_rx(struct mlx5e_priv *priv) {}
+#endif
+#endif /* __MLX5E_NVMEOTCP_H__ */
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c b/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c
index cc51ce16df14..fddadc896c26 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c
@@ -38,6 +38,7 @@
#include "en/ptp.h"
#include "lib/clock.h"
#include "en/fs_ethtool.h"
+#include "en_accel/nvmeotcp.h"
void mlx5e_ethtool_get_drvinfo(struct mlx5e_priv *priv,
struct ethtool_drvinfo *drvinfo)
@@ -1918,6 +1919,11 @@ int mlx5e_modify_rx_cqe_compression_locked(struct mlx5e_priv *priv, bool new_val
return -EINVAL;
}
+ if (priv->channels.params.nvmeotcp) {
+ netdev_warn(priv->netdev, "Can't set CQE compression after ULP DDP NVMe-TCP offload\n");
+ return -EINVAL;
+ }
+
new_params = priv->channels.params;
MLX5E_SET_PFLAG(&new_params, MLX5E_PFLAG_RX_CQE_COMPRESS, new_val);
if (rx_filter)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_fs.c b/drivers/net/ethernet/mellanox/mlx5/core/en_fs.c
index 777d311d44ef..853e30f221c8 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_fs.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_fs.c
@@ -62,7 +62,7 @@ struct mlx5e_flow_steering {
#ifdef CONFIG_MLX5_EN_ARFS
struct mlx5e_arfs_tables *arfs;
#endif
-#ifdef CONFIG_MLX5_EN_TLS
+#if defined(CONFIG_MLX5_EN_TLS) || defined(CONFIG_MLX5_EN_NVMEOTCP)
struct mlx5e_accel_fs_tcp *accel_tcp;
#endif
struct mlx5e_fs_udp *udp;
@@ -1557,7 +1557,7 @@ void mlx5e_fs_set_any(struct mlx5e_flow_steering *fs, struct mlx5e_fs_any *any)
fs->any = any;
}
-#ifdef CONFIG_MLX5_EN_TLS
+#if defined(CONFIG_MLX5_EN_TLS) || defined(CONFIG_MLX5_EN_NVMEOTCP)
struct mlx5e_accel_fs_tcp *mlx5e_fs_get_accel_tcp(struct mlx5e_flow_steering *fs)
{
return fs->accel_tcp;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index a467a011c599..54a3df934807 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -50,6 +50,7 @@
#include "en_accel/macsec.h"
#include "en_accel/en_accel.h"
#include "en_accel/ktls.h"
+#include "en_accel/nvmeotcp.h"
#include "lib/vxlan.h"
#include "lib/clock.h"
#include "en/port.h"
@@ -4247,6 +4248,13 @@ static netdev_features_t mlx5e_fix_features(struct net_device *netdev,
features &= ~NETIF_F_NETNS_LOCAL;
}
+ if (features & (NETIF_F_LRO | NETIF_F_GRO_HW)) {
+ if (params->nvmeotcp) {
+ netdev_warn(netdev, "Disabling HW-GRO/LRO, not supported after ULP DDP NVMe-TCP offload\n");
+ features &= ~(NETIF_F_LRO | NETIF_F_GRO_HW);
+ }
+ }
+
mutex_unlock(&priv->state_lock);
return features;
@@ -5005,6 +5013,9 @@ const struct net_device_ops mlx5e_netdev_ops = {
.ndo_has_offload_stats = mlx5e_has_offload_stats,
.ndo_get_offload_stats = mlx5e_get_offload_stats,
#endif
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+ .ulp_ddp_ops = &mlx5e_nvmeotcp_ops,
+#endif
};
static u32 mlx5e_choose_lro_timeout(struct mlx5_core_dev *mdev, u32 wanted_timeout)
@@ -5353,6 +5364,10 @@ static int mlx5e_nic_init(struct mlx5_core_dev *mdev,
if (err)
mlx5_core_err(mdev, "TLS initialization failed, %d\n", err);
+ err = mlx5e_nvmeotcp_init(priv);
+ if (err)
+ mlx5_core_err(mdev, "NVMEoTCP initialization failed, %d\n", err);
+
mlx5e_health_create_reporters(priv);
/* If netdev is already registered (e.g. move from uplink to nic profile),
@@ -5373,6 +5388,7 @@ static int mlx5e_nic_init(struct mlx5_core_dev *mdev,
static void mlx5e_nic_cleanup(struct mlx5e_priv *priv)
{
mlx5e_health_destroy_reporters(priv);
+ mlx5e_nvmeotcp_cleanup(priv);
mlx5e_ktls_cleanup(priv);
mlx5e_fs_cleanup(priv->fs);
debugfs_remove_recursive(priv->dfs_root);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/main.c b/drivers/net/ethernet/mellanox/mlx5/core/main.c
index c2593625c09a..2578482f9afa 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c
@@ -1744,6 +1744,7 @@ static const int types[] = {
MLX5_CAP_MACSEC,
MLX5_CAP_ADV_VIRTUALIZATION,
MLX5_CAP_CRYPTO,
+ MLX5_CAP_DEV_NVMEOTCP,
};
static void mlx5_hca_caps_free(struct mlx5_core_dev *dev)
--
2.34.1
^ permalink raw reply related [flat|nested] 44+ messages in thread
* [PATCH v24 14/20] net/mlx5e: TCP flow steering for nvme-tcp acceleration
2024-04-04 12:36 [PATCH v24 00/20] nvme-tcp receive offloads Aurelien Aptel
` (12 preceding siblings ...)
2024-04-04 12:37 ` [PATCH v24 13/20] net/mlx5e: NVMEoTCP, offload initialization Aurelien Aptel
@ 2024-04-04 12:37 ` Aurelien Aptel
2024-04-04 12:37 ` [PATCH v24 15/20] net/mlx5e: NVMEoTCP, use KLM UMRs for buffer registration Aurelien Aptel
` (6 subsequent siblings)
20 siblings, 0 replies; 44+ messages in thread
From: Aurelien Aptel @ 2024-04-04 12:37 UTC (permalink / raw)
To: linux-nvme, netdev, sagi, hch, kbusch, axboe, chaitanyak, davem, kuba
Cc: Boris Pismenny, aaptel, aurelien.aptel, smalin, malin1024,
ogerlitz, yorayz, galshalom, mgurtovoy
From: Boris Pismenny <borisp@nvidia.com>
Both nvme-tcp and tls acceleration require tcp flow steering.
Add reference counter to share TCP flow steering structure.
Signed-off-by: Boris Pismenny <borisp@nvidia.com>
Signed-off-by: Ben Ben-Ishay <benishay@nvidia.com>
Signed-off-by: Or Gerlitz <ogerlitz@nvidia.com>
Signed-off-by: Yoray Zack <yorayz@nvidia.com>
Signed-off-by: Aurelien Aptel <aaptel@nvidia.com>
Reviewed-by: Tariq Toukan <tariqt@nvidia.com>
---
.../ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c | 12 +++++++++++-
1 file changed, 11 insertions(+), 1 deletion(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c
index c7d191f66ad1..82a9e2a4f58b 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c
@@ -14,6 +14,7 @@ enum accel_fs_tcp_type {
struct mlx5e_accel_fs_tcp {
struct mlx5e_flow_table tables[ACCEL_FS_TCP_NUM_TYPES];
struct mlx5_flow_handle *default_rules[ACCEL_FS_TCP_NUM_TYPES];
+ refcount_t user_count;
};
static enum mlx5_traffic_types fs_accel2tt(enum accel_fs_tcp_type i)
@@ -361,6 +362,9 @@ void mlx5e_accel_fs_tcp_destroy(struct mlx5e_flow_steering *fs)
if (!accel_tcp)
return;
+ if (!refcount_dec_and_test(&accel_tcp->user_count))
+ return;
+
accel_fs_tcp_disable(fs);
for (i = 0; i < ACCEL_FS_TCP_NUM_TYPES; i++)
@@ -372,12 +376,17 @@ void mlx5e_accel_fs_tcp_destroy(struct mlx5e_flow_steering *fs)
int mlx5e_accel_fs_tcp_create(struct mlx5e_flow_steering *fs)
{
- struct mlx5e_accel_fs_tcp *accel_tcp;
+ struct mlx5e_accel_fs_tcp *accel_tcp = mlx5e_fs_get_accel_tcp(fs);
int i, err;
if (!MLX5_CAP_FLOWTABLE_NIC_RX(mlx5e_fs_get_mdev(fs), ft_field_support.outer_ip_version))
return -EOPNOTSUPP;
+ if (accel_tcp) {
+ refcount_inc(&accel_tcp->user_count);
+ return 0;
+ }
+
accel_tcp = kzalloc(sizeof(*accel_tcp), GFP_KERNEL);
if (!accel_tcp)
return -ENOMEM;
@@ -393,6 +402,7 @@ int mlx5e_accel_fs_tcp_create(struct mlx5e_flow_steering *fs)
if (err)
goto err_destroy_tables;
+ refcount_set(&accel_tcp->user_count, 1);
return 0;
err_destroy_tables:
--
2.34.1
^ permalink raw reply related [flat|nested] 44+ messages in thread
* [PATCH v24 15/20] net/mlx5e: NVMEoTCP, use KLM UMRs for buffer registration
2024-04-04 12:36 [PATCH v24 00/20] nvme-tcp receive offloads Aurelien Aptel
` (13 preceding siblings ...)
2024-04-04 12:37 ` [PATCH v24 14/20] net/mlx5e: TCP flow steering for nvme-tcp acceleration Aurelien Aptel
@ 2024-04-04 12:37 ` Aurelien Aptel
2024-04-04 12:37 ` [PATCH v24 16/20] net/mlx5e: NVMEoTCP, queue init/teardown Aurelien Aptel
` (5 subsequent siblings)
20 siblings, 0 replies; 44+ messages in thread
From: Aurelien Aptel @ 2024-04-04 12:37 UTC (permalink / raw)
To: linux-nvme, netdev, sagi, hch, kbusch, axboe, chaitanyak, davem, kuba
Cc: aaptel, aurelien.aptel, smalin, malin1024, ogerlitz, yorayz,
borisp, galshalom, mgurtovoy
From: Ben Ben-Ishay <benishay@nvidia.com>
NVMEoTCP offload uses buffer registration for ddp operation.
Every request comprises from SG list that might consist from elements
with multiple combination sizes, thus the appropriate way to perform
buffer registration is with KLM UMRs.
UMR stands for user-mode memory registration, it is a mechanism to alter
address translation properties of MKEY by posting WorkQueueElement
aka WQE on send queue.
MKEY stands for memory key, MKEY are used to describe a region in memory
that can be later used by HW.
KLM stands for {Key, Length, MemVa}, KLM_MKEY is indirect MKEY that
enables to map multiple memory spaces with different sizes in unified MKEY.
KLM UMR is a UMR that use to update a KLM_MKEY.
Nothing needs to be done on memory registration completion and this
notification is expensive so we add a wrapper to be able to ring the
doorbell without generating any.
Signed-off-by: Ben Ben-Ishay <benishay@nvidia.com>
Signed-off-by: Boris Pismenny <borisp@nvidia.com>
Signed-off-by: Or Gerlitz <ogerlitz@nvidia.com>
Signed-off-by: Yoray Zack <yorayz@nvidia.com>
Signed-off-by: Aurelien Aptel <aaptel@nvidia.com>
Reviewed-by: Tariq Toukan <tariqt@nvidia.com>
---
.../net/ethernet/mellanox/mlx5/core/en/txrx.h | 16 ++-
.../mellanox/mlx5/core/en_accel/nvmeotcp.c | 123 ++++++++++++++++++
.../mlx5/core/en_accel/nvmeotcp_utils.h | 25 ++++
.../net/ethernet/mellanox/mlx5/core/en_rx.c | 4 +
4 files changed, 165 insertions(+), 3 deletions(-)
create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_utils.h
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h b/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h
index cdd7fbf218ae..294fdcdb0f6c 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h
@@ -50,6 +50,9 @@ enum mlx5e_icosq_wqe_type {
MLX5E_ICOSQ_WQE_SET_PSV_TLS,
MLX5E_ICOSQ_WQE_GET_PSV_TLS,
#endif
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+ MLX5E_ICOSQ_WQE_UMR_NVMEOTCP,
+#endif
};
/* General */
@@ -256,10 +259,10 @@ static inline u16 mlx5e_icosq_get_next_pi(struct mlx5e_icosq *sq, u16 size)
}
static inline void
-mlx5e_notify_hw(struct mlx5_wq_cyc *wq, u16 pc, void __iomem *uar_map,
- struct mlx5_wqe_ctrl_seg *ctrl)
+__mlx5e_notify_hw(struct mlx5_wq_cyc *wq, u16 pc, void __iomem *uar_map,
+ struct mlx5_wqe_ctrl_seg *ctrl, u8 cq_update)
{
- ctrl->fm_ce_se |= MLX5_WQE_CTRL_CQ_UPDATE;
+ ctrl->fm_ce_se |= cq_update;
/* ensure wqe is visible to device before updating doorbell record */
dma_wmb();
@@ -273,6 +276,13 @@ mlx5e_notify_hw(struct mlx5_wq_cyc *wq, u16 pc, void __iomem *uar_map,
mlx5_write64((__be32 *)ctrl, uar_map);
}
+static inline void
+mlx5e_notify_hw(struct mlx5_wq_cyc *wq, u16 pc, void __iomem *uar_map,
+ struct mlx5_wqe_ctrl_seg *ctrl)
+{
+ __mlx5e_notify_hw(wq, pc, uar_map, ctrl, MLX5_WQE_CTRL_CQ_UPDATE);
+}
+
static inline void mlx5e_cq_arm(struct mlx5e_cq *cq)
{
struct mlx5_core_cq *mcq;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c
index 9965757873f9..c36bcc230455 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c
@@ -4,6 +4,7 @@
#include <linux/netdevice.h>
#include <linux/idr.h>
#include "en_accel/nvmeotcp.h"
+#include "en_accel/nvmeotcp_utils.h"
#include "en_accel/fs_tcp.h"
#include "en/txrx.h"
@@ -19,6 +20,120 @@ static const struct rhashtable_params rhash_queues = {
.max_size = MAX_NUM_NVMEOTCP_QUEUES,
};
+static void
+fill_nvmeotcp_klm_wqe(struct mlx5e_nvmeotcp_queue *queue, struct mlx5e_umr_wqe *wqe, u16 ccid,
+ u32 klm_entries, u16 klm_offset)
+{
+ struct scatterlist *sgl_mkey;
+ u32 lkey, i;
+
+ lkey = queue->priv->mdev->mlx5e_res.hw_objs.mkey;
+ for (i = 0; i < klm_entries; i++) {
+ sgl_mkey = &queue->ccid_table[ccid].sgl[i + klm_offset];
+ wqe->inline_klms[i].bcount = cpu_to_be32(sg_dma_len(sgl_mkey));
+ wqe->inline_klms[i].key = cpu_to_be32(lkey);
+ wqe->inline_klms[i].va = cpu_to_be64(sgl_mkey->dma_address);
+ }
+
+ for (; i < ALIGN(klm_entries, MLX5_UMR_KLM_NUM_ENTRIES_ALIGNMENT); i++) {
+ wqe->inline_klms[i].bcount = 0;
+ wqe->inline_klms[i].key = 0;
+ wqe->inline_klms[i].va = 0;
+ }
+}
+
+static void
+build_nvmeotcp_klm_umr(struct mlx5e_nvmeotcp_queue *queue, struct mlx5e_umr_wqe *wqe,
+ u16 ccid, int klm_entries, u32 klm_offset, u32 len,
+ enum wqe_type klm_type)
+{
+ u32 id = (klm_type == KLM_UMR) ? queue->ccid_table[ccid].klm_mkey :
+ (mlx5e_tir_get_tirn(&queue->tir) << MLX5_WQE_CTRL_TIR_TIS_INDEX_SHIFT);
+ u8 opc_mod = (klm_type == KLM_UMR) ? MLX5_CTRL_SEGMENT_OPC_MOD_UMR_UMR :
+ MLX5_OPC_MOD_TRANSPORT_TIR_STATIC_PARAMS;
+ u32 ds_cnt = MLX5E_KLM_UMR_DS_CNT(ALIGN(klm_entries, MLX5_UMR_KLM_NUM_ENTRIES_ALIGNMENT));
+ struct mlx5_wqe_umr_ctrl_seg *ucseg = &wqe->uctrl;
+ struct mlx5_wqe_ctrl_seg *cseg = &wqe->ctrl;
+ struct mlx5_mkey_seg *mkc = &wqe->mkc;
+ u32 sqn = queue->sq.sqn;
+ u16 pc = queue->sq.pc;
+
+ cseg->opmod_idx_opcode = cpu_to_be32((pc << MLX5_WQE_CTRL_WQE_INDEX_SHIFT) |
+ MLX5_OPCODE_UMR | (opc_mod) << 24);
+ cseg->qpn_ds = cpu_to_be32((sqn << MLX5_WQE_CTRL_QPN_SHIFT) | ds_cnt);
+ cseg->general_id = cpu_to_be32(id);
+
+ if (klm_type == KLM_UMR && !klm_offset) {
+ ucseg->mkey_mask = cpu_to_be64(MLX5_MKEY_MASK_XLT_OCT_SIZE |
+ MLX5_MKEY_MASK_LEN | MLX5_MKEY_MASK_FREE);
+ mkc->xlt_oct_size = cpu_to_be32(ALIGN(len, MLX5_UMR_KLM_NUM_ENTRIES_ALIGNMENT));
+ mkc->len = cpu_to_be64(queue->ccid_table[ccid].size);
+ }
+
+ ucseg->flags = MLX5_UMR_INLINE | MLX5_UMR_TRANSLATION_OFFSET_EN;
+ ucseg->xlt_octowords = cpu_to_be16(ALIGN(klm_entries, MLX5_UMR_KLM_NUM_ENTRIES_ALIGNMENT));
+ ucseg->xlt_offset = cpu_to_be16(klm_offset);
+ fill_nvmeotcp_klm_wqe(queue, wqe, ccid, klm_entries, klm_offset);
+}
+
+static void
+mlx5e_nvmeotcp_fill_wi(struct mlx5e_icosq *sq, u32 wqebbs, u16 pi)
+{
+ struct mlx5e_icosq_wqe_info *wi = &sq->db.wqe_info[pi];
+
+ memset(wi, 0, sizeof(*wi));
+
+ wi->num_wqebbs = wqebbs;
+ wi->wqe_type = MLX5E_ICOSQ_WQE_UMR_NVMEOTCP;
+}
+
+static u32
+post_klm_wqe(struct mlx5e_nvmeotcp_queue *queue,
+ enum wqe_type wqe_type,
+ u16 ccid,
+ u32 klm_length,
+ u32 klm_offset)
+{
+ struct mlx5e_icosq *sq = &queue->sq;
+ u32 wqebbs, cur_klm_entries;
+ struct mlx5e_umr_wqe *wqe;
+ u16 pi, wqe_sz;
+
+ cur_klm_entries = min_t(int, queue->max_klms_per_wqe, klm_length - klm_offset);
+ wqe_sz = MLX5E_KLM_UMR_WQE_SZ(ALIGN(cur_klm_entries, MLX5_UMR_KLM_NUM_ENTRIES_ALIGNMENT));
+ wqebbs = DIV_ROUND_UP(wqe_sz, MLX5_SEND_WQE_BB);
+ pi = mlx5e_icosq_get_next_pi(sq, wqebbs);
+ wqe = MLX5E_NVMEOTCP_FETCH_KLM_WQE(sq, pi);
+ mlx5e_nvmeotcp_fill_wi(sq, wqebbs, pi);
+ build_nvmeotcp_klm_umr(queue, wqe, ccid, cur_klm_entries, klm_offset,
+ klm_length, wqe_type);
+ sq->pc += wqebbs;
+ sq->doorbell_cseg = &wqe->ctrl;
+ return cur_klm_entries;
+}
+
+static void
+mlx5e_nvmeotcp_post_klm_wqe(struct mlx5e_nvmeotcp_queue *queue, enum wqe_type wqe_type,
+ u16 ccid, u32 klm_length)
+{
+ struct mlx5e_icosq *sq = &queue->sq;
+ u32 klm_offset = 0, wqes, i;
+
+ wqes = DIV_ROUND_UP(klm_length, queue->max_klms_per_wqe);
+
+ spin_lock_bh(&queue->sq_lock);
+
+ for (i = 0; i < wqes; i++)
+ klm_offset += post_klm_wqe(queue, wqe_type, ccid, klm_length, klm_offset);
+
+ if (wqe_type == KLM_UMR) /* not asking for completion on ddp_setup UMRs */
+ __mlx5e_notify_hw(&sq->wq, sq->pc, sq->uar_map, sq->doorbell_cseg, 0);
+ else
+ mlx5e_notify_hw(&sq->wq, sq->pc, sq->uar_map, sq->doorbell_cseg);
+
+ spin_unlock_bh(&queue->sq_lock);
+}
+
static int
mlx5e_nvmeotcp_offload_limits(struct net_device *netdev,
struct ulp_ddp_limits *limits)
@@ -45,6 +160,14 @@ mlx5e_nvmeotcp_ddp_setup(struct net_device *netdev,
struct sock *sk,
struct ulp_ddp_io *ddp)
{
+ struct mlx5e_nvmeotcp_queue *queue;
+
+ queue = container_of(ulp_ddp_get_ctx(sk),
+ struct mlx5e_nvmeotcp_queue, ulp_ddp_ctx);
+
+ /* Placeholder - map_sg and initializing the count */
+
+ mlx5e_nvmeotcp_post_klm_wqe(queue, KLM_UMR, ddp->command_id, 0);
return 0;
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_utils.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_utils.h
new file mode 100644
index 000000000000..6ef92679c5d0
--- /dev/null
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_utils.h
@@ -0,0 +1,25 @@
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/* Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. */
+#ifndef __MLX5E_NVMEOTCP_UTILS_H__
+#define __MLX5E_NVMEOTCP_UTILS_H__
+
+#include "en.h"
+
+#define MLX5E_NVMEOTCP_FETCH_KLM_WQE(sq, pi) \
+ ((struct mlx5e_umr_wqe *)\
+ mlx5e_fetch_wqe(&(sq)->wq, pi, sizeof(struct mlx5e_umr_wqe)))
+
+#define MLX5_CTRL_SEGMENT_OPC_MOD_UMR_NVMEOTCP_TIR_PROGRESS_PARAMS 0x4
+
+#define MLX5_CTRL_SEGMENT_OPC_MOD_UMR_TIR_PARAMS 0x2
+#define MLX5_CTRL_SEGMENT_OPC_MOD_UMR_UMR 0x0
+
+enum wqe_type {
+ KLM_UMR,
+ BSF_KLM_UMR,
+ SET_PSV_UMR,
+ BSF_UMR,
+ KLM_INV_UMR,
+};
+
+#endif /* __MLX5E_NVMEOTCP_UTILS_H__ */
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
index 780902f63823..ff8b648534db 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
@@ -1061,6 +1061,10 @@ int mlx5e_poll_ico_cq(struct mlx5e_cq *cq, int budget)
case MLX5E_ICOSQ_WQE_GET_PSV_TLS:
mlx5e_ktls_handle_get_psv_completion(wi, sq);
break;
+#endif
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+ case MLX5E_ICOSQ_WQE_UMR_NVMEOTCP:
+ break;
#endif
default:
netdev_WARN_ONCE(cq->netdev,
--
2.34.1
^ permalink raw reply related [flat|nested] 44+ messages in thread
* [PATCH v24 16/20] net/mlx5e: NVMEoTCP, queue init/teardown
2024-04-04 12:36 [PATCH v24 00/20] nvme-tcp receive offloads Aurelien Aptel
` (14 preceding siblings ...)
2024-04-04 12:37 ` [PATCH v24 15/20] net/mlx5e: NVMEoTCP, use KLM UMRs for buffer registration Aurelien Aptel
@ 2024-04-04 12:37 ` Aurelien Aptel
2024-04-04 12:37 ` [PATCH v24 17/20] net/mlx5e: NVMEoTCP, ddp setup and resync Aurelien Aptel
` (4 subsequent siblings)
20 siblings, 0 replies; 44+ messages in thread
From: Aurelien Aptel @ 2024-04-04 12:37 UTC (permalink / raw)
To: linux-nvme, netdev, sagi, hch, kbusch, axboe, chaitanyak, davem, kuba
Cc: aaptel, aurelien.aptel, smalin, malin1024, ogerlitz, yorayz,
borisp, galshalom, mgurtovoy
From: Ben Ben-Ishay <benishay@nvidia.com>
Adds the ddp ops of sk_add, sk_del and offload limits.
When nvme-tcp establishes new queue/connection, the sk_add op is called.
We allocate a hardware context to offload operations for this queue:
- use a steering rule based on the connection 5-tuple to mark packets
of this queue/connection with a flow-tag in their completion (cqe)
- use a dedicated TIR to identify the queue and maintain the HW context
- use a dedicated ICOSQ to maintain the HW context by UMR postings
- use a dedicated tag buffer for buffer registration
- maintain static and progress HW contexts by posting the proper WQEs.
When nvme-tcp teardowns a queue/connection, the sk_del op is called.
We teardown the queue and free the corresponding contexts.
The offload limits we advertise deal with the max SG supported.
[Re-enabled calling open/close icosq out of en_main.c]
Signed-off-by: Ben Ben-Ishay <benishay@nvidia.com>
Signed-off-by: Boris Pismenny <borisp@nvidia.com>
Signed-off-by: Or Gerlitz <ogerlitz@nvidia.com>
Signed-off-by: Yoray Zack <yorayz@nvidia.com>
Signed-off-by: Aurelien Aptel <aaptel@nvidia.com>
Reviewed-by: Tariq Toukan <tariqt@nvidia.com>
---
drivers/net/ethernet/mellanox/mlx5/core/en.h | 4 +
.../ethernet/mellanox/mlx5/core/en/rx_res.c | 28 +
.../ethernet/mellanox/mlx5/core/en/rx_res.h | 4 +
.../net/ethernet/mellanox/mlx5/core/en/tir.c | 15 +
.../net/ethernet/mellanox/mlx5/core/en/tir.h | 2 +
.../net/ethernet/mellanox/mlx5/core/en/txrx.h | 6 +
.../mellanox/mlx5/core/en_accel/nvmeotcp.c | 566 +++++++++++++++++-
.../mellanox/mlx5/core/en_accel/nvmeotcp.h | 4 +
.../mlx5/core/en_accel/nvmeotcp_utils.h | 41 ++
.../net/ethernet/mellanox/mlx5/core/en_main.c | 8 +-
.../net/ethernet/mellanox/mlx5/core/en_rx.c | 15 +-
11 files changed, 682 insertions(+), 11 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h
index 6f2a1d419f64..2220aeba7d50 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h
@@ -1046,6 +1046,10 @@ int mlx5e_create_rq(struct mlx5e_rq *rq, struct mlx5e_rq_param *param, u16 q_cou
void mlx5e_destroy_rq(struct mlx5e_rq *rq);
struct mlx5e_sq_param;
+int mlx5e_open_icosq(struct mlx5e_channel *c, struct mlx5e_params *params,
+ struct mlx5e_sq_param *param, struct mlx5e_icosq *sq,
+ work_func_t recover_work_func);
+void mlx5e_close_icosq(struct mlx5e_icosq *sq);
int mlx5e_open_xdpsq(struct mlx5e_channel *c, struct mlx5e_params *params,
struct mlx5e_sq_param *param, struct xsk_buff_pool *xsk_pool,
struct mlx5e_xdpsq *sq, bool is_redirect);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/rx_res.c b/drivers/net/ethernet/mellanox/mlx5/core/en/rx_res.c
index a86eade9a9e0..fc25cfd21026 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/rx_res.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/rx_res.c
@@ -675,6 +675,34 @@ struct mlx5e_rss_params_hash mlx5e_rx_res_get_current_hash(struct mlx5e_rx_res *
return mlx5e_rss_get_hash(res->rss[0]);
}
+int mlx5e_rx_res_nvmeotcp_tir_create(struct mlx5e_rx_res *res, unsigned int rxq, bool crc_rx,
+ u32 tag_buf_id, struct mlx5e_tir *tir)
+{
+ bool inner_ft_support = res->features & MLX5E_RX_RES_FEATURE_INNER_FT;
+ struct mlx5e_tir_builder *builder;
+ u32 rqtn;
+ int err;
+
+ builder = mlx5e_tir_builder_alloc(false);
+ if (!builder)
+ return -ENOMEM;
+
+ rqtn = mlx5e_rx_res_get_rqtn_direct(res, rxq);
+
+ mlx5e_tir_builder_build_rqt(builder, res->mdev->mlx5e_res.hw_objs.td.tdn, rqtn,
+ inner_ft_support);
+ mlx5e_tir_builder_build_direct(builder);
+ mlx5e_tir_builder_build_nvmeotcp(builder, crc_rx, tag_buf_id);
+ down_read(&res->pkt_merge_param_sem);
+ mlx5e_tir_builder_build_packet_merge(builder, &res->pkt_merge_param);
+ err = mlx5e_tir_init(tir, builder, res->mdev, false);
+ up_read(&res->pkt_merge_param_sem);
+
+ mlx5e_tir_builder_free(builder);
+
+ return err;
+}
+
int mlx5e_rx_res_tls_tir_create(struct mlx5e_rx_res *res, unsigned int rxq,
struct mlx5e_tir *tir)
{
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/rx_res.h b/drivers/net/ethernet/mellanox/mlx5/core/en/rx_res.h
index 7b1a9f0f1874..26accc96a46b 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/rx_res.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/rx_res.h
@@ -68,4 +68,8 @@ struct mlx5e_rss_params_hash mlx5e_rx_res_get_current_hash(struct mlx5e_rx_res *
/* Accel TIRs */
int mlx5e_rx_res_tls_tir_create(struct mlx5e_rx_res *res, unsigned int rxq,
struct mlx5e_tir *tir);
+
+int mlx5e_rx_res_nvmeotcp_tir_create(struct mlx5e_rx_res *res, unsigned int rxq, bool crc_rx,
+ u32 tag_buf_id, struct mlx5e_tir *tir);
+
#endif /* __MLX5_EN_RX_RES_H__ */
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/tir.c b/drivers/net/ethernet/mellanox/mlx5/core/en/tir.c
index d4239e3b3c88..8bdf74cbd8cd 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/tir.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/tir.c
@@ -143,6 +143,21 @@ void mlx5e_tir_builder_build_direct(struct mlx5e_tir_builder *builder)
MLX5_SET(tirc, tirc, rx_hash_fn, MLX5_RX_HASH_FN_INVERTED_XOR8);
}
+void mlx5e_tir_builder_build_nvmeotcp(struct mlx5e_tir_builder *builder, bool crc_rx,
+ u32 tag_buf_id)
+{
+ void *tirc = mlx5e_tir_builder_get_tirc(builder);
+
+ WARN_ON(builder->modify);
+
+ MLX5_SET(tirc, tirc, nvmeotcp_zero_copy_en, 1);
+ MLX5_SET(tirc, tirc, nvmeotcp_tag_buffer_table_id, tag_buf_id);
+ MLX5_SET(tirc, tirc, nvmeotcp_crc_en, !!crc_rx);
+ MLX5_SET(tirc, tirc, self_lb_block,
+ MLX5_TIRC_SELF_LB_BLOCK_BLOCK_UNICAST |
+ MLX5_TIRC_SELF_LB_BLOCK_BLOCK_MULTICAST);
+}
+
void mlx5e_tir_builder_build_tls(struct mlx5e_tir_builder *builder)
{
void *tirc = mlx5e_tir_builder_get_tirc(builder);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/tir.h b/drivers/net/ethernet/mellanox/mlx5/core/en/tir.h
index 857a84bcd53a..bdec6931444b 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/tir.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/tir.h
@@ -35,6 +35,8 @@ void mlx5e_tir_builder_build_rss(struct mlx5e_tir_builder *builder,
bool inner);
void mlx5e_tir_builder_build_direct(struct mlx5e_tir_builder *builder);
void mlx5e_tir_builder_build_tls(struct mlx5e_tir_builder *builder);
+void mlx5e_tir_builder_build_nvmeotcp(struct mlx5e_tir_builder *builder, bool crc_rx,
+ u32 tag_buf_id);
struct mlx5_core_dev;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h b/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h
index 294fdcdb0f6c..c87dca17d5c8 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h
@@ -52,6 +52,7 @@ enum mlx5e_icosq_wqe_type {
#endif
#ifdef CONFIG_MLX5_EN_NVMEOTCP
MLX5E_ICOSQ_WQE_UMR_NVMEOTCP,
+ MLX5E_ICOSQ_WQE_SET_PSV_NVMEOTCP,
#endif
};
@@ -224,6 +225,11 @@ struct mlx5e_icosq_wqe_info {
struct {
struct mlx5e_ktls_rx_resync_buf *buf;
} tls_get_params;
+#endif
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+ struct {
+ struct mlx5e_nvmeotcp_queue *queue;
+ } nvmeotcp_q;
#endif
};
};
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c
index c36bcc230455..2878ac0b7355 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c
@@ -3,6 +3,7 @@
#include <linux/netdevice.h>
#include <linux/idr.h>
+#include <linux/nvme-tcp.h>
#include "en_accel/nvmeotcp.h"
#include "en_accel/nvmeotcp_utils.h"
#include "en_accel/fs_tcp.h"
@@ -11,6 +12,11 @@
#define MAX_NUM_NVMEOTCP_QUEUES (4000)
#define MIN_NUM_NVMEOTCP_QUEUES (1)
+/* Max PDU data will be 512K */
+#define MLX5E_NVMEOTCP_MAX_SEGMENTS (128)
+#define MLX5E_NVMEOTCP_IO_THRESHOLD (32 * 1024)
+#define MLX5E_NVMEOTCP_FULL_CCID_RANGE (0)
+
static const struct rhashtable_params rhash_queues = {
.key_len = sizeof(int),
.key_offset = offsetof(struct mlx5e_nvmeotcp_queue, id),
@@ -20,6 +26,95 @@ static const struct rhashtable_params rhash_queues = {
.max_size = MAX_NUM_NVMEOTCP_QUEUES,
};
+static u32 mlx5e_get_max_sgl(struct mlx5_core_dev *mdev)
+{
+ return min_t(u32,
+ MLX5E_NVMEOTCP_MAX_SEGMENTS,
+ 1 << MLX5_CAP_GEN(mdev, log_max_klm_list_size));
+}
+
+static u32
+mlx5e_get_channel_ix_from_io_cpu(struct mlx5e_params *params, u32 io_cpu)
+{
+ int num_channels = params->num_channels;
+ u32 channel_ix = io_cpu;
+
+ if (channel_ix >= num_channels)
+ channel_ix = channel_ix % num_channels;
+
+ return channel_ix;
+}
+
+static
+int mlx5e_create_nvmeotcp_tag_buf_table(struct mlx5_core_dev *mdev,
+ struct mlx5e_nvmeotcp_queue *queue,
+ u8 log_table_size)
+{
+ u32 in[MLX5_ST_SZ_DW(create_nvmeotcp_tag_buf_table_in)] = {};
+ u32 out[MLX5_ST_SZ_DW(general_obj_out_cmd_hdr)];
+ u64 general_obj_types;
+ void *obj;
+ int err;
+
+ obj = MLX5_ADDR_OF(create_nvmeotcp_tag_buf_table_in, in,
+ nvmeotcp_tag_buf_table_obj);
+
+ general_obj_types = MLX5_CAP_GEN_64(mdev, general_obj_types);
+ if (!(general_obj_types &
+ MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_NVMEOTCP_TAG_BUFFER_TABLE))
+ return -EINVAL;
+
+ MLX5_SET(general_obj_in_cmd_hdr, in, opcode,
+ MLX5_CMD_OP_CREATE_GENERAL_OBJECT);
+ MLX5_SET(general_obj_in_cmd_hdr, in, obj_type,
+ MLX5_GENERAL_OBJECT_TYPES_NVMEOTCP_TAG_BUFFER_TABLE);
+ MLX5_SET(nvmeotcp_tag_buf_table_obj, obj,
+ log_tag_buffer_table_size, log_table_size);
+
+ err = mlx5_cmd_exec(mdev, in, sizeof(in), out, sizeof(out));
+ if (!err)
+ queue->tag_buf_table_id = MLX5_GET(general_obj_out_cmd_hdr,
+ out, obj_id);
+ return err;
+}
+
+static
+void mlx5_destroy_nvmeotcp_tag_buf_table(struct mlx5_core_dev *mdev, u32 uid)
+{
+ u32 in[MLX5_ST_SZ_DW(general_obj_in_cmd_hdr)] = {};
+ u32 out[MLX5_ST_SZ_DW(general_obj_out_cmd_hdr)];
+
+ MLX5_SET(general_obj_in_cmd_hdr, in, opcode,
+ MLX5_CMD_OP_DESTROY_GENERAL_OBJECT);
+ MLX5_SET(general_obj_in_cmd_hdr, in, obj_type,
+ MLX5_GENERAL_OBJECT_TYPES_NVMEOTCP_TAG_BUFFER_TABLE);
+ MLX5_SET(general_obj_in_cmd_hdr, in, obj_id, uid);
+
+ mlx5_cmd_exec(mdev, in, sizeof(in), out, sizeof(out));
+}
+
+static void
+fill_nvmeotcp_bsf_klm_wqe(struct mlx5e_nvmeotcp_queue *queue, struct mlx5e_umr_wqe *wqe,
+ u16 ccid, u32 klm_entries, u16 klm_offset)
+{
+ u32 i;
+
+ /* BSF_KLM_UMR is used to update the tag_buffer. To spare the
+ * need to update both mkc.length and tag_buffer[i].len in two
+ * different UMRs we initialize the tag_buffer[*].len to the
+ * maximum size of an entry so the HW check will pass and the
+ * validity of the MKEY len will be checked against the
+ * updated mkey context field.
+ */
+ for (i = 0; i < klm_entries; i++) {
+ u32 lkey = queue->ccid_table[i + klm_offset].klm_mkey;
+
+ wqe->inline_klms[i].bcount = cpu_to_be32(U32_MAX);
+ wqe->inline_klms[i].key = cpu_to_be32(lkey);
+ wqe->inline_klms[i].va = 0;
+ }
+}
+
static void
fill_nvmeotcp_klm_wqe(struct mlx5e_nvmeotcp_queue *queue, struct mlx5e_umr_wqe *wqe, u16 ccid,
u32 klm_entries, u16 klm_offset)
@@ -73,18 +168,149 @@ build_nvmeotcp_klm_umr(struct mlx5e_nvmeotcp_queue *queue, struct mlx5e_umr_wqe
ucseg->flags = MLX5_UMR_INLINE | MLX5_UMR_TRANSLATION_OFFSET_EN;
ucseg->xlt_octowords = cpu_to_be16(ALIGN(klm_entries, MLX5_UMR_KLM_NUM_ENTRIES_ALIGNMENT));
ucseg->xlt_offset = cpu_to_be16(klm_offset);
- fill_nvmeotcp_klm_wqe(queue, wqe, ccid, klm_entries, klm_offset);
+ if (klm_type == BSF_KLM_UMR)
+ fill_nvmeotcp_bsf_klm_wqe(queue, wqe, ccid, klm_entries, klm_offset);
+ else
+ fill_nvmeotcp_klm_wqe(queue, wqe, ccid, klm_entries, klm_offset);
+}
+
+static void
+fill_nvmeotcp_progress_params(struct mlx5e_nvmeotcp_queue *queue,
+ struct mlx5_seg_nvmeotcp_progress_params *params,
+ u32 seq)
+{
+ void *ctx = params->ctx;
+
+ params->tir_num = cpu_to_be32(mlx5e_tir_get_tirn(&queue->tir));
+
+ MLX5_SET(nvmeotcp_progress_params, ctx, next_pdu_tcp_sn, seq);
+ MLX5_SET(nvmeotcp_progress_params, ctx, pdu_tracker_state,
+ MLX5E_NVMEOTCP_PROGRESS_PARAMS_PDU_TRACKER_STATE_START);
+}
+
+void
+build_nvmeotcp_progress_params(struct mlx5e_nvmeotcp_queue *queue,
+ struct mlx5e_set_nvmeotcp_progress_params_wqe *wqe,
+ u32 seq)
+{
+ struct mlx5_wqe_ctrl_seg *cseg = &wqe->ctrl;
+ u32 sqn = queue->sq.sqn;
+ u16 pc = queue->sq.pc;
+ u8 opc_mod;
+
+ memset(wqe, 0, MLX5E_NVMEOTCP_PROGRESS_PARAMS_WQE_SZ);
+ opc_mod = MLX5_CTRL_SEGMENT_OPC_MOD_UMR_NVMEOTCP_TIR_PROGRESS_PARAMS;
+ cseg->opmod_idx_opcode = cpu_to_be32((pc << MLX5_WQE_CTRL_WQE_INDEX_SHIFT) |
+ MLX5_OPCODE_SET_PSV | (opc_mod << 24));
+ cseg->qpn_ds = cpu_to_be32((sqn << MLX5_WQE_CTRL_QPN_SHIFT) |
+ PROGRESS_PARAMS_DS_CNT);
+ fill_nvmeotcp_progress_params(queue, &wqe->params, seq);
+}
+
+static void
+fill_nvmeotcp_static_params(struct mlx5e_nvmeotcp_queue *queue,
+ struct mlx5_wqe_transport_static_params_seg *params,
+ u32 resync_seq, bool ddgst_offload_en)
+{
+ void *ctx = params->ctx;
+
+ MLX5_SET(transport_static_params, ctx, const_1, 1);
+ MLX5_SET(transport_static_params, ctx, const_2, 2);
+ MLX5_SET(transport_static_params, ctx, acc_type,
+ MLX5_TRANSPORT_STATIC_PARAMS_ACC_TYPE_NVMETCP);
+ MLX5_SET(transport_static_params, ctx, nvme_resync_tcp_sn, resync_seq);
+ MLX5_SET(transport_static_params, ctx, pda, queue->pda);
+ MLX5_SET(transport_static_params, ctx, ddgst_en,
+ !!(queue->dgst & NVME_TCP_DATA_DIGEST_ENABLE));
+ MLX5_SET(transport_static_params, ctx, ddgst_offload_en, ddgst_offload_en);
+ MLX5_SET(transport_static_params, ctx, hddgst_en,
+ !!(queue->dgst & NVME_TCP_HDR_DIGEST_ENABLE));
+ MLX5_SET(transport_static_params, ctx, hdgst_offload_en, 0);
+ MLX5_SET(transport_static_params, ctx, ti,
+ MLX5_TRANSPORT_STATIC_PARAMS_TI_INITIATOR);
+ MLX5_SET(transport_static_params, ctx, cccid_ttag, 1);
+ MLX5_SET(transport_static_params, ctx, zero_copy_en, 1);
+}
+
+void
+build_nvmeotcp_static_params(struct mlx5e_nvmeotcp_queue *queue,
+ struct mlx5e_set_transport_static_params_wqe *wqe,
+ u32 resync_seq, bool crc_rx)
+{
+ u8 opc_mod = MLX5_OPC_MOD_TRANSPORT_TIR_STATIC_PARAMS;
+ struct mlx5_wqe_umr_ctrl_seg *ucseg = &wqe->uctrl;
+ struct mlx5_wqe_ctrl_seg *cseg = &wqe->ctrl;
+ u32 sqn = queue->sq.sqn;
+ u16 pc = queue->sq.pc;
+
+ memset(wqe, 0, MLX5E_TRANSPORT_STATIC_PARAMS_WQE_SZ);
+
+ cseg->opmod_idx_opcode = cpu_to_be32((pc << MLX5_WQE_CTRL_WQE_INDEX_SHIFT) |
+ MLX5_OPCODE_UMR | (opc_mod) << 24);
+ cseg->qpn_ds = cpu_to_be32((sqn << MLX5_WQE_CTRL_QPN_SHIFT) |
+ MLX5E_TRANSPORT_STATIC_PARAMS_DS_CNT);
+ cseg->imm = cpu_to_be32(mlx5e_tir_get_tirn(&queue->tir)
+ << MLX5_WQE_CTRL_TIR_TIS_INDEX_SHIFT);
+
+ ucseg->flags = MLX5_UMR_INLINE;
+ ucseg->bsf_octowords = cpu_to_be16(MLX5E_TRANSPORT_STATIC_PARAMS_OCTWORD_SIZE);
+ fill_nvmeotcp_static_params(queue, &wqe->params, resync_seq, crc_rx);
}
static void
-mlx5e_nvmeotcp_fill_wi(struct mlx5e_icosq *sq, u32 wqebbs, u16 pi)
+mlx5e_nvmeotcp_fill_wi(struct mlx5e_nvmeotcp_queue *nvmeotcp_queue,
+ struct mlx5e_icosq *sq, u32 wqebbs, u16 pi,
+ enum wqe_type type)
{
struct mlx5e_icosq_wqe_info *wi = &sq->db.wqe_info[pi];
memset(wi, 0, sizeof(*wi));
wi->num_wqebbs = wqebbs;
- wi->wqe_type = MLX5E_ICOSQ_WQE_UMR_NVMEOTCP;
+ switch (type) {
+ case SET_PSV_UMR:
+ wi->wqe_type = MLX5E_ICOSQ_WQE_SET_PSV_NVMEOTCP;
+ wi->nvmeotcp_q.queue = nvmeotcp_queue;
+ break;
+ default:
+ /* cases where no further action is required upon completion, such as ddp setup */
+ wi->wqe_type = MLX5E_ICOSQ_WQE_UMR_NVMEOTCP;
+ break;
+ }
+}
+
+static void
+mlx5e_nvmeotcp_rx_post_static_params_wqe(struct mlx5e_nvmeotcp_queue *queue, u32 resync_seq)
+{
+ struct mlx5e_set_transport_static_params_wqe *wqe;
+ struct mlx5e_icosq *sq = &queue->sq;
+ u16 pi, wqebbs;
+
+ spin_lock_bh(&queue->sq_lock);
+ wqebbs = MLX5E_TRANSPORT_SET_STATIC_PARAMS_WQEBBS;
+ pi = mlx5e_icosq_get_next_pi(sq, wqebbs);
+ wqe = MLX5E_TRANSPORT_FETCH_SET_STATIC_PARAMS_WQE(sq, pi);
+ mlx5e_nvmeotcp_fill_wi(NULL, sq, wqebbs, pi, BSF_UMR);
+ build_nvmeotcp_static_params(queue, wqe, resync_seq, queue->crc_rx);
+ sq->pc += wqebbs;
+ mlx5e_notify_hw(&sq->wq, sq->pc, sq->uar_map, &wqe->ctrl);
+ spin_unlock_bh(&queue->sq_lock);
+}
+
+static void
+mlx5e_nvmeotcp_rx_post_progress_params_wqe(struct mlx5e_nvmeotcp_queue *queue, u32 seq)
+{
+ struct mlx5e_set_nvmeotcp_progress_params_wqe *wqe;
+ struct mlx5e_icosq *sq = &queue->sq;
+ u16 pi, wqebbs;
+
+ wqebbs = MLX5E_NVMEOTCP_PROGRESS_PARAMS_WQEBBS;
+ pi = mlx5e_icosq_get_next_pi(sq, wqebbs);
+ wqe = MLX5E_NVMEOTCP_FETCH_PROGRESS_PARAMS_WQE(sq, pi);
+ mlx5e_nvmeotcp_fill_wi(queue, sq, wqebbs, pi, SET_PSV_UMR);
+ build_nvmeotcp_progress_params(queue, wqe, seq);
+ sq->pc += wqebbs;
+ mlx5e_notify_hw(&sq->wq, sq->pc, sq->uar_map, &wqe->ctrl);
}
static u32
@@ -104,7 +330,7 @@ post_klm_wqe(struct mlx5e_nvmeotcp_queue *queue,
wqebbs = DIV_ROUND_UP(wqe_sz, MLX5_SEND_WQE_BB);
pi = mlx5e_icosq_get_next_pi(sq, wqebbs);
wqe = MLX5E_NVMEOTCP_FETCH_KLM_WQE(sq, pi);
- mlx5e_nvmeotcp_fill_wi(sq, wqebbs, pi);
+ mlx5e_nvmeotcp_fill_wi(queue, sq, wqebbs, pi, wqe_type);
build_nvmeotcp_klm_umr(queue, wqe, ccid, cur_klm_entries, klm_offset,
klm_length, wqe_type);
sq->pc += wqebbs;
@@ -134,25 +360,328 @@ mlx5e_nvmeotcp_post_klm_wqe(struct mlx5e_nvmeotcp_queue *queue, enum wqe_type wq
spin_unlock_bh(&queue->sq_lock);
}
+static int mlx5e_create_nvmeotcp_mkey(struct mlx5_core_dev *mdev, u8 access_mode,
+ u32 translation_octword_size, u32 *mkey)
+{
+ int inlen = MLX5_ST_SZ_BYTES(create_mkey_in);
+ void *mkc;
+ u32 *in;
+ int err;
+
+ in = kvzalloc(inlen, GFP_KERNEL);
+ if (!in)
+ return -ENOMEM;
+
+ mkc = MLX5_ADDR_OF(create_mkey_in, in, memory_key_mkey_entry);
+ MLX5_SET(mkc, mkc, free, 1);
+ MLX5_SET(mkc, mkc, translations_octword_size, translation_octword_size);
+ MLX5_SET(mkc, mkc, umr_en, 1);
+ MLX5_SET(mkc, mkc, lw, 1);
+ MLX5_SET(mkc, mkc, lr, 1);
+ MLX5_SET(mkc, mkc, access_mode_1_0, access_mode);
+
+ MLX5_SET(mkc, mkc, qpn, 0xffffff);
+ MLX5_SET(mkc, mkc, pd, mdev->mlx5e_res.hw_objs.pdn);
+
+ err = mlx5_core_create_mkey(mdev, mkey, in, inlen);
+
+ kvfree(in);
+ return err;
+}
+
static int
mlx5e_nvmeotcp_offload_limits(struct net_device *netdev,
struct ulp_ddp_limits *limits)
{
+ struct mlx5e_priv *priv = netdev_priv(netdev);
+ struct mlx5_core_dev *mdev = priv->mdev;
+
+ if (limits->type != ULP_DDP_NVME)
+ return -EOPNOTSUPP;
+
+ limits->max_ddp_sgl_len = mlx5e_get_max_sgl(mdev);
+ limits->io_threshold = MLX5E_NVMEOTCP_IO_THRESHOLD;
+ limits->tls = false;
+ limits->nvmeotcp.full_ccid_range = MLX5E_NVMEOTCP_FULL_CCID_RANGE;
return 0;
}
+static int mlx5e_nvmeotcp_queue_handler_poll(struct napi_struct *napi, int budget)
+{
+ struct mlx5e_nvmeotcp_queue_handler *qh;
+ int work_done;
+
+ qh = container_of(napi, struct mlx5e_nvmeotcp_queue_handler, napi);
+
+ work_done = mlx5e_poll_ico_cq(qh->cq, budget);
+
+ if (work_done == budget || !napi_complete_done(napi, work_done))
+ goto out;
+
+ mlx5e_cq_arm(qh->cq);
+
+out:
+ return work_done;
+}
+
+static void
+mlx5e_nvmeotcp_destroy_icosq(struct mlx5e_icosq *sq)
+{
+ mlx5e_close_icosq(sq);
+ mlx5e_close_cq(&sq->cq);
+}
+
+static void mlx5e_nvmeotcp_icosq_err_cqe_work(struct work_struct *recover_work)
+{
+ struct mlx5e_icosq *sq = container_of(recover_work, struct mlx5e_icosq, recover_work);
+
+ /* Not implemented yet. */
+
+ netdev_warn(sq->channel->netdev, "nvmeotcp icosq recovery is not implemented\n");
+}
+
+static int
+mlx5e_nvmeotcp_build_icosq(struct mlx5e_nvmeotcp_queue *queue, struct mlx5e_priv *priv, int io_cpu)
+{
+ u16 max_sgl, max_klm_per_wqe, max_umr_per_ccid, sgl_rest, wqebbs_rest;
+ struct mlx5e_channel *c = priv->channels.c[queue->channel_ix];
+ struct mlx5e_sq_param icosq_param = {};
+ struct mlx5e_create_cq_param ccp = {};
+ struct dim_cq_moder icocq_moder = {};
+ struct mlx5e_icosq *icosq;
+ int err = -ENOMEM;
+ u16 log_icosq_sz;
+ u32 max_wqebbs;
+
+ icosq = &queue->sq;
+ max_sgl = mlx5e_get_max_sgl(priv->mdev);
+ max_klm_per_wqe = queue->max_klms_per_wqe;
+ max_umr_per_ccid = max_sgl / max_klm_per_wqe;
+ sgl_rest = max_sgl % max_klm_per_wqe;
+ wqebbs_rest = sgl_rest ? MLX5E_KLM_UMR_WQEBBS(sgl_rest) : 0;
+ max_wqebbs = (MLX5E_KLM_UMR_WQEBBS(max_klm_per_wqe) *
+ max_umr_per_ccid + wqebbs_rest) * queue->size;
+ log_icosq_sz = order_base_2(max_wqebbs);
+
+ mlx5e_build_icosq_param(priv->mdev, log_icosq_sz, &icosq_param);
+ ccp.napi = &queue->qh.napi;
+ ccp.ch_stats = &priv->channel_stats[queue->channel_ix]->ch;
+ ccp.node = cpu_to_node(io_cpu);
+ ccp.ix = queue->channel_ix;
+
+ err = mlx5e_open_cq(priv->mdev, icocq_moder, &icosq_param.cqp, &ccp, &icosq->cq);
+ if (err)
+ goto err_nvmeotcp_sq;
+ err = mlx5e_open_icosq(c, &priv->channels.params, &icosq_param, icosq,
+ mlx5e_nvmeotcp_icosq_err_cqe_work);
+ if (err)
+ goto close_cq;
+
+ spin_lock_init(&queue->sq_lock);
+ return 0;
+
+close_cq:
+ mlx5e_close_cq(&icosq->cq);
+err_nvmeotcp_sq:
+ return err;
+}
+
+static void
+mlx5e_nvmeotcp_destroy_rx(struct mlx5e_priv *priv, struct mlx5e_nvmeotcp_queue *queue,
+ struct mlx5_core_dev *mdev)
+{
+ int i;
+
+ mlx5e_accel_fs_del_sk(queue->fh);
+
+ for (i = 0; i < queue->size; i++)
+ mlx5_core_destroy_mkey(mdev, queue->ccid_table[i].klm_mkey);
+
+ mlx5e_tir_destroy(&queue->tir);
+ mlx5_destroy_nvmeotcp_tag_buf_table(mdev, queue->tag_buf_table_id);
+
+ mlx5e_deactivate_icosq(&queue->sq);
+ napi_disable(&queue->qh.napi);
+ mlx5e_nvmeotcp_destroy_icosq(&queue->sq);
+ netif_napi_del(&queue->qh.napi);
+}
+
+static int
+mlx5e_nvmeotcp_queue_rx_init(struct mlx5e_nvmeotcp_queue *queue,
+ struct ulp_ddp_config *config,
+ struct net_device *netdev)
+{
+ struct nvme_tcp_ddp_config *nvme_config = &config->nvmeotcp;
+ u8 log_queue_size = order_base_2(nvme_config->queue_size);
+ struct mlx5e_priv *priv = netdev_priv(netdev);
+ struct mlx5_core_dev *mdev = priv->mdev;
+ struct sock *sk = queue->sk;
+ int err, max_sgls, i;
+
+ if (nvme_config->queue_size >
+ BIT(MLX5_CAP_DEV_NVMEOTCP(mdev, log_max_nvmeotcp_tag_buffer_size)))
+ return -EINVAL;
+
+ err = mlx5e_create_nvmeotcp_tag_buf_table(mdev, queue, log_queue_size);
+ if (err)
+ return err;
+
+ queue->qh.cq = &queue->sq.cq;
+ netif_napi_add(priv->netdev, &queue->qh.napi, mlx5e_nvmeotcp_queue_handler_poll);
+
+ mutex_lock(&priv->state_lock);
+ err = mlx5e_nvmeotcp_build_icosq(queue, priv, config->io_cpu);
+ mutex_unlock(&priv->state_lock);
+ if (err)
+ goto del_napi;
+
+ napi_enable(&queue->qh.napi);
+ mlx5e_activate_icosq(&queue->sq);
+
+ /* initializes queue->tir */
+ err = mlx5e_rx_res_nvmeotcp_tir_create(priv->rx_res, queue->channel_ix, queue->crc_rx,
+ queue->tag_buf_table_id, &queue->tir);
+ if (err)
+ goto destroy_icosq;
+
+ mlx5e_nvmeotcp_rx_post_static_params_wqe(queue, 0);
+ mlx5e_nvmeotcp_rx_post_progress_params_wqe(queue, tcp_sk(sk)->copied_seq);
+
+ queue->ccid_table = kcalloc(queue->size, sizeof(struct mlx5e_nvmeotcp_queue_entry),
+ GFP_KERNEL);
+ if (!queue->ccid_table) {
+ err = -ENOMEM;
+ goto destroy_tir;
+ }
+
+ max_sgls = mlx5e_get_max_sgl(mdev);
+ for (i = 0; i < queue->size; i++) {
+ err = mlx5e_create_nvmeotcp_mkey(mdev, MLX5_MKC_ACCESS_MODE_KLMS, max_sgls,
+ &queue->ccid_table[i].klm_mkey);
+ if (err)
+ goto free_ccid_table;
+ }
+
+ mlx5e_nvmeotcp_post_klm_wqe(queue, BSF_KLM_UMR, 0, queue->size);
+
+ if (!(WARN_ON(!wait_for_completion_timeout(&queue->static_params_done,
+ msecs_to_jiffies(3000)))))
+ queue->fh = mlx5e_accel_fs_add_sk(priv->fs, sk, mlx5e_tir_get_tirn(&queue->tir),
+ queue->id);
+
+ if (IS_ERR_OR_NULL(queue->fh)) {
+ err = -EINVAL;
+ goto destroy_mkeys;
+ }
+
+ return 0;
+
+destroy_mkeys:
+ while ((i--))
+ mlx5_core_destroy_mkey(mdev, queue->ccid_table[i].klm_mkey);
+free_ccid_table:
+ kfree(queue->ccid_table);
+destroy_tir:
+ mlx5e_tir_destroy(&queue->tir);
+destroy_icosq:
+ mlx5e_deactivate_icosq(&queue->sq);
+ napi_disable(&queue->qh.napi);
+ mlx5e_nvmeotcp_destroy_icosq(&queue->sq);
+del_napi:
+ netif_napi_del(&queue->qh.napi);
+ mlx5_destroy_nvmeotcp_tag_buf_table(mdev, queue->tag_buf_table_id);
+
+ return err;
+}
+
static int
mlx5e_nvmeotcp_queue_init(struct net_device *netdev,
struct sock *sk,
- struct ulp_ddp_config *tconfig)
+ struct ulp_ddp_config *config)
{
+ struct nvme_tcp_ddp_config *nvme_config = &config->nvmeotcp;
+ struct mlx5e_priv *priv = netdev_priv(netdev);
+ struct mlx5_core_dev *mdev = priv->mdev;
+ struct mlx5e_nvmeotcp_queue *queue;
+ int queue_id, err;
+
+ if (config->type != ULP_DDP_NVME) {
+ err = -EOPNOTSUPP;
+ goto out;
+ }
+
+ queue = kzalloc(sizeof(*queue), GFP_KERNEL);
+ if (!queue) {
+ err = -ENOMEM;
+ goto out;
+ }
+
+ queue_id = ida_simple_get(&priv->nvmeotcp->queue_ids,
+ MIN_NUM_NVMEOTCP_QUEUES, MAX_NUM_NVMEOTCP_QUEUES,
+ GFP_KERNEL);
+ if (queue_id < 0) {
+ err = -ENOSPC;
+ goto free_queue;
+ }
+
+ queue->crc_rx = !!(nvme_config->dgst & NVME_TCP_DATA_DIGEST_ENABLE);
+ queue->ulp_ddp_ctx.type = ULP_DDP_NVME;
+ queue->sk = sk;
+ queue->id = queue_id;
+ queue->dgst = nvme_config->dgst;
+ queue->pda = nvme_config->cpda;
+ queue->channel_ix = mlx5e_get_channel_ix_from_io_cpu(&priv->channels.params,
+ config->io_cpu);
+ queue->size = nvme_config->queue_size;
+ queue->max_klms_per_wqe = MLX5E_MAX_KLM_PER_WQE(mdev);
+ queue->priv = priv;
+ init_completion(&queue->static_params_done);
+
+ err = mlx5e_nvmeotcp_queue_rx_init(queue, config, netdev);
+ if (err)
+ goto remove_queue_id;
+
+ err = rhashtable_insert_fast(&priv->nvmeotcp->queue_hash, &queue->hash,
+ rhash_queues);
+ if (err)
+ goto destroy_rx;
+
+ write_lock_bh(&sk->sk_callback_lock);
+ ulp_ddp_set_ctx(sk, queue);
+ write_unlock_bh(&sk->sk_callback_lock);
+ refcount_set(&queue->ref_count, 1);
return 0;
+
+destroy_rx:
+ mlx5e_nvmeotcp_destroy_rx(priv, queue, mdev);
+remove_queue_id:
+ ida_simple_remove(&priv->nvmeotcp->queue_ids, queue_id);
+free_queue:
+ kfree(queue);
+out:
+ return err;
}
static void
mlx5e_nvmeotcp_queue_teardown(struct net_device *netdev,
struct sock *sk)
{
+ struct mlx5e_priv *priv = netdev_priv(netdev);
+ struct mlx5_core_dev *mdev = priv->mdev;
+ struct mlx5e_nvmeotcp_queue *queue;
+
+ queue = container_of(ulp_ddp_get_ctx(sk), struct mlx5e_nvmeotcp_queue, ulp_ddp_ctx);
+
+ WARN_ON(refcount_read(&queue->ref_count) != 1);
+ mlx5e_nvmeotcp_destroy_rx(priv, queue, mdev);
+
+ rhashtable_remove_fast(&priv->nvmeotcp->queue_hash, &queue->hash,
+ rhash_queues);
+ ida_simple_remove(&priv->nvmeotcp->queue_ids, queue->id);
+ write_lock_bh(&sk->sk_callback_lock);
+ ulp_ddp_set_ctx(sk, NULL);
+ write_unlock_bh(&sk->sk_callback_lock);
+ mlx5e_nvmeotcp_put_queue(queue);
}
static int
@@ -171,6 +700,13 @@ mlx5e_nvmeotcp_ddp_setup(struct net_device *netdev,
return 0;
}
+void mlx5e_nvmeotcp_ctx_complete(struct mlx5e_icosq_wqe_info *wi)
+{
+ struct mlx5e_nvmeotcp_queue *queue = wi->nvmeotcp_q.queue;
+
+ complete(&queue->static_params_done);
+}
+
static void
mlx5e_nvmeotcp_ddp_teardown(struct net_device *netdev,
struct sock *sk,
@@ -185,6 +721,26 @@ mlx5e_nvmeotcp_ddp_resync(struct net_device *netdev,
{
}
+struct mlx5e_nvmeotcp_queue *
+mlx5e_nvmeotcp_get_queue(struct mlx5e_nvmeotcp *nvmeotcp, int id)
+{
+ struct mlx5e_nvmeotcp_queue *queue;
+
+ queue = rhashtable_lookup_fast(&nvmeotcp->queue_hash,
+ &id, rhash_queues);
+ if (!IS_ERR_OR_NULL(queue))
+ refcount_inc(&queue->ref_count);
+ return queue;
+}
+
+void mlx5e_nvmeotcp_put_queue(struct mlx5e_nvmeotcp_queue *queue)
+{
+ if (refcount_dec_and_test(&queue->ref_count)) {
+ kfree(queue->ccid_table);
+ kfree(queue);
+ }
+}
+
int set_ulp_ddp_nvme_tcp(struct net_device *netdev, bool enable)
{
struct mlx5e_priv *priv = netdev_priv(netdev);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h
index 29546992791f..8b29f3fde7f2 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h
@@ -106,6 +106,10 @@ struct mlx5e_nvmeotcp {
int mlx5e_nvmeotcp_init(struct mlx5e_priv *priv);
int set_ulp_ddp_nvme_tcp(struct net_device *netdev, bool enable);
void mlx5e_nvmeotcp_cleanup(struct mlx5e_priv *priv);
+struct mlx5e_nvmeotcp_queue *
+mlx5e_nvmeotcp_get_queue(struct mlx5e_nvmeotcp *nvmeotcp, int id);
+void mlx5e_nvmeotcp_put_queue(struct mlx5e_nvmeotcp_queue *queue);
+void mlx5e_nvmeotcp_ctx_complete(struct mlx5e_icosq_wqe_info *wi);
static inline void mlx5e_nvmeotcp_init_rx(struct mlx5e_priv *priv) {}
void mlx5e_nvmeotcp_cleanup_rx(struct mlx5e_priv *priv);
extern const struct ulp_ddp_dev_ops mlx5e_nvmeotcp_ops;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_utils.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_utils.h
index 6ef92679c5d0..fdb194c30e3b 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_utils.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_utils.h
@@ -4,6 +4,35 @@
#define __MLX5E_NVMEOTCP_UTILS_H__
#include "en.h"
+#include "en_accel/nvmeotcp.h"
+#include "en_accel/common_utils.h"
+
+enum {
+ MLX5E_NVMEOTCP_PROGRESS_PARAMS_PDU_TRACKER_STATE_START = 0,
+ MLX5E_NVMEOTCP_PROGRESS_PARAMS_PDU_TRACKER_STATE_TRACKING = 1,
+ MLX5E_NVMEOTCP_PROGRESS_PARAMS_PDU_TRACKER_STATE_SEARCHING = 2,
+};
+
+struct mlx5_seg_nvmeotcp_progress_params {
+ __be32 tir_num;
+ u8 ctx[MLX5_ST_SZ_BYTES(nvmeotcp_progress_params)];
+};
+
+struct mlx5e_set_nvmeotcp_progress_params_wqe {
+ struct mlx5_wqe_ctrl_seg ctrl;
+ struct mlx5_seg_nvmeotcp_progress_params params;
+};
+
+/* macros for wqe handling */
+#define MLX5E_NVMEOTCP_PROGRESS_PARAMS_WQE_SZ \
+ (sizeof(struct mlx5e_set_nvmeotcp_progress_params_wqe))
+
+#define MLX5E_NVMEOTCP_PROGRESS_PARAMS_WQEBBS \
+ (DIV_ROUND_UP(MLX5E_NVMEOTCP_PROGRESS_PARAMS_WQE_SZ, MLX5_SEND_WQE_BB))
+
+#define MLX5E_NVMEOTCP_FETCH_PROGRESS_PARAMS_WQE(sq, pi) \
+ ((struct mlx5e_set_nvmeotcp_progress_params_wqe *)\
+ mlx5e_fetch_wqe(&(sq)->wq, pi, sizeof(struct mlx5e_set_nvmeotcp_progress_params_wqe)))
#define MLX5E_NVMEOTCP_FETCH_KLM_WQE(sq, pi) \
((struct mlx5e_umr_wqe *)\
@@ -14,6 +43,9 @@
#define MLX5_CTRL_SEGMENT_OPC_MOD_UMR_TIR_PARAMS 0x2
#define MLX5_CTRL_SEGMENT_OPC_MOD_UMR_UMR 0x0
+#define PROGRESS_PARAMS_DS_CNT \
+ DIV_ROUND_UP(MLX5E_NVMEOTCP_PROGRESS_PARAMS_WQE_SZ, MLX5_SEND_WQE_DS)
+
enum wqe_type {
KLM_UMR,
BSF_KLM_UMR,
@@ -22,4 +54,13 @@ enum wqe_type {
KLM_INV_UMR,
};
+void
+build_nvmeotcp_progress_params(struct mlx5e_nvmeotcp_queue *queue,
+ struct mlx5e_set_nvmeotcp_progress_params_wqe *wqe, u32 seq);
+
+void
+build_nvmeotcp_static_params(struct mlx5e_nvmeotcp_queue *queue,
+ struct mlx5e_set_transport_static_params_wqe *wqe,
+ u32 resync_seq, bool crc_rx);
+
#endif /* __MLX5E_NVMEOTCP_UTILS_H__ */
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index 54a3df934807..764762e169a8 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -1868,9 +1868,9 @@ void mlx5e_tx_err_cqe_work(struct work_struct *recover_work)
mlx5e_reporter_tx_err_cqe(sq);
}
-static int mlx5e_open_icosq(struct mlx5e_channel *c, struct mlx5e_params *params,
- struct mlx5e_sq_param *param, struct mlx5e_icosq *sq,
- work_func_t recover_work_func)
+int mlx5e_open_icosq(struct mlx5e_channel *c, struct mlx5e_params *params,
+ struct mlx5e_sq_param *param, struct mlx5e_icosq *sq,
+ work_func_t recover_work_func)
{
struct mlx5e_create_sq_param csp = {};
int err;
@@ -1914,7 +1914,7 @@ void mlx5e_deactivate_icosq(struct mlx5e_icosq *icosq)
synchronize_net(); /* Sync with NAPI. */
}
-static void mlx5e_close_icosq(struct mlx5e_icosq *sq)
+void mlx5e_close_icosq(struct mlx5e_icosq *sq)
{
if (sq->ktls_resync)
mlx5e_ktls_rx_resync_destroy_resp_list(sq->ktls_resync);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
index ff8b648534db..c88c4972f708 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
@@ -53,6 +53,7 @@
#include "en_accel/macsec.h"
#include "en_accel/ipsec_rxtx.h"
#include "en_accel/ktls_txrx.h"
+#include "en_accel/nvmeotcp.h"
#include "en/xdp.h"
#include "en/xsk/rx.h"
#include "en/health.h"
@@ -957,16 +958,23 @@ void mlx5e_free_icosq_descs(struct mlx5e_icosq *sq)
ci = mlx5_wq_cyc_ctr2ix(&sq->wq, sqcc);
wi = &sq->db.wqe_info[ci];
sqcc += wi->num_wqebbs;
-#ifdef CONFIG_MLX5_EN_TLS
switch (wi->wqe_type) {
+#ifdef CONFIG_MLX5_EN_TLS
case MLX5E_ICOSQ_WQE_SET_PSV_TLS:
mlx5e_ktls_handle_ctx_completion(wi);
break;
case MLX5E_ICOSQ_WQE_GET_PSV_TLS:
mlx5e_ktls_handle_get_psv_completion(wi, sq);
break;
- }
#endif
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+ case MLX5E_ICOSQ_WQE_SET_PSV_NVMEOTCP:
+ mlx5e_nvmeotcp_ctx_complete(wi);
+ break;
+#endif
+ default:
+ break;
+ }
}
sq->cc = sqcc;
}
@@ -1065,6 +1073,9 @@ int mlx5e_poll_ico_cq(struct mlx5e_cq *cq, int budget)
#ifdef CONFIG_MLX5_EN_NVMEOTCP
case MLX5E_ICOSQ_WQE_UMR_NVMEOTCP:
break;
+ case MLX5E_ICOSQ_WQE_SET_PSV_NVMEOTCP:
+ mlx5e_nvmeotcp_ctx_complete(wi);
+ break;
#endif
default:
netdev_WARN_ONCE(cq->netdev,
--
2.34.1
^ permalink raw reply related [flat|nested] 44+ messages in thread
* [PATCH v24 17/20] net/mlx5e: NVMEoTCP, ddp setup and resync
2024-04-04 12:36 [PATCH v24 00/20] nvme-tcp receive offloads Aurelien Aptel
` (15 preceding siblings ...)
2024-04-04 12:37 ` [PATCH v24 16/20] net/mlx5e: NVMEoTCP, queue init/teardown Aurelien Aptel
@ 2024-04-04 12:37 ` Aurelien Aptel
2024-04-04 12:37 ` [PATCH v24 18/20] net/mlx5e: NVMEoTCP, async ddp invalidation Aurelien Aptel
` (3 subsequent siblings)
20 siblings, 0 replies; 44+ messages in thread
From: Aurelien Aptel @ 2024-04-04 12:37 UTC (permalink / raw)
To: linux-nvme, netdev, sagi, hch, kbusch, axboe, chaitanyak, davem, kuba
Cc: aaptel, aurelien.aptel, smalin, malin1024, ogerlitz, yorayz,
borisp, galshalom, mgurtovoy
From: Ben Ben-Ishay <benishay@nvidia.com>
NVMEoTCP offload uses buffer registration for every NVME request to perform
direct data placement. This is achieved by creating a NIC HW mapping
between the CCID (command capsule ID) to the set of buffers that compose
the request. The registration is implemented via MKEY for which we do
fast/async mapping using KLM UMR WQE.
The buffer registration takes place when the ULP calls the ddp_setup op
which is done before they send their corresponding request to the other
side (e.g nvmf target). We don't wait for the completion of the
registration before returning back to the ulp. The reason being that
the HW mapping should be in place fast enough vs the RTT it would take
for the request to be responded. If this doesn't happen, some IO may not
be ddp-offloaded, but that doesn't stop the overall offloading session.
When the offloading HW gets out of sync with the protocol session, a
hardware/software handshake takes place to resync. The ddp_resync op is the
part of the handshake where the SW confirms to the HW that a indeed they
identified correctly a PDU header at a certain TCP sequence number. This
allows the HW to resume the offload.
The 1st part of the handshake is when the HW identifies such sequence
number in an arriving packet. A special mark is made on the completion
(cqe) and then the mlx5 driver invokes the ddp resync_request callback
advertised by the ULP in the ddp context - this is in downstream patch.
Signed-off-by: Ben Ben-Ishay <benishay@nvidia.com>
Signed-off-by: Boris Pismenny <borisp@nvidia.com>
Signed-off-by: Or Gerlitz <ogerlitz@nvidia.com>
Signed-off-by: Yoray Zack <yorayz@nvidia.com>
Signed-off-by: Aurelien Aptel <aaptel@nvidia.com>
Reviewed-by: Tariq Toukan <tariqt@nvidia.com>
---
.../mellanox/mlx5/core/en_accel/nvmeotcp.c | 146 +++++++++++++++++-
1 file changed, 144 insertions(+), 2 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c
index 2878ac0b7355..d7b416f77c3f 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c
@@ -684,19 +684,156 @@ mlx5e_nvmeotcp_queue_teardown(struct net_device *netdev,
mlx5e_nvmeotcp_put_queue(queue);
}
+static bool
+mlx5e_nvmeotcp_validate_small_sgl_suffix(struct scatterlist *sg, int sg_len, int mtu)
+{
+ int i, hole_size, hole_len, chunk_size = 0;
+
+ for (i = 1; i < sg_len; i++)
+ chunk_size += sg_dma_len(&sg[i]);
+
+ if (chunk_size >= mtu)
+ return true;
+
+ hole_size = mtu - chunk_size - 1;
+ hole_len = DIV_ROUND_UP(hole_size, PAGE_SIZE);
+
+ if (sg_len + hole_len > MAX_SKB_FRAGS)
+ return false;
+
+ return true;
+}
+
+static bool
+mlx5e_nvmeotcp_validate_big_sgl_suffix(struct scatterlist *sg, int sg_len, int mtu)
+{
+ int i, j, last_elem, window_idx, window_size = MAX_SKB_FRAGS - 1;
+ int chunk_size = 0;
+
+ last_elem = sg_len - window_size;
+ window_idx = window_size;
+
+ for (j = 1; j < window_size; j++)
+ chunk_size += sg_dma_len(&sg[j]);
+
+ for (i = 1; i <= last_elem; i++, window_idx++) {
+ chunk_size += sg_dma_len(&sg[window_idx]);
+ if (chunk_size < mtu - 1)
+ return false;
+
+ chunk_size -= sg_dma_len(&sg[i]);
+ }
+
+ return true;
+}
+
+/* This function makes sure that the middle/suffix of a PDU SGL meets the
+ * restriction of MAX_SKB_FRAGS. There are two cases here:
+ * 1. sg_len < MAX_SKB_FRAGS - the extreme case here is a packet that consists
+ * of one byte from the first SG element + the rest of the SGL and the remaining
+ * space of the packet will be scattered to the WQE and will be pointed by
+ * SKB frags.
+ * 2. sg_len => MAX_SKB_FRAGS - the extreme case here is a packet that consists
+ * of one byte from middle SG element + 15 continuous SG elements + one byte
+ * from a sequential SG element or the rest of the packet.
+ */
+static bool
+mlx5e_nvmeotcp_validate_sgl_suffix(struct scatterlist *sg, int sg_len, int mtu)
+{
+ int ret;
+
+ if (sg_len < MAX_SKB_FRAGS)
+ ret = mlx5e_nvmeotcp_validate_small_sgl_suffix(sg, sg_len, mtu);
+ else
+ ret = mlx5e_nvmeotcp_validate_big_sgl_suffix(sg, sg_len, mtu);
+
+ return ret;
+}
+
+static bool
+mlx5e_nvmeotcp_validate_sgl_prefix(struct scatterlist *sg, int sg_len, int mtu)
+{
+ int i, hole_size, hole_len, tmp_len, chunk_size = 0;
+
+ tmp_len = min_t(int, sg_len, MAX_SKB_FRAGS);
+
+ for (i = 0; i < tmp_len; i++)
+ chunk_size += sg_dma_len(&sg[i]);
+
+ if (chunk_size >= mtu)
+ return true;
+
+ hole_size = mtu - chunk_size;
+ hole_len = DIV_ROUND_UP(hole_size, PAGE_SIZE);
+
+ if (tmp_len + hole_len > MAX_SKB_FRAGS)
+ return false;
+
+ return true;
+}
+
+/* This function is responsible to ensure that a PDU could be offloaded.
+ * PDU is offloaded by building a non-linear SKB such that each SGL element is
+ * placed in frag, thus this function should ensure that all packets that
+ * represent part of the PDU won't exaggerate from MAX_SKB_FRAGS SGL.
+ * In addition NVMEoTCP offload has one PDU offload for packet restriction.
+ * Packet could start with a new PDU and then we should check that the prefix
+ * of the PDU meets the requirement or a packet can start in the middle of SG
+ * element and then we should check that the suffix of PDU meets the requirement.
+ */
+static bool
+mlx5e_nvmeotcp_validate_sgl(struct scatterlist *sg, int sg_len, int mtu)
+{
+ int max_hole_frags;
+
+ max_hole_frags = DIV_ROUND_UP(mtu, PAGE_SIZE);
+ if (sg_len + max_hole_frags <= MAX_SKB_FRAGS)
+ return true;
+
+ if (!mlx5e_nvmeotcp_validate_sgl_prefix(sg, sg_len, mtu) ||
+ !mlx5e_nvmeotcp_validate_sgl_suffix(sg, sg_len, mtu))
+ return false;
+
+ return true;
+}
+
static int
mlx5e_nvmeotcp_ddp_setup(struct net_device *netdev,
struct sock *sk,
struct ulp_ddp_io *ddp)
{
+ struct scatterlist *sg = ddp->sg_table.sgl;
+ struct mlx5e_nvmeotcp_queue_entry *nvqt;
struct mlx5e_nvmeotcp_queue *queue;
+ struct mlx5_core_dev *mdev;
+ int i, size = 0, count = 0;
queue = container_of(ulp_ddp_get_ctx(sk),
struct mlx5e_nvmeotcp_queue, ulp_ddp_ctx);
+ mdev = queue->priv->mdev;
+ count = dma_map_sg(mdev->device, ddp->sg_table.sgl, ddp->nents,
+ DMA_FROM_DEVICE);
+
+ if (count <= 0)
+ return -EINVAL;
- /* Placeholder - map_sg and initializing the count */
+ if (WARN_ON(count > mlx5e_get_max_sgl(mdev)))
+ return -ENOSPC;
+
+ if (!mlx5e_nvmeotcp_validate_sgl(sg, count, READ_ONCE(netdev->mtu)))
+ return -EOPNOTSUPP;
+
+ for (i = 0; i < count; i++)
+ size += sg_dma_len(&sg[i]);
+
+ nvqt = &queue->ccid_table[ddp->command_id];
+ nvqt->size = size;
+ nvqt->ddp = ddp;
+ nvqt->sgl = sg;
+ nvqt->ccid_gen++;
+ nvqt->sgl_length = count;
+ mlx5e_nvmeotcp_post_klm_wqe(queue, KLM_UMR, ddp->command_id, count);
- mlx5e_nvmeotcp_post_klm_wqe(queue, KLM_UMR, ddp->command_id, 0);
return 0;
}
@@ -719,6 +856,11 @@ static void
mlx5e_nvmeotcp_ddp_resync(struct net_device *netdev,
struct sock *sk, u32 seq)
{
+ struct mlx5e_nvmeotcp_queue *queue =
+ container_of(ulp_ddp_get_ctx(sk), struct mlx5e_nvmeotcp_queue, ulp_ddp_ctx);
+
+ queue->after_resync_cqe = 1;
+ mlx5e_nvmeotcp_rx_post_static_params_wqe(queue, seq);
}
struct mlx5e_nvmeotcp_queue *
--
2.34.1
^ permalink raw reply related [flat|nested] 44+ messages in thread
* [PATCH v24 18/20] net/mlx5e: NVMEoTCP, async ddp invalidation
2024-04-04 12:36 [PATCH v24 00/20] nvme-tcp receive offloads Aurelien Aptel
` (16 preceding siblings ...)
2024-04-04 12:37 ` [PATCH v24 17/20] net/mlx5e: NVMEoTCP, ddp setup and resync Aurelien Aptel
@ 2024-04-04 12:37 ` Aurelien Aptel
2024-04-04 12:37 ` [PATCH v24 19/20] net/mlx5e: NVMEoTCP, data-path for DDP+DDGST offload Aurelien Aptel
` (2 subsequent siblings)
20 siblings, 0 replies; 44+ messages in thread
From: Aurelien Aptel @ 2024-04-04 12:37 UTC (permalink / raw)
To: linux-nvme, netdev, sagi, hch, kbusch, axboe, chaitanyak, davem, kuba
Cc: aaptel, aurelien.aptel, smalin, malin1024, ogerlitz, yorayz,
borisp, galshalom, mgurtovoy
From: Ben Ben-Ishay <benishay@nvidia.com>
After the ULP consumed the buffers of the offloaded request, it calls the
ddp_teardown op to release the NIC mapping for them and allow the NIC to
reuse the HW contexts associated with offloading this IO. We do a
fast/async un-mapping via UMR WQE. In this case, the ULP does holds off
with completing the request towards the upper/application layers until the
HW unmapping is done.
When the corresponding CQE is received, a notification is done via the
the teardown_done ddp callback advertised by the ULP in the ddp context.
Signed-off-by: Ben Ben-Ishay <benishay@nvidia.com>
Signed-off-by: Boris Pismenny <borisp@nvidia.com>
Signed-off-by: Or Gerlitz <ogerlitz@nvidia.com>
Signed-off-by: Yoray Zack <yorayz@nvidia.com>
Signed-off-by: Aurelien Aptel <aaptel@nvidia.com>
Reviewed-by: Tariq Toukan <tariqt@nvidia.com>
---
.../net/ethernet/mellanox/mlx5/core/en/txrx.h | 4 ++
.../mellanox/mlx5/core/en_accel/nvmeotcp.c | 66 ++++++++++++++++---
.../mellanox/mlx5/core/en_accel/nvmeotcp.h | 1 +
.../net/ethernet/mellanox/mlx5/core/en_rx.c | 6 ++
4 files changed, 67 insertions(+), 10 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h b/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h
index c87dca17d5c8..3c124f708afc 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h
@@ -52,6 +52,7 @@ enum mlx5e_icosq_wqe_type {
#endif
#ifdef CONFIG_MLX5_EN_NVMEOTCP
MLX5E_ICOSQ_WQE_UMR_NVMEOTCP,
+ MLX5E_ICOSQ_WQE_UMR_NVMEOTCP_INVALIDATE,
MLX5E_ICOSQ_WQE_SET_PSV_NVMEOTCP,
#endif
};
@@ -230,6 +231,9 @@ struct mlx5e_icosq_wqe_info {
struct {
struct mlx5e_nvmeotcp_queue *queue;
} nvmeotcp_q;
+ struct {
+ struct mlx5e_nvmeotcp_queue_entry *entry;
+ } nvmeotcp_qe;
#endif
};
};
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c
index d7b416f77c3f..505f4e6d5098 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c
@@ -142,10 +142,11 @@ build_nvmeotcp_klm_umr(struct mlx5e_nvmeotcp_queue *queue, struct mlx5e_umr_wqe
u16 ccid, int klm_entries, u32 klm_offset, u32 len,
enum wqe_type klm_type)
{
- u32 id = (klm_type == KLM_UMR) ? queue->ccid_table[ccid].klm_mkey :
- (mlx5e_tir_get_tirn(&queue->tir) << MLX5_WQE_CTRL_TIR_TIS_INDEX_SHIFT);
- u8 opc_mod = (klm_type == KLM_UMR) ? MLX5_CTRL_SEGMENT_OPC_MOD_UMR_UMR :
- MLX5_OPC_MOD_TRANSPORT_TIR_STATIC_PARAMS;
+ u32 id = (klm_type == BSF_KLM_UMR) ?
+ (mlx5e_tir_get_tirn(&queue->tir) << MLX5_WQE_CTRL_TIR_TIS_INDEX_SHIFT) :
+ queue->ccid_table[ccid].klm_mkey;
+ u8 opc_mod = (klm_type == BSF_KLM_UMR) ? MLX5_OPC_MOD_TRANSPORT_TIR_STATIC_PARAMS :
+ MLX5_CTRL_SEGMENT_OPC_MOD_UMR_UMR;
u32 ds_cnt = MLX5E_KLM_UMR_DS_CNT(ALIGN(klm_entries, MLX5_UMR_KLM_NUM_ENTRIES_ALIGNMENT));
struct mlx5_wqe_umr_ctrl_seg *ucseg = &wqe->uctrl;
struct mlx5_wqe_ctrl_seg *cseg = &wqe->ctrl;
@@ -158,6 +159,13 @@ build_nvmeotcp_klm_umr(struct mlx5e_nvmeotcp_queue *queue, struct mlx5e_umr_wqe
cseg->qpn_ds = cpu_to_be32((sqn << MLX5_WQE_CTRL_QPN_SHIFT) | ds_cnt);
cseg->general_id = cpu_to_be32(id);
+ if (!klm_entries) { /* this is invalidate */
+ ucseg->mkey_mask = cpu_to_be64(MLX5_MKEY_MASK_FREE);
+ ucseg->flags = MLX5_UMR_INLINE;
+ mkc->status = MLX5_MKEY_STATUS_FREE;
+ return;
+ }
+
if (klm_type == KLM_UMR && !klm_offset) {
ucseg->mkey_mask = cpu_to_be64(MLX5_MKEY_MASK_XLT_OCT_SIZE |
MLX5_MKEY_MASK_LEN | MLX5_MKEY_MASK_FREE);
@@ -259,8 +267,8 @@ build_nvmeotcp_static_params(struct mlx5e_nvmeotcp_queue *queue,
static void
mlx5e_nvmeotcp_fill_wi(struct mlx5e_nvmeotcp_queue *nvmeotcp_queue,
- struct mlx5e_icosq *sq, u32 wqebbs, u16 pi,
- enum wqe_type type)
+ struct mlx5e_icosq *sq, u32 wqebbs,
+ u16 pi, u16 ccid, enum wqe_type type)
{
struct mlx5e_icosq_wqe_info *wi = &sq->db.wqe_info[pi];
@@ -272,6 +280,10 @@ mlx5e_nvmeotcp_fill_wi(struct mlx5e_nvmeotcp_queue *nvmeotcp_queue,
wi->wqe_type = MLX5E_ICOSQ_WQE_SET_PSV_NVMEOTCP;
wi->nvmeotcp_q.queue = nvmeotcp_queue;
break;
+ case KLM_INV_UMR:
+ wi->wqe_type = MLX5E_ICOSQ_WQE_UMR_NVMEOTCP_INVALIDATE;
+ wi->nvmeotcp_qe.entry = &nvmeotcp_queue->ccid_table[ccid];
+ break;
default:
/* cases where no further action is required upon completion, such as ddp setup */
wi->wqe_type = MLX5E_ICOSQ_WQE_UMR_NVMEOTCP;
@@ -290,7 +302,7 @@ mlx5e_nvmeotcp_rx_post_static_params_wqe(struct mlx5e_nvmeotcp_queue *queue, u32
wqebbs = MLX5E_TRANSPORT_SET_STATIC_PARAMS_WQEBBS;
pi = mlx5e_icosq_get_next_pi(sq, wqebbs);
wqe = MLX5E_TRANSPORT_FETCH_SET_STATIC_PARAMS_WQE(sq, pi);
- mlx5e_nvmeotcp_fill_wi(NULL, sq, wqebbs, pi, BSF_UMR);
+ mlx5e_nvmeotcp_fill_wi(NULL, sq, wqebbs, pi, 0, BSF_UMR);
build_nvmeotcp_static_params(queue, wqe, resync_seq, queue->crc_rx);
sq->pc += wqebbs;
mlx5e_notify_hw(&sq->wq, sq->pc, sq->uar_map, &wqe->ctrl);
@@ -307,7 +319,7 @@ mlx5e_nvmeotcp_rx_post_progress_params_wqe(struct mlx5e_nvmeotcp_queue *queue, u
wqebbs = MLX5E_NVMEOTCP_PROGRESS_PARAMS_WQEBBS;
pi = mlx5e_icosq_get_next_pi(sq, wqebbs);
wqe = MLX5E_NVMEOTCP_FETCH_PROGRESS_PARAMS_WQE(sq, pi);
- mlx5e_nvmeotcp_fill_wi(queue, sq, wqebbs, pi, SET_PSV_UMR);
+ mlx5e_nvmeotcp_fill_wi(queue, sq, wqebbs, pi, 0, SET_PSV_UMR);
build_nvmeotcp_progress_params(queue, wqe, seq);
sq->pc += wqebbs;
mlx5e_notify_hw(&sq->wq, sq->pc, sq->uar_map, &wqe->ctrl);
@@ -330,7 +342,7 @@ post_klm_wqe(struct mlx5e_nvmeotcp_queue *queue,
wqebbs = DIV_ROUND_UP(wqe_sz, MLX5_SEND_WQE_BB);
pi = mlx5e_icosq_get_next_pi(sq, wqebbs);
wqe = MLX5E_NVMEOTCP_FETCH_KLM_WQE(sq, pi);
- mlx5e_nvmeotcp_fill_wi(queue, sq, wqebbs, pi, wqe_type);
+ mlx5e_nvmeotcp_fill_wi(queue, sq, wqebbs, pi, ccid, wqe_type);
build_nvmeotcp_klm_umr(queue, wqe, ccid, cur_klm_entries, klm_offset,
klm_length, wqe_type);
sq->pc += wqebbs;
@@ -345,7 +357,10 @@ mlx5e_nvmeotcp_post_klm_wqe(struct mlx5e_nvmeotcp_queue *queue, enum wqe_type wq
struct mlx5e_icosq *sq = &queue->sq;
u32 klm_offset = 0, wqes, i;
- wqes = DIV_ROUND_UP(klm_length, queue->max_klms_per_wqe);
+ if (wqe_type == KLM_INV_UMR)
+ wqes = 1;
+ else
+ wqes = DIV_ROUND_UP(klm_length, queue->max_klms_per_wqe);
spin_lock_bh(&queue->sq_lock);
@@ -844,12 +859,43 @@ void mlx5e_nvmeotcp_ctx_complete(struct mlx5e_icosq_wqe_info *wi)
complete(&queue->static_params_done);
}
+void mlx5e_nvmeotcp_ddp_inv_done(struct mlx5e_icosq_wqe_info *wi)
+{
+ struct mlx5e_nvmeotcp_queue_entry *q_entry = wi->nvmeotcp_qe.entry;
+ struct mlx5e_nvmeotcp_queue *queue = q_entry->queue;
+ struct mlx5_core_dev *mdev = queue->priv->mdev;
+ struct ulp_ddp_io *ddp = q_entry->ddp;
+ const struct ulp_ddp_ulp_ops *ulp_ops;
+
+ dma_unmap_sg(mdev->device, ddp->sg_table.sgl,
+ q_entry->sgl_length, DMA_FROM_DEVICE);
+
+ q_entry->sgl_length = 0;
+
+ ulp_ops = inet_csk(queue->sk)->icsk_ulp_ddp_ops;
+ if (ulp_ops && ulp_ops->ddp_teardown_done)
+ ulp_ops->ddp_teardown_done(q_entry->ddp_ctx);
+}
+
static void
mlx5e_nvmeotcp_ddp_teardown(struct net_device *netdev,
struct sock *sk,
struct ulp_ddp_io *ddp,
void *ddp_ctx)
{
+ struct mlx5e_nvmeotcp_queue_entry *q_entry;
+ struct mlx5e_nvmeotcp_queue *queue;
+
+ queue = container_of(ulp_ddp_get_ctx(sk), struct mlx5e_nvmeotcp_queue, ulp_ddp_ctx);
+ q_entry = &queue->ccid_table[ddp->command_id];
+ WARN_ONCE(q_entry->sgl_length == 0,
+ "Invalidation of empty sgl (CID 0x%x, queue 0x%x)\n",
+ ddp->command_id, queue->id);
+
+ q_entry->ddp_ctx = ddp_ctx;
+ q_entry->queue = queue;
+
+ mlx5e_nvmeotcp_post_klm_wqe(queue, KLM_INV_UMR, ddp->command_id, 0);
}
static void
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h
index 8b29f3fde7f2..13817d8a0aae 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h
@@ -109,6 +109,7 @@ void mlx5e_nvmeotcp_cleanup(struct mlx5e_priv *priv);
struct mlx5e_nvmeotcp_queue *
mlx5e_nvmeotcp_get_queue(struct mlx5e_nvmeotcp *nvmeotcp, int id);
void mlx5e_nvmeotcp_put_queue(struct mlx5e_nvmeotcp_queue *queue);
+void mlx5e_nvmeotcp_ddp_inv_done(struct mlx5e_icosq_wqe_info *wi);
void mlx5e_nvmeotcp_ctx_complete(struct mlx5e_icosq_wqe_info *wi);
static inline void mlx5e_nvmeotcp_init_rx(struct mlx5e_priv *priv) {}
void mlx5e_nvmeotcp_cleanup_rx(struct mlx5e_priv *priv);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
index c88c4972f708..89ef317bdd2f 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
@@ -968,6 +968,9 @@ void mlx5e_free_icosq_descs(struct mlx5e_icosq *sq)
break;
#endif
#ifdef CONFIG_MLX5_EN_NVMEOTCP
+ case MLX5E_ICOSQ_WQE_UMR_NVMEOTCP_INVALIDATE:
+ mlx5e_nvmeotcp_ddp_inv_done(wi);
+ break;
case MLX5E_ICOSQ_WQE_SET_PSV_NVMEOTCP:
mlx5e_nvmeotcp_ctx_complete(wi);
break;
@@ -1073,6 +1076,9 @@ int mlx5e_poll_ico_cq(struct mlx5e_cq *cq, int budget)
#ifdef CONFIG_MLX5_EN_NVMEOTCP
case MLX5E_ICOSQ_WQE_UMR_NVMEOTCP:
break;
+ case MLX5E_ICOSQ_WQE_UMR_NVMEOTCP_INVALIDATE:
+ mlx5e_nvmeotcp_ddp_inv_done(wi);
+ break;
case MLX5E_ICOSQ_WQE_SET_PSV_NVMEOTCP:
mlx5e_nvmeotcp_ctx_complete(wi);
break;
--
2.34.1
^ permalink raw reply related [flat|nested] 44+ messages in thread
* [PATCH v24 19/20] net/mlx5e: NVMEoTCP, data-path for DDP+DDGST offload
2024-04-04 12:36 [PATCH v24 00/20] nvme-tcp receive offloads Aurelien Aptel
` (17 preceding siblings ...)
2024-04-04 12:37 ` [PATCH v24 18/20] net/mlx5e: NVMEoTCP, async ddp invalidation Aurelien Aptel
@ 2024-04-04 12:37 ` Aurelien Aptel
2024-04-04 12:37 ` [PATCH v24 20/20] net/mlx5e: NVMEoTCP, statistics Aurelien Aptel
2024-04-06 5:45 ` [PATCH v24 00/20] nvme-tcp receive offloads Jakub Kicinski
20 siblings, 0 replies; 44+ messages in thread
From: Aurelien Aptel @ 2024-04-04 12:37 UTC (permalink / raw)
To: linux-nvme, netdev, sagi, hch, kbusch, axboe, chaitanyak, davem, kuba
Cc: aaptel, aurelien.aptel, smalin, malin1024, ogerlitz, yorayz,
borisp, galshalom, mgurtovoy
From: Ben Ben-Ishay <benishay@nvidia.com>
This patch implements the data-path for direct data placement (DDP)
and DDGST offloads. NVMEoTCP DDP constructs an SKB from each CQE, while
pointing at NVME destination buffers. In turn, this enables the offload,
as the NVMe-TCP layer will skip the copy when src == dst.
Additionally, this patch adds support for DDGST (CRC32) offload.
HW will report DDGST offload only if it has not encountered an error
in the received packet. We pass this indication in skb->ulp_crc
up the stack to NVMe-TCP to skip computing the DDGST if all
corresponding SKBs were verified by HW.
This patch also handles context resynchronization requests made by
NIC HW. The resync request is passed to the NVMe-TCP layer
to be handled at a later point in time.
Finally, we also use the skb->no_condense bit to avoid skb_condense.
This is critical as every SKB that uses DDP has a hole that fits
perfectly with skb_condense's policy, but filling this hole is
counter-productive as the data there already resides in its
destination buffer.
Signed-off-by: Ben Ben-Ishay <benishay@nvidia.com>
Signed-off-by: Boris Pismenny <borisp@nvidia.com>
Signed-off-by: Or Gerlitz <ogerlitz@nvidia.com>
Signed-off-by: Yoray Zack <yorayz@nvidia.com>
Signed-off-by: Aurelien Aptel <aaptel@nvidia.com>
Reviewed-by: Tariq Toukan <tariqt@nvidia.com>
---
.../net/ethernet/mellanox/mlx5/core/Makefile | 2 +-
.../net/ethernet/mellanox/mlx5/core/en/txrx.h | 6 +
.../mlx5/core/en_accel/nvmeotcp_rxtx.c | 345 ++++++++++++++++++
.../mlx5/core/en_accel/nvmeotcp_rxtx.h | 37 ++
.../net/ethernet/mellanox/mlx5/core/en_rx.c | 44 ++-
5 files changed, 419 insertions(+), 15 deletions(-)
create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.c
create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.h
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/Makefile b/drivers/net/ethernet/mellanox/mlx5/core/Makefile
index 41cb8f831632..19691917682e 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/Makefile
+++ b/drivers/net/ethernet/mellanox/mlx5/core/Makefile
@@ -109,7 +109,7 @@ mlx5_core-$(CONFIG_MLX5_EN_TLS) += en_accel/ktls_stats.o \
en_accel/fs_tcp.o en_accel/ktls.o en_accel/ktls_txrx.o \
en_accel/ktls_tx.o en_accel/ktls_rx.o
-mlx5_core-$(CONFIG_MLX5_EN_NVMEOTCP) += en_accel/fs_tcp.o en_accel/nvmeotcp.o
+mlx5_core-$(CONFIG_MLX5_EN_NVMEOTCP) += en_accel/fs_tcp.o en_accel/nvmeotcp.o en_accel/nvmeotcp_rxtx.o
mlx5_core-$(CONFIG_MLX5_SW_STEERING) += steering/dr_domain.o steering/dr_table.o \
steering/dr_matcher.o steering/dr_rule.o \
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h b/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h
index 3c124f708afc..516054e480d9 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h
@@ -526,4 +526,10 @@ static inline struct mlx5e_mpw_info *mlx5e_get_mpw_info(struct mlx5e_rq *rq, int
return (struct mlx5e_mpw_info *)((char *)rq->mpwqe.info + array_size(i, isz));
}
+
+static inline struct mlx5e_wqe_frag_info *get_frag(struct mlx5e_rq *rq, u16 ix)
+{
+ return &rq->wqe.frags[ix << rq->wqe.info.log_num_frags];
+}
+
#endif
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.c
new file mode 100644
index 000000000000..269d8075f3c2
--- /dev/null
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.c
@@ -0,0 +1,345 @@
+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
+// Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES.
+
+#include "en_accel/nvmeotcp_rxtx.h"
+#include <linux/mlx5/mlx5_ifc.h>
+#include "en/txrx.h"
+
+#define MLX5E_TC_FLOW_ID_MASK 0x00ffffff
+
+static struct mlx5e_frag_page *mlx5e_get_frag(struct mlx5e_rq *rq,
+ struct mlx5_cqe64 *cqe)
+{
+ struct mlx5e_frag_page *fp;
+
+ if (rq->wq_type == MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ) {
+ u16 wqe_id = be16_to_cpu(cqe->wqe_id);
+ u16 stride_ix = mpwrq_get_cqe_stride_index(cqe);
+ u32 wqe_offset = stride_ix << rq->mpwqe.log_stride_sz;
+ u32 page_idx = wqe_offset >> rq->mpwqe.page_shift;
+ struct mlx5e_mpw_info *wi = mlx5e_get_mpw_info(rq, wqe_id);
+ union mlx5e_alloc_units *au = &wi->alloc_units;
+
+ fp = &au->frag_pages[page_idx];
+ } else {
+ /* Legacy */
+ struct mlx5_wq_cyc *wq = &rq->wqe.wq;
+ u16 ci = mlx5_wq_cyc_ctr2ix(wq, be16_to_cpu(cqe->wqe_counter));
+ struct mlx5e_wqe_frag_info *wi = get_frag(rq, ci);
+
+ fp = wi->frag_page;
+ }
+
+ return fp;
+}
+
+static void nvmeotcp_update_resync(struct mlx5e_nvmeotcp_queue *queue,
+ struct mlx5e_cqe128 *cqe128)
+{
+ const struct ulp_ddp_ulp_ops *ulp_ops;
+ u32 seq;
+
+ seq = be32_to_cpu(cqe128->resync_tcp_sn);
+ ulp_ops = inet_csk(queue->sk)->icsk_ulp_ddp_ops;
+ if (ulp_ops && ulp_ops->resync_request)
+ ulp_ops->resync_request(queue->sk, seq, ULP_DDP_RESYNC_PENDING);
+}
+
+static void mlx5e_nvmeotcp_advance_sgl_iter(struct mlx5e_nvmeotcp_queue *queue)
+{
+ struct mlx5e_nvmeotcp_queue_entry *nqe = &queue->ccid_table[queue->ccid];
+
+ queue->ccoff += nqe->sgl[queue->ccsglidx].length;
+ queue->ccoff_inner = 0;
+ queue->ccsglidx++;
+}
+
+static inline void
+mlx5e_nvmeotcp_add_skb_frag(struct net_device *netdev, struct sk_buff *skb,
+ struct mlx5e_nvmeotcp_queue *queue,
+ struct mlx5e_nvmeotcp_queue_entry *nqe, u32 fragsz)
+{
+ dma_sync_single_for_cpu(&netdev->dev,
+ nqe->sgl[queue->ccsglidx].offset + queue->ccoff_inner,
+ fragsz, DMA_FROM_DEVICE);
+
+ page_ref_inc(compound_head(sg_page(&nqe->sgl[queue->ccsglidx])));
+
+ skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags,
+ sg_page(&nqe->sgl[queue->ccsglidx]),
+ nqe->sgl[queue->ccsglidx].offset + queue->ccoff_inner,
+ fragsz,
+ fragsz);
+}
+
+static inline void
+mlx5_nvmeotcp_add_tail_nonlinear(struct sk_buff *skb, skb_frag_t *org_frags,
+ int org_nr_frags, int frag_index)
+{
+ while (org_nr_frags != frag_index) {
+ skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags,
+ skb_frag_page(&org_frags[frag_index]),
+ skb_frag_off(&org_frags[frag_index]),
+ skb_frag_size(&org_frags[frag_index]),
+ skb_frag_size(&org_frags[frag_index]));
+ frag_index++;
+ }
+}
+
+static void
+mlx5_nvmeotcp_add_tail(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe,
+ struct mlx5e_nvmeotcp_queue *queue, struct sk_buff *skb,
+ int offset, int len)
+{
+ struct mlx5e_frag_page *frag_page = mlx5e_get_frag(rq, cqe);
+
+ frag_page->frags++;
+ skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags,
+ virt_to_page(skb->data), offset, len, len);
+}
+
+static void mlx5_nvmeotcp_trim_nonlinear(struct sk_buff *skb, skb_frag_t *org_frags,
+ int *frag_index, int remaining)
+{
+ unsigned int frag_size;
+ int nr_frags;
+
+ /* skip @remaining bytes in frags */
+ *frag_index = 0;
+ while (remaining) {
+ frag_size = skb_frag_size(&skb_shinfo(skb)->frags[*frag_index]);
+ if (frag_size > remaining) {
+ skb_frag_off_add(&skb_shinfo(skb)->frags[*frag_index],
+ remaining);
+ skb_frag_size_sub(&skb_shinfo(skb)->frags[*frag_index],
+ remaining);
+ remaining = 0;
+ } else {
+ remaining -= frag_size;
+ skb_frag_unref(skb, *frag_index);
+ *frag_index += 1;
+ }
+ }
+
+ /* save original frags for the tail and unref */
+ nr_frags = skb_shinfo(skb)->nr_frags;
+ memcpy(&org_frags[*frag_index], &skb_shinfo(skb)->frags[*frag_index],
+ (nr_frags - *frag_index) * sizeof(skb_frag_t));
+
+ /* remove frags from skb */
+ skb_shinfo(skb)->nr_frags = 0;
+ skb->len -= skb->data_len;
+ skb->truesize -= skb->data_len;
+ skb->data_len = 0;
+}
+
+static bool
+mlx5e_nvmeotcp_rebuild_rx_skb_nonlinear(struct mlx5e_rq *rq, struct sk_buff *skb,
+ struct mlx5_cqe64 *cqe, u32 cqe_bcnt)
+{
+ int ccoff, cclen, hlen, ccid, remaining, fragsz, to_copy = 0;
+ struct net_device *netdev = rq->netdev;
+ struct mlx5e_priv *priv = netdev_priv(netdev);
+ struct mlx5e_nvmeotcp_queue_entry *nqe;
+ skb_frag_t org_frags[MAX_SKB_FRAGS];
+ struct mlx5e_nvmeotcp_queue *queue;
+ int org_nr_frags, frag_index;
+ struct mlx5e_cqe128 *cqe128;
+ u32 queue_id;
+
+ queue_id = (be32_to_cpu(cqe->sop_drop_qpn) & MLX5E_TC_FLOW_ID_MASK);
+ queue = mlx5e_nvmeotcp_get_queue(priv->nvmeotcp, queue_id);
+ if (unlikely(!queue)) {
+ dev_kfree_skb_any(skb);
+ return false;
+ }
+
+ cqe128 = container_of(cqe, struct mlx5e_cqe128, cqe64);
+ if (cqe_is_nvmeotcp_resync(cqe)) {
+ nvmeotcp_update_resync(queue, cqe128);
+ mlx5e_nvmeotcp_put_queue(queue);
+ return true;
+ }
+
+ /* If a resync occurred in the previous cqe,
+ * the current cqe.crcvalid bit may not be valid,
+ * so we will treat it as 0
+ */
+ if (unlikely(queue->after_resync_cqe) && cqe_is_nvmeotcp_crcvalid(cqe)) {
+ skb->ulp_crc = 0;
+ queue->after_resync_cqe = 0;
+ } else {
+ if (queue->crc_rx)
+ skb->ulp_crc = cqe_is_nvmeotcp_crcvalid(cqe);
+ }
+
+ skb->no_condense = cqe_is_nvmeotcp_zc(cqe);
+ if (!cqe_is_nvmeotcp_zc(cqe)) {
+ mlx5e_nvmeotcp_put_queue(queue);
+ return true;
+ }
+
+ /* cc ddp from cqe */
+ ccid = be16_to_cpu(cqe128->ccid);
+ ccoff = be32_to_cpu(cqe128->ccoff);
+ cclen = be16_to_cpu(cqe128->cclen);
+ hlen = be16_to_cpu(cqe128->hlen);
+
+ /* carve a hole in the skb for DDP data */
+ org_nr_frags = skb_shinfo(skb)->nr_frags;
+ mlx5_nvmeotcp_trim_nonlinear(skb, org_frags, &frag_index, cclen);
+ nqe = &queue->ccid_table[ccid];
+
+ /* packet starts new ccid? */
+ if (queue->ccid != ccid || queue->ccid_gen != nqe->ccid_gen) {
+ queue->ccid = ccid;
+ queue->ccoff = 0;
+ queue->ccoff_inner = 0;
+ queue->ccsglidx = 0;
+ queue->ccid_gen = nqe->ccid_gen;
+ }
+
+ /* skip inside cc until the ccoff in the cqe */
+ while (queue->ccoff + queue->ccoff_inner < ccoff) {
+ remaining = nqe->sgl[queue->ccsglidx].length - queue->ccoff_inner;
+ fragsz = min_t(off_t, remaining,
+ ccoff - (queue->ccoff + queue->ccoff_inner));
+
+ if (fragsz == remaining)
+ mlx5e_nvmeotcp_advance_sgl_iter(queue);
+ else
+ queue->ccoff_inner += fragsz;
+ }
+
+ /* adjust the skb according to the cqe cc */
+ while (to_copy < cclen) {
+ remaining = nqe->sgl[queue->ccsglidx].length - queue->ccoff_inner;
+ fragsz = min_t(int, remaining, cclen - to_copy);
+
+ mlx5e_nvmeotcp_add_skb_frag(netdev, skb, queue, nqe, fragsz);
+ to_copy += fragsz;
+ if (fragsz == remaining)
+ mlx5e_nvmeotcp_advance_sgl_iter(queue);
+ else
+ queue->ccoff_inner += fragsz;
+ }
+
+ if (cqe_bcnt > hlen + cclen) {
+ remaining = cqe_bcnt - hlen - cclen;
+ mlx5_nvmeotcp_add_tail_nonlinear(skb, org_frags,
+ org_nr_frags,
+ frag_index);
+ }
+
+ mlx5e_nvmeotcp_put_queue(queue);
+ return true;
+}
+
+static bool
+mlx5e_nvmeotcp_rebuild_rx_skb_linear(struct mlx5e_rq *rq, struct sk_buff *skb,
+ struct mlx5_cqe64 *cqe, u32 cqe_bcnt)
+{
+ int ccoff, cclen, hlen, ccid, remaining, fragsz, to_copy = 0;
+ struct net_device *netdev = rq->netdev;
+ struct mlx5e_priv *priv = netdev_priv(netdev);
+ struct mlx5e_nvmeotcp_queue_entry *nqe;
+ struct mlx5e_nvmeotcp_queue *queue;
+ struct mlx5e_cqe128 *cqe128;
+ u32 queue_id;
+
+ queue_id = (be32_to_cpu(cqe->sop_drop_qpn) & MLX5E_TC_FLOW_ID_MASK);
+ queue = mlx5e_nvmeotcp_get_queue(priv->nvmeotcp, queue_id);
+ if (unlikely(!queue)) {
+ dev_kfree_skb_any(skb);
+ return false;
+ }
+
+ cqe128 = container_of(cqe, struct mlx5e_cqe128, cqe64);
+ if (cqe_is_nvmeotcp_resync(cqe)) {
+ nvmeotcp_update_resync(queue, cqe128);
+ mlx5e_nvmeotcp_put_queue(queue);
+ return true;
+ }
+
+ /* If a resync occurred in the previous cqe,
+ * the current cqe.crcvalid bit may not be valid,
+ * so we will treat it as 0
+ */
+ if (unlikely(queue->after_resync_cqe) && cqe_is_nvmeotcp_crcvalid(cqe)) {
+ skb->ulp_crc = 0;
+ queue->after_resync_cqe = 0;
+ } else {
+ if (queue->crc_rx)
+ skb->ulp_crc = cqe_is_nvmeotcp_crcvalid(cqe);
+ }
+
+ skb->no_condense = cqe_is_nvmeotcp_zc(cqe);
+ if (!cqe_is_nvmeotcp_zc(cqe)) {
+ mlx5e_nvmeotcp_put_queue(queue);
+ return true;
+ }
+
+ /* cc ddp from cqe */
+ ccid = be16_to_cpu(cqe128->ccid);
+ ccoff = be32_to_cpu(cqe128->ccoff);
+ cclen = be16_to_cpu(cqe128->cclen);
+ hlen = be16_to_cpu(cqe128->hlen);
+
+ /* carve a hole in the skb for DDP data */
+ skb_trim(skb, hlen);
+ nqe = &queue->ccid_table[ccid];
+
+ /* packet starts new ccid? */
+ if (queue->ccid != ccid || queue->ccid_gen != nqe->ccid_gen) {
+ queue->ccid = ccid;
+ queue->ccoff = 0;
+ queue->ccoff_inner = 0;
+ queue->ccsglidx = 0;
+ queue->ccid_gen = nqe->ccid_gen;
+ }
+
+ /* skip inside cc until the ccoff in the cqe */
+ while (queue->ccoff + queue->ccoff_inner < ccoff) {
+ remaining = nqe->sgl[queue->ccsglidx].length - queue->ccoff_inner;
+ fragsz = min_t(off_t, remaining,
+ ccoff - (queue->ccoff + queue->ccoff_inner));
+
+ if (fragsz == remaining)
+ mlx5e_nvmeotcp_advance_sgl_iter(queue);
+ else
+ queue->ccoff_inner += fragsz;
+ }
+
+ /* adjust the skb according to the cqe cc */
+ while (to_copy < cclen) {
+ remaining = nqe->sgl[queue->ccsglidx].length - queue->ccoff_inner;
+ fragsz = min_t(int, remaining, cclen - to_copy);
+
+ mlx5e_nvmeotcp_add_skb_frag(netdev, skb, queue, nqe, fragsz);
+ to_copy += fragsz;
+ if (fragsz == remaining)
+ mlx5e_nvmeotcp_advance_sgl_iter(queue);
+ else
+ queue->ccoff_inner += fragsz;
+ }
+
+ if (cqe_bcnt > hlen + cclen) {
+ remaining = cqe_bcnt - hlen - cclen;
+ mlx5_nvmeotcp_add_tail(rq, cqe, queue, skb,
+ offset_in_page(skb->data) +
+ hlen + cclen, remaining);
+ }
+
+ mlx5e_nvmeotcp_put_queue(queue);
+ return true;
+}
+
+bool
+mlx5e_nvmeotcp_rebuild_rx_skb(struct mlx5e_rq *rq, struct sk_buff *skb,
+ struct mlx5_cqe64 *cqe, u32 cqe_bcnt)
+{
+ if (skb->data_len)
+ return mlx5e_nvmeotcp_rebuild_rx_skb_nonlinear(rq, skb, cqe, cqe_bcnt);
+ else
+ return mlx5e_nvmeotcp_rebuild_rx_skb_linear(rq, skb, cqe, cqe_bcnt);
+}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.h
new file mode 100644
index 000000000000..a8ca8a53bac6
--- /dev/null
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.h
@@ -0,0 +1,37 @@
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/* Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. */
+#ifndef __MLX5E_NVMEOTCP_RXTX_H__
+#define __MLX5E_NVMEOTCP_RXTX_H__
+
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+
+#include <linux/skbuff.h>
+#include "en_accel/nvmeotcp.h"
+
+bool
+mlx5e_nvmeotcp_rebuild_rx_skb(struct mlx5e_rq *rq, struct sk_buff *skb,
+ struct mlx5_cqe64 *cqe, u32 cqe_bcnt);
+
+static inline int mlx5_nvmeotcp_get_headlen(struct mlx5_cqe64 *cqe, u32 cqe_bcnt)
+{
+ struct mlx5e_cqe128 *cqe128;
+
+ if (!cqe_is_nvmeotcp_zc(cqe))
+ return cqe_bcnt;
+
+ cqe128 = container_of(cqe, struct mlx5e_cqe128, cqe64);
+ return be16_to_cpu(cqe128->hlen);
+}
+
+#else
+
+static inline bool
+mlx5e_nvmeotcp_rebuild_rx_skb(struct mlx5e_rq *rq, struct sk_buff *skb,
+ struct mlx5_cqe64 *cqe, u32 cqe_bcnt)
+{ return true; }
+
+static inline int mlx5_nvmeotcp_get_headlen(struct mlx5_cqe64 *cqe, u32 cqe_bcnt)
+{ return cqe_bcnt; }
+
+#endif /* CONFIG_MLX5_EN_NVMEOTCP */
+#endif /* __MLX5E_NVMEOTCP_RXTX_H__ */
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
index 89ef317bdd2f..a319a1494b0a 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
@@ -53,7 +53,7 @@
#include "en_accel/macsec.h"
#include "en_accel/ipsec_rxtx.h"
#include "en_accel/ktls_txrx.h"
-#include "en_accel/nvmeotcp.h"
+#include "en_accel/nvmeotcp_rxtx.h"
#include "en/xdp.h"
#include "en/xsk/rx.h"
#include "en/health.h"
@@ -336,10 +336,6 @@ static inline void mlx5e_put_rx_frag(struct mlx5e_rq *rq,
mlx5e_page_release_fragmented(rq, frag->frag_page);
}
-static inline struct mlx5e_wqe_frag_info *get_frag(struct mlx5e_rq *rq, u16 ix)
-{
- return &rq->wqe.frags[ix << rq->wqe.info.log_num_frags];
-}
static int mlx5e_alloc_rx_wqe(struct mlx5e_rq *rq, struct mlx5e_rx_wqe_cyc *wqe,
u16 ix)
@@ -1566,7 +1562,7 @@ static inline void mlx5e_handle_csum(struct net_device *netdev,
#define MLX5E_CE_BIT_MASK 0x80
-static inline void mlx5e_build_rx_skb(struct mlx5_cqe64 *cqe,
+static inline bool mlx5e_build_rx_skb(struct mlx5_cqe64 *cqe,
u32 cqe_bcnt,
struct mlx5e_rq *rq,
struct sk_buff *skb)
@@ -1577,6 +1573,13 @@ static inline void mlx5e_build_rx_skb(struct mlx5_cqe64 *cqe,
skb->mac_len = ETH_HLEN;
+ if (IS_ENABLED(CONFIG_MLX5_EN_NVMEOTCP) && cqe_is_nvmeotcp(cqe)) {
+ bool ret = mlx5e_nvmeotcp_rebuild_rx_skb(rq, skb, cqe, cqe_bcnt);
+
+ if (unlikely(!ret))
+ return ret;
+ }
+
if (unlikely(get_cqe_tls_offload(cqe)))
mlx5e_ktls_handle_rx_skb(rq, skb, cqe, &cqe_bcnt);
@@ -1623,6 +1626,8 @@ static inline void mlx5e_build_rx_skb(struct mlx5_cqe64 *cqe,
if (unlikely(mlx5e_skb_is_multicast(skb)))
stats->mcast_packets++;
+
+ return true;
}
static void mlx5e_shampo_complete_rx_cqe(struct mlx5e_rq *rq,
@@ -1646,7 +1651,7 @@ static void mlx5e_shampo_complete_rx_cqe(struct mlx5e_rq *rq,
}
}
-static inline void mlx5e_complete_rx_cqe(struct mlx5e_rq *rq,
+static inline bool mlx5e_complete_rx_cqe(struct mlx5e_rq *rq,
struct mlx5_cqe64 *cqe,
u32 cqe_bcnt,
struct sk_buff *skb)
@@ -1655,7 +1660,7 @@ static inline void mlx5e_complete_rx_cqe(struct mlx5e_rq *rq,
stats->packets++;
stats->bytes += cqe_bcnt;
- mlx5e_build_rx_skb(cqe, cqe_bcnt, rq, skb);
+ return mlx5e_build_rx_skb(cqe, cqe_bcnt, rq, skb);
}
static inline
@@ -1869,7 +1874,8 @@ static void mlx5e_handle_rx_cqe(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe)
goto wq_cyc_pop;
}
- mlx5e_complete_rx_cqe(rq, cqe, cqe_bcnt, skb);
+ if (unlikely(!mlx5e_complete_rx_cqe(rq, cqe, cqe_bcnt, skb)))
+ goto wq_cyc_pop;
if (mlx5e_cqe_regb_chain(cqe))
if (!mlx5e_tc_update_skb_nic(cqe, skb)) {
@@ -1916,7 +1922,8 @@ static void mlx5e_handle_rx_cqe_rep(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe)
goto wq_cyc_pop;
}
- mlx5e_complete_rx_cqe(rq, cqe, cqe_bcnt, skb);
+ if (unlikely(!mlx5e_complete_rx_cqe(rq, cqe, cqe_bcnt, skb)))
+ goto wq_cyc_pop;
if (rep->vlan && skb_vlan_tag_present(skb))
skb_vlan_pop(skb);
@@ -1965,7 +1972,8 @@ static void mlx5e_handle_rx_cqe_mpwrq_rep(struct mlx5e_rq *rq, struct mlx5_cqe64
if (!skb)
goto mpwrq_cqe_out;
- mlx5e_complete_rx_cqe(rq, cqe, cqe_bcnt, skb);
+ if (unlikely(!mlx5e_complete_rx_cqe(rq, cqe, cqe_bcnt, skb)))
+ goto mpwrq_cqe_out;
mlx5e_rep_tc_receive(cqe, rq, skb);
@@ -2011,13 +2019,18 @@ mlx5e_fill_skb_data(struct sk_buff *skb, struct mlx5e_rq *rq,
}
}
+static inline u16 mlx5e_get_headlen_hint(struct mlx5_cqe64 *cqe, u32 cqe_bcnt)
+{
+ return min_t(u32, MLX5E_RX_MAX_HEAD, mlx5_nvmeotcp_get_headlen(cqe, cqe_bcnt));
+}
+
static struct sk_buff *
mlx5e_skb_from_cqe_mpwrq_nonlinear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *wi,
struct mlx5_cqe64 *cqe, u16 cqe_bcnt, u32 head_offset,
u32 page_idx)
{
struct mlx5e_frag_page *frag_page = &wi->alloc_units.frag_pages[page_idx];
- u16 headlen = min_t(u16, MLX5E_RX_MAX_HEAD, cqe_bcnt);
+ u16 headlen = mlx5e_get_headlen_hint(cqe, cqe_bcnt);
struct mlx5e_frag_page *head_page = frag_page;
u32 frag_offset = head_offset;
u32 byte_cnt = cqe_bcnt;
@@ -2440,7 +2453,8 @@ static void mlx5e_handle_rx_cqe_mpwrq(struct mlx5e_rq *rq, struct mlx5_cqe64 *cq
if (!skb)
goto mpwrq_cqe_out;
- mlx5e_complete_rx_cqe(rq, cqe, cqe_bcnt, skb);
+ if (unlikely(!mlx5e_complete_rx_cqe(rq, cqe, cqe_bcnt, skb)))
+ goto mpwrq_cqe_out;
if (mlx5e_cqe_regb_chain(cqe))
if (!mlx5e_tc_update_skb_nic(cqe, skb)) {
@@ -2773,7 +2787,9 @@ static void mlx5e_trap_handle_rx_cqe(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe
if (!skb)
goto wq_cyc_pop;
- mlx5e_complete_rx_cqe(rq, cqe, cqe_bcnt, skb);
+ if (unlikely(!mlx5e_complete_rx_cqe(rq, cqe, cqe_bcnt, skb)))
+ goto wq_cyc_pop;
+
skb_push(skb, ETH_HLEN);
mlx5_devlink_trap_report(rq->mdev, trap_id, skb,
--
2.34.1
^ permalink raw reply related [flat|nested] 44+ messages in thread
* [PATCH v24 20/20] net/mlx5e: NVMEoTCP, statistics
2024-04-04 12:36 [PATCH v24 00/20] nvme-tcp receive offloads Aurelien Aptel
` (18 preceding siblings ...)
2024-04-04 12:37 ` [PATCH v24 19/20] net/mlx5e: NVMEoTCP, data-path for DDP+DDGST offload Aurelien Aptel
@ 2024-04-04 12:37 ` Aurelien Aptel
2024-04-06 5:45 ` [PATCH v24 00/20] nvme-tcp receive offloads Jakub Kicinski
20 siblings, 0 replies; 44+ messages in thread
From: Aurelien Aptel @ 2024-04-04 12:37 UTC (permalink / raw)
To: linux-nvme, netdev, sagi, hch, kbusch, axboe, chaitanyak, davem, kuba
Cc: Aurelien Aptel, aurelien.aptel, smalin, malin1024, ogerlitz,
yorayz, borisp, galshalom, mgurtovoy
NVMEoTCP offload statistics include both control and data path
statistic: counters for the netdev ddp ops, offloaded packets/bytes,
resync and dropped packets.
Expose the statistics using ulp_ddp_ops->get_stats()
instead of the regular statistics flow.
Signed-off-by: Ben Ben-Ishay <benishay@nvidia.com>
Signed-off-by: Boris Pismenny <borisp@nvidia.com>
Signed-off-by: Or Gerlitz <ogerlitz@nvidia.com>
Signed-off-by: Yoray Zack <yorayz@nvidia.com>
Signed-off-by: Shai Malin <smalin@nvidia.com>
Signed-off-by: Aurelien Aptel <aaptel@nvidia.com>
Reviewed-by: Tariq Toukan <tariqt@nvidia.com>
---
.../net/ethernet/mellanox/mlx5/core/Makefile | 3 +-
.../mellanox/mlx5/core/en_accel/nvmeotcp.c | 52 ++++++++++++---
.../mellanox/mlx5/core/en_accel/nvmeotcp.h | 16 +++++
.../mlx5/core/en_accel/nvmeotcp_rxtx.c | 11 +++-
.../mlx5/core/en_accel/nvmeotcp_stats.c | 66 +++++++++++++++++++
.../ethernet/mellanox/mlx5/core/en_stats.h | 8 +++
6 files changed, 145 insertions(+), 11 deletions(-)
create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_stats.c
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/Makefile b/drivers/net/ethernet/mellanox/mlx5/core/Makefile
index 19691917682e..65d6555bb8a5 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/Makefile
+++ b/drivers/net/ethernet/mellanox/mlx5/core/Makefile
@@ -109,7 +109,8 @@ mlx5_core-$(CONFIG_MLX5_EN_TLS) += en_accel/ktls_stats.o \
en_accel/fs_tcp.o en_accel/ktls.o en_accel/ktls_txrx.o \
en_accel/ktls_tx.o en_accel/ktls_rx.o
-mlx5_core-$(CONFIG_MLX5_EN_NVMEOTCP) += en_accel/fs_tcp.o en_accel/nvmeotcp.o en_accel/nvmeotcp_rxtx.o
+mlx5_core-$(CONFIG_MLX5_EN_NVMEOTCP) += en_accel/fs_tcp.o en_accel/nvmeotcp.o \
+ en_accel/nvmeotcp_rxtx.o en_accel/nvmeotcp_stats.o
mlx5_core-$(CONFIG_MLX5_SW_STEERING) += steering/dr_domain.o steering/dr_table.o \
steering/dr_matcher.o steering/dr_rule.o \
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c
index 505f4e6d5098..67c33915ca50 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.c
@@ -616,9 +616,15 @@ mlx5e_nvmeotcp_queue_init(struct net_device *netdev,
{
struct nvme_tcp_ddp_config *nvme_config = &config->nvmeotcp;
struct mlx5e_priv *priv = netdev_priv(netdev);
+ struct mlx5e_nvmeotcp_sw_stats *sw_stats;
struct mlx5_core_dev *mdev = priv->mdev;
struct mlx5e_nvmeotcp_queue *queue;
int queue_id, err;
+ u32 channel_ix;
+
+ channel_ix = mlx5e_get_channel_ix_from_io_cpu(&priv->channels.params,
+ config->io_cpu);
+ sw_stats = &priv->nvmeotcp->sw_stats;
if (config->type != ULP_DDP_NVME) {
err = -EOPNOTSUPP;
@@ -645,11 +651,11 @@ mlx5e_nvmeotcp_queue_init(struct net_device *netdev,
queue->id = queue_id;
queue->dgst = nvme_config->dgst;
queue->pda = nvme_config->cpda;
- queue->channel_ix = mlx5e_get_channel_ix_from_io_cpu(&priv->channels.params,
- config->io_cpu);
+ queue->channel_ix = channel_ix;
queue->size = nvme_config->queue_size;
queue->max_klms_per_wqe = MLX5E_MAX_KLM_PER_WQE(mdev);
queue->priv = priv;
+ queue->sw_stats = sw_stats;
init_completion(&queue->static_params_done);
err = mlx5e_nvmeotcp_queue_rx_init(queue, config, netdev);
@@ -661,6 +667,7 @@ mlx5e_nvmeotcp_queue_init(struct net_device *netdev,
if (err)
goto destroy_rx;
+ atomic64_inc(&sw_stats->rx_nvmeotcp_sk_add);
write_lock_bh(&sk->sk_callback_lock);
ulp_ddp_set_ctx(sk, queue);
write_unlock_bh(&sk->sk_callback_lock);
@@ -674,6 +681,7 @@ mlx5e_nvmeotcp_queue_init(struct net_device *netdev,
free_queue:
kfree(queue);
out:
+ atomic64_inc(&sw_stats->rx_nvmeotcp_sk_add_fail);
return err;
}
@@ -687,6 +695,8 @@ mlx5e_nvmeotcp_queue_teardown(struct net_device *netdev,
queue = container_of(ulp_ddp_get_ctx(sk), struct mlx5e_nvmeotcp_queue, ulp_ddp_ctx);
+ atomic64_inc(&queue->sw_stats->rx_nvmeotcp_sk_del);
+
WARN_ON(refcount_read(&queue->ref_count) != 1);
mlx5e_nvmeotcp_destroy_rx(priv, queue, mdev);
@@ -818,25 +828,34 @@ mlx5e_nvmeotcp_ddp_setup(struct net_device *netdev,
struct ulp_ddp_io *ddp)
{
struct scatterlist *sg = ddp->sg_table.sgl;
+ struct mlx5e_nvmeotcp_sw_stats *sw_stats;
struct mlx5e_nvmeotcp_queue_entry *nvqt;
struct mlx5e_nvmeotcp_queue *queue;
struct mlx5_core_dev *mdev;
int i, size = 0, count = 0;
+ int ret = 0;
queue = container_of(ulp_ddp_get_ctx(sk),
struct mlx5e_nvmeotcp_queue, ulp_ddp_ctx);
+ sw_stats = queue->sw_stats;
mdev = queue->priv->mdev;
count = dma_map_sg(mdev->device, ddp->sg_table.sgl, ddp->nents,
DMA_FROM_DEVICE);
- if (count <= 0)
- return -EINVAL;
+ if (count <= 0) {
+ ret = -EINVAL;
+ goto ddp_setup_fail;
+ }
- if (WARN_ON(count > mlx5e_get_max_sgl(mdev)))
- return -ENOSPC;
+ if (WARN_ON(count > mlx5e_get_max_sgl(mdev))) {
+ ret = -ENOSPC;
+ goto ddp_setup_fail;
+ }
- if (!mlx5e_nvmeotcp_validate_sgl(sg, count, READ_ONCE(netdev->mtu)))
- return -EOPNOTSUPP;
+ if (!mlx5e_nvmeotcp_validate_sgl(sg, count, READ_ONCE(netdev->mtu))) {
+ ret = -EOPNOTSUPP;
+ goto ddp_setup_fail;
+ }
for (i = 0; i < count; i++)
size += sg_dma_len(&sg[i]);
@@ -848,8 +867,13 @@ mlx5e_nvmeotcp_ddp_setup(struct net_device *netdev,
nvqt->ccid_gen++;
nvqt->sgl_length = count;
mlx5e_nvmeotcp_post_klm_wqe(queue, KLM_UMR, ddp->command_id, count);
-
+ atomic64_inc(&sw_stats->rx_nvmeotcp_ddp_setup);
return 0;
+
+ddp_setup_fail:
+ dma_unmap_sg(mdev->device, ddp->sg_table.sgl, count, DMA_FROM_DEVICE);
+ atomic64_inc(&sw_stats->rx_nvmeotcp_ddp_setup_fail);
+ return ret;
}
void mlx5e_nvmeotcp_ctx_complete(struct mlx5e_icosq_wqe_info *wi)
@@ -896,6 +920,7 @@ mlx5e_nvmeotcp_ddp_teardown(struct net_device *netdev,
q_entry->queue = queue;
mlx5e_nvmeotcp_post_klm_wqe(queue, KLM_INV_UMR, ddp->command_id, 0);
+ atomic64_inc(&queue->sw_stats->rx_nvmeotcp_ddp_teardown);
}
static void
@@ -929,6 +954,14 @@ void mlx5e_nvmeotcp_put_queue(struct mlx5e_nvmeotcp_queue *queue)
}
}
+static int mlx5e_ulp_ddp_get_stats(struct net_device *dev,
+ struct ulp_ddp_stats *stats)
+{
+ struct mlx5e_priv *priv = netdev_priv(dev);
+
+ return mlx5e_nvmeotcp_get_stats(priv, stats);
+}
+
int set_ulp_ddp_nvme_tcp(struct net_device *netdev, bool enable)
{
struct mlx5e_priv *priv = netdev_priv(netdev);
@@ -1028,6 +1061,7 @@ const struct ulp_ddp_dev_ops mlx5e_nvmeotcp_ops = {
.resync = mlx5e_nvmeotcp_ddp_resync,
.set_caps = mlx5e_ulp_ddp_set_caps,
.get_caps = mlx5e_ulp_ddp_get_caps,
+ .get_stats = mlx5e_ulp_ddp_get_stats,
};
void mlx5e_nvmeotcp_cleanup_rx(struct mlx5e_priv *priv)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h
index 13817d8a0aae..41b5b304e598 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp.h
@@ -9,6 +9,15 @@
#include "en.h"
#include "en/params.h"
+struct mlx5e_nvmeotcp_sw_stats {
+ atomic64_t rx_nvmeotcp_sk_add;
+ atomic64_t rx_nvmeotcp_sk_add_fail;
+ atomic64_t rx_nvmeotcp_sk_del;
+ atomic64_t rx_nvmeotcp_ddp_setup;
+ atomic64_t rx_nvmeotcp_ddp_setup_fail;
+ atomic64_t rx_nvmeotcp_ddp_teardown;
+};
+
struct mlx5e_nvmeotcp_queue_entry {
struct mlx5e_nvmeotcp_queue *queue;
u32 sgl_length;
@@ -52,6 +61,7 @@ struct mlx5e_nvmeotcp_queue_handler {
* @sk: The socket used by the NVMe-TCP queue
* @crc_rx: CRC Rx offload indication for this queue
* @priv: mlx5e netdev priv
+ * @sw_stats: Global software statistics for nvmeotcp offload
* @static_params_done: Async completion structure for the initial umr mapping
* synchronization
* @sq_lock: Spin lock for the icosq
@@ -88,6 +98,7 @@ struct mlx5e_nvmeotcp_queue {
u8 crc_rx:1;
/* for ddp invalidate flow */
struct mlx5e_priv *priv;
+ struct mlx5e_nvmeotcp_sw_stats *sw_stats;
/* end of data-path section */
struct completion static_params_done;
@@ -97,6 +108,7 @@ struct mlx5e_nvmeotcp_queue {
};
struct mlx5e_nvmeotcp {
+ struct mlx5e_nvmeotcp_sw_stats sw_stats;
struct ida queue_ids;
struct rhashtable queue_hash;
struct ulp_ddp_dev_caps ddp_caps;
@@ -113,6 +125,7 @@ void mlx5e_nvmeotcp_ddp_inv_done(struct mlx5e_icosq_wqe_info *wi);
void mlx5e_nvmeotcp_ctx_complete(struct mlx5e_icosq_wqe_info *wi);
static inline void mlx5e_nvmeotcp_init_rx(struct mlx5e_priv *priv) {}
void mlx5e_nvmeotcp_cleanup_rx(struct mlx5e_priv *priv);
+int mlx5e_nvmeotcp_get_stats(struct mlx5e_priv *priv, struct ulp_ddp_stats *stats);
extern const struct ulp_ddp_dev_ops mlx5e_nvmeotcp_ops;
#else
@@ -121,5 +134,8 @@ static inline void mlx5e_nvmeotcp_cleanup(struct mlx5e_priv *priv) {}
static inline int set_ulp_ddp_nvme_tcp(struct net_device *dev, bool en) { return -EOPNOTSUPP; }
static inline void mlx5e_nvmeotcp_init_rx(struct mlx5e_priv *priv) {}
static inline void mlx5e_nvmeotcp_cleanup_rx(struct mlx5e_priv *priv) {}
+static inline int mlx5e_nvmeotcp_get_stats(struct mlx5e_priv *priv,
+ struct ulp_ddp_stats *stats)
+{ return 0; }
#endif
#endif /* __MLX5E_NVMEOTCP_H__ */
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.c
index 269d8075f3c2..6ed9acdec376 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_rxtx.c
@@ -140,6 +140,7 @@ mlx5e_nvmeotcp_rebuild_rx_skb_nonlinear(struct mlx5e_rq *rq, struct sk_buff *skb
int ccoff, cclen, hlen, ccid, remaining, fragsz, to_copy = 0;
struct net_device *netdev = rq->netdev;
struct mlx5e_priv *priv = netdev_priv(netdev);
+ struct mlx5e_rq_stats *stats = rq->stats;
struct mlx5e_nvmeotcp_queue_entry *nqe;
skb_frag_t org_frags[MAX_SKB_FRAGS];
struct mlx5e_nvmeotcp_queue *queue;
@@ -151,12 +152,14 @@ mlx5e_nvmeotcp_rebuild_rx_skb_nonlinear(struct mlx5e_rq *rq, struct sk_buff *skb
queue = mlx5e_nvmeotcp_get_queue(priv->nvmeotcp, queue_id);
if (unlikely(!queue)) {
dev_kfree_skb_any(skb);
+ stats->nvmeotcp_drop++;
return false;
}
cqe128 = container_of(cqe, struct mlx5e_cqe128, cqe64);
if (cqe_is_nvmeotcp_resync(cqe)) {
nvmeotcp_update_resync(queue, cqe128);
+ stats->nvmeotcp_resync++;
mlx5e_nvmeotcp_put_queue(queue);
return true;
}
@@ -230,7 +233,8 @@ mlx5e_nvmeotcp_rebuild_rx_skb_nonlinear(struct mlx5e_rq *rq, struct sk_buff *skb
org_nr_frags,
frag_index);
}
-
+ stats->nvmeotcp_packets++;
+ stats->nvmeotcp_bytes += cclen;
mlx5e_nvmeotcp_put_queue(queue);
return true;
}
@@ -242,6 +246,7 @@ mlx5e_nvmeotcp_rebuild_rx_skb_linear(struct mlx5e_rq *rq, struct sk_buff *skb,
int ccoff, cclen, hlen, ccid, remaining, fragsz, to_copy = 0;
struct net_device *netdev = rq->netdev;
struct mlx5e_priv *priv = netdev_priv(netdev);
+ struct mlx5e_rq_stats *stats = rq->stats;
struct mlx5e_nvmeotcp_queue_entry *nqe;
struct mlx5e_nvmeotcp_queue *queue;
struct mlx5e_cqe128 *cqe128;
@@ -251,12 +256,14 @@ mlx5e_nvmeotcp_rebuild_rx_skb_linear(struct mlx5e_rq *rq, struct sk_buff *skb,
queue = mlx5e_nvmeotcp_get_queue(priv->nvmeotcp, queue_id);
if (unlikely(!queue)) {
dev_kfree_skb_any(skb);
+ stats->nvmeotcp_drop++;
return false;
}
cqe128 = container_of(cqe, struct mlx5e_cqe128, cqe64);
if (cqe_is_nvmeotcp_resync(cqe)) {
nvmeotcp_update_resync(queue, cqe128);
+ stats->nvmeotcp_resync++;
mlx5e_nvmeotcp_put_queue(queue);
return true;
}
@@ -330,6 +337,8 @@ mlx5e_nvmeotcp_rebuild_rx_skb_linear(struct mlx5e_rq *rq, struct sk_buff *skb,
hlen + cclen, remaining);
}
+ stats->nvmeotcp_packets++;
+ stats->nvmeotcp_bytes += cclen;
mlx5e_nvmeotcp_put_queue(queue);
return true;
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_stats.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_stats.c
new file mode 100644
index 000000000000..af1838154bf8
--- /dev/null
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/nvmeotcp_stats.c
@@ -0,0 +1,66 @@
+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
+// Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES.
+
+#include "en_accel/nvmeotcp.h"
+
+struct ulp_ddp_counter_map {
+ size_t eth_offset;
+ size_t mlx_offset;
+};
+
+#define DECLARE_ULP_SW_STAT(fld) \
+ { offsetof(struct ulp_ddp_stats, fld), \
+ offsetof(struct mlx5e_nvmeotcp_sw_stats, fld) }
+
+#define DECLARE_ULP_RQ_STAT(fld) \
+ { offsetof(struct ulp_ddp_stats, rx_ ## fld), \
+ offsetof(struct mlx5e_rq_stats, fld) }
+
+#define READ_CTR_ATOMIC64(ptr, dsc, i) \
+ atomic64_read((atomic64_t *)((char *)(ptr) + (dsc)[i].mlx_offset))
+
+#define READ_CTR(ptr, desc, i) \
+ (*((u64 *)((char *)(ptr) + (desc)[i].mlx_offset)))
+
+#define SET_ULP_STAT(ptr, desc, i, val) \
+ (*(u64 *)((char *)(ptr) + (desc)[i].eth_offset) = (val))
+
+/* Global counters */
+static const struct ulp_ddp_counter_map sw_stats_desc[] = {
+ DECLARE_ULP_SW_STAT(rx_nvmeotcp_sk_add),
+ DECLARE_ULP_SW_STAT(rx_nvmeotcp_sk_del),
+ DECLARE_ULP_SW_STAT(rx_nvmeotcp_ddp_setup),
+ DECLARE_ULP_SW_STAT(rx_nvmeotcp_ddp_setup_fail),
+ DECLARE_ULP_SW_STAT(rx_nvmeotcp_ddp_teardown),
+};
+
+/* Per-rx-queue counters */
+static const struct ulp_ddp_counter_map rq_stats_desc[] = {
+ DECLARE_ULP_RQ_STAT(nvmeotcp_drop),
+ DECLARE_ULP_RQ_STAT(nvmeotcp_resync),
+ DECLARE_ULP_RQ_STAT(nvmeotcp_packets),
+ DECLARE_ULP_RQ_STAT(nvmeotcp_bytes),
+};
+
+int mlx5e_nvmeotcp_get_stats(struct mlx5e_priv *priv, struct ulp_ddp_stats *stats)
+{
+ unsigned int i, ch, n = 0;
+
+ if (!priv->nvmeotcp)
+ return 0;
+
+ for (i = 0; i < ARRAY_SIZE(sw_stats_desc); i++, n++)
+ SET_ULP_STAT(stats, sw_stats_desc, i,
+ READ_CTR_ATOMIC64(&priv->nvmeotcp->sw_stats, sw_stats_desc, i));
+
+ for (i = 0; i < ARRAY_SIZE(rq_stats_desc); i++, n++) {
+ u64 sum = 0;
+
+ for (ch = 0; ch < priv->stats_nch; ch++)
+ sum += READ_CTR(&priv->channel_stats[ch]->rq, rq_stats_desc, i);
+
+ SET_ULP_STAT(stats, rq_stats_desc, i, sum);
+ }
+
+ return n;
+}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_stats.h b/drivers/net/ethernet/mellanox/mlx5/core/en_stats.h
index 12b3607afecd..929a0723812f 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_stats.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_stats.h
@@ -128,6 +128,8 @@ void mlx5e_stats_rmon_get(struct mlx5e_priv *priv,
const struct ethtool_rmon_hist_range **ranges);
void mlx5e_get_link_ext_stats(struct net_device *dev,
struct ethtool_link_ext_stats *stats);
+struct ulp_ddp_stats;
+void mlx5e_stats_ulp_ddp_get(struct mlx5e_priv *priv, struct ulp_ddp_stats *stats);
/* Concrete NIC Stats */
@@ -396,6 +398,12 @@ struct mlx5e_rq_stats {
u64 tls_resync_res_skip;
u64 tls_err;
#endif
+#ifdef CONFIG_MLX5_EN_NVMEOTCP
+ u64 nvmeotcp_drop;
+ u64 nvmeotcp_resync;
+ u64 nvmeotcp_packets;
+ u64 nvmeotcp_bytes;
+#endif
};
struct mlx5e_sq_stats {
--
2.34.1
^ permalink raw reply related [flat|nested] 44+ messages in thread
* Re: [PATCH v24 00/20] nvme-tcp receive offloads
2024-04-04 12:36 [PATCH v24 00/20] nvme-tcp receive offloads Aurelien Aptel
` (19 preceding siblings ...)
2024-04-04 12:37 ` [PATCH v24 20/20] net/mlx5e: NVMEoTCP, statistics Aurelien Aptel
@ 2024-04-06 5:45 ` Jakub Kicinski
2024-04-07 22:21 ` Sagi Grimberg
20 siblings, 1 reply; 44+ messages in thread
From: Jakub Kicinski @ 2024-04-06 5:45 UTC (permalink / raw)
To: Aurelien Aptel
Cc: linux-nvme, netdev, sagi, hch, kbusch, axboe, chaitanyak, davem,
aurelien.aptel, smalin, malin1024, ogerlitz, yorayz, borisp,
galshalom, mgurtovoy, edumazet
Doesn't apply, again, unfortunately.
On Thu, 4 Apr 2024 12:36:57 +0000 Aurelien Aptel wrote:
> Testing
> =======
> This series was tested on ConnectX-7 HW using various configurations
> of IO sizes, queue depths, MTUs, and with both the SPDK and kernel NVMe-TCP
> targets.
About testing, what do you have in terms of a testing setup?
As you said this is similar to the TLS offload:
> Note:
> These offloads are similar in nature to the packet-based NIC TLS offloads,
> which are already upstream (see net/tls/tls_device.c).
> You can read more about TLS offload here:
> https://www.kernel.org/doc/html/latest/networking/tls-offload.html
and our experience trying to maintain and extend the very much used SW
kernel TLS implementation in presence of the device offload is mixed :(
We can't test it, we break it, get CVEs for it :( In all fairness
the inline offload is probably not as bad as the crypto accelerator
path, but still it breaks. So assuming that this gets all the necessary
acks can we expect you to plug some testing into the netdev CI so that
we see whether any incoming change is breaking this offload?
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v24 05/20] nvme-tcp: Add DDP offload control path
2024-04-04 12:37 ` [PATCH v24 05/20] nvme-tcp: Add DDP offload control path Aurelien Aptel
@ 2024-04-07 22:08 ` Sagi Grimberg
2024-04-10 6:31 ` Aurelien Aptel
0 siblings, 1 reply; 44+ messages in thread
From: Sagi Grimberg @ 2024-04-07 22:08 UTC (permalink / raw)
To: Aurelien Aptel, linux-nvme, netdev, hch, kbusch, axboe,
chaitanyak, davem, kuba
Cc: Boris Pismenny, aurelien.aptel, smalin, malin1024, ogerlitz,
yorayz, galshalom, mgurtovoy, brauner
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v24 06/20] nvme-tcp: Add DDP data-path
2024-04-04 12:37 ` [PATCH v24 06/20] nvme-tcp: Add DDP data-path Aurelien Aptel
@ 2024-04-07 22:08 ` Sagi Grimberg
2024-04-10 6:31 ` Aurelien Aptel
0 siblings, 1 reply; 44+ messages in thread
From: Sagi Grimberg @ 2024-04-07 22:08 UTC (permalink / raw)
To: Aurelien Aptel, linux-nvme, netdev, hch, kbusch, axboe,
chaitanyak, davem, kuba
Cc: Boris Pismenny, aurelien.aptel, smalin, malin1024, ogerlitz,
yorayz, galshalom, mgurtovoy
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v24 00/20] nvme-tcp receive offloads
2024-04-06 5:45 ` [PATCH v24 00/20] nvme-tcp receive offloads Jakub Kicinski
@ 2024-04-07 22:21 ` Sagi Grimberg
2024-04-09 22:35 ` Chaitanya Kulkarni
0 siblings, 1 reply; 44+ messages in thread
From: Sagi Grimberg @ 2024-04-07 22:21 UTC (permalink / raw)
To: Jakub Kicinski, Aurelien Aptel
Cc: linux-nvme, netdev, hch, kbusch, axboe, chaitanyak, davem,
aurelien.aptel, smalin, malin1024, ogerlitz, yorayz, borisp,
galshalom, mgurtovoy, edumazet
On 06/04/2024 8:45, Jakub Kicinski wrote:
> Doesn't apply, again, unfortunately.
>
> On Thu, 4 Apr 2024 12:36:57 +0000 Aurelien Aptel wrote:
>> Testing
>> =======
>> This series was tested on ConnectX-7 HW using various configurations
>> of IO sizes, queue depths, MTUs, and with both the SPDK and kernel NVMe-TCP
>> targets.
> About testing, what do you have in terms of a testing setup?
> As you said this is similar to the TLS offload:
>
>> Note:
>> These offloads are similar in nature to the packet-based NIC TLS offloads,
>> which are already upstream (see net/tls/tls_device.c).
>> You can read more about TLS offload here:
>> https://www.kernel.org/doc/html/latest/networking/tls-offload.html
> and our experience trying to maintain and extend the very much used SW
> kernel TLS implementation in presence of the device offload is mixed :(
> We can't test it, we break it, get CVEs for it :( In all fairness
Especially when nvme-tcp can also run on tls, but that is incompatible with
this offload at the moment (I've told the nvidia folks that I do not expect
this incompatibility to be permanent).
> the inline offload is probably not as bad as the crypto accelerator
> path, but still it breaks. So assuming that this gets all the necessary
> acks can we expect you to plug some testing into the netdev CI so that
> we see whether any incoming change is breaking this offload?
Agree, also given that there is an effort to extend blktests to run on
real controllers, perhaps we should add a few tests there as well?
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v24 09/20] Documentation: add ULP DDP offload documentation
2024-04-04 12:37 ` [PATCH v24 09/20] Documentation: add ULP DDP offload documentation Aurelien Aptel
@ 2024-04-09 8:49 ` Bagas Sanjaya
0 siblings, 0 replies; 44+ messages in thread
From: Bagas Sanjaya @ 2024-04-09 8:49 UTC (permalink / raw)
To: Aurelien Aptel, linux-nvme, netdev, sagi, hch, kbusch, axboe,
chaitanyak, davem, kuba
Cc: Yoray Zack, aurelien.aptel, smalin, malin1024, ogerlitz, borisp,
galshalom, mgurtovoy, linux-doc, edumazet, pabeni, corbet
[-- Attachment #1: Type: text/plain, Size: 16027 bytes --]
On Thu, Apr 04, 2024 at 12:37:06PM +0000, Aurelien Aptel wrote:
> diff --git a/Documentation/networking/ulp-ddp-offload.rst b/Documentation/networking/ulp-ddp-offload.rst
> new file mode 100644
> index 000000000000..4133e5094ff5
> --- /dev/null
> +++ b/Documentation/networking/ulp-ddp-offload.rst
> @@ -0,0 +1,372 @@
> +.. SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
> +
> +=================================
> +ULP direct data placement offload
> +=================================
> +
> +Overview
> +========
> +
> +The Linux kernel ULP direct data placement (DDP) offload infrastructure
> +provides tagged request-response protocols, such as NVMe-TCP, the ability to
> +place response data directly in pre-registered buffers according to header
> +tags. DDP is particularly useful for data-intensive pipelined protocols whose
> +responses may be reordered.
> +
> +For example, in NVMe-TCP numerous read requests are sent together and each
> +request is tagged using the PDU header CID field. Receiving servers process
> +requests as fast as possible and sometimes responses for smaller requests
> +bypasses responses to larger requests, e.g., 4KB reads bypass 1GB reads.
> +Thereafter, clients correlate responses to requests using PDU header CID tags.
> +The processing of each response requires copying data from SKBs to read
> +request destination buffers; The offload avoids this copy. The offload is
> +oblivious to destination buffers which can reside either in userspace
> +(O_DIRECT) or in kernel pagecache.
> +
> +Request TCP byte-stream:
> +
> +.. parsed-literal::
> +
> + +---------------+-------+---------------+-------+---------------+-------+
> + | PDU hdr CID=1 | Req 1 | PDU hdr CID=2 | Req 2 | PDU hdr CID=3 | Req 3 |
> + +---------------+-------+---------------+-------+---------------+-------+
> +
> +Response TCP byte-stream:
> +
> +.. parsed-literal::
> +
> + +---------------+--------+---------------+--------+---------------+--------+
> + | PDU hdr CID=2 | Resp 2 | PDU hdr CID=3 | Resp 3 | PDU hdr CID=1 | Resp 1 |
> + +---------------+--------+---------------+--------+---------------+--------+
> +
> +The driver builds SKB page fragments that point to destination buffers.
> +Consequently, SKBs represent the original data on the wire, which enables
> +*transparent* inter-operation with the network stack. To avoid copies between
> +SKBs and destination buffers, the layer-5 protocol (L5P) will check
> +``if (src == dst)`` for SKB page fragments, success indicates that data is
> +already placed there by NIC hardware and copy should be skipped.
> +
> +In addition, L5P might have DDGST which ensures data integrity over
> +the network. If not offloaded, ULP DDP might not be efficient as L5P
> +will need to go over the data and calculate it by itself, cancelling
> +out the benefits of the DDP copy skip. ULP DDP has support for Rx/Tx
> +DDGST offload. On the received side the NIC will verify DDGST for
> +received PDUs and update SKB->ulp_ddp and SKB->ulp_crc bits. If all the SKBs
> +making up a L5P PDU have crc on, L5P will skip on calculating and
> +verifying the DDGST for the corresponding PDU. On the Tx side, the NIC
> +will be responsible for calculating and filling the DDGST fields in
> +the sent PDUs.
> +
> +Offloading does require NIC hardware to track L5P protocol framing, similarly
> +to RX TLS offload (see Documentation/networking/tls-offload.rst). NIC hardware
> +will parse PDU headers, extract fields such as operation type, length, tag
> +identifier, etc. and only offload segments that correspond to tags registered
> +with the NIC, see the :ref:`buf_reg` section.
> +
> +Device configuration
> +====================
> +
> +During driver initialization the driver sets the ULP DDP operations
> +for the :c:type:`struct net_device <net_device>` via
> +`netdev->netdev_ops->ulp_ddp_ops`.
> +
> +The :c:member:`get_caps` operation returns the ULP DDP capabilities
> +enabled and/or supported by the device to the caller. The current list
> +of capabilities is represented as a bitset:
> +
> +.. code-block:: c
> +
> + enum ulp_ddp_cap {
> + ULP_DDP_CAP_NVME_TCP,
> + ULP_DDP_CAP_NVME_TCP_DDGST,
> + };
> +
> +The enablement of capabilities can be controlled via the
> +:c:member:`set_caps` operation. This operation is exposed to userspace
> +via netlink. See Documentation/netlink/specs/ulp_ddp.yaml for more
> +details.
> +
> +Later, after the L5P completes its handshake, the L5P queries the
> +driver for its runtime limitations via the :c:member:`limits` operation:
> +
> +.. code-block:: c
> +
> + int (*limits)(struct net_device *netdev,
> + struct ulp_ddp_limits *lim);
> +
> +
> +All L5P share a common set of limits and parameters (:c:type:`struct ulp_ddp_limits <ulp_ddp_limits>`):
> +
> +.. code-block:: c
> +
> + /**
> + * struct ulp_ddp_limits - Generic ulp ddp limits: tcp ddp
> + * protocol limits.
> + * Add new instances of ulp_ddp_limits in the union below (nvme-tcp, etc.).
> + *
> + * @type: type of this limits struct
> + * @max_ddp_sgl_len: maximum sgl size supported (zero means no limit)
> + * @io_threshold: minimum payload size required to offload
> + * @tls: support for ULP over TLS
> + * @nvmeotcp: NVMe-TCP specific limits
> + */
> + struct ulp_ddp_limits {
> + enum ulp_ddp_type type;
> + int max_ddp_sgl_len;
> + int io_threshold;
> + bool tls:1;
> + union {
> + /* ... protocol-specific limits ... */
> + struct nvme_tcp_ddp_limits nvmeotcp;
> + };
> + };
> +
> +But each L5P can also add protocol-specific limits e.g.:
> +
> +.. code-block:: c
> +
> + /**
> + * struct nvme_tcp_ddp_limits - nvme tcp driver limitations
> + *
> + * @full_ccid_range: true if the driver supports the full CID range
> + */
> + struct nvme_tcp_ddp_limits {
> + bool full_ccid_range;
> + };
> +
> +Once the L5P has made sure the device is supported the offload
> +operations are installed on the socket.
> +
> +If offload installation fails, then the connection is handled by software as if
> +offload was not attempted.
> +
> +To request offload for a socket `sk`, the L5P calls :c:member:`sk_add`:
> +
> +.. code-block:: c
> +
> + int (*sk_add)(struct net_device *netdev,
> + struct sock *sk,
> + struct ulp_ddp_config *config);
> +
> +The function return 0 for success. In case of failure, L5P software should
> +fallback to normal non-offloaded operations. The `config` parameter indicates
> +the L5P type and any metadata relevant for that protocol. For example, in
> +NVMe-TCP the following config is used:
> +
> +.. code-block:: c
> +
> + /**
> + * struct nvme_tcp_ddp_config - nvme tcp ddp configuration for an IO queue
> + *
> + * @pfv: pdu version (e.g., NVME_TCP_PFV_1_0)
> + * @cpda: controller pdu data alignment (dwords, 0's based)
> + * @dgst: digest types enabled.
> + * The netdev will offload crc if L5P data digest is supported.
> + * @queue_size: number of nvme-tcp IO queue elements
> + */
> + struct nvme_tcp_ddp_config {
> + u16 pfv;
> + u8 cpda;
> + u8 dgst;
> + int queue_size;
> + };
> +
> +When offload is not needed anymore, e.g. when the socket is being released, the L5P
> +calls :c:member:`sk_del` to release device contexts:
> +
> +.. code-block:: c
> +
> + void (*sk_del)(struct net_device *netdev,
> + struct sock *sk);
> +
> +Normal operation
> +================
> +
> +At the very least, the device maintains the following state for each connection:
> +
> + * 5-tuple
> + * expected TCP sequence number
> + * mapping between tags and corresponding buffers
> + * current offset within PDU, PDU length, current PDU tag
> +
> +NICs should not assume any correlation between PDUs and TCP packets.
> +If TCP packets arrive in-order, offload will place PDU payloads
> +directly inside corresponding registered buffers. NIC offload should
> +not delay packets. If offload is not possible, than the packet is
> +passed as-is to software. To perform offload on incoming packets
> +without buffering packets in the NIC, the NIC stores some inter-packet
> +state, such as partial PDU headers.
> +
> +RX data-path
> +------------
> +
> +After the device validates TCP checksums, it can perform DDP offload. The
> +packet is steered to the DDP offload context according to the 5-tuple.
> +Thereafter, the expected TCP sequence number is checked against the packet
> +TCP sequence number. If there is a match, offload is performed: the PDU payload
> +is DMA written to the corresponding destination buffer according to the PDU header
> +tag. The data should be DMAed only once, and the NIC receive ring will only
> +store the remaining TCP and PDU headers.
> +
> +We remark that a single TCP packet may have numerous PDUs embedded inside. NICs
> +can choose to offload one or more of these PDUs according to various
> +trade-offs. Possibly, offloading such small PDUs is of little value, and it is
> +better to leave it to software.
> +
> +Upon receiving a DDP offloaded packet, the driver reconstructs the original SKB
> +using page frags, while pointing to the destination buffers whenever possible.
> +This method enables seamless integration with the network stack, which can
> +inspect and modify packet fields transparently to the offload.
> +
> +.. _buf_reg:
> +
> +Destination buffer registration
> +-------------------------------
> +
> +To register the mapping between tags and destination buffers for a socket
> +`sk`, the L5P calls :c:member:`setup` of :c:type:`struct ulp_ddp_dev_ops
> +<ulp_ddp_dev_ops>`:
> +
> +.. code-block:: c
> +
> + int (*setup)(struct net_device *netdev,
> + struct sock *sk,
> + struct ulp_ddp_io *io);
> +
> +
> +The `io` provides the buffer via scatter-gather list (`sg_table`) and
> +corresponding tag (`command_id`):
> +
> +.. code-block:: c
> +
> + /**
> + * struct ulp_ddp_io - tcp ddp configuration for an IO request.
> + *
> + * @command_id: identifier on the wire associated with these buffers
> + * @nents: number of entries in the sg_table
> + * @sg_table: describing the buffers for this IO request
> + * @first_sgl: first SGL in sg_table
> + */
> + struct ulp_ddp_io {
> + u32 command_id;
> + int nents;
> + struct sg_table sg_table;
> + struct scatterlist first_sgl[SG_CHUNK_SIZE];
> + };
> +
> +After the buffers have been consumed by the L5P, to release the NIC mapping of
> +buffers the L5P calls :c:member:`teardown` of :c:type:`struct
> +ulp_ddp_dev_ops <ulp_ddp_dev_ops>`:
> +
> +.. code-block:: c
> +
> + void (*teardown)(struct net_device *netdev,
> + struct sock *sk,
> + struct ulp_ddp_io *io,
> + void *ddp_ctx);
> +
> +`teardown` receives the same `io` context and an additional opaque
> +`ddp_ctx` that is used for asynchronous teardown, see the :ref:`async_release`
> +section.
> +
> +.. _async_release:
> +
> +Asynchronous teardown
> +---------------------
> +
> +To teardown the association between tags and buffers and allow tag reuse NIC HW
> +is called by the NIC driver during `teardown`. This operation may be
> +performed either synchronously or asynchronously. In asynchronous teardown,
> +`teardown` returns immediately without unmapping NIC HW buffers. Later,
> +when the unmapping completes by NIC HW, the NIC driver will call up to L5P
> +using :c:member:`ddp_teardown_done` of :c:type:`struct ulp_ddp_ulp_ops <ulp_ddp_ulp_ops>`:
> +
> +.. code-block:: c
> +
> + void (*ddp_teardown_done)(void *ddp_ctx);
> +
> +The `ddp_ctx` parameter passed in `ddp_teardown_done` is the same on provided
> +in `teardown` and it is used to carry some context about the buffers
> +and tags that are released.
> +
> +Resync handling
> +===============
> +
> +RX
> +--
> +In presence of packet drops or network packet reordering, the device may lose
> +synchronization between the TCP stream and the L5P framing, and require a
> +resync with the kernel's TCP stack. When the device is out of sync, no offload
> +takes place, and packets are passed as-is to software. Resync is very similar
> +to TLS offload (see documentation at Documentation/networking/tls-offload.rst)
> +
> +If only packets with L5P data are lost or reordered, then resynchronization may
> +be avoided by NIC HW that keeps tracking PDU headers. If, however, PDU headers
> +are reordered, then resynchronization is necessary.
> +
> +To resynchronize hardware during traffic, we use a handshake between hardware
> +and software. The NIC HW searches for a sequence of bytes that identifies L5P
> +headers (i.e., magic pattern). For example, in NVMe-TCP, the PDU operation
> +type can be used for this purpose. Using the PDU header length field, the NIC
> +HW will continue to find and match magic patterns in subsequent PDU headers. If
> +the pattern is missing in an expected position, then searching for the pattern
> +starts anew.
> +
> +The NIC will not resume offload when the magic pattern is first identified.
> +Instead, it will request L5P software to confirm that indeed this is a PDU
> +header. To request confirmation the NIC driver calls up to L5P using
> +:c:member:`resync_request` of :c:type:`struct ulp_ddp_ulp_ops <ulp_ddp_ulp_ops>`:
> +
> +.. code-block:: c
> +
> + bool (*resync_request)(struct sock *sk, u32 seq, u32 flags);
> +
> +The `seq` parameter contains the TCP sequence of the last byte in the PDU header.
> +The `flags` parameter contains a flag (`ULP_DDP_RESYNC_PENDING`) indicating whether
> +a request is pending or not.
> +L5P software will respond to this request after observing the packet containing
> +TCP sequence `seq` in-order. If the PDU header is indeed there, then L5P
> +software calls the NIC driver using the :c:member:`resync` function of
> +the :c:type:`struct ulp_ddp_dev_ops <ulp_ddp_ops>` inside the :c:type:`struct
> +net_device <net_device>` while passing the same `seq` to confirm it is a PDU
> +header.
> +
> +.. code-block:: c
> +
> + void (*resync)(struct net_device *netdev,
> + struct sock *sk, u32 seq);
> +
> +Statistics
> +==========
> +
> +Per L5P protocol, the NIC driver must report statistics for the above
> +netdevice operations and packets processed by offload.
> +These statistics are per-device and can be retrieved from userspace
> +via netlink (see Documentation/netlink/specs/ulp_ddp.yaml).
> +
> +For example, NVMe-TCP offload reports:
> +
> + * ``rx_nvme_tcp_sk_add`` - number of NVMe-TCP Rx offload contexts created.
> + * ``rx_nvme_tcp_sk_add_fail`` - number of NVMe-TCP Rx offload context creation
> + failures.
> + * ``rx_nvme_tcp_sk_del`` - number of NVMe-TCP Rx offload contexts destroyed.
> + * ``rx_nvme_tcp_setup`` - number of DDP buffers mapped.
> + * ``rx_nvme_tcp_setup_fail`` - number of DDP buffers mapping that failed.
> + * ``rx_nvme_tcp_teardown`` - number of DDP buffers unmapped.
> + * ``rx_nvme_tcp_drop`` - number of packets dropped in the driver due to fatal
> + errors.
> + * ``rx_nvme_tcp_resync`` - number of packets with resync requests.
> + * ``rx_nvme_tcp_packets`` - number of packets that used offload.
> + * ``rx_nvme_tcp_bytes`` - number of bytes placed in DDP buffers.
> +
> +NIC requirements
> +================
> +
> +NIC hardware should meet the following requirements to provide this offload:
> +
> + * Offload must never buffer TCP packets.
> + * Offload must never modify TCP packet headers.
> + * Offload must never reorder TCP packets within a flow.
> + * Offload must never drop TCP packets.
> + * Offload must not depend on any TCP fields beyond the
> + 5-tuple and TCP sequence number.
The doc LGTM, thanks!
Reviewed-by: Bagas Sanjaya <bagasdotme@gmail.com>
--
An old man doll... just what I always wanted! - Clara
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v24 00/20] nvme-tcp receive offloads
2024-04-07 22:21 ` Sagi Grimberg
@ 2024-04-09 22:35 ` Chaitanya Kulkarni
2024-04-09 22:59 ` Jakub Kicinski
0 siblings, 1 reply; 44+ messages in thread
From: Chaitanya Kulkarni @ 2024-04-09 22:35 UTC (permalink / raw)
To: Sagi Grimberg, Jakub Kicinski, Aurelien Aptel
Cc: linux-nvme, netdev, hch, kbusch, axboe, davem, aurelien.aptel,
Shai Malin, malin1024, Or Gerlitz, Yoray Zack, Boris Pismenny,
Gal Shalom, Max Gurtovoy, edumazet
On 4/7/2024 3:21 PM, Sagi Grimberg wrote:
>
>
> On 06/04/2024 8:45, Jakub Kicinski wrote:
>> Doesn't apply, again, unfortunately.
>>
>> On Thu, 4 Apr 2024 12:36:57 +0000 Aurelien Aptel wrote:
>>> Testing
>>> =======
>>> This series was tested on ConnectX-7 HW using various configurations
>>> of IO sizes, queue depths, MTUs, and with both the SPDK and kernel
>>> NVMe-TCP
>>> targets.
>> About testing, what do you have in terms of a testing setup?
>> As you said this is similar to the TLS offload:
>>
>>> Note:
>>> These offloads are similar in nature to the packet-based NIC TLS
>>> offloads,
>>> which are already upstream (see net/tls/tls_device.c).
>>> You can read more about TLS offload here:
>>> https://www.kernel.org/doc/html/latest/networking/tls-offload.html
>> and our experience trying to maintain and extend the very much used SW
>> kernel TLS implementation in presence of the device offload is mixed :(
>> We can't test it, we break it, get CVEs for it :( In all fairness
>
> Especially when nvme-tcp can also run on tls, but that is incompatible with
> this offload at the moment (I've told the nvidia folks that I do not expect
> this incompatibility to be permanent).
>
>> the inline offload is probably not as bad as the crypto accelerator
>> path, but still it breaks. So assuming that this gets all the necessary
>> acks can we expect you to plug some testing into the netdev CI so that
>> we see whether any incoming change is breaking this offload?
>
> Agree, also given that there is an effort to extend blktests to run on
> real controllers, perhaps we should add a few tests there as well?
blktests seems to be the right framework to add all the testcases to
cover the targeted subsystem(s) for this patchset. Daniel from Suse has
already posted an RFC (see [1]) to add support for blktests so we can
use real controllers for better test coverage. We will be discussing
that at LSFMM session this year in detail.
With this support in the blktest framework, we can definitely generate
right test-coverage for the tcp-offload that can be run by anyone who
has this H/W. Just like I run NVMe tests on the code going from NVMe
tree to block tree for every pull request, we are planning to run new
nvme tcp offload specific tests regularly on NVMe tree. We will be happy
to provide the H/W to distros who are supporting this feature in order
to make testing easier for others as well.
Hope this answers any questions regrading ongoing testing on this
patchset when this code is merged.
-ck
[1] https://lists.infradead.org/pipermail/linux-nvme/2024-March/046056.html
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v24 00/20] nvme-tcp receive offloads
2024-04-09 22:35 ` Chaitanya Kulkarni
@ 2024-04-09 22:59 ` Jakub Kicinski
2024-04-18 8:29 ` Chaitanya Kulkarni
0 siblings, 1 reply; 44+ messages in thread
From: Jakub Kicinski @ 2024-04-09 22:59 UTC (permalink / raw)
To: Chaitanya Kulkarni
Cc: Sagi Grimberg, Aurelien Aptel, linux-nvme, netdev, hch, kbusch,
axboe, davem, aurelien.aptel, Shai Malin, malin1024, Or Gerlitz,
Yoray Zack, Boris Pismenny, Gal Shalom, Max Gurtovoy, edumazet
On Tue, 9 Apr 2024 22:35:51 +0000 Chaitanya Kulkarni wrote:
> blktests seems to be the right framework to add all the testcases to
> cover the targeted subsystem(s) for this patchset. Daniel from Suse has
> already posted an RFC (see [1]) to add support for blktests so we can
> use real controllers for better test coverage. We will be discussing
> that at LSFMM session this year in detail.
No preference on the framework or where the tests live, FWIW.
> With this support in the blktest framework, we can definitely generate
> right test-coverage for the tcp-offload that can be run by anyone who
> has this H/W. Just like I run NVMe tests on the code going from NVMe
> tree to block tree for every pull request, we are planning to run new
> nvme tcp offload specific tests regularly on NVMe tree. We will be happy
> to provide the H/W to distros who are supporting this feature in order
> to make testing easier for others as well.
You're not sending these patches to the distros, you're sending them
to the upstream Linux kernel. And unfortunately we don't have a test
lab where we could put your HW, so it's on you. To be clear all you
need to do is periodically build and test certain upstream branches
and report results. By "report" all I mean is put a JSON file with the
result somewhere we can HTTP GET. KernelCI has been around for a while,
I don't think this is a crazy ask.
^ permalink raw reply [flat|nested] 44+ messages in thread
* RE: [PATCH v24 06/20] nvme-tcp: Add DDP data-path
2024-04-07 22:08 ` Sagi Grimberg
@ 2024-04-10 6:31 ` Aurelien Aptel
0 siblings, 0 replies; 44+ messages in thread
From: Aurelien Aptel @ 2024-04-10 6:31 UTC (permalink / raw)
To: Sagi Grimberg, linux-nvme, netdev, hch, kbusch, axboe,
Chaitanya Kulkarni, davem, kuba
Cc: Boris Pismenny, aurelien.aptel, Shai Malin, malin1024,
Or Gerlitz, Yoray Zack, Gal Shalom, Max Gurtovoy
Thanks!
^ permalink raw reply [flat|nested] 44+ messages in thread
* RE: [PATCH v24 05/20] nvme-tcp: Add DDP offload control path
2024-04-07 22:08 ` Sagi Grimberg
@ 2024-04-10 6:31 ` Aurelien Aptel
0 siblings, 0 replies; 44+ messages in thread
From: Aurelien Aptel @ 2024-04-10 6:31 UTC (permalink / raw)
To: Sagi Grimberg, linux-nvme, netdev, hch, kbusch, axboe,
Chaitanya Kulkarni, davem, kuba
Cc: Boris Pismenny, aurelien.aptel, Shai Malin, malin1024,
Or Gerlitz, Yoray Zack, Gal Shalom, Max Gurtovoy, brauner
Thanks for the reviews!
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v24 03/20] iov_iter: skip copy if src == dst for direct data placement
2024-04-04 12:37 ` [PATCH v24 03/20] iov_iter: skip copy if src == dst for direct data placement Aurelien Aptel
@ 2024-04-15 14:28 ` Max Gurtovoy
2024-04-16 20:30 ` David Laight
1 sibling, 0 replies; 44+ messages in thread
From: Max Gurtovoy @ 2024-04-15 14:28 UTC (permalink / raw)
To: Aurelien Aptel, linux-nvme, netdev, sagi, hch, kbusch, axboe,
chaitanyak, davem, kuba
Cc: aurelien.aptel, smalin, malin1024, ogerlitz, yorayz, borisp,
galshalom, viro
On 04/04/2024 15:37, Aurelien Aptel wrote:
> From: Ben Ben-Ishay <benishay@nvidia.com>
>
> When using direct data placement (DDP) the NIC could write the payload
> directly into the destination buffer and constructs SKBs such that
> they point to this data. To skip copies when SKB data already resides
> in the destination buffer we check if (src == dst), and skip the copy
> when it's true.
>
> Signed-off-by: Ben Ben-Ishay <benishay@nvidia.com>
> Signed-off-by: Boris Pismenny <borisp@nvidia.com>
> Signed-off-by: Or Gerlitz <ogerlitz@nvidia.com>
> Signed-off-by: Yoray Zack <yorayz@nvidia.com>
> Signed-off-by: Shai Malin <smalin@nvidia.com>
> Signed-off-by: Aurelien Aptel <aaptel@nvidia.com>
> Reviewed-by: Chaitanya Kulkarni <kch@nvidia.com>
> ---
> lib/iov_iter.c | 9 ++++++++-
> 1 file changed, 8 insertions(+), 1 deletion(-)
>
> diff --git a/lib/iov_iter.c b/lib/iov_iter.c
> index 4a6a9f419bd7..a85125485174 100644
> --- a/lib/iov_iter.c
> +++ b/lib/iov_iter.c
> @@ -62,7 +62,14 @@ static __always_inline
> size_t memcpy_to_iter(void *iter_to, size_t progress,
> size_t len, void *from, void *priv2)
> {
> - memcpy(iter_to, from + progress, len);
> + /*
> + * When using direct data placement (DDP) the hardware writes
> + * data directly to the destination buffer, and constructs
> + * IOVs such that they point to this data.
> + * Thus, when the src == dst we skip the memcpy.
> + */
> + if (iter_to != from + progress)
> + memcpy(iter_to, from + progress, len);
> return 0;
> }
>
Looks good,
Reviewed-by: Max Gurtovoy <mgurtovoy@nvidia.com>
^ permalink raw reply [flat|nested] 44+ messages in thread
* RE: [PATCH v24 03/20] iov_iter: skip copy if src == dst for direct data placement
2024-04-04 12:37 ` [PATCH v24 03/20] iov_iter: skip copy if src == dst for direct data placement Aurelien Aptel
2024-04-15 14:28 ` Max Gurtovoy
@ 2024-04-16 20:30 ` David Laight
2024-04-18 8:22 ` Aurelien Aptel
1 sibling, 1 reply; 44+ messages in thread
From: David Laight @ 2024-04-16 20:30 UTC (permalink / raw)
To: 'Aurelien Aptel',
linux-nvme, netdev, sagi, hch, kbusch, axboe, chaitanyak, davem,
kuba
Cc: aurelien.aptel, smalin, malin1024, ogerlitz, yorayz, borisp,
galshalom, mgurtovoy, viro
From: Aurelien Aptel
> Sent: 04 April 2024 13:37
>
> From: Ben Ben-Ishay <benishay@nvidia.com>
>
> When using direct data placement (DDP) the NIC could write the payload
> directly into the destination buffer and constructs SKBs such that
> they point to this data. To skip copies when SKB data already resides
> in the destination buffer we check if (src == dst), and skip the copy
> when it's true.
...
> diff --git a/lib/iov_iter.c b/lib/iov_iter.c
> index 4a6a9f419bd7..a85125485174 100644
> --- a/lib/iov_iter.c
> +++ b/lib/iov_iter.c
> @@ -62,7 +62,14 @@ static __always_inline
> size_t memcpy_to_iter(void *iter_to, size_t progress,
> size_t len, void *from, void *priv2)
> {
> - memcpy(iter_to, from + progress, len);
> + /*
> + * When using direct data placement (DDP) the hardware writes
> + * data directly to the destination buffer, and constructs
> + * IOVs such that they point to this data.
> + * Thus, when the src == dst we skip the memcpy.
> + */
> + if (iter_to != from + progress)
> + memcpy(iter_to, from + progress, len);
How must does this conditional cost for the normal case
when it is true?
I suspect it is mispredicted 50% of the time.
So, while it may speed up your test, the overall system
impact will be negative.
David
> return 0;
> }
>
> --
> 2.34.1
>
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)
^ permalink raw reply [flat|nested] 44+ messages in thread
* RE: [PATCH v24 03/20] iov_iter: skip copy if src == dst for direct data placement
2024-04-16 20:30 ` David Laight
@ 2024-04-18 8:22 ` Aurelien Aptel
0 siblings, 0 replies; 44+ messages in thread
From: Aurelien Aptel @ 2024-04-18 8:22 UTC (permalink / raw)
To: David Laight, linux-nvme, netdev, sagi, hch, kbusch, axboe,
chaitanyak, davem, kuba
Cc: aurelien.aptel, smalin, malin1024, ogerlitz, yorayz, borisp,
galshalom, mgurtovoy, viro
Hi David,
David Laight <David.Laight@ACULAB.COM> writes:
> How must does this conditional cost for the normal case
> when it is true?
> I suspect it is mispredicted 50% of the time.
> So, while it may speed up your test, the overall system
> impact will be negative.
We have done some measures to evaluate the cost of the additional test
and we don't see any noticeable impact on performances.
If this is still a concern, we can add compilation-time check on
CONFIG_ULP_DDP i.e. ifdef or IS_ENABLED() (which gets optimized out).
Thanks
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v24 00/20] nvme-tcp receive offloads
2024-04-09 22:59 ` Jakub Kicinski
@ 2024-04-18 8:29 ` Chaitanya Kulkarni
2024-04-18 15:28 ` Jakub Kicinski
0 siblings, 1 reply; 44+ messages in thread
From: Chaitanya Kulkarni @ 2024-04-18 8:29 UTC (permalink / raw)
To: Jakub Kicinski
Cc: Sagi Grimberg, Aurelien Aptel, linux-nvme, netdev, hch, kbusch,
axboe, davem, aurelien.aptel, Shai Malin, malin1024, Or Gerlitz,
Yoray Zack, Boris Pismenny, Gal Shalom, Max Gurtovoy, edumazet
On 4/9/2024 3:59 PM, Jakub Kicinski wrote:
> On Tue, 9 Apr 2024 22:35:51 +0000 Chaitanya Kulkarni wrote:
>> blktests seems to be the right framework to add all the testcases to
>> cover the targeted subsystem(s) for this patchset. Daniel from Suse has
>> already posted an RFC (see [1]) to add support for blktests so we can
>> use real controllers for better test coverage. We will be discussing
>> that at LSFMM session this year in detail.
>
> No preference on the framework or where the tests live, FWIW.
>
>> With this support in the blktest framework, we can definitely generate
>> right test-coverage for the tcp-offload that can be run by anyone who
>> has this H/W. Just like I run NVMe tests on the code going from NVMe
>> tree to block tree for every pull request, we are planning to run new
>> nvme tcp offload specific tests regularly on NVMe tree. We will be happy
>> to provide the H/W to distros who are supporting this feature in order
>> to make testing easier for others as well.
>
> You're not sending these patches to the distros, you're sending them
> to the upstream Linux kernel. And unfortunately we don't have a test
> lab where we could put your HW, so it's on you. To be clear all you
> need to do is periodically build and test certain upstream branches
> and report results. By "report" all I mean is put a JSON file with the
> result somewhere we can HTTP GET. KernelCI has been around for a while,
> I don't think this is a crazy ask.
That should be doable, we can run the tests and make the results
available for others to access in JASON format. Just to
clarify what you mean by the Kernel CI ? what I understood you want
our tests to run and provide the results, still not clear about the
Kernel CI involvement in this process, can you please elaborate ?
-ck
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v24 00/20] nvme-tcp receive offloads
2024-04-18 8:29 ` Chaitanya Kulkarni
@ 2024-04-18 15:28 ` Jakub Kicinski
0 siblings, 0 replies; 44+ messages in thread
From: Jakub Kicinski @ 2024-04-18 15:28 UTC (permalink / raw)
To: Chaitanya Kulkarni
Cc: Sagi Grimberg, Aurelien Aptel, linux-nvme, netdev, hch, kbusch,
axboe, davem, aurelien.aptel, Shai Malin, malin1024, Or Gerlitz,
Yoray Zack, Boris Pismenny, Gal Shalom, Max Gurtovoy, edumazet
On Thu, 18 Apr 2024 08:29:27 +0000 Chaitanya Kulkarni wrote:
> > You're not sending these patches to the distros, you're sending them
> > to the upstream Linux kernel. And unfortunately we don't have a test
> > lab where we could put your HW, so it's on you. To be clear all you
> > need to do is periodically build and test certain upstream branches
> > and report results. By "report" all I mean is put a JSON file with the
> > result somewhere we can HTTP GET. KernelCI has been around for a while,
> > I don't think this is a crazy ask.
>
> That should be doable, we can run the tests and make the results
> available for others to access in JASON format. Just to
> clarify what you mean by the Kernel CI ? what I understood you want
> our tests to run and provide the results, still not clear about the
> Kernel CI involvement in this process, can you please elaborate ?
Kernel CI was just an example of CI being a thing upstream.
For us you'd need to generate a much simplified JSON which netdev CI
can consume. Could you reach out to Petr Machata or Ido Schimmel
internally at nVidia? They are involved.
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v24 04/20] net/tls,core: export get_netdev_for_sock
2024-04-04 12:37 ` [PATCH v24 04/20] net/tls,core: export get_netdev_for_sock Aurelien Aptel
@ 2024-04-21 11:45 ` Sagi Grimberg
0 siblings, 0 replies; 44+ messages in thread
From: Sagi Grimberg @ 2024-04-21 11:45 UTC (permalink / raw)
To: Aurelien Aptel, linux-nvme, netdev, hch, kbusch, axboe,
chaitanyak, davem, kuba
Cc: aurelien.aptel, smalin, malin1024, ogerlitz, yorayz, borisp,
galshalom, mgurtovoy, edumazet, pabeni, john.fastabend, daniel
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v24 01/20] net: Introduce direct data placement tcp offload
2024-04-04 12:36 ` [PATCH v24 01/20] net: Introduce direct data placement tcp offload Aurelien Aptel
@ 2024-04-21 11:47 ` Sagi Grimberg
2024-04-26 7:21 ` Aurelien Aptel
0 siblings, 1 reply; 44+ messages in thread
From: Sagi Grimberg @ 2024-04-21 11:47 UTC (permalink / raw)
To: Aurelien Aptel, linux-nvme, netdev, hch, kbusch, axboe,
chaitanyak, davem, kuba
Cc: Boris Pismenny, aurelien.aptel, smalin, malin1024, ogerlitz,
yorayz, galshalom, mgurtovoy, edumazet, pabeni, dsahern, ast,
jacob.e.keller
> +int ulp_ddp_sk_add(struct net_device *netdev,
> + struct sock *sk,
> + struct ulp_ddp_config *config,
> + const struct ulp_ddp_ulp_ops *ops)
> +{
> + int ret;
> +
> + /* put in ulp_ddp_sk_del() */
> + dev_hold(netdev);
> +
> + config->io_cpu = sk->sk_incoming_cpu;
> + ret = netdev->netdev_ops->ulp_ddp_ops->sk_add(netdev, sk, config);
Still don't understand why you need the io_cpu config if you are passing
the sk to the driver...
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v24 01/20] net: Introduce direct data placement tcp offload
2024-04-21 11:47 ` Sagi Grimberg
@ 2024-04-26 7:21 ` Aurelien Aptel
2024-04-28 8:15 ` Sagi Grimberg
0 siblings, 1 reply; 44+ messages in thread
From: Aurelien Aptel @ 2024-04-26 7:21 UTC (permalink / raw)
To: Sagi Grimberg, linux-nvme, netdev, hch, kbusch, axboe,
chaitanyak, davem, kuba
Cc: Boris Pismenny, aurelien.aptel, smalin, malin1024, ogerlitz,
yorayz, galshalom, mgurtovoy, edumazet, pabeni, dsahern, ast,
jacob.e.keller
Hi Sagi,
Sagi Grimberg <sagi@grimberg.me> writes:
>> + config->io_cpu = sk->sk_incoming_cpu;
>> + ret = netdev->netdev_ops->ulp_ddp_ops->sk_add(netdev, sk, config);
>
> Still don't understand why you need the io_cpu config if you are passing
> the sk to the driver...
With our HW we cannot move the offload queues to a different CPU without
destroying and recreating the offload resources on the new CPU.
Since the connection is created from a different CPU then the io queue
thread, we cannot predict which CPU we should create our offload context
on.
Ideally, io_cpu should be set to nvme_queue->io_cpu or it should be removed
and the socket should be offloaded from the io thread. What do you
prefer?
Thanks
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v24 01/20] net: Introduce direct data placement tcp offload
2024-04-26 7:21 ` Aurelien Aptel
@ 2024-04-28 8:15 ` Sagi Grimberg
2024-04-29 11:35 ` Aurelien Aptel
0 siblings, 1 reply; 44+ messages in thread
From: Sagi Grimberg @ 2024-04-28 8:15 UTC (permalink / raw)
To: Aurelien Aptel, linux-nvme, netdev, hch, kbusch, axboe,
chaitanyak, davem, kuba
Cc: Boris Pismenny, aurelien.aptel, smalin, malin1024, ogerlitz,
yorayz, galshalom, mgurtovoy, edumazet, pabeni, dsahern, ast,
jacob.e.keller
On 26/04/2024 10:21, Aurelien Aptel wrote:
> Hi Sagi,
>
> Sagi Grimberg <sagi@grimberg.me> writes:
>>> + config->io_cpu = sk->sk_incoming_cpu;
>>> + ret = netdev->netdev_ops->ulp_ddp_ops->sk_add(netdev, sk, config);
>> Still don't understand why you need the io_cpu config if you are passing
>> the sk to the driver...
> With our HW we cannot move the offload queues to a different CPU without
> destroying and recreating the offload resources on the new CPU.
This is not simply a steering rule that can be overwritten at any point?
>
> Since the connection is created from a different CPU then the io queue
> thread, we cannot predict which CPU we should create our offload context
> on.
>
> Ideally, io_cpu should be set to nvme_queue->io_cpu or it should be removed
> and the socket should be offloaded from the io thread. What do you
> prefer?
I was simply referring to the fact that you set config->io_cpu from
sk->sk_incoming_cpu
and then you pass sk (and config) to .sk_add, so why does this
assignment need to
exist here and not below the interface down at the driver?
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v24 01/20] net: Introduce direct data placement tcp offload
2024-04-28 8:15 ` Sagi Grimberg
@ 2024-04-29 11:35 ` Aurelien Aptel
2024-04-30 11:54 ` Sagi Grimberg
0 siblings, 1 reply; 44+ messages in thread
From: Aurelien Aptel @ 2024-04-29 11:35 UTC (permalink / raw)
To: Sagi Grimberg, linux-nvme, netdev, hch, kbusch, axboe,
chaitanyak, davem, kuba
Cc: Boris Pismenny, aurelien.aptel, smalin, malin1024, ogerlitz,
yorayz, galshalom, mgurtovoy, edumazet, pabeni, dsahern, ast,
jacob.e.keller
Sagi Grimberg <sagi@grimberg.me> writes:
> This is not simply a steering rule that can be overwritten at any point?
No, unlike steering rules, the offload resources cannot be moved to a
different queue.
In order to move it we will need to re-create the queue and the
resources assigned to it. We will consider to improve the HW/FW/SW to
allow this in future versions.
> I was simply referring to the fact that you set config->io_cpu from
> sk->sk_incoming_cpu
> and then you pass sk (and config) to .sk_add, so why does this
> assignment need to
> exist here and not below the interface down at the driver?
You're correct, it doesn't need to exist *if* we use sk->incoming_cpu,
which at the time it is used, is the wrong value.
The right value for cfg->io_cpu is nvme_queue->io_cpu.
So either:
- we do that and thus keep cfg->io_cpu.
- or we remove cfg->io_cpu, and we offload the socket from
nvme_tcp_io_work() where the io_cpu is implicitly going to be
the current CPU.
Thanks
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v24 01/20] net: Introduce direct data placement tcp offload
2024-04-29 11:35 ` Aurelien Aptel
@ 2024-04-30 11:54 ` Sagi Grimberg
2024-05-02 7:04 ` Aurelien Aptel
0 siblings, 1 reply; 44+ messages in thread
From: Sagi Grimberg @ 2024-04-30 11:54 UTC (permalink / raw)
To: Aurelien Aptel, linux-nvme, netdev, hch, kbusch, axboe,
chaitanyak, davem, kuba
Cc: Boris Pismenny, aurelien.aptel, smalin, malin1024, ogerlitz,
yorayz, galshalom, mgurtovoy, edumazet, pabeni, dsahern, ast,
jacob.e.keller
On 29/04/2024 14:35, Aurelien Aptel wrote:
> Sagi Grimberg <sagi@grimberg.me> writes:
>> This is not simply a steering rule that can be overwritten at any point?
> No, unlike steering rules, the offload resources cannot be moved to a
> different queue.
>
> In order to move it we will need to re-create the queue and the
> resources assigned to it. We will consider to improve the HW/FW/SW to
> allow this in future versions.
Well, you cannot rely on the fact that the application will be pinned to a
specific cpu core. That may be the case by accident, but you must not and
cannot assume it.
Even today, nvme-tcp has an option to run from an unbound wq context,
where queue->io_cpu is set to WORK_CPU_UNBOUND. What are you going
to do there?
nvme-tcp may handle rx side directly from .data_ready() in the future, what
will the offload do in that case?
>
>> I was simply referring to the fact that you set config->io_cpu from
>> sk->sk_incoming_cpu
>> and then you pass sk (and config) to .sk_add, so why does this
>> assignment need to
>> exist here and not below the interface down at the driver?
> You're correct, it doesn't need to exist *if* we use sk->incoming_cpu,
> which at the time it is used, is the wrong value.
> The right value for cfg->io_cpu is nvme_queue->io_cpu.
io_cpu may or may not mean anything. You cannot rely on it, nor dictate it.
>
> So either:
> - we do that and thus keep cfg->io_cpu.
> - or we remove cfg->io_cpu, and we offload the socket from
> nvme_tcp_io_work() where the io_cpu is implicitly going to be
> the current CPU.
What do you mean offload the socket from nvme_tcp_io_work? I do not
understand what this means.
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v24 01/20] net: Introduce direct data placement tcp offload
2024-04-30 11:54 ` Sagi Grimberg
@ 2024-05-02 7:04 ` Aurelien Aptel
2024-05-03 7:31 ` Sagi Grimberg
0 siblings, 1 reply; 44+ messages in thread
From: Aurelien Aptel @ 2024-05-02 7:04 UTC (permalink / raw)
To: Sagi Grimberg, linux-nvme, netdev, hch, kbusch, axboe,
chaitanyak, davem, kuba
Cc: Boris Pismenny, aurelien.aptel, smalin, malin1024, ogerlitz,
yorayz, galshalom, mgurtovoy, edumazet, pabeni, dsahern, ast,
jacob.e.keller
Sagi Grimberg <sagi@grimberg.me> writes:
> Well, you cannot rely on the fact that the application will be pinned to a
> specific cpu core. That may be the case by accident, but you must not and
> cannot assume it.
Just to be clear, any CPU can read from the socket and benefit from the
offload but there will be an extra cost if the queue CPU is different
from the offload CPU. We use cfg->io_cpu as a hint.
> Even today, nvme-tcp has an option to run from an unbound wq context,
> where queue->io_cpu is set to WORK_CPU_UNBOUND. What are you going to
> do there?
When the CPU is not bound to a specific core, we will most likely always
have CPU misalignment and the extra cost that goes with it.
But when it is bound, which is still the default common case, we will
benefit from the alignment. To not lose that benefit for the default
most common case, we would like to keep cfg->io_cpu.
Could you clarify what are the advantages of running unbounded queues,
or to handle RX on a different cpu than the current io_cpu?
> nvme-tcp may handle rx side directly from .data_ready() in the future, what
> will the offload do in that case?
It is not clear to us what the benefit of handling rx in .data_ready()
will achieve. From our experiment, ->sk_data_ready() is called either
from queue->io_cpu, or sk->sk_incoming_cpu. Unless you enable aRFS,
sk_incoming_cpu will be constant for the whole connection. Can you
clarify would handling RX from data_ready() provide?
> io_cpu may or may not mean anything. You cannot rely on it, nor dictate it.
We are just interested in optimizing the bounded case, where io_cpu has
meaning.
> > - or we remove cfg->io_cpu, and we offload the socket from
> > nvme_tcp_io_work() where the io_cpu is implicitly going to be
> > the current CPU.
> What do you mean offload the socket from nvme_tcp_io_work? I do not
> understand what this means.
We meant setting up the offload from the io thread instead, by calling
nvme_tcp_offload_socket() from nvme_tcp_io_work(), and making sure it's
only called once. Something like this:
+ if (queue->ctrl->ddp_netdev && !nvme_tcp_admin_queue(queue) && !test_bit(NVME_TCP_Q_OFF_DDP, &queue->flags)) {
+ int ret;
+
+ ret = nvme_tcp_offload_socket(queue);
+ if (ret) {
+ printk("XXX offload setup failed\n");
+ }
+ }
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v24 01/20] net: Introduce direct data placement tcp offload
2024-05-02 7:04 ` Aurelien Aptel
@ 2024-05-03 7:31 ` Sagi Grimberg
2024-05-06 12:28 ` Aurelien Aptel
0 siblings, 1 reply; 44+ messages in thread
From: Sagi Grimberg @ 2024-05-03 7:31 UTC (permalink / raw)
To: Aurelien Aptel, linux-nvme, netdev, hch, kbusch, axboe,
chaitanyak, davem, kuba
Cc: Boris Pismenny, aurelien.aptel, smalin, malin1024, ogerlitz,
yorayz, galshalom, mgurtovoy, edumazet, pabeni, dsahern, ast,
jacob.e.keller
On 5/2/24 10:04, Aurelien Aptel wrote:
> Sagi Grimberg <sagi@grimberg.me> writes:
>> Well, you cannot rely on the fact that the application will be pinned to a
>> specific cpu core. That may be the case by accident, but you must not and
>> cannot assume it.
> Just to be clear, any CPU can read from the socket and benefit from the
> offload but there will be an extra cost if the queue CPU is different
> from the offload CPU. We use cfg->io_cpu as a hint.
Understood. It is usually the case as io threads are not aligned to the
rss steering rules (unless
arfs is used).
>
>> Even today, nvme-tcp has an option to run from an unbound wq context,
>> where queue->io_cpu is set to WORK_CPU_UNBOUND. What are you going to
>> do there?
> When the CPU is not bound to a specific core, we will most likely always
> have CPU misalignment and the extra cost that goes with it.
Yes, as done today.
>
> But when it is bound, which is still the default common case, we will
> benefit from the alignment. To not lose that benefit for the default
> most common case, we would like to keep cfg->io_cpu.
Well, this explanation is much more reasonable. Setting .affinity_hint
argument
seems like a proper argument to the interface and nvme-tcp can set it to
queue->io_cpu.
>
> Could you clarify what are the advantages of running unbounded queues,
> or to handle RX on a different cpu than the current io_cpu?
See the discussion related to the patch from Li Feng:
https://lore.kernel.org/lkml/20230413062339.2454616-1-fengli@smartx.com/
>
>> nvme-tcp may handle rx side directly from .data_ready() in the future, what
>> will the offload do in that case?
> It is not clear to us what the benefit of handling rx in .data_ready()
> will achieve. From our experiment, ->sk_data_ready() is called either
> from queue->io_cpu, or sk->sk_incoming_cpu. Unless you enable aRFS,
> sk_incoming_cpu will be constant for the whole connection. Can you
> clarify would handling RX from data_ready() provide?
Save the context switching to a kthread from softirq, can reduce latency
substantially
for some workloads.
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v24 01/20] net: Introduce direct data placement tcp offload
2024-05-03 7:31 ` Sagi Grimberg
@ 2024-05-06 12:28 ` Aurelien Aptel
0 siblings, 0 replies; 44+ messages in thread
From: Aurelien Aptel @ 2024-05-06 12:28 UTC (permalink / raw)
To: Sagi Grimberg, linux-nvme, netdev, hch, kbusch, axboe,
chaitanyak, davem, kuba
Cc: Boris Pismenny, aurelien.aptel, smalin, malin1024, ogerlitz,
yorayz, galshalom, mgurtovoy, edumazet, pabeni, dsahern, ast,
jacob.e.keller
Sagi Grimberg <sagi@grimberg.me> writes:
> Understood. It is usually the case as io threads are not aligned to the
> rss steering rules (unless
> arfs is used).
The steering rule we add has 2 effects:
1) steer the flow to the offload object on the HW
2) provide CPU alignement with the io threads (Rx affinity similar to
aRFS)
We understand point (2) might not be achieved with unbounded queue
scenarios. That's fine.
>> But when it is bound, which is still the default common case, we will
>> benefit from the alignment. To not lose that benefit for the default
>> most common case, we would like to keep cfg->io_cpu.
>
> Well, this explanation is much more reasonable. Setting .affinity_hint
> argument
> seems like a proper argument to the interface and nvme-tcp can set it to
> queue->io_cpu.
Ok, we will rename cfg->io_cpu to ->affinity_hint.
>> Could you clarify what are the advantages of running unbounded queues,
>> or to handle RX on a different cpu than the current io_cpu?
>
> See the discussion related to the patch from Li Feng:
> https://lore.kernel.org/lkml/20230413062339.2454616-1-fengli@smartx.com/
Thanks. As said previously, we are fine with decreased perfs in this
edge case.
>>> nvme-tcp may handle rx side directly from .data_ready() in the future, what
>>> will the offload do in that case?
>> It is not clear to us what the benefit of handling rx in .data_ready()
>> will achieve. From our experiment, ->sk_data_ready() is called either
>> from queue->io_cpu, or sk->sk_incoming_cpu. Unless you enable aRFS,
>> sk_incoming_cpu will be constant for the whole connection. Can you
>> clarify would handling RX from data_ready() provide?
>
> Save the context switching to a kthread from softirq, can reduce latency
> substantially
> for some workloads.
Ok, thanks for the explanation. With bounded queues and steering the
softirq CPU will be aligned with the offload CPU, so we think this is
also OK.
Thanks
^ permalink raw reply [flat|nested] 44+ messages in thread
end of thread, other threads:[~2024-05-06 12:28 UTC | newest]
Thread overview: 44+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-04-04 12:36 [PATCH v24 00/20] nvme-tcp receive offloads Aurelien Aptel
2024-04-04 12:36 ` [PATCH v24 01/20] net: Introduce direct data placement tcp offload Aurelien Aptel
2024-04-21 11:47 ` Sagi Grimberg
2024-04-26 7:21 ` Aurelien Aptel
2024-04-28 8:15 ` Sagi Grimberg
2024-04-29 11:35 ` Aurelien Aptel
2024-04-30 11:54 ` Sagi Grimberg
2024-05-02 7:04 ` Aurelien Aptel
2024-05-03 7:31 ` Sagi Grimberg
2024-05-06 12:28 ` Aurelien Aptel
2024-04-04 12:36 ` [PATCH v24 02/20] netlink: add new family to manage ULP_DDP enablement and stats Aurelien Aptel
2024-04-04 12:37 ` [PATCH v24 03/20] iov_iter: skip copy if src == dst for direct data placement Aurelien Aptel
2024-04-15 14:28 ` Max Gurtovoy
2024-04-16 20:30 ` David Laight
2024-04-18 8:22 ` Aurelien Aptel
2024-04-04 12:37 ` [PATCH v24 04/20] net/tls,core: export get_netdev_for_sock Aurelien Aptel
2024-04-21 11:45 ` Sagi Grimberg
2024-04-04 12:37 ` [PATCH v24 05/20] nvme-tcp: Add DDP offload control path Aurelien Aptel
2024-04-07 22:08 ` Sagi Grimberg
2024-04-10 6:31 ` Aurelien Aptel
2024-04-04 12:37 ` [PATCH v24 06/20] nvme-tcp: Add DDP data-path Aurelien Aptel
2024-04-07 22:08 ` Sagi Grimberg
2024-04-10 6:31 ` Aurelien Aptel
2024-04-04 12:37 ` [PATCH v24 07/20] nvme-tcp: RX DDGST offload Aurelien Aptel
2024-04-04 12:37 ` [PATCH v24 08/20] nvme-tcp: Deal with netdevice DOWN events Aurelien Aptel
2024-04-04 12:37 ` [PATCH v24 09/20] Documentation: add ULP DDP offload documentation Aurelien Aptel
2024-04-09 8:49 ` Bagas Sanjaya
2024-04-04 12:37 ` [PATCH v24 10/20] net/mlx5e: Rename from tls to transport static params Aurelien Aptel
2024-04-04 12:37 ` [PATCH v24 11/20] net/mlx5e: Refactor ico sq polling to get budget Aurelien Aptel
2024-04-04 12:37 ` [PATCH v24 12/20] net/mlx5: Add NVMEoTCP caps, HW bits, 128B CQE and enumerations Aurelien Aptel
2024-04-04 12:37 ` [PATCH v24 13/20] net/mlx5e: NVMEoTCP, offload initialization Aurelien Aptel
2024-04-04 12:37 ` [PATCH v24 14/20] net/mlx5e: TCP flow steering for nvme-tcp acceleration Aurelien Aptel
2024-04-04 12:37 ` [PATCH v24 15/20] net/mlx5e: NVMEoTCP, use KLM UMRs for buffer registration Aurelien Aptel
2024-04-04 12:37 ` [PATCH v24 16/20] net/mlx5e: NVMEoTCP, queue init/teardown Aurelien Aptel
2024-04-04 12:37 ` [PATCH v24 17/20] net/mlx5e: NVMEoTCP, ddp setup and resync Aurelien Aptel
2024-04-04 12:37 ` [PATCH v24 18/20] net/mlx5e: NVMEoTCP, async ddp invalidation Aurelien Aptel
2024-04-04 12:37 ` [PATCH v24 19/20] net/mlx5e: NVMEoTCP, data-path for DDP+DDGST offload Aurelien Aptel
2024-04-04 12:37 ` [PATCH v24 20/20] net/mlx5e: NVMEoTCP, statistics Aurelien Aptel
2024-04-06 5:45 ` [PATCH v24 00/20] nvme-tcp receive offloads Jakub Kicinski
2024-04-07 22:21 ` Sagi Grimberg
2024-04-09 22:35 ` Chaitanya Kulkarni
2024-04-09 22:59 ` Jakub Kicinski
2024-04-18 8:29 ` Chaitanya Kulkarni
2024-04-18 15:28 ` Jakub Kicinski
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).