From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,UNPARSEABLE_RELAY,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0BFA2C433E0 for ; Mon, 1 Feb 2021 10:10:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BABB964DB2 for ; Mon, 1 Feb 2021 10:10:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233195AbhBAKKd (ORCPT ); Mon, 1 Feb 2021 05:10:33 -0500 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:59722 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S232981AbhBAKGJ (ORCPT ); Mon, 1 Feb 2021 05:06:09 -0500 Received: from Internal Mail-Server by MTLPINE1 (envelope-from borisp@mellanox.com) with SMTP; 1 Feb 2021 12:05:13 +0200 Received: from gen-l-vrt-133.mtl.labs.mlnx. (gen-l-vrt-133.mtl.labs.mlnx [10.237.11.160]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 111A5Cxu029353; Mon, 1 Feb 2021 12:05:13 +0200 From: Boris Pismenny To: dsahern@gmail.com, kuba@kernel.org, davem@davemloft.net, saeedm@nvidia.com, hch@lst.de, sagi@grimberg.me, axboe@fb.com, kbusch@kernel.org, viro@zeniv.linux.org.uk, edumazet@google.com, smalin@marvell.com Cc: boris.pismenny@gmail.com, linux-nvme@lists.infradead.org, netdev@vger.kernel.org, benishay@nvidia.com, ogerlitz@nvidia.com, yorayz@nvidia.com, Ben Ben-Ishay , Or Gerlitz , Yoray Zack Subject: [PATCH v3 net-next 03/21] net: Introduce crc offload for tcp ddp ulp Date: Mon, 1 Feb 2021 12:04:51 +0200 Message-Id: <20210201100509.27351-4-borisp@mellanox.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20210201100509.27351-1-borisp@mellanox.com> References: <20210201100509.27351-1-borisp@mellanox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org This commit introduces support for CRC offload to direct data placement ULP on the receive side. Both DDP and CRC share a common API to initialize the offload for a TCP socket. But otherwise, both can be executed independently. On the receive side, CRC offload relies on a new SKB bit that indicates that no CRC error was encountered while processing this packet. If all packets of a ULP message have this bit set, then the CRC verification for the message can be skipped, as hardware already checked it. The following patches will set and use this bit to perform NVME-TCP CRC offload. A subsequent series, will add NVMe-TCP transmit side CRC support. Signed-off-by: Boris Pismenny Signed-off-by: Ben Ben-Ishay Signed-off-by: Or Gerlitz Signed-off-by: Yoray Zack Reviewed-by: Sagi Grimberg --- include/linux/netdev_features.h | 2 ++ include/linux/netdevice.h | 2 +- include/linux/skbuff.h | 2 +- net/Kconfig | 8 ++++++++ net/ethtool/common.c | 1 + net/ipv4/tcp_input.c | 4 ++-- net/ipv4/tcp_ipv4.c | 2 +- net/ipv4/tcp_offload.c | 2 +- 8 files changed, 17 insertions(+), 6 deletions(-) diff --git a/include/linux/netdev_features.h b/include/linux/netdev_features.h index 7977371d2dd1..77fb2cb99b78 100644 --- a/include/linux/netdev_features.h +++ b/include/linux/netdev_features.h @@ -86,6 +86,7 @@ enum { NETIF_F_HW_MACSEC_BIT, /* Offload MACsec operations */ NETIF_F_GRO_UDP_FWD_BIT, /* Allow UDP GRO for forwarding */ NETIF_F_HW_TCP_DDP_BIT, /* TCP direct data placement offload */ + NETIF_F_HW_TCP_DDP_CRC_RX_BIT, /* TCP DDP CRC RX offload */ /* * Add your fresh new feature above and remember to update @@ -161,6 +162,7 @@ enum { #define NETIF_F_HW_MACSEC __NETIF_F(HW_MACSEC) #define NETIF_F_GRO_UDP_FWD __NETIF_F(GRO_UDP_FWD) #define NETIF_F_HW_TCP_DDP __NETIF_F(HW_TCP_DDP) +#define NETIF_F_HW_TCP_DDP_CRC_RX __NETIF_F(HW_TCP_DDP_CRC_RX) /* Finds the next feature with the highest number of the range of start till 0. */ diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index bd28520e30f2..1065ddaa8e6b 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -1943,7 +1943,7 @@ struct net_device { const struct tlsdev_ops *tlsdev_ops; #endif -#ifdef CONFIG_TCP_DDP +#if IS_ENABLED(CONFIG_TCP_DDP) || IS_ENABLED(CONFIG_TCP_DDP_CRC) const struct tcp_ddp_dev_ops *tcp_ddp_ops; #endif diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 6d00d62628c8..32c5ab8e2f67 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -861,7 +861,7 @@ struct sk_buff { #ifdef CONFIG_TLS_DEVICE __u8 decrypted:1; #endif -#ifdef CONFIG_TCP_DDP +#if defined(CONFIG_TCP_DDP) || defined(CONFIG_TCP_DDP_CRC) __u8 ddp_crc:1; #endif diff --git a/net/Kconfig b/net/Kconfig index 3876861cdc90..80ed9f038968 100644 --- a/net/Kconfig +++ b/net/Kconfig @@ -465,6 +465,14 @@ config TCP_DDP NVMe-TCP/iSCSI, to request the NIC to place TCP payload data of a command response directly into kernel pages. +config TCP_DDP_CRC + bool "TCP direct data placement CRC offload" + default n + help + Direct Data Placement (DDP) CRC32C offload for TCP enables ULP, such as + NVMe-TCP/iSCSI, to request the NIC to calculate/verify the data digest + of commands as they go through the NIC. Thus avoiding the costly + per-byte overhead. endif # if NET diff --git a/net/ethtool/common.c b/net/ethtool/common.c index bbf83cd01106..14286afb2a69 100644 --- a/net/ethtool/common.c +++ b/net/ethtool/common.c @@ -70,6 +70,7 @@ const char netdev_features_strings[NETDEV_FEATURE_COUNT][ETH_GSTRING_LEN] = { [NETIF_F_HW_MACSEC_BIT] = "macsec-hw-offload", [NETIF_F_GRO_UDP_FWD_BIT] = "rx-udp-gro-forwarding", [NETIF_F_HW_TCP_DDP_BIT] = "tcp-ddp-offload", + [NETIF_F_HW_TCP_DDP_CRC_RX_BIT] = "tcp-ddp-crc-rx-offload", }; const char diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c index 0ae1ffca090d..cd490bd548a3 100644 --- a/net/ipv4/tcp_input.c +++ b/net/ipv4/tcp_input.c @@ -5150,7 +5150,7 @@ tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root, #ifdef CONFIG_TLS_DEVICE nskb->decrypted = skb->decrypted; #endif -#ifdef CONFIG_TCP_DDP +#if defined(CONFIG_TCP_DDP) || defined(CONFIG_TCP_DDP_CRC) nskb->ddp_crc = skb->ddp_crc; #endif TCP_SKB_CB(nskb)->seq = TCP_SKB_CB(nskb)->end_seq = start; @@ -5186,7 +5186,7 @@ tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root, if (skb->decrypted != nskb->decrypted) goto end; #endif -#ifdef CONFIG_TCP_DDP +#if defined(CONFIG_TCP_DDP) || defined(CONFIG_TCP_DDP_CRC) if (skb->ddp_crc != nskb->ddp_crc) goto end; diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c index b114fc870513..4c903c1a6c90 100644 --- a/net/ipv4/tcp_ipv4.c +++ b/net/ipv4/tcp_ipv4.c @@ -1811,7 +1811,7 @@ bool tcp_add_backlog(struct sock *sk, struct sk_buff *skb) #ifdef CONFIG_TLS_DEVICE tail->decrypted != skb->decrypted || #endif -#ifdef CONFIG_TCP_DDP +#if defined(CONFIG_TCP_DDP) || defined(CONFIG_TCP_DDP_CRC) tail->ddp_crc != skb->ddp_crc || #endif thtail->doff != th->doff || diff --git a/net/ipv4/tcp_offload.c b/net/ipv4/tcp_offload.c index 3ce196375d94..9e43a044e730 100644 --- a/net/ipv4/tcp_offload.c +++ b/net/ipv4/tcp_offload.c @@ -262,7 +262,7 @@ struct sk_buff *tcp_gro_receive(struct list_head *head, struct sk_buff *skb) #ifdef CONFIG_TLS_DEVICE flush |= p->decrypted ^ skb->decrypted; #endif -#ifdef CONFIG_TCP_DDP +#if defined(CONFIG_TCP_DDP) || defined(CONFIG_TCP_DDP_CRC) flush |= p->ddp_crc ^ skb->ddp_crc; #endif -- 2.24.1 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,UNPARSEABLE_RELAY, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 484D5C433DB for ; Mon, 1 Feb 2021 10:16:41 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E3B9464DB1 for ; Mon, 1 Feb 2021 10:16:40 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E3B9464DB1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=mellanox.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-nvme-bounces+linux-nvme=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=APtlezPbfcXFUcGCUUJxuD4DeOfGzdk+uKVYiFaWTGk=; b=vV07NR3QNM92akN7kh3/aa9fa uD2cmlUnmjHNdMgAXSBtRGDZvJCNyOT+cyvooqJns/lQ0Vr0vbyBZoQjJ4wDy3ewIMWPVFrjaHMCB H+v0nuLB5Jz7RArYA9xWO2d0W8GD7mpzxtRnjM2HPGIPEBJOfIUb+kfk6yxVOp86eVj1+pRo9/F7y tYwZKhOO70xsBDQdmkSHb1iG8j3kjmG+RyWy3jHwOhIPD2rMoHPsx0oA9BzQEg5tFe5Ao5kLXikCa 1ZxI2oYSE6q85uk8SFtUMXJ4qsooxk7TtMMw2/yajqSBIQujhK54EBjEl++nJDV2RKgl4DyVwthMT gp8kBJgxA==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l6WFs-0000K4-Ky; Mon, 01 Feb 2021 10:16:24 +0000 Received: from mail-il-dmz.mellanox.com ([193.47.165.129] helo=mellanox.co.il) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l6W5B-00042Q-VR for linux-nvme@lists.infradead.org; Mon, 01 Feb 2021 10:05:32 +0000 Received: from Internal Mail-Server by MTLPINE1 (envelope-from borisp@mellanox.com) with SMTP; 1 Feb 2021 12:05:13 +0200 Received: from gen-l-vrt-133.mtl.labs.mlnx. (gen-l-vrt-133.mtl.labs.mlnx [10.237.11.160]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 111A5Cxu029353; Mon, 1 Feb 2021 12:05:13 +0200 From: Boris Pismenny To: dsahern@gmail.com, kuba@kernel.org, davem@davemloft.net, saeedm@nvidia.com, hch@lst.de, sagi@grimberg.me, axboe@fb.com, kbusch@kernel.org, viro@zeniv.linux.org.uk, edumazet@google.com, smalin@marvell.com Subject: [PATCH v3 net-next 03/21] net: Introduce crc offload for tcp ddp ulp Date: Mon, 1 Feb 2021 12:04:51 +0200 Message-Id: <20210201100509.27351-4-borisp@mellanox.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20210201100509.27351-1-borisp@mellanox.com> References: <20210201100509.27351-1-borisp@mellanox.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210201_050522_650980_C8A2D134 X-CRM114-Status: GOOD ( 17.68 ) X-BeenThere: linux-nvme@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Yoray Zack , yorayz@nvidia.com, boris.pismenny@gmail.com, Ben Ben-Ishay , benishay@nvidia.com, linux-nvme@lists.infradead.org, netdev@vger.kernel.org, Or Gerlitz , ogerlitz@nvidia.com Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "Linux-nvme" Errors-To: linux-nvme-bounces+linux-nvme=archiver.kernel.org@lists.infradead.org This commit introduces support for CRC offload to direct data placement ULP on the receive side. Both DDP and CRC share a common API to initialize the offload for a TCP socket. But otherwise, both can be executed independently. On the receive side, CRC offload relies on a new SKB bit that indicates that no CRC error was encountered while processing this packet. If all packets of a ULP message have this bit set, then the CRC verification for the message can be skipped, as hardware already checked it. The following patches will set and use this bit to perform NVME-TCP CRC offload. A subsequent series, will add NVMe-TCP transmit side CRC support. Signed-off-by: Boris Pismenny Signed-off-by: Ben Ben-Ishay Signed-off-by: Or Gerlitz Signed-off-by: Yoray Zack Reviewed-by: Sagi Grimberg --- include/linux/netdev_features.h | 2 ++ include/linux/netdevice.h | 2 +- include/linux/skbuff.h | 2 +- net/Kconfig | 8 ++++++++ net/ethtool/common.c | 1 + net/ipv4/tcp_input.c | 4 ++-- net/ipv4/tcp_ipv4.c | 2 +- net/ipv4/tcp_offload.c | 2 +- 8 files changed, 17 insertions(+), 6 deletions(-) diff --git a/include/linux/netdev_features.h b/include/linux/netdev_features.h index 7977371d2dd1..77fb2cb99b78 100644 --- a/include/linux/netdev_features.h +++ b/include/linux/netdev_features.h @@ -86,6 +86,7 @@ enum { NETIF_F_HW_MACSEC_BIT, /* Offload MACsec operations */ NETIF_F_GRO_UDP_FWD_BIT, /* Allow UDP GRO for forwarding */ NETIF_F_HW_TCP_DDP_BIT, /* TCP direct data placement offload */ + NETIF_F_HW_TCP_DDP_CRC_RX_BIT, /* TCP DDP CRC RX offload */ /* * Add your fresh new feature above and remember to update @@ -161,6 +162,7 @@ enum { #define NETIF_F_HW_MACSEC __NETIF_F(HW_MACSEC) #define NETIF_F_GRO_UDP_FWD __NETIF_F(GRO_UDP_FWD) #define NETIF_F_HW_TCP_DDP __NETIF_F(HW_TCP_DDP) +#define NETIF_F_HW_TCP_DDP_CRC_RX __NETIF_F(HW_TCP_DDP_CRC_RX) /* Finds the next feature with the highest number of the range of start till 0. */ diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index bd28520e30f2..1065ddaa8e6b 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -1943,7 +1943,7 @@ struct net_device { const struct tlsdev_ops *tlsdev_ops; #endif -#ifdef CONFIG_TCP_DDP +#if IS_ENABLED(CONFIG_TCP_DDP) || IS_ENABLED(CONFIG_TCP_DDP_CRC) const struct tcp_ddp_dev_ops *tcp_ddp_ops; #endif diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 6d00d62628c8..32c5ab8e2f67 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -861,7 +861,7 @@ struct sk_buff { #ifdef CONFIG_TLS_DEVICE __u8 decrypted:1; #endif -#ifdef CONFIG_TCP_DDP +#if defined(CONFIG_TCP_DDP) || defined(CONFIG_TCP_DDP_CRC) __u8 ddp_crc:1; #endif diff --git a/net/Kconfig b/net/Kconfig index 3876861cdc90..80ed9f038968 100644 --- a/net/Kconfig +++ b/net/Kconfig @@ -465,6 +465,14 @@ config TCP_DDP NVMe-TCP/iSCSI, to request the NIC to place TCP payload data of a command response directly into kernel pages. +config TCP_DDP_CRC + bool "TCP direct data placement CRC offload" + default n + help + Direct Data Placement (DDP) CRC32C offload for TCP enables ULP, such as + NVMe-TCP/iSCSI, to request the NIC to calculate/verify the data digest + of commands as they go through the NIC. Thus avoiding the costly + per-byte overhead. endif # if NET diff --git a/net/ethtool/common.c b/net/ethtool/common.c index bbf83cd01106..14286afb2a69 100644 --- a/net/ethtool/common.c +++ b/net/ethtool/common.c @@ -70,6 +70,7 @@ const char netdev_features_strings[NETDEV_FEATURE_COUNT][ETH_GSTRING_LEN] = { [NETIF_F_HW_MACSEC_BIT] = "macsec-hw-offload", [NETIF_F_GRO_UDP_FWD_BIT] = "rx-udp-gro-forwarding", [NETIF_F_HW_TCP_DDP_BIT] = "tcp-ddp-offload", + [NETIF_F_HW_TCP_DDP_CRC_RX_BIT] = "tcp-ddp-crc-rx-offload", }; const char diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c index 0ae1ffca090d..cd490bd548a3 100644 --- a/net/ipv4/tcp_input.c +++ b/net/ipv4/tcp_input.c @@ -5150,7 +5150,7 @@ tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root, #ifdef CONFIG_TLS_DEVICE nskb->decrypted = skb->decrypted; #endif -#ifdef CONFIG_TCP_DDP +#if defined(CONFIG_TCP_DDP) || defined(CONFIG_TCP_DDP_CRC) nskb->ddp_crc = skb->ddp_crc; #endif TCP_SKB_CB(nskb)->seq = TCP_SKB_CB(nskb)->end_seq = start; @@ -5186,7 +5186,7 @@ tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root, if (skb->decrypted != nskb->decrypted) goto end; #endif -#ifdef CONFIG_TCP_DDP +#if defined(CONFIG_TCP_DDP) || defined(CONFIG_TCP_DDP_CRC) if (skb->ddp_crc != nskb->ddp_crc) goto end; diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c index b114fc870513..4c903c1a6c90 100644 --- a/net/ipv4/tcp_ipv4.c +++ b/net/ipv4/tcp_ipv4.c @@ -1811,7 +1811,7 @@ bool tcp_add_backlog(struct sock *sk, struct sk_buff *skb) #ifdef CONFIG_TLS_DEVICE tail->decrypted != skb->decrypted || #endif -#ifdef CONFIG_TCP_DDP +#if defined(CONFIG_TCP_DDP) || defined(CONFIG_TCP_DDP_CRC) tail->ddp_crc != skb->ddp_crc || #endif thtail->doff != th->doff || diff --git a/net/ipv4/tcp_offload.c b/net/ipv4/tcp_offload.c index 3ce196375d94..9e43a044e730 100644 --- a/net/ipv4/tcp_offload.c +++ b/net/ipv4/tcp_offload.c @@ -262,7 +262,7 @@ struct sk_buff *tcp_gro_receive(struct list_head *head, struct sk_buff *skb) #ifdef CONFIG_TLS_DEVICE flush |= p->decrypted ^ skb->decrypted; #endif -#ifdef CONFIG_TCP_DDP +#if defined(CONFIG_TCP_DDP) || defined(CONFIG_TCP_DDP_CRC) flush |= p->ddp_crc ^ skb->ddp_crc; #endif -- 2.24.1 _______________________________________________ Linux-nvme mailing list Linux-nvme@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-nvme