kernel-tls-handshake.lists.linux.dev archive mirror
 help / color / mirror / Atom feed
* [PATCHv2 00/18] nvme: In-kernel TLS support for TCP
@ 2023-03-29 13:59 Hannes Reinecke
  2023-03-29 13:59 ` [PATCH 01/18] nvme-keyring: register '.nvme' keyring and add CONFIG_NVME_TLS Hannes Reinecke
                   ` (17 more replies)
  0 siblings, 18 replies; 74+ messages in thread
From: Hannes Reinecke @ 2023-03-29 13:59 UTC (permalink / raw)
  To: Christoph Hellwig
  Cc: Sagi Grimberg, Keith Busch, linux-nvme, Chuck Lever,
	kernel-tls-handshake, Hannes Reinecke

Hi all,

finally I've managed to put all things together and enable in-kernel
TLS support for NVMe-over-TCP.

The patchset is based on the TLS upcall mechanism from Chuck Lever
(cf '[PATCH v7 0/2] Another crack at a handshake upcall mechanism'
posted to the linux netdev list), and requires the 'tlshd' userspace
daemon (https://github.com/oracle/ktls-utils) for the actual TLS handshake.
Changes for nvme-cli are already included in the upstream repository.

Theory of operation:
A dedicated '.nvme' keyring is created to hold the pre-shared keys (PSKs)
for the TLS handshake. Keys will have to be provisioned before TLS handshake
is attempted; that can be done with the 'nvme gen-tls-key' command for nvme-cli
(patches are already merged upstream).
After connection to the remote TCP port the client side will use the
'best' PSK (as inferred from the NVMe TCP spec) or the PSK specified
by the '--tls_key' option to nvme-cli and call the TLS userspace daemon
to initiate a TLS handshake.
The server side will then invoke the TLS userspace daemon to run the TLS
handshake.
If the TLS handshake succeeds the userspace daemon will be activating
kTLS on the socket, and control is passed back to the kernel.

To make this work I had to implement the 'read_sock()' functionality
for TLS; it seems to be holding up well enough (for me), but it really
could do with reviews from persons with more network stack knowledge.

As usual, comments and reviews are welcome.

Changes to the original RFC:
- Add a CONFIG_NVME_TLS config option
- Use a single PSK for the TLS handshake
- Make TLS connections mandatory
- Do not peek messages for the server
- Simplify data_ready callback
- Implement read_sock() for TLS

Hannes Reinecke (18):
  nvme-keyring: register '.nvme' keyring and add CONFIG_NVME_TLS
  nvme-keyring: define a 'psk' keytype
  nvme: add TCP TSAS definitions
  nvme-tcp: add definitions for TLS cipher suites
  net/tls: implement ->read_sock()
  nvme/tcp: allocate socket file
  nvme-keyring: implement nvme_tls_psk_default()
  security/keys: export key_lookup()
  nvme-tcp: enable TLS handshake upcall
  nvme-tcp: fixup send workflow for kTLS
  nvme-tcp: control message handling for recvmsg()
  nvme-fabrics: parse options 'keyring' and 'tls_key'
  nvmet: make TCP sectype settable via configfs
  nvmet-tcp: allocate socket file
  nvmet-tcp: enable TLS handshake upcall
  nvmet-tcp: rework sendpage for kTLS
  nvmet-tcp: control messages for recvmsg()
  nvmet-tcp: add configfs attribute 'param_keyring'

 drivers/nvme/common/Kconfig    |   9 ++
 drivers/nvme/common/Makefile   |   1 +
 drivers/nvme/common/keyring.c  | 181 ++++++++++++++++++++++
 drivers/nvme/host/core.c       |  44 +++++-
 drivers/nvme/host/fabrics.c    |  90 ++++++++++-
 drivers/nvme/host/fabrics.h    |   9 ++
 drivers/nvme/host/nvme.h       |   4 +-
 drivers/nvme/host/tcp.c        | 213 +++++++++++++++++++++++---
 drivers/nvme/target/configfs.c | 165 ++++++++++++++++++++-
 drivers/nvme/target/nvmet.h    |   1 +
 drivers/nvme/target/tcp.c      | 264 ++++++++++++++++++++++++++++++---
 include/linux/nvme-keyring.h   |  20 +++
 include/linux/nvme-tcp.h       |   6 +
 include/linux/nvme.h           |  10 ++
 net/tls/tls.h                  |   2 +
 net/tls/tls_main.c             |   2 +
 net/tls/tls_sw.c               |  71 +++++++++
 security/keys/key.c            |   1 +
 18 files changed, 1050 insertions(+), 43 deletions(-)
 create mode 100644 drivers/nvme/common/keyring.c
 create mode 100644 include/linux/nvme-keyring.h

-- 
2.35.3


^ permalink raw reply	[flat|nested] 74+ messages in thread

* [PATCH 01/18] nvme-keyring: register '.nvme' keyring and add CONFIG_NVME_TLS
  2023-03-29 13:59 [PATCHv2 00/18] nvme: In-kernel TLS support for TCP Hannes Reinecke
@ 2023-03-29 13:59 ` Hannes Reinecke
  2023-03-29 14:49   ` Sagi Grimberg
                     ` (2 more replies)
  2023-03-29 13:59 ` [PATCH 02/18] nvme-keyring: define a 'psk' keytype Hannes Reinecke
                   ` (16 subsequent siblings)
  17 siblings, 3 replies; 74+ messages in thread
From: Hannes Reinecke @ 2023-03-29 13:59 UTC (permalink / raw)
  To: Christoph Hellwig
  Cc: Sagi Grimberg, Keith Busch, linux-nvme, Chuck Lever,
	kernel-tls-handshake, Hannes Reinecke

Register a '.nvme' keyring to hold keys for TLS and DH-HMAC-CHAP and
add a new config option NVME_TLS to enable support for NVMe-TCP/TLS.
We need a separate keyring for NVMe as the configuration is done
via individual commands (eg for configfs), and the usual per-session
or per-process keyrings can't be used.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/nvme/common/Kconfig   |  9 +++++++++
 drivers/nvme/common/Makefile  |  1 +
 drivers/nvme/common/keyring.c | 36 +++++++++++++++++++++++++++++++++++
 drivers/nvme/host/core.c      | 19 +++++++++++++++---
 include/linux/nvme-keyring.h  | 12 ++++++++++++
 5 files changed, 74 insertions(+), 3 deletions(-)
 create mode 100644 drivers/nvme/common/keyring.c
 create mode 100644 include/linux/nvme-keyring.h

diff --git a/drivers/nvme/common/Kconfig b/drivers/nvme/common/Kconfig
index 4514f44362dd..b6fff16da1fb 100644
--- a/drivers/nvme/common/Kconfig
+++ b/drivers/nvme/common/Kconfig
@@ -2,3 +2,12 @@
 
 config NVME_COMMON
        tristate
+
+config NVME_TLS
+	bool "NVMe/TCP TLS encryption support"
+	depends on NVME_COMMON
+	select KEYS
+	help
+	  Enables TLS encryption for NVMe/TCP using the netlink handshake API.
+
+	  If unsure, say N.
diff --git a/drivers/nvme/common/Makefile b/drivers/nvme/common/Makefile
index 720c625b8a52..4bf8c08293f3 100644
--- a/drivers/nvme/common/Makefile
+++ b/drivers/nvme/common/Makefile
@@ -5,3 +5,4 @@ ccflags-y			+= -I$(src)
 obj-$(CONFIG_NVME_COMMON)	+= nvme-common.o
 
 nvme-common-y			+= auth.o
+nvme-common-$(CONFIG_NVME_TLS)	+= keyring.o
diff --git a/drivers/nvme/common/keyring.c b/drivers/nvme/common/keyring.c
new file mode 100644
index 000000000000..3a6e8a0b38e2
--- /dev/null
+++ b/drivers/nvme/common/keyring.c
@@ -0,0 +1,36 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2020 Hannes Reinecke, SUSE Linux
+ */
+
+#include <linux/module.h>
+#include <linux/nvme.h>
+#include <linux/seq_file.h>
+#include <linux/key-type.h>
+#include <keys/user-type.h>
+
+static struct key *nvme_keyring;
+
+int nvme_keyring_init(void)
+{
+	int err;
+
+	nvme_keyring = keyring_alloc(".nvme",
+				     GLOBAL_ROOT_UID, GLOBAL_ROOT_GID,
+				     current_cred(),
+				     (KEY_POS_ALL & ~KEY_POS_SETATTR) |
+				     (KEY_USR_ALL & ~KEY_USR_SETATTR),
+				     KEY_ALLOC_NOT_IN_QUOTA, NULL, NULL);
+	if (IS_ERR(nvme_keyring))
+		return PTR_ERR(nvme_keyring);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(nvme_keyring_init);
+
+void nvme_keyring_exit(void)
+{
+	key_revoke(nvme_keyring);
+	key_put(nvme_keyring);
+}
+EXPORT_SYMBOL_GPL(nvme_keyring_exit);
diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index d4be525f8100..416d0a898f56 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -25,6 +25,9 @@
 #include "nvme.h"
 #include "fabrics.h"
 #include <linux/nvme-auth.h>
+#ifdef CONFIG_NVME_TLS
+#include <linux/nvme-keyring.h>
+#endif
 
 #define CREATE_TRACE_POINTS
 #include "trace.h"
@@ -3954,7 +3957,6 @@ static umode_t nvme_dev_attrs_are_visible(struct kobject *kobj,
 	if (a == &dev_attr_dhchap_ctrl_secret.attr && !ctrl->opts)
 		return 0;
 #endif
-
 	return a->mode;
 }
 
@@ -5414,13 +5416,21 @@ static int __init nvme_core_init(void)
 		result = PTR_ERR(nvme_ns_chr_class);
 		goto unregister_generic_ns;
 	}
-
-	result = nvme_init_auth();
+#ifdef CONFIG_NVME_TLS
+	result = nvme_keyring_init();
 	if (result)
 		goto destroy_ns_chr;
+#endif
+	result = nvme_init_auth();
+	if (result)
+		goto keyring_exit;
 	return 0;
 
+keyring_exit:
+#ifdef CONFIG_NVME_TLS
+	nvme_keyring_exit();
 destroy_ns_chr:
+#endif
 	class_destroy(nvme_ns_chr_class);
 unregister_generic_ns:
 	unregister_chrdev_region(nvme_ns_chr_devt, NVME_MINORS);
@@ -5443,6 +5453,9 @@ static int __init nvme_core_init(void)
 static void __exit nvme_core_exit(void)
 {
 	nvme_exit_auth();
+#ifdef CONFIG_NVME_TLS
+	nvme_keyring_exit();
+#endif
 	class_destroy(nvme_ns_chr_class);
 	class_destroy(nvme_subsys_class);
 	class_destroy(nvme_class);
diff --git a/include/linux/nvme-keyring.h b/include/linux/nvme-keyring.h
new file mode 100644
index 000000000000..a875c06cc922
--- /dev/null
+++ b/include/linux/nvme-keyring.h
@@ -0,0 +1,12 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2021 Hannes Reinecke, SUSE Software Solutions
+ */
+
+#ifndef _NVME_KEYRING_H
+#define _NVME_KEYRING_H
+
+int nvme_keyring_init(void);
+void nvme_keyring_exit(void);
+
+#endif
-- 
2.35.3


^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [PATCH 02/18] nvme-keyring: define a 'psk' keytype
  2023-03-29 13:59 [PATCHv2 00/18] nvme: In-kernel TLS support for TCP Hannes Reinecke
  2023-03-29 13:59 ` [PATCH 01/18] nvme-keyring: register '.nvme' keyring and add CONFIG_NVME_TLS Hannes Reinecke
@ 2023-03-29 13:59 ` Hannes Reinecke
  2023-03-29 13:59 ` [PATCH 03/18] nvme: add TCP TSAS definitions Hannes Reinecke
                   ` (15 subsequent siblings)
  17 siblings, 0 replies; 74+ messages in thread
From: Hannes Reinecke @ 2023-03-29 13:59 UTC (permalink / raw)
  To: Christoph Hellwig
  Cc: Sagi Grimberg, Keith Busch, linux-nvme, Chuck Lever,
	kernel-tls-handshake, Hannes Reinecke

Define a 'psk' keytype to hold the NVMe TLS PSKs.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/nvme/common/keyring.c | 98 +++++++++++++++++++++++++++++++++++
 include/linux/nvme-keyring.h  |  6 +++
 2 files changed, 104 insertions(+)

diff --git a/drivers/nvme/common/keyring.c b/drivers/nvme/common/keyring.c
index 3a6e8a0b38e2..4ac33538f839 100644
--- a/drivers/nvme/common/keyring.c
+++ b/drivers/nvme/common/keyring.c
@@ -5,12 +5,104 @@
 
 #include <linux/module.h>
 #include <linux/nvme.h>
+#include <linux/nvme-tcp.h>
 #include <linux/seq_file.h>
 #include <linux/key-type.h>
 #include <keys/user-type.h>
 
 static struct key *nvme_keyring;
 
+key_serial_t nvme_keyring_id(void)
+{
+	return nvme_keyring->serial;
+}
+EXPORT_SYMBOL_GPL(nvme_keyring_id);
+
+static void nvme_tls_psk_describe(const struct key *key, struct seq_file *m)
+{
+	seq_puts(m, key->description);
+	seq_printf(m, ": %u", key->datalen);
+}
+
+static bool nvme_tls_psk_match(const struct key *key,
+			       const struct key_match_data *match_data)
+{
+	const char *match_id;
+	size_t match_len;
+
+	if (!key->description) {
+		pr_debug("%s: no key description\n", __func__);
+		return false;
+	}
+	match_len = strlen(key->description);
+	pr_debug("%s: id %s len %zd\n", __func__, key->description, match_len);
+
+	if (!match_data->raw_data) {
+		pr_debug("%s: no match data\n", __func__);
+		return false;
+	}
+	match_id = match_data->raw_data;
+	pr_debug("%s: match '%s' '%s' len %zd\n",
+		 __func__, match_id, key->description, match_len);
+	return !memcmp(key->description, match_id, match_len);
+}
+
+static int nvme_tls_psk_match_preparse(struct key_match_data *match_data)
+{
+	match_data->lookup_type = KEYRING_SEARCH_LOOKUP_ITERATE;
+	match_data->cmp = nvme_tls_psk_match;
+	return 0;
+}
+
+static struct key_type nvme_tls_psk_key_type = {
+	.name           = "psk",
+	.flags          = KEY_TYPE_NET_DOMAIN,
+	.preparse       = user_preparse,
+	.free_preparse  = user_free_preparse,
+	.match_preparse = nvme_tls_psk_match_preparse,
+	.instantiate    = generic_key_instantiate,
+	.revoke         = user_revoke,
+	.destroy        = user_destroy,
+	.describe       = nvme_tls_psk_describe,
+	.read           = user_read,
+};
+
+struct key *nvme_tls_psk_lookup(struct key *keyring,
+		const char *hostnqn, const char *subnqn,
+		int hmac, bool generated)
+{
+	char *identity;
+	size_t identity_len = (NVMF_NQN_SIZE) * 2 + 11;
+	key_ref_t keyref;
+	key_serial_t keyring_id;
+
+	identity = kzalloc(identity_len, GFP_KERNEL);
+	if (!identity)
+		return ERR_PTR(-ENOMEM);
+
+	snprintf(identity, identity_len, "NVMe0%c%02d %s %s",
+		 generated ? 'G' : 'R', hmac, hostnqn, subnqn);
+
+	if (!keyring)
+		keyring = nvme_keyring;
+	keyring_id = key_serial(keyring);
+	pr_debug("keyring %x lookup tls psk '%s'\n",
+		 keyring_id, identity);
+	keyref = keyring_search(make_key_ref(keyring, true),
+				&nvme_tls_psk_key_type,
+				identity, false);
+	if (IS_ERR(keyref)) {
+		pr_debug("lookup tls psk '%s' failed, error %ld\n",
+			 identity, PTR_ERR(keyref));
+		kfree(identity);
+		return ERR_PTR(-ENOKEY);
+	}
+	kfree(identity);
+
+	return key_ref_to_ptr(keyref);
+}
+EXPORT_SYMBOL_GPL(nvme_tls_psk_lookup);
+
 int nvme_keyring_init(void)
 {
 	int err;
@@ -24,12 +116,18 @@ int nvme_keyring_init(void)
 	if (IS_ERR(nvme_keyring))
 		return PTR_ERR(nvme_keyring);
 
+	err = register_key_type(&nvme_tls_psk_key_type);
+	if (err) {
+		key_put(nvme_keyring);
+		return err;
+	}
 	return 0;
 }
 EXPORT_SYMBOL_GPL(nvme_keyring_init);
 
 void nvme_keyring_exit(void)
 {
+	unregister_key_type(&nvme_tls_psk_key_type);
 	key_revoke(nvme_keyring);
 	key_put(nvme_keyring);
 }
diff --git a/include/linux/nvme-keyring.h b/include/linux/nvme-keyring.h
index a875c06cc922..5be60485ddfa 100644
--- a/include/linux/nvme-keyring.h
+++ b/include/linux/nvme-keyring.h
@@ -6,6 +6,12 @@
 #ifndef _NVME_KEYRING_H
 #define _NVME_KEYRING_H
 
+struct key *nvme_tls_psk_lookup(struct key *keyring,
+				const char *hostnqn, const char *subnqn,
+				int hmac, bool generated);
+
+key_serial_t nvme_keyring_id(void);
+
 int nvme_keyring_init(void);
 void nvme_keyring_exit(void);
 
-- 
2.35.3


^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [PATCH 03/18] nvme: add TCP TSAS definitions
  2023-03-29 13:59 [PATCHv2 00/18] nvme: In-kernel TLS support for TCP Hannes Reinecke
  2023-03-29 13:59 ` [PATCH 01/18] nvme-keyring: register '.nvme' keyring and add CONFIG_NVME_TLS Hannes Reinecke
  2023-03-29 13:59 ` [PATCH 02/18] nvme-keyring: define a 'psk' keytype Hannes Reinecke
@ 2023-03-29 13:59 ` Hannes Reinecke
  2023-03-29 13:59 ` [PATCH 04/18] nvme-tcp: add definitions for TLS cipher suites Hannes Reinecke
                   ` (14 subsequent siblings)
  17 siblings, 0 replies; 74+ messages in thread
From: Hannes Reinecke @ 2023-03-29 13:59 UTC (permalink / raw)
  To: Christoph Hellwig
  Cc: Sagi Grimberg, Keith Busch, linux-nvme, Chuck Lever,
	kernel-tls-handshake, Hannes Reinecke

Signed-off-by: Hannes Reinecke <hare@suse.de>
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
---
 include/linux/nvme.h | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/include/linux/nvme.h b/include/linux/nvme.h
index 779507ac750b..ea961ca2022d 100644
--- a/include/linux/nvme.h
+++ b/include/linux/nvme.h
@@ -108,6 +108,13 @@ enum {
 	NVMF_RDMA_CMS_RDMA_CM	= 1, /* Sockets based endpoint addressing */
 };
 
+/* TSAS SECTYPE for TCP transport */
+enum {
+	NVMF_TCP_SECTYPE_NONE = 0, /* No Security */
+	NVMF_TCP_SECTYPE_TLS12 = 1, /* TLSv1.2, NVMe-oF 1.1 and NVMe-TCP 3.6.1.1 */
+	NVMF_TCP_SECTYPE_TLS13 = 2, /* TLSv1.3, NVMe-oF 1.1 and NVMe-TCP 3.6.1.1 */
+};
+
 #define NVME_AQ_DEPTH		32
 #define NVME_NR_AEN_COMMANDS	1
 #define NVME_AQ_BLK_MQ_DEPTH	(NVME_AQ_DEPTH - NVME_NR_AEN_COMMANDS)
@@ -1458,6 +1465,9 @@ struct nvmf_disc_rsp_page_entry {
 			__u16	pkey;
 			__u8	resv10[246];
 		} rdma;
+		struct tcp {
+			__u8	sectype;
+		} tcp;
 	} tsas;
 };
 
-- 
2.35.3


^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [PATCH 04/18] nvme-tcp: add definitions for TLS cipher suites
  2023-03-29 13:59 [PATCHv2 00/18] nvme: In-kernel TLS support for TCP Hannes Reinecke
                   ` (2 preceding siblings ...)
  2023-03-29 13:59 ` [PATCH 03/18] nvme: add TCP TSAS definitions Hannes Reinecke
@ 2023-03-29 13:59 ` Hannes Reinecke
  2023-03-29 13:59 ` [PATCH 05/18] net/tls: implement ->read_sock() Hannes Reinecke
                   ` (13 subsequent siblings)
  17 siblings, 0 replies; 74+ messages in thread
From: Hannes Reinecke @ 2023-03-29 13:59 UTC (permalink / raw)
  To: Christoph Hellwig
  Cc: Sagi Grimberg, Keith Busch, linux-nvme, Chuck Lever,
	kernel-tls-handshake, Hannes Reinecke

Signed-off-by: Hannes Reinecke <hare@suse.de>
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
---
 include/linux/nvme-tcp.h | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/include/linux/nvme-tcp.h b/include/linux/nvme-tcp.h
index 75470159a194..19ca863813f1 100644
--- a/include/linux/nvme-tcp.h
+++ b/include/linux/nvme-tcp.h
@@ -18,6 +18,12 @@ enum nvme_tcp_pfv {
 	NVME_TCP_PFV_1_0 = 0x0,
 };
 
+enum nvme_tcp_tls_cipher {
+	NVME_TCP_TLS_CIPHER_INVALID     = 0,
+	NVME_TCP_TLS_CIPHER_SHA256      = 1,
+	NVME_TCP_TLS_CIPHER_SHA384      = 2,
+};
+
 enum nvme_tcp_fatal_error_status {
 	NVME_TCP_FES_INVALID_PDU_HDR		= 0x01,
 	NVME_TCP_FES_PDU_SEQ_ERR		= 0x02,
-- 
2.35.3


^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [PATCH 05/18] net/tls: implement ->read_sock()
  2023-03-29 13:59 [PATCHv2 00/18] nvme: In-kernel TLS support for TCP Hannes Reinecke
                   ` (3 preceding siblings ...)
  2023-03-29 13:59 ` [PATCH 04/18] nvme-tcp: add definitions for TLS cipher suites Hannes Reinecke
@ 2023-03-29 13:59 ` Hannes Reinecke
  2023-03-29 15:37   ` Sagi Grimberg
  2023-03-29 15:44   ` Sagi Grimberg
  2023-03-29 13:59 ` [PATCH 06/18] nvme/tcp: allocate socket file Hannes Reinecke
                   ` (12 subsequent siblings)
  17 siblings, 2 replies; 74+ messages in thread
From: Hannes Reinecke @ 2023-03-29 13:59 UTC (permalink / raw)
  To: Christoph Hellwig
  Cc: Sagi Grimberg, Keith Busch, linux-nvme, Chuck Lever,
	kernel-tls-handshake, Hannes Reinecke, Boris Pismenny

Implement ->read_sock() function for use with nvme-tcp.

Signed-off-by: Hannes Reinecke <hare@suse.de>
Cc: Boris Pismenny <boris.pismenny@gmail.com>
---
 net/tls/tls.h      |  2 ++
 net/tls/tls_main.c |  2 ++
 net/tls/tls_sw.c   | 71 ++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 75 insertions(+)

diff --git a/net/tls/tls.h b/net/tls/tls.h
index 804c3880d028..a5bf3a9ce142 100644
--- a/net/tls/tls.h
+++ b/net/tls/tls.h
@@ -113,6 +113,8 @@ bool tls_sw_sock_is_readable(struct sock *sk);
 ssize_t tls_sw_splice_read(struct socket *sock, loff_t *ppos,
 			   struct pipe_inode_info *pipe,
 			   size_t len, unsigned int flags);
+int tls_sw_read_sock(struct sock *sk, read_descriptor_t *desc,
+		     sk_read_actor_t read_actor);
 
 int tls_device_sendmsg(struct sock *sk, struct msghdr *msg, size_t size);
 int tls_device_sendpage(struct sock *sk, struct page *page,
diff --git a/net/tls/tls_main.c b/net/tls/tls_main.c
index 3735cb00905d..9b097f2902f6 100644
--- a/net/tls/tls_main.c
+++ b/net/tls/tls_main.c
@@ -934,9 +934,11 @@ static void build_proto_ops(struct proto_ops ops[TLS_NUM_CONFIG][TLS_NUM_CONFIG]
 
 	ops[TLS_BASE][TLS_SW  ] = ops[TLS_BASE][TLS_BASE];
 	ops[TLS_BASE][TLS_SW  ].splice_read	= tls_sw_splice_read;
+	ops[TLS_BASE][TLS_SW  ].read_sock	= tls_sw_read_sock;
 
 	ops[TLS_SW  ][TLS_SW  ] = ops[TLS_SW  ][TLS_BASE];
 	ops[TLS_SW  ][TLS_SW  ].splice_read	= tls_sw_splice_read;
+	ops[TLS_SW  ][TLS_SW  ].read_sock	= tls_sw_read_sock;
 
 #ifdef CONFIG_TLS_DEVICE
 	ops[TLS_HW  ][TLS_BASE] = ops[TLS_BASE][TLS_BASE];
diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c
index 782d3701b86f..77d57066daf4 100644
--- a/net/tls/tls_sw.c
+++ b/net/tls/tls_sw.c
@@ -2210,6 +2210,77 @@ ssize_t tls_sw_splice_read(struct socket *sock,  loff_t *ppos,
 	goto splice_read_end;
 }
 
+int tls_sw_read_sock(struct sock *sk, read_descriptor_t *desc,
+		     sk_read_actor_t read_actor)
+{
+	struct tls_context *tls_ctx = tls_get_ctx(sk);
+	struct tls_sw_context_rx *ctx = tls_sw_ctx_rx(tls_ctx);
+	struct strp_msg *rxm = NULL;
+	struct tls_msg *tlm;
+	struct sk_buff *skb;
+	ssize_t copied = 0;
+	int err, used;
+
+	if (!skb_queue_empty(&ctx->rx_list)) {
+		skb = __skb_dequeue(&ctx->rx_list);
+	} else {
+		struct tls_decrypt_arg darg;
+
+		err = tls_rx_rec_wait(sk, NULL, true, true);
+		if (err <= 0)
+			return err;
+
+		memset(&darg.inargs, 0, sizeof(darg.inargs));
+
+		err = tls_rx_one_record(sk, NULL, &darg);
+		if (err < 0) {
+			tls_err_abort(sk, -EBADMSG);
+			return err;
+		}
+
+		tls_rx_rec_done(ctx);
+		skb = darg.skb;
+	}
+
+	do {
+		rxm = strp_msg(skb);
+		tlm = tls_msg(skb);
+
+		/* read_sock does not support reading control messages */
+		if (tlm->control != TLS_RECORD_TYPE_DATA) {
+			err = -EINVAL;
+			goto read_sock_requeue;
+		}
+
+		used = read_actor(desc, skb, rxm->offset, rxm->full_len);
+		if (used <= 0) {
+			err = used;
+			goto read_sock_end;
+		}
+
+		copied += used;
+		if (used < rxm->full_len) {
+			rxm->offset += used;
+			rxm->full_len -= used;
+			if (!desc->count)
+				goto read_sock_requeue;
+		} else {
+			consume_skb(skb);
+			if (desc->count && !skb_queue_empty(&ctx->rx_list))
+				skb = __skb_dequeue(&ctx->rx_list);
+			else
+				skb = NULL;
+		}
+	} while (skb);
+
+read_sock_end:
+	return copied ? : err;
+
+read_sock_requeue:
+	__skb_queue_head(&ctx->rx_list, skb);
+	goto read_sock_end;
+}
+
 bool tls_sw_sock_is_readable(struct sock *sk)
 {
 	struct tls_context *tls_ctx = tls_get_ctx(sk);
-- 
2.35.3


^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [PATCH 06/18] nvme/tcp: allocate socket file
  2023-03-29 13:59 [PATCHv2 00/18] nvme: In-kernel TLS support for TCP Hannes Reinecke
                   ` (4 preceding siblings ...)
  2023-03-29 13:59 ` [PATCH 05/18] net/tls: implement ->read_sock() Hannes Reinecke
@ 2023-03-29 13:59 ` Hannes Reinecke
  2023-03-29 15:57   ` Sagi Grimberg
  2023-03-29 13:59 ` [PATCH 07/18] nvme-keyring: implement nvme_tls_psk_default() Hannes Reinecke
                   ` (11 subsequent siblings)
  17 siblings, 1 reply; 74+ messages in thread
From: Hannes Reinecke @ 2023-03-29 13:59 UTC (permalink / raw)
  To: Christoph Hellwig
  Cc: Sagi Grimberg, Keith Busch, linux-nvme, Chuck Lever,
	kernel-tls-handshake, Hannes Reinecke

When using the TLS upcall we need to allocate a socket file such
that the userspace daemon is able to use the socket.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/nvme/host/tcp.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index 42c0598c31f2..fddf785aba74 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -115,6 +115,7 @@ enum nvme_tcp_recv_state {
 struct nvme_tcp_ctrl;
 struct nvme_tcp_queue {
 	struct socket		*sock;
+	struct file		*sock_file;
 	struct work_struct	io_work;
 	int			io_cpu;
 
@@ -1330,7 +1331,10 @@ static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid)
 	}
 
 	noreclaim_flag = memalloc_noreclaim_save();
-	sock_release(queue->sock);
+	/* ->sock will be released by fput() */
+	fput(queue->sock_file);
+	queue->sock_file = NULL;
+	queue->sock = NULL;
 	memalloc_noreclaim_restore(noreclaim_flag);
 
 	kfree(queue->pdu);
@@ -1526,6 +1530,13 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid)
 		goto err_destroy_mutex;
 	}
 
+	queue->sock_file = sock_alloc_file(queue->sock, O_CLOEXEC, NULL);
+	if (IS_ERR(queue->sock_file)) {
+		sock_release(queue->sock);
+		ret = PTR_ERR(queue->sock_file);
+		queue->sock_file = NULL;
+		goto err_sock;
+	}
 	nvme_tcp_reclassify_socket(queue->sock);
 
 	/* Single syn retry */
@@ -1647,7 +1658,9 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid)
 	if (queue->hdr_digest || queue->data_digest)
 		nvme_tcp_free_crypto(queue);
 err_sock:
-	sock_release(queue->sock);
+	/* ->sock will be released by fput() */
+	fput(queue->sock_file);
+	queue->sock_file = NULL;
 	queue->sock = NULL;
 err_destroy_mutex:
 	mutex_destroy(&queue->send_mutex);
-- 
2.35.3


^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [PATCH 07/18] nvme-keyring: implement nvme_tls_psk_default()
  2023-03-29 13:59 [PATCHv2 00/18] nvme: In-kernel TLS support for TCP Hannes Reinecke
                   ` (5 preceding siblings ...)
  2023-03-29 13:59 ` [PATCH 06/18] nvme/tcp: allocate socket file Hannes Reinecke
@ 2023-03-29 13:59 ` Hannes Reinecke
  2023-03-29 15:35   ` Sagi Grimberg
  2023-03-29 13:59 ` [PATCH 08/18] security/keys: export key_lookup() Hannes Reinecke
                   ` (10 subsequent siblings)
  17 siblings, 1 reply; 74+ messages in thread
From: Hannes Reinecke @ 2023-03-29 13:59 UTC (permalink / raw)
  To: Christoph Hellwig
  Cc: Sagi Grimberg, Keith Busch, linux-nvme, Chuck Lever,
	kernel-tls-handshake, Hannes Reinecke

Implement a function to select the 'best' PSK for TLS.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/nvme/common/keyring.c | 47 +++++++++++++++++++++++++++++++++++
 include/linux/nvme-keyring.h  |  2 ++
 2 files changed, 49 insertions(+)

diff --git a/drivers/nvme/common/keyring.c b/drivers/nvme/common/keyring.c
index 4ac33538f839..ca36a061bd48 100644
--- a/drivers/nvme/common/keyring.c
+++ b/drivers/nvme/common/keyring.c
@@ -103,6 +103,53 @@ struct key *nvme_tls_psk_lookup(struct key *keyring,
 }
 EXPORT_SYMBOL_GPL(nvme_tls_psk_lookup);
 
+/*
+ * NVMe PSK priority list
+ *
+ * 'Retained' PSKs (ie 'generated == false')
+ * should be preferred to 'generated' PSKs,
+ * and SHA-384 should be preferred to SHA-256.
+ */
+struct nvme_psk_priority_list {
+	bool generated;
+	enum nvme_tcp_tls_cipher cipher;
+} nvme_psk_prio[] = {
+	{ .generated = false,
+	  .cipher = NVME_TCP_TLS_CIPHER_SHA384, },
+	{ .generated = false,
+	  .cipher = NVME_TCP_TLS_CIPHER_SHA256, },
+	{ .generated = true,
+	  .cipher = NVME_TCP_TLS_CIPHER_SHA384, },
+	{ .generated = true,
+	  .cipher = NVME_TCP_TLS_CIPHER_SHA256, },
+};
+
+/*
+ * nvme_tls_psk_default - Return 'best' PSK to use for TLS ClientHello
+ */
+key_serial_t nvme_tls_psk_default(struct key *keyring,
+		      const char *hostnqn, const char *subnqn)
+{
+	struct key *tls_key;
+	key_serial_t tls_key_id;
+	int prio;
+
+	for (prio = 0; prio < ARRAY_SIZE(nvme_psk_prio); prio++) {
+		bool generated = nvme_psk_prio[prio].generated;
+		enum nvme_tcp_tls_cipher cipher = nvme_psk_prio[prio].cipher;
+
+		tls_key = nvme_tls_psk_lookup(keyring, hostnqn, subnqn,
+					      cipher, generated);
+		if (!IS_ERR(tls_key)) {
+			tls_key_id = tls_key->serial;
+			key_put(tls_key);
+			return tls_key_id;
+		}
+	}
+	return 0;
+}
+EXPORT_SYMBOL_GPL(nvme_tls_psk_default);
+
 int nvme_keyring_init(void)
 {
 	int err;
diff --git a/include/linux/nvme-keyring.h b/include/linux/nvme-keyring.h
index 5be60485ddfa..5293e0a90167 100644
--- a/include/linux/nvme-keyring.h
+++ b/include/linux/nvme-keyring.h
@@ -9,6 +9,8 @@
 struct key *nvme_tls_psk_lookup(struct key *keyring,
 				const char *hostnqn, const char *subnqn,
 				int hmac, bool generated);
+key_serial_t nvme_tls_psk_default(struct key *keyring,
+				  const char *hostnqn, const char *subnqn);
 
 key_serial_t nvme_keyring_id(void);
 
-- 
2.35.3


^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [PATCH 08/18] security/keys: export key_lookup()
  2023-03-29 13:59 [PATCHv2 00/18] nvme: In-kernel TLS support for TCP Hannes Reinecke
                   ` (6 preceding siblings ...)
  2023-03-29 13:59 ` [PATCH 07/18] nvme-keyring: implement nvme_tls_psk_default() Hannes Reinecke
@ 2023-03-29 13:59 ` Hannes Reinecke
  2023-03-29 13:59 ` [PATCH 09/18] nvme-tcp: enable TLS handshake upcall Hannes Reinecke
                   ` (9 subsequent siblings)
  17 siblings, 0 replies; 74+ messages in thread
From: Hannes Reinecke @ 2023-03-29 13:59 UTC (permalink / raw)
  To: Christoph Hellwig
  Cc: Sagi Grimberg, Keith Busch, linux-nvme, Chuck Lever,
	kernel-tls-handshake, Hannes Reinecke

For in-kernel consumers one cannot readily assign a user (eg when
running from a workqueue), so the normal key search permissions
cannot be applied.
This patch exports the 'key_lookup()' function for a simple lookup
of keys without checking for permissions.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 security/keys/key.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/security/keys/key.c b/security/keys/key.c
index 5c0c7df833f8..bd1b7d45df90 100644
--- a/security/keys/key.c
+++ b/security/keys/key.c
@@ -693,6 +693,7 @@ struct key *key_lookup(key_serial_t id)
 	spin_unlock(&key_serial_lock);
 	return key;
 }
+EXPORT_SYMBOL_GPL(key_lookup);
 
 /*
  * Find and lock the specified key type against removal.
-- 
2.35.3


^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [PATCH 09/18] nvme-tcp: enable TLS handshake upcall
  2023-03-29 13:59 [PATCHv2 00/18] nvme: In-kernel TLS support for TCP Hannes Reinecke
                   ` (7 preceding siblings ...)
  2023-03-29 13:59 ` [PATCH 08/18] security/keys: export key_lookup() Hannes Reinecke
@ 2023-03-29 13:59 ` Hannes Reinecke
  2023-03-30 12:54   ` Daniel Wagner
  2023-03-30 15:03   ` Sagi Grimberg
  2023-03-29 13:59 ` [PATCH 10/18] nvme-tcp: fixup send workflow for kTLS Hannes Reinecke
                   ` (8 subsequent siblings)
  17 siblings, 2 replies; 74+ messages in thread
From: Hannes Reinecke @ 2023-03-29 13:59 UTC (permalink / raw)
  To: Christoph Hellwig
  Cc: Sagi Grimberg, Keith Busch, linux-nvme, Chuck Lever,
	kernel-tls-handshake, Hannes Reinecke

Add a fabrics option 'tls' and start the TLS handshake upcall
with the default PSK. When TLS is started the PSK key serial
number is displayed in the sysfs attribute 'tls_key'

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/nvme/host/core.c    |  25 ++++++-
 drivers/nvme/host/fabrics.c |  11 +++
 drivers/nvme/host/fabrics.h |   3 +
 drivers/nvme/host/nvme.h    |   4 +-
 drivers/nvme/host/tcp.c     | 141 ++++++++++++++++++++++++++++++++++--
 5 files changed, 174 insertions(+), 10 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 416d0a898f56..869396058e2e 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -3901,6 +3901,19 @@ static DEVICE_ATTR(dhchap_ctrl_secret, S_IRUGO | S_IWUSR,
 	nvme_ctrl_dhchap_ctrl_secret_show, nvme_ctrl_dhchap_ctrl_secret_store);
 #endif
 
+#ifdef CONFIG_NVME_TLS
+static ssize_t tls_key_show(struct device *dev,
+			    struct device_attribute *attr, char *buf)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+
+	if (!ctrl->tls_key)
+		return 0;
+	return sysfs_emit(buf, "%08x", key_serial(ctrl->tls_key));
+}
+static DEVICE_ATTR_RO(tls_key);
+#endif
+
 static struct attribute *nvme_dev_attrs[] = {
 	&dev_attr_reset_controller.attr,
 	&dev_attr_rescan_controller.attr,
@@ -3927,6 +3940,9 @@ static struct attribute *nvme_dev_attrs[] = {
 #ifdef CONFIG_NVME_AUTH
 	&dev_attr_dhchap_secret.attr,
 	&dev_attr_dhchap_ctrl_secret.attr,
+#endif
+#ifdef CONFIG_NVME_TLS
+	&dev_attr_tls_key.attr,
 #endif
 	NULL
 };
@@ -3956,6 +3972,10 @@ static umode_t nvme_dev_attrs_are_visible(struct kobject *kobj,
 		return 0;
 	if (a == &dev_attr_dhchap_ctrl_secret.attr && !ctrl->opts)
 		return 0;
+#endif
+#ifdef CONFIG_NVME_TLS
+	if (a == &dev_attr_tls_key.attr && !ctrl->opts)
+		return 0;
 #endif
 	return a->mode;
 }
@@ -5094,7 +5114,10 @@ static void nvme_free_ctrl(struct device *dev)
 
 	if (!subsys || ctrl->instance != subsys->instance)
 		ida_free(&nvme_instance_ida, ctrl->instance);
-
+#ifdef CONFIG_NVME_TLS
+	if (ctrl->tls_key)
+		key_put(ctrl->tls_key);
+#endif
 	nvme_free_cels(ctrl);
 	nvme_mpath_uninit(ctrl);
 	nvme_auth_stop(ctrl);
diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index bbaa04a0c502..3e4f0e45b58f 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -609,6 +609,7 @@ static const match_table_t opt_tokens = {
 	{ NVMF_OPT_DISCOVERY,		"discovery"		},
 	{ NVMF_OPT_DHCHAP_SECRET,	"dhchap_secret=%s"	},
 	{ NVMF_OPT_DHCHAP_CTRL_SECRET,	"dhchap_ctrl_secret=%s"	},
+	{ NVMF_OPT_TLS,			"tls"			},
 	{ NVMF_OPT_ERR,			NULL			}
 };
 
@@ -632,6 +633,7 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
 	opts->hdr_digest = false;
 	opts->data_digest = false;
 	opts->tos = -1; /* < 0 == use transport default */
+	opts->tls = false;
 
 	options = o = kstrdup(buf, GFP_KERNEL);
 	if (!options)
@@ -918,6 +920,15 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
 			kfree(opts->dhchap_ctrl_secret);
 			opts->dhchap_ctrl_secret = p;
 			break;
+		case NVMF_OPT_TLS:
+#ifdef CONFIG_NVME_TLS
+			opts->tls = true;
+			break;
+#else
+			pr_err("TLS is not supported\n");
+			ret = -EINVAL;
+			goto out;
+#endif
 		default:
 			pr_warn("unknown parameter or missing value '%s' in ctrl creation request\n",
 				p);
diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h
index dcac3df8a5f7..5db36e250e7a 100644
--- a/drivers/nvme/host/fabrics.h
+++ b/drivers/nvme/host/fabrics.h
@@ -70,6 +70,7 @@ enum {
 	NVMF_OPT_DISCOVERY	= 1 << 22,
 	NVMF_OPT_DHCHAP_SECRET	= 1 << 23,
 	NVMF_OPT_DHCHAP_CTRL_SECRET = 1 << 24,
+	NVMF_OPT_TLS		= 1 << 25,
 };
 
 /**
@@ -102,6 +103,7 @@ enum {
  * @dhchap_secret: DH-HMAC-CHAP secret
  * @dhchap_ctrl_secret: DH-HMAC-CHAP controller secret for bi-directional
  *              authentication
+ * @tls:        Start TLS encrypted connections (TCP)
  * @disable_sqflow: disable controller sq flow control
  * @hdr_digest: generate/verify header digest (TCP)
  * @data_digest: generate/verify data digest (TCP)
@@ -128,6 +130,7 @@ struct nvmf_ctrl_options {
 	int			max_reconnects;
 	char			*dhchap_secret;
 	char			*dhchap_ctrl_secret;
+	bool			tls;
 	bool			disable_sqflow;
 	bool			hdr_digest;
 	bool			data_digest;
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index bf46f122e9e1..421d01f849d1 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -347,7 +347,9 @@ struct nvme_ctrl {
 	struct nvme_dhchap_key *ctrl_key;
 	u16 transaction;
 #endif
-
+#ifdef CONFIG_NVME_TLS
+	struct key *tls_key;
+#endif
 	/* Power saving configuration */
 	u64 ps_max_latency_us;
 	bool apst_enabled;
diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index fddf785aba74..fdb564d0b9f4 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -8,9 +8,15 @@
 #include <linux/init.h>
 #include <linux/slab.h>
 #include <linux/err.h>
+#include <linux/key.h>
 #include <linux/nvme-tcp.h>
 #include <net/sock.h>
 #include <net/tcp.h>
+#ifdef CONFIG_NVME_TLS
+#include <linux/nvme-keyring.h>
+#include <net/tls.h>
+#include <net/handshake.h>
+#endif
 #include <linux/blk-mq.h>
 #include <crypto/hash.h>
 #include <net/busy_poll.h>
@@ -31,6 +37,16 @@ static int so_priority;
 module_param(so_priority, int, 0644);
 MODULE_PARM_DESC(so_priority, "nvme tcp socket optimize priority");
 
+#ifdef CONFIG_NVME_TLS
+/*
+ * TLS handshake timeout
+ */
+static int tls_handshake_timeout = 10;
+module_param(tls_handshake_timeout, int, 0644);
+MODULE_PARM_DESC(tls_handshake_timeout,
+		 "nvme TLS handshake timeout in seconds (default 10)");
+#endif
+
 #ifdef CONFIG_DEBUG_LOCK_ALLOC
 /* lockdep can detect a circular dependency of the form
  *   sk_lock -> mmap_lock (page fault) -> fs locks -> sk_lock
@@ -147,7 +163,10 @@ struct nvme_tcp_queue {
 	struct ahash_request	*snd_hash;
 	__le32			exp_ddgst;
 	__le32			recv_ddgst;
-
+#ifdef CONFIG_NVME_TLS
+	struct completion       *tls_complete;
+	int                     tls_err;
+#endif
 	struct page_frag_cache	pf_cache;
 
 	void (*state_change)(struct sock *);
@@ -1503,7 +1522,80 @@ static void nvme_tcp_set_queue_io_cpu(struct nvme_tcp_queue *queue)
 	queue->io_cpu = cpumask_next_wrap(n - 1, cpu_online_mask, -1, false);
 }
 
-static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid)
+#ifdef CONFIG_NVME_TLS
+static void nvme_tcp_tls_done(void *data, int status, key_serial_t peerid)
+{
+	struct nvme_tcp_queue *queue = data;
+	struct nvme_tcp_ctrl *ctrl = queue->ctrl;
+	int qid = nvme_tcp_queue_id(queue);
+
+	dev_dbg(ctrl->ctrl.device, "queue %d: TLS handshake done, key %x, status %d\n",
+		qid, peerid, status);
+
+	if (status)
+		queue->tls_err = -status;
+	else {
+		struct key *tls_key = key_lookup(peerid);
+		if (IS_ERR(tls_key)) {
+			dev_warn(ctrl->ctrl.device, "queue %d: Invalid key %x\n",
+				 qid, peerid);
+			queue->tls_err = -ENOKEY;
+		} else {
+			ctrl->ctrl.tls_key = tls_key;
+			queue->tls_err = 0;
+		}
+	}
+
+	if (queue->tls_complete)
+		complete(queue->tls_complete);
+}
+
+static int nvme_tcp_start_tls(struct nvme_ctrl *nctrl,
+			      struct nvme_tcp_queue *queue,
+			      key_serial_t peerid)
+{
+	int qid = nvme_tcp_queue_id(queue);
+	int ret;
+	struct tls_handshake_args args;
+	unsigned long tmo = tls_handshake_timeout * HZ;
+	DECLARE_COMPLETION_ONSTACK(tls_complete);
+	key_serial_t keyring = nvme_keyring_id();
+
+	dev_dbg(nctrl->device, "queue %d: start TLS with key %x\n",
+		qid, peerid);
+	args.ta_sock = queue->sock;
+	args.ta_done = nvme_tcp_tls_done;
+	args.ta_data = queue;
+	args.ta_my_peerids[0] = peerid;
+	args.ta_num_peerids = 1;
+	args.ta_keyring = keyring;
+	args.ta_timeout_ms = tls_handshake_timeout * 2 * 1000;
+	queue->tls_err = -EOPNOTSUPP;
+	queue->tls_complete = &tls_complete;
+	ret = tls_client_hello_psk(&args, GFP_KERNEL);
+	if (ret) {
+		dev_dbg(nctrl->device, "queue %d: failed to start TLS: %d\n",
+			qid, ret);
+		return ret;
+	}
+	if (wait_for_completion_timeout(queue->tls_complete, tmo) == 0) {
+		dev_dbg(nctrl->device,
+			"queue %d: TLS handshake timeout\n", qid);
+		queue->tls_complete = NULL;
+		ret = -ETIMEDOUT;
+	} else {
+		dev_dbg(nctrl->device,
+			"queue %d: TLS handshake complete, error %d\n",
+			qid, queue->tls_err);
+		ret = queue->tls_err;
+	}
+	queue->tls_complete = NULL;
+	return ret;
+}
+#endif
+
+static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid,
+				key_serial_t peerid)
 {
 	struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(nctrl);
 	struct nvme_tcp_queue *queue = &ctrl->queues[qid];
@@ -1627,6 +1719,14 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid)
 		goto err_rcv_pdu;
 	}
 
+#ifdef CONFIG_NVME_TLS
+	/* If PSKs are configured try to start TLS */
+	if (peerid) {
+		ret = nvme_tcp_start_tls(nctrl, queue, peerid);
+		if (ret)
+			goto err_init_connect;
+	}
+#endif
 	ret = nvme_tcp_init_connection(queue);
 	if (ret)
 		goto err_init_connect;
@@ -1771,10 +1871,23 @@ static int nvme_tcp_start_io_queues(struct nvme_ctrl *ctrl,
 static int nvme_tcp_alloc_admin_queue(struct nvme_ctrl *ctrl)
 {
 	int ret;
-
-	ret = nvme_tcp_alloc_queue(ctrl, 0);
+	key_serial_t psk_id = 0;
+
+#ifdef CONFIG_NVME_TLS
+	if (ctrl->opts->tls) {
+		psk_id = nvme_tls_psk_default(NULL,
+					      ctrl->opts->host->nqn,
+					      ctrl->opts->subsysnqn);
+		if (!psk_id) {
+			dev_err(ctrl->device, "no valid PSK found\n");
+			ret = -ENOKEY;
+			goto out_free_queue;
+		}
+	}
+#endif
+	ret = nvme_tcp_alloc_queue(ctrl, 0, psk_id);
 	if (ret)
-		return ret;
+		goto out_free_queue;
 
 	ret = nvme_tcp_alloc_async_req(to_tcp_ctrl(ctrl));
 	if (ret)
@@ -1790,9 +1903,21 @@ static int nvme_tcp_alloc_admin_queue(struct nvme_ctrl *ctrl)
 static int __nvme_tcp_alloc_io_queues(struct nvme_ctrl *ctrl)
 {
 	int i, ret;
-
+	key_serial_t psk_id = 0;
+
+#ifdef CONFIG_NVME_TLS
+	if (ctrl->opts->tls) {
+		psk_id = nvme_tls_psk_default(NULL,
+					      ctrl->opts->host->nqn,
+					      ctrl->opts->subsysnqn);
+		if (!psk_id) {
+			dev_err(ctrl->device, "no valid PSK found\n");
+			return -ENOKEY;
+		}
+	}
+#endif
 	for (i = 1; i < ctrl->queue_count; i++) {
-		ret = nvme_tcp_alloc_queue(ctrl, i);
+		ret = nvme_tcp_alloc_queue(ctrl, i, psk_id);
 		if (ret)
 			goto out_free_queues;
 	}
@@ -2701,7 +2826,7 @@ static struct nvmf_transport_ops nvme_tcp_transport = {
 			  NVMF_OPT_HOST_TRADDR | NVMF_OPT_CTRL_LOSS_TMO |
 			  NVMF_OPT_HDR_DIGEST | NVMF_OPT_DATA_DIGEST |
 			  NVMF_OPT_NR_WRITE_QUEUES | NVMF_OPT_NR_POLL_QUEUES |
-			  NVMF_OPT_TOS | NVMF_OPT_HOST_IFACE,
+			  NVMF_OPT_TOS | NVMF_OPT_HOST_IFACE | NVMF_OPT_TLS,
 	.create_ctrl	= nvme_tcp_create_ctrl,
 };
 
-- 
2.35.3


^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [PATCH 10/18] nvme-tcp: fixup send workflow for kTLS
  2023-03-29 13:59 [PATCHv2 00/18] nvme: In-kernel TLS support for TCP Hannes Reinecke
                   ` (8 preceding siblings ...)
  2023-03-29 13:59 ` [PATCH 09/18] nvme-tcp: enable TLS handshake upcall Hannes Reinecke
@ 2023-03-29 13:59 ` Hannes Reinecke
  2023-03-30 15:24   ` Sagi Grimberg
  2023-03-29 13:59 ` [PATCH 11/18] nvme-tcp: control message handling for recvmsg() Hannes Reinecke
                   ` (7 subsequent siblings)
  17 siblings, 1 reply; 74+ messages in thread
From: Hannes Reinecke @ 2023-03-29 13:59 UTC (permalink / raw)
  To: Christoph Hellwig
  Cc: Sagi Grimberg, Keith Busch, linux-nvme, Chuck Lever,
	kernel-tls-handshake, Hannes Reinecke

kTLS does not support MSG_EOR flag for sendmsg(), and in general
is really picky about invalid MSG_XXX flags.
So ensure that the MSG_EOR flags is blanked out for TLS, and that
the MSG_SENDPAGE_LAST is only set if we actually do sendpage().

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/nvme/host/tcp.c | 28 ++++++++++++++++++++--------
 1 file changed, 20 insertions(+), 8 deletions(-)

diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index fdb564d0b9f4..54d27873227a 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -218,6 +218,15 @@ static inline struct blk_mq_tags *nvme_tcp_tagset(struct nvme_tcp_queue *queue)
 	return queue->ctrl->tag_set.tags[queue_idx - 1];
 }
 
+static inline bool nvme_tcp_tls_enabled(struct nvme_tcp_queue *queue)
+{
+#ifdef CONFIG_NVME_TLS
+	return (queue->ctrl->ctrl.tls_key != NULL);
+#else
+	return false;
+#endif
+}
+
 static inline u8 nvme_tcp_hdgst_len(struct nvme_tcp_queue *queue)
 {
 	return queue->hdr_digest ? NVME_TCP_DIGEST_LENGTH : 0;
@@ -1021,12 +1030,14 @@ static int nvme_tcp_try_send_data(struct nvme_tcp_request *req)
 		int req_data_sent = req->data_sent;
 		int ret, flags = MSG_DONTWAIT;
 
-		if (last && !queue->data_digest && !nvme_tcp_queue_more(queue))
+		if (!last || queue->data_digest || nvme_tcp_queue_more(queue))
+			flags |= MSG_MORE;
+		else if (!nvme_tcp_tls_enabled(queue))
 			flags |= MSG_EOR;
-		else
-			flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST;
 
 		if (sendpage_ok(page)) {
+			if (flags & MSG_MORE)
+				flags |= MSG_SENDPAGE_NOTLAST;
 			ret = kernel_sendpage(queue->sock, page, offset, len,
 					flags);
 		} else {
@@ -1077,9 +1088,11 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req)
 	int flags = MSG_DONTWAIT;
 	int ret;
 
-	if (inline_data || nvme_tcp_queue_more(queue))
-		flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST;
-	else
+	if (inline_data || nvme_tcp_queue_more(queue)) {
+		flags |= MSG_MORE;
+		if (!nvme_tcp_tls_enabled(queue))
+			flags |= MSG_SENDPAGE_NOTLAST;
+	} else if (!nvme_tcp_tls_enabled(queue))
 		flags |= MSG_EOR;
 
 	if (queue->hdr_digest && !req->offset)
@@ -1154,9 +1167,8 @@ static int nvme_tcp_try_send_ddgst(struct nvme_tcp_request *req)
 
 	if (nvme_tcp_queue_more(queue))
 		msg.msg_flags |= MSG_MORE;
-	else
+	else if (!nvme_tcp_tls_enabled(queue))
 		msg.msg_flags |= MSG_EOR;
-
 	ret = kernel_sendmsg(queue->sock, &msg, &iov, 1, iov.iov_len);
 	if (unlikely(ret <= 0))
 		return ret;
-- 
2.35.3


^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [PATCH 11/18] nvme-tcp: control message handling for recvmsg()
  2023-03-29 13:59 [PATCHv2 00/18] nvme: In-kernel TLS support for TCP Hannes Reinecke
                   ` (9 preceding siblings ...)
  2023-03-29 13:59 ` [PATCH 10/18] nvme-tcp: fixup send workflow for kTLS Hannes Reinecke
@ 2023-03-29 13:59 ` Hannes Reinecke
  2023-03-30 15:25   ` Sagi Grimberg
  2023-03-29 13:59 ` [PATCH 12/18] nvme-fabrics: parse options 'keyring' and 'tls_key' Hannes Reinecke
                   ` (6 subsequent siblings)
  17 siblings, 1 reply; 74+ messages in thread
From: Hannes Reinecke @ 2023-03-29 13:59 UTC (permalink / raw)
  To: Christoph Hellwig
  Cc: Sagi Grimberg, Keith Busch, linux-nvme, Chuck Lever,
	kernel-tls-handshake, Hannes Reinecke

kTLS is sending TLS ALERT messages as control messages for recvmsg().
As we can't do anything sensible with it just abort the connection
and let the userspace agent to a re-negotiation.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/nvme/host/tcp.c | 23 ++++++++++++++++++++++-
 1 file changed, 22 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index 54d27873227a..2e15fa83b725 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -1377,6 +1377,11 @@ static int nvme_tcp_init_connection(struct nvme_tcp_queue *queue)
 {
 	struct nvme_tcp_icreq_pdu *icreq;
 	struct nvme_tcp_icresp_pdu *icresp;
+#ifdef CONFIG_NVME_TLS
+	char cbuf[CMSG_LEN(sizeof(char))] = {};
+	struct cmsghdr *cmsg;
+	unsigned char ctype;
+#endif
 	struct msghdr msg = {};
 	struct kvec iov;
 	bool ctrl_hdgst, ctrl_ddgst;
@@ -1414,11 +1419,27 @@ static int nvme_tcp_init_connection(struct nvme_tcp_queue *queue)
 	memset(&msg, 0, sizeof(msg));
 	iov.iov_base = icresp;
 	iov.iov_len = sizeof(*icresp);
+#ifdef CONFIG_NVME_TLS
+	msg.msg_control = cbuf;
+	msg.msg_controllen = sizeof(cbuf);
+#endif
 	ret = kernel_recvmsg(queue->sock, &msg, &iov, 1,
 			iov.iov_len, msg.msg_flags);
 	if (ret < 0)
 		goto free_icresp;
-
+#ifdef CONFIG_NVME_TLS
+	cmsg = (struct cmsghdr *)cbuf;
+	if (CMSG_OK(&msg, cmsg) &&
+	    cmsg->cmsg_level == SOL_TLS &&
+	    cmsg->cmsg_type == TLS_GET_RECORD_TYPE) {
+		ctype = *((unsigned char *)CMSG_DATA(cmsg));
+		if (ctype != TLS_RECORD_TYPE_DATA) {
+			pr_err("queue %d: unhandled TLS record %d\n",
+			       nvme_tcp_queue_id(queue), ctype);
+			return -ENOTCONN;
+		}
+	}
+#endif
 	ret = -EINVAL;
 	if (icresp->hdr.type != nvme_tcp_icresp) {
 		pr_err("queue %d: bad type returned %d\n",
-- 
2.35.3


^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [PATCH 12/18] nvme-fabrics: parse options 'keyring' and 'tls_key'
  2023-03-29 13:59 [PATCHv2 00/18] nvme: In-kernel TLS support for TCP Hannes Reinecke
                   ` (10 preceding siblings ...)
  2023-03-29 13:59 ` [PATCH 11/18] nvme-tcp: control message handling for recvmsg() Hannes Reinecke
@ 2023-03-29 13:59 ` Hannes Reinecke
  2023-03-30 15:33   ` Sagi Grimberg
  2023-03-29 13:59 ` [PATCH 13/18] nvmet: make TCP sectype settable via configfs Hannes Reinecke
                   ` (5 subsequent siblings)
  17 siblings, 1 reply; 74+ messages in thread
From: Hannes Reinecke @ 2023-03-29 13:59 UTC (permalink / raw)
  To: Christoph Hellwig
  Cc: Sagi Grimberg, Keith Busch, linux-nvme, Chuck Lever,
	kernel-tls-handshake, Hannes Reinecke

Parse the fabrics options 'keyring' and 'tls_key' and store the
referenced keys in the options structure.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/nvme/host/fabrics.c | 79 ++++++++++++++++++++++++++++++++++++-
 drivers/nvme/host/fabrics.h |  6 +++
 drivers/nvme/host/tcp.c     | 20 +++++++---
 3 files changed, 98 insertions(+), 7 deletions(-)

diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index 3e4f0e45b58f..5f5e487d498c 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -605,6 +605,8 @@ static const match_table_t opt_tokens = {
 	{ NVMF_OPT_NR_WRITE_QUEUES,	"nr_write_queues=%d"	},
 	{ NVMF_OPT_NR_POLL_QUEUES,	"nr_poll_queues=%d"	},
 	{ NVMF_OPT_TOS,			"tos=%d"		},
+	{ NVMF_OPT_KEYRING,		"keyring=%d"		},
+	{ NVMF_OPT_TLS_KEY,		"tls_key=%d"		},
 	{ NVMF_OPT_FAIL_FAST_TMO,	"fast_io_fail_tmo=%d"	},
 	{ NVMF_OPT_DISCOVERY,		"discovery"		},
 	{ NVMF_OPT_DHCHAP_SECRET,	"dhchap_secret=%s"	},
@@ -620,8 +622,9 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
 	char *options, *o, *p;
 	int token, ret = 0;
 	size_t nqnlen  = 0;
-	int ctrl_loss_tmo = NVMF_DEF_CTRL_LOSS_TMO;
+	int ctrl_loss_tmo = NVMF_DEF_CTRL_LOSS_TMO, key_id;
 	uuid_t hostid;
+	struct key *key = NULL;
 
 	/* Set defaults */
 	opts->queue_size = NVMF_DEF_QUEUE_SIZE;
@@ -889,6 +892,74 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
 			}
 			opts->tos = token;
 			break;
+		case NVMF_OPT_KEYRING:
+#ifdef CONFIG_NVME_TLS
+			if (match_int(args, &key_id)) {
+				ret = -EINVAL;
+				goto out;
+			}
+			if (key_id < 0) {
+				pr_err("Invalid keyring id %d\n", key_id);
+				ret = -EINVAL;
+				goto out;
+			}
+			if (!key_id) {
+				pr_debug("Using default keyring\n");
+				if (opts->keyring) {
+					key_put(opts->keyring);
+					opts->keyring = NULL;
+				}
+				break;
+			}
+			key = key_lookup(key_id);
+			if (!key) {
+				pr_err("Keyring id %08x not found\n", key_id);
+				ret = -ENOKEY;
+				goto out;
+			}
+			if (opts->keyring)
+				key_put(opts->keyring);
+			opts->keyring = key;
+			break;
+#else
+			pr_err("TLS is not supported\n");
+			ret = -EINVAL;
+			goto out;
+#endif
+		case NVMF_OPT_TLS_KEY:
+#ifdef CONFIG_NVME_TLS
+			if (match_int(args, &key_id)) {
+				ret = -EINVAL;
+				goto out;
+			}
+			if (key_id < 0) {
+				pr_err("Invalid key id %d\n", key_id);
+				ret = -EINVAL;
+				goto out;
+			}
+			if (!key_id) {
+				pr_debug("Using 'best' PSK\n");
+				if (opts->tls_key) {
+					key_put(opts->tls_key);
+					opts->tls_key = NULL;
+				}
+				break;
+			}
+			key = key_lookup(key_id);
+			if (!key) {
+				pr_err("Key id %08x not found\n", key_id);
+				ret = -ENOKEY;
+				goto out;
+			}
+			if (opts->tls_key)
+				key_put(opts->tls_key);
+			opts->tls_key = key;
+#else
+			pr_err("TLS is not supported\n");
+			ret = -EINVAL;
+			goto out;
+#endif
+			break;
 		case NVMF_OPT_DISCOVERY:
 			opts->discovery_nqn = true;
 			break;
@@ -1054,6 +1125,12 @@ static int nvmf_check_allowed_opts(struct nvmf_ctrl_options *opts,
 void nvmf_free_options(struct nvmf_ctrl_options *opts)
 {
 	nvmf_host_put(opts->host);
+#ifdef CONFIG_NVME_TLS
+	if (opts->keyring)
+		key_put(opts->keyring);
+	if (opts->tls_key)
+		key_put(opts->tls_key);
+#endif
 	kfree(opts->transport);
 	kfree(opts->traddr);
 	kfree(opts->trsvcid);
diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h
index 5db36e250e7a..2ff7b7168a40 100644
--- a/drivers/nvme/host/fabrics.h
+++ b/drivers/nvme/host/fabrics.h
@@ -71,6 +71,8 @@ enum {
 	NVMF_OPT_DHCHAP_SECRET	= 1 << 23,
 	NVMF_OPT_DHCHAP_CTRL_SECRET = 1 << 24,
 	NVMF_OPT_TLS		= 1 << 25,
+	NVMF_OPT_KEYRING	= 1 << 26,
+	NVMF_OPT_TLS_KEY	= 1 << 27,
 };
 
 /**
@@ -103,6 +105,8 @@ enum {
  * @dhchap_secret: DH-HMAC-CHAP secret
  * @dhchap_ctrl_secret: DH-HMAC-CHAP controller secret for bi-directional
  *              authentication
+ * @keyring:    Keyring to use for key lookups
+ * @tls_key:    TLS key for encrypted connections (TCP)
  * @tls:        Start TLS encrypted connections (TCP)
  * @disable_sqflow: disable controller sq flow control
  * @hdr_digest: generate/verify header digest (TCP)
@@ -130,6 +134,8 @@ struct nvmf_ctrl_options {
 	int			max_reconnects;
 	char			*dhchap_secret;
 	char			*dhchap_ctrl_secret;
+	struct key		*keyring;
+	struct key		*tls_key;
 	bool			tls;
 	bool			disable_sqflow;
 	bool			hdr_digest;
diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index 2e15fa83b725..00a3f18a69af 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -1596,6 +1596,8 @@ static int nvme_tcp_start_tls(struct nvme_ctrl *nctrl,
 
 	dev_dbg(nctrl->device, "queue %d: start TLS with key %x\n",
 		qid, peerid);
+	if (nctrl->opts->keyring)
+		keyring = key_serial(nctrl->opts->keyring);
 	args.ta_sock = queue->sock;
 	args.ta_done = nvme_tcp_tls_done;
 	args.ta_data = queue;
@@ -1908,9 +1910,12 @@ static int nvme_tcp_alloc_admin_queue(struct nvme_ctrl *ctrl)
 
 #ifdef CONFIG_NVME_TLS
 	if (ctrl->opts->tls) {
-		psk_id = nvme_tls_psk_default(NULL,
-					      ctrl->opts->host->nqn,
-					      ctrl->opts->subsysnqn);
+		if (ctrl->opts->tls_key)
+			psk_id = key_serial(ctrl->opts->tls_key);
+		else
+			psk_id = nvme_tls_psk_default(ctrl->opts->keyring,
+						      ctrl->opts->host->nqn,
+						      ctrl->opts->subsysnqn);
 		if (!psk_id) {
 			dev_err(ctrl->device, "no valid PSK found\n");
 			ret = -ENOKEY;
@@ -1940,9 +1945,12 @@ static int __nvme_tcp_alloc_io_queues(struct nvme_ctrl *ctrl)
 
 #ifdef CONFIG_NVME_TLS
 	if (ctrl->opts->tls) {
-		psk_id = nvme_tls_psk_default(NULL,
-					      ctrl->opts->host->nqn,
-					      ctrl->opts->subsysnqn);
+		if (ctrl->opts->tls_key)
+			psk_id = key_serial(ctrl->opts->tls_key);
+		else
+			psk_id = nvme_tls_psk_default(ctrl->opts->keyring,
+						      ctrl->opts->host->nqn,
+						      ctrl->opts->subsysnqn);
 		if (!psk_id) {
 			dev_err(ctrl->device, "no valid PSK found\n");
 			return -ENOKEY;
-- 
2.35.3


^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [PATCH 13/18] nvmet: make TCP sectype settable via configfs
  2023-03-29 13:59 [PATCHv2 00/18] nvme: In-kernel TLS support for TCP Hannes Reinecke
                   ` (11 preceding siblings ...)
  2023-03-29 13:59 ` [PATCH 12/18] nvme-fabrics: parse options 'keyring' and 'tls_key' Hannes Reinecke
@ 2023-03-29 13:59 ` Hannes Reinecke
  2023-03-30 16:07   ` Sagi Grimberg
  2023-03-29 13:59 ` [PATCH 14/18] nvmet-tcp: allocate socket file Hannes Reinecke
                   ` (4 subsequent siblings)
  17 siblings, 1 reply; 74+ messages in thread
From: Hannes Reinecke @ 2023-03-29 13:59 UTC (permalink / raw)
  To: Christoph Hellwig
  Cc: Sagi Grimberg, Keith Busch, linux-nvme, Chuck Lever,
	kernel-tls-handshake, Hannes Reinecke

Add a new configfs attribute 'addr_tsas' to make the TCP sectype
settable via configfs.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/nvme/target/configfs.c | 67 ++++++++++++++++++++++++++++++++++
 1 file changed, 67 insertions(+)

diff --git a/drivers/nvme/target/configfs.c b/drivers/nvme/target/configfs.c
index 907143870da5..ca66ee6dc153 100644
--- a/drivers/nvme/target/configfs.c
+++ b/drivers/nvme/target/configfs.c
@@ -303,6 +303,11 @@ static void nvmet_port_init_tsas_rdma(struct nvmet_port *port)
 	port->disc_addr.tsas.rdma.cms = NVMF_RDMA_CMS_RDMA_CM;
 }
 
+static void nvmet_port_init_tsas_tcp(struct nvmet_port *port, int sectype)
+{
+	port->disc_addr.tsas.tcp.sectype = sectype;
+}
+
 static ssize_t nvmet_addr_trtype_store(struct config_item *item,
 		const char *page, size_t count)
 {
@@ -325,11 +330,72 @@ static ssize_t nvmet_addr_trtype_store(struct config_item *item,
 	port->disc_addr.trtype = nvmet_transport[i].type;
 	if (port->disc_addr.trtype == NVMF_TRTYPE_RDMA)
 		nvmet_port_init_tsas_rdma(port);
+	else if (port->disc_addr.trtype == NVMF_TRTYPE_TCP)
+		nvmet_port_init_tsas_tcp(port, NVMF_TCP_SECTYPE_NONE);
 	return count;
 }
 
 CONFIGFS_ATTR(nvmet_, addr_trtype);
 
+static const struct nvmet_type_name_map nvmet_addr_tsas_tcp[] = {
+	{ NVMF_TCP_SECTYPE_NONE,	"none" },
+	{ NVMF_TCP_SECTYPE_TLS13,	"tls1.3" },
+};
+
+static const struct nvmet_type_name_map nvmet_addr_tsas_rdma[] = {
+	{ NVMF_RDMA_QPTYPE_CONNECTED,	"connected" },
+	{ NVMF_RDMA_QPTYPE_DATAGRAM,	"datagram"  },
+};
+
+static ssize_t nvmet_addr_tsas_show(struct config_item *item,
+		char *page)
+{
+	struct nvmet_port *port = to_nvmet_port(item);
+	int i;
+
+	if (port->disc_addr.trtype == NVMF_TRTYPE_TCP) {
+		for (i = 0; i < ARRAY_SIZE(nvmet_addr_tsas_tcp); i++) {
+			if (port->disc_addr.tsas.tcp.sectype == nvmet_addr_tsas_tcp[i].type)
+				return sprintf(page, "%s\n", nvmet_addr_tsas_tcp[i].name);
+		}
+		return sprintf(page, "reserved\n");
+	} else if (port->disc_addr.trtype == NVMF_TRTYPE_RDMA) {
+		for (i = 0; i < ARRAY_SIZE(nvmet_addr_tsas_rdma); i++) {
+			if (port->disc_addr.tsas.rdma.qptype == nvmet_addr_tsas_rdma[i].type)
+				return sprintf(page, "%s\n", nvmet_addr_tsas_rdma[i].name);
+		}
+		return sprintf(page, "reserved\n");
+	}
+	return sprintf(page, "not required\n");
+}
+
+static ssize_t nvmet_addr_tsas_store(struct config_item *item,
+		const char *page, size_t count)
+{
+	struct nvmet_port *port = to_nvmet_port(item);
+	int i;
+
+	if (nvmet_is_port_enabled(port, __func__))
+		return -EACCES;
+
+	if (port->disc_addr.trtype != NVMF_TRTYPE_TCP)
+		return -EINVAL;
+
+	for (i = 0; i < ARRAY_SIZE(nvmet_addr_tsas_tcp); i++) {
+		if (sysfs_streq(page, nvmet_addr_tsas_tcp[i].name))
+			goto found;
+	}
+
+	pr_err("Invalid value '%s' for tsas\n", page);
+	return -EINVAL;
+
+found:
+	nvmet_port_init_tsas_tcp(port, nvmet_addr_tsas_tcp[i].type);
+	return count;
+}
+
+CONFIGFS_ATTR(nvmet_, addr_tsas);
+
 /*
  * Namespace structures & file operation functions below
  */
@@ -1741,6 +1807,7 @@ static struct configfs_attribute *nvmet_port_attrs[] = {
 	&nvmet_attr_addr_traddr,
 	&nvmet_attr_addr_trsvcid,
 	&nvmet_attr_addr_trtype,
+	&nvmet_attr_addr_tsas,
 	&nvmet_attr_param_inline_data_size,
 #ifdef CONFIG_BLK_DEV_INTEGRITY
 	&nvmet_attr_param_pi_enable,
-- 
2.35.3


^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [PATCH 14/18] nvmet-tcp: allocate socket file
  2023-03-29 13:59 [PATCHv2 00/18] nvme: In-kernel TLS support for TCP Hannes Reinecke
                   ` (12 preceding siblings ...)
  2023-03-29 13:59 ` [PATCH 13/18] nvmet: make TCP sectype settable via configfs Hannes Reinecke
@ 2023-03-29 13:59 ` Hannes Reinecke
  2023-03-30 16:08   ` Sagi Grimberg
  2023-03-29 13:59 ` [PATCH 15/18] nvmet-tcp: enable TLS handshake upcall Hannes Reinecke
                   ` (3 subsequent siblings)
  17 siblings, 1 reply; 74+ messages in thread
From: Hannes Reinecke @ 2023-03-29 13:59 UTC (permalink / raw)
  To: Christoph Hellwig
  Cc: Sagi Grimberg, Keith Busch, linux-nvme, Chuck Lever,
	kernel-tls-handshake, Hannes Reinecke

When using the TLS upcall we need to allocate a socket file such
that the userspace daemon is able to use the socket.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/nvme/target/tcp.c | 51 +++++++++++++++++++++++++++++----------
 1 file changed, 38 insertions(+), 13 deletions(-)

diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c
index 66e8f9fd0ca7..5931971d715f 100644
--- a/drivers/nvme/target/tcp.c
+++ b/drivers/nvme/target/tcp.c
@@ -96,12 +96,14 @@ struct nvmet_tcp_cmd {
 
 enum nvmet_tcp_queue_state {
 	NVMET_TCP_Q_CONNECTING,
+	NVMET_TCP_Q_TLS_HANDSHAKE,
 	NVMET_TCP_Q_LIVE,
 	NVMET_TCP_Q_DISCONNECTING,
 };
 
 struct nvmet_tcp_queue {
 	struct socket		*sock;
+	struct file		*sock_file;
 	struct nvmet_tcp_port	*port;
 	struct work_struct	io_work;
 	struct nvmet_cq		nvme_cq;
@@ -1406,6 +1408,19 @@ static void nvmet_tcp_restore_socket_callbacks(struct nvmet_tcp_queue *queue)
 	write_unlock_bh(&sock->sk->sk_callback_lock);
 }
 
+static void nvmet_tcp_close_sock(struct nvmet_tcp_queue *queue)
+{
+	if (queue->sock_file) {
+		fput(queue->sock_file);
+		queue->sock_file = NULL;
+		queue->sock = NULL;
+	} else {
+		WARN_ON(!queue->sock->ops);
+		sock_release(queue->sock);
+		queue->sock = NULL;
+	}
+}
+
 static void nvmet_tcp_uninit_data_in_cmds(struct nvmet_tcp_queue *queue)
 {
 	struct nvmet_tcp_cmd *cmd = queue->cmds;
@@ -1455,12 +1470,11 @@ static void nvmet_tcp_release_queue_work(struct work_struct *w)
 	nvmet_sq_destroy(&queue->nvme_sq);
 	cancel_work_sync(&queue->io_work);
 	nvmet_tcp_free_cmd_data_in_buffers(queue);
-	sock_release(queue->sock);
+	nvmet_tcp_close_sock(queue);
 	nvmet_tcp_free_cmds(queue);
 	if (queue->hdr_digest || queue->data_digest)
 		nvmet_tcp_free_crypto(queue);
 	ida_free(&nvmet_tcp_queue_ida, queue->idx);
-
 	page = virt_to_head_page(queue->pf_cache.va);
 	__page_frag_cache_drain(page, queue->pf_cache.pagecnt_bias);
 	kfree(queue);
@@ -1583,7 +1597,7 @@ static int nvmet_tcp_set_queue_sock(struct nvmet_tcp_queue *queue)
 	return ret;
 }
 
-static int nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port,
+static void nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port,
 		struct socket *newsock)
 {
 	struct nvmet_tcp_queue *queue;
@@ -1591,7 +1605,7 @@ static int nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port,
 
 	queue = kzalloc(sizeof(*queue), GFP_KERNEL);
 	if (!queue)
-		return -ENOMEM;
+		return;
 
 	INIT_WORK(&queue->release_work, nvmet_tcp_release_queue_work);
 	INIT_WORK(&queue->io_work, nvmet_tcp_io_work);
@@ -1599,15 +1613,28 @@ static int nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port,
 	queue->port = port;
 	queue->nr_cmds = 0;
 	spin_lock_init(&queue->state_lock);
-	queue->state = NVMET_TCP_Q_CONNECTING;
+	if (queue->port->nport->disc_addr.tsas.tcp.sectype ==
+	    NVMF_TCP_SECTYPE_TLS13)
+		queue->state = NVMET_TCP_Q_TLS_HANDSHAKE;
+	else
+		queue->state = NVMET_TCP_Q_CONNECTING;
 	INIT_LIST_HEAD(&queue->free_list);
 	init_llist_head(&queue->resp_list);
 	INIT_LIST_HEAD(&queue->resp_send_list);
 
+	if (queue->state == NVMET_TCP_Q_TLS_HANDSHAKE) {
+		queue->sock_file = sock_alloc_file(queue->sock, O_CLOEXEC, NULL);
+		if (IS_ERR(queue->sock_file)) {
+			ret = PTR_ERR(queue->sock_file);
+			queue->sock_file = NULL;
+			goto out_free_queue;
+		}
+	}
+
 	queue->idx = ida_alloc(&nvmet_tcp_queue_ida, GFP_KERNEL);
 	if (queue->idx < 0) {
 		ret = queue->idx;
-		goto out_free_queue;
+		goto out_sock;
 	}
 
 	ret = nvmet_tcp_alloc_cmd(queue, &queue->connect);
@@ -1628,7 +1655,7 @@ static int nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port,
 	if (ret)
 		goto out_destroy_sq;
 
-	return 0;
+	return;
 out_destroy_sq:
 	mutex_lock(&nvmet_tcp_queue_mutex);
 	list_del_init(&queue->queue_list);
@@ -1638,9 +1665,11 @@ static int nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port,
 	nvmet_tcp_free_cmd(&queue->connect);
 out_ida_remove:
 	ida_free(&nvmet_tcp_queue_ida, queue->idx);
+out_sock:
+	nvmet_tcp_close_sock(queue);
 out_free_queue:
 	kfree(queue);
-	return ret;
+	pr_err("failed to allocate queue");
 }
 
 static void nvmet_tcp_accept_work(struct work_struct *w)
@@ -1657,11 +1686,7 @@ static void nvmet_tcp_accept_work(struct work_struct *w)
 				pr_warn("failed to accept err=%d\n", ret);
 			return;
 		}
-		ret = nvmet_tcp_alloc_queue(port, newsock);
-		if (ret) {
-			pr_err("failed to allocate queue\n");
-			sock_release(newsock);
-		}
+		nvmet_tcp_alloc_queue(port, newsock);
 	}
 }
 
-- 
2.35.3


^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [PATCH 15/18] nvmet-tcp: enable TLS handshake upcall
  2023-03-29 13:59 [PATCHv2 00/18] nvme: In-kernel TLS support for TCP Hannes Reinecke
                   ` (13 preceding siblings ...)
  2023-03-29 13:59 ` [PATCH 14/18] nvmet-tcp: allocate socket file Hannes Reinecke
@ 2023-03-29 13:59 ` Hannes Reinecke
  2023-04-03 12:51   ` Sagi Grimberg
  2023-03-29 13:59 ` [PATCH 16/18] nvmet-tcp: rework sendpage for kTLS Hannes Reinecke
                   ` (2 subsequent siblings)
  17 siblings, 1 reply; 74+ messages in thread
From: Hannes Reinecke @ 2023-03-29 13:59 UTC (permalink / raw)
  To: Christoph Hellwig
  Cc: Sagi Grimberg, Keith Busch, linux-nvme, Chuck Lever,
	kernel-tls-handshake, Hannes Reinecke

Add functions to start the TLS handshake upcall when
the TCP RSAS sectype is set to 'tls1.3'.

Signed-off-by: Hannes Reincke <hare@suse.de>
---
 drivers/nvme/target/configfs.c |  32 +++++++-
 drivers/nvme/target/tcp.c      | 135 ++++++++++++++++++++++++++++++++-
 2 files changed, 163 insertions(+), 4 deletions(-)

diff --git a/drivers/nvme/target/configfs.c b/drivers/nvme/target/configfs.c
index ca66ee6dc153..36fbf6a22d09 100644
--- a/drivers/nvme/target/configfs.c
+++ b/drivers/nvme/target/configfs.c
@@ -159,10 +159,12 @@ static const struct nvmet_type_name_map nvmet_addr_treq[] = {
 	{ NVMF_TREQ_NOT_REQUIRED,	"not required" },
 };
 
+#define NVMET_PORT_TREQ(port) ((port)->disc_addr.treq & NVME_TREQ_SECURE_CHANNEL_MASK)
+
 static ssize_t nvmet_addr_treq_show(struct config_item *item, char *page)
 {
-	u8 treq = to_nvmet_port(item)->disc_addr.treq &
-		NVME_TREQ_SECURE_CHANNEL_MASK;
+	struct nvmet_port *port = to_nvmet_port(item);
+	u8 treq = NVMET_PORT_TREQ(port);
 	int i;
 
 	for (i = 0; i < ARRAY_SIZE(nvmet_addr_treq); i++) {
@@ -193,6 +195,17 @@ static ssize_t nvmet_addr_treq_store(struct config_item *item,
 	return -EINVAL;
 
 found:
+#ifdef CONFIG_NVME_TLS
+	if (port->disc_addr.trtype == NVMF_TRTYPE_TCP) {
+		if (port->disc_addr.tsas.tcp.sectype != NVMF_TCP_SECTYPE_TLS13) {
+			pr_warn("cannot change TREQ when TLS is not enabled\n");
+			return -EINVAL;
+		} else if (nvmet_addr_treq[i].type == NVMF_TREQ_NOT_SPECIFIED) {
+			pr_warn("cannot set TREQ to 'not specified' when TLS is enabled\n");
+			return -EINVAL;
+		}
+	}
+#endif
 	treq |= nvmet_addr_treq[i].type;
 	port->disc_addr.treq = treq;
 	return count;
@@ -373,6 +386,7 @@ static ssize_t nvmet_addr_tsas_store(struct config_item *item,
 		const char *page, size_t count)
 {
 	struct nvmet_port *port = to_nvmet_port(item);
+	u8 treq = port->disc_addr.treq & ~NVME_TREQ_SECURE_CHANNEL_MASK;
 	int i;
 
 	if (nvmet_is_port_enabled(port, __func__))
@@ -391,6 +405,20 @@ static ssize_t nvmet_addr_tsas_store(struct config_item *item,
 
 found:
 	nvmet_port_init_tsas_tcp(port, nvmet_addr_tsas_tcp[i].type);
+	if (nvmet_addr_tsas_tcp[i].type == NVMF_TCP_SECTYPE_TLS13) {
+#ifdef CONFIG_NVME_TLS
+		if (NVMET_PORT_TREQ(port) == NVMF_TREQ_NOT_SPECIFIED)
+			treq |= NVMF_TREQ_REQUIRED;
+		else
+			treq |= NVMET_PORT_TREQ(port);
+#else
+		pr_err("TLS not supported\n");
+		return -EINVAL;
+#endif
+	} else {
+		/* Set to 'not specified' if TLS is not enabled */
+		treq |= NVMF_TREQ_NOT_SPECIFIED;
+	}
 	return count;
 }
 
diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c
index 5931971d715f..ebec882120fd 100644
--- a/drivers/nvme/target/tcp.c
+++ b/drivers/nvme/target/tcp.c
@@ -11,6 +11,10 @@
 #include <linux/nvme-tcp.h>
 #include <net/sock.h>
 #include <net/tcp.h>
+#ifdef CONFIG_NVME_TLS
+#include <net/handshake.h>
+#include <linux/nvme-keyring.h>
+#endif
 #include <linux/inet.h>
 #include <linux/llist.h>
 #include <crypto/hash.h>
@@ -40,6 +44,16 @@ module_param(idle_poll_period_usecs, int, 0644);
 MODULE_PARM_DESC(idle_poll_period_usecs,
 		"nvmet tcp io_work poll till idle time period in usecs");
 
+#ifdef CONFIG_NVME_TLS
+/*
+ * TLS handshake timeout
+ */
+static int tls_handshake_timeout = 30;
+module_param(tls_handshake_timeout, int, 0644);
+MODULE_PARM_DESC(tls_handshake_timeout,
+		 "nvme TLS handshake timeout in seconds (default 30)");
+#endif
+
 #define NVMET_TCP_RECV_BUDGET		8
 #define NVMET_TCP_SEND_BUDGET		8
 #define NVMET_TCP_IO_WORK_BUDGET	64
@@ -130,6 +144,10 @@ struct nvmet_tcp_queue {
 	bool			data_digest;
 	struct ahash_request	*snd_hash;
 	struct ahash_request	*rcv_hash;
+#ifdef CONFIG_NVME_TLS
+	struct key		*tls_psk;
+	struct delayed_work	tls_handshake_work;
+#endif
 
 	unsigned long           poll_end;
 
@@ -1474,6 +1492,10 @@ static void nvmet_tcp_release_queue_work(struct work_struct *w)
 	nvmet_tcp_free_cmds(queue);
 	if (queue->hdr_digest || queue->data_digest)
 		nvmet_tcp_free_crypto(queue);
+#ifdef CONFIG_NVME_TLS
+	if (queue->tls_psk)
+		key_put(queue->tls_psk);
+#endif
 	ida_free(&nvmet_tcp_queue_ida, queue->idx);
 	page = virt_to_head_page(queue->pf_cache.va);
 	__page_frag_cache_drain(page, queue->pf_cache.pagecnt_bias);
@@ -1488,8 +1510,12 @@ static void nvmet_tcp_data_ready(struct sock *sk)
 
 	read_lock_bh(&sk->sk_callback_lock);
 	queue = sk->sk_user_data;
-	if (likely(queue))
-		queue_work_on(queue_cpu(queue), nvmet_tcp_wq, &queue->io_work);
+	if (queue->data_ready)
+		queue->data_ready(sk);
+	if (likely(queue) &&
+	    queue->state != NVMET_TCP_Q_TLS_HANDSHAKE)
+		queue_work_on(queue_cpu(queue), nvmet_tcp_wq,
+			      &queue->io_work);
 	read_unlock_bh(&sk->sk_callback_lock);
 }
 
@@ -1597,6 +1623,89 @@ static int nvmet_tcp_set_queue_sock(struct nvmet_tcp_queue *queue)
 	return ret;
 }
 
+#ifdef CONFIG_NVME_TLS
+static void nvmet_tcp_tls_queue_restart(struct nvmet_tcp_queue *queue)
+{
+	spin_lock(&queue->state_lock);
+	if (queue->state != NVMET_TCP_Q_TLS_HANDSHAKE) {
+		pr_warn("queue %d: TLS handshake already completed\n",
+			queue->idx);
+		spin_unlock(&queue->state_lock);
+		return;
+	}
+	queue->state = NVMET_TCP_Q_CONNECTING;
+	spin_unlock(&queue->state_lock);
+
+	pr_debug("queue %d: restarting queue after TLS handshake\n",
+		 queue->idx);
+	/*
+	 * Set callbacks after handshake; TLS implementation
+	 * might have changed the socket callbacks.
+	 */
+	nvmet_tcp_set_queue_sock(queue);
+}
+
+static void nvmet_tcp_tls_handshake_done(void *data, int status,
+					 key_serial_t peerid)
+{
+	struct nvmet_tcp_queue *queue = data;
+
+	pr_debug("queue %d: TLS handshake done, key %x, status %d\n",
+		 queue->idx, peerid, status);
+	if (!status) {
+		spin_lock(&queue->state_lock);
+		queue->tls_psk = key_lookup(peerid);
+		if (IS_ERR(queue->tls_psk)) {
+			pr_warn("queue %d: TLS key %x not found\n",
+				queue->idx, peerid);
+			queue->tls_psk = NULL;
+		}
+		spin_unlock(&queue->state_lock);
+	}
+	cancel_delayed_work_sync(&queue->tls_handshake_work);
+	if (status)
+		nvmet_tcp_schedule_release_queue(queue);
+	else
+		nvmet_tcp_tls_queue_restart(queue);
+}
+
+static void nvmet_tcp_tls_handshake_timeout_work(struct work_struct *w)
+{
+	struct nvmet_tcp_queue *queue = container_of(to_delayed_work(w),
+			struct nvmet_tcp_queue, tls_handshake_work);
+
+	pr_debug("queue %d: TLS handshake timeout\n", queue->idx);
+	nvmet_tcp_schedule_release_queue(queue);
+}
+
+static int nvmet_tcp_tls_handshake(struct nvmet_tcp_queue *queue)
+{
+	int ret = -EOPNOTSUPP;
+	struct tls_handshake_args args;
+
+	if (queue->state != NVMET_TCP_Q_TLS_HANDSHAKE) {
+		pr_warn("cannot start TLS in state %d\n", queue->state);
+		return -EINVAL;
+	}
+
+	pr_debug("queue %d: TLS ServerHello\n", queue->idx);
+	args.ta_sock = queue->sock;
+	args.ta_done = nvmet_tcp_tls_handshake_done;
+	args.ta_data = queue;
+	args.ta_keyring = nvme_keyring_id();
+	args.ta_timeout_ms = tls_handshake_timeout * 2 * 1024;
+
+	ret = tls_server_hello_psk(&args, GFP_KERNEL);
+	if (ret) {
+		pr_err("failed to start TLS, err=%d\n", ret);
+	} else {
+		queue_delayed_work(nvmet_wq, &queue->tls_handshake_work,
+				   tls_handshake_timeout * HZ);
+	}
+	return ret;
+}
+#endif
+
 static void nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port,
 		struct socket *newsock)
 {
@@ -1609,6 +1718,10 @@ static void nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port,
 
 	INIT_WORK(&queue->release_work, nvmet_tcp_release_queue_work);
 	INIT_WORK(&queue->io_work, nvmet_tcp_io_work);
+#ifdef CONFIG_NVME_TLS
+	INIT_DELAYED_WORK(&queue->tls_handshake_work,
+			  nvmet_tcp_tls_handshake_timeout_work);
+#endif
 	queue->sock = newsock;
 	queue->port = port;
 	queue->nr_cmds = 0;
@@ -1622,6 +1735,7 @@ static void nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port,
 	init_llist_head(&queue->resp_list);
 	INIT_LIST_HEAD(&queue->resp_send_list);
 
+#ifdef CONFIG_NVME_TLS
 	if (queue->state == NVMET_TCP_Q_TLS_HANDSHAKE) {
 		queue->sock_file = sock_alloc_file(queue->sock, O_CLOEXEC, NULL);
 		if (IS_ERR(queue->sock_file)) {
@@ -1630,6 +1744,7 @@ static void nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port,
 			goto out_free_queue;
 		}
 	}
+#endif
 
 	queue->idx = ida_alloc(&nvmet_tcp_queue_ida, GFP_KERNEL);
 	if (queue->idx < 0) {
@@ -1651,6 +1766,22 @@ static void nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port,
 	list_add_tail(&queue->queue_list, &nvmet_tcp_queue_list);
 	mutex_unlock(&nvmet_tcp_queue_mutex);
 
+#ifdef CONFIG_NVME_TLS
+	if (queue->state == NVMET_TCP_Q_TLS_HANDSHAKE) {
+		struct sock *sk = queue->sock->sk;
+
+		/* Restore the default callbacks before starting upcall */
+		read_lock_bh(&sk->sk_callback_lock);
+		sk->sk_user_data = NULL;
+		sk->sk_data_ready = port->data_ready;
+		read_unlock_bh(&sk->sk_callback_lock);
+		if (!nvmet_tcp_tls_handshake(queue))
+			return;
+
+		/* TLS handshake failed, terminate the connection */
+		goto out_destroy_sq;
+	}
+#endif
 	ret = nvmet_tcp_set_queue_sock(queue);
 	if (ret)
 		goto out_destroy_sq;
-- 
2.35.3


^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [PATCH 16/18] nvmet-tcp: rework sendpage for kTLS
  2023-03-29 13:59 [PATCHv2 00/18] nvme: In-kernel TLS support for TCP Hannes Reinecke
                   ` (14 preceding siblings ...)
  2023-03-29 13:59 ` [PATCH 15/18] nvmet-tcp: enable TLS handshake upcall Hannes Reinecke
@ 2023-03-29 13:59 ` Hannes Reinecke
  2023-04-03 12:52   ` Sagi Grimberg
  2023-03-29 13:59 ` [PATCH 17/18] nvmet-tcp: control messages for recvmsg() Hannes Reinecke
  2023-03-29 13:59 ` [PATCH 18/18] nvmet-tcp: add configfs attribute 'param_keyring' Hannes Reinecke
  17 siblings, 1 reply; 74+ messages in thread
From: Hannes Reinecke @ 2023-03-29 13:59 UTC (permalink / raw)
  To: Christoph Hellwig
  Cc: Sagi Grimberg, Keith Busch, linux-nvme, Chuck Lever,
	kernel-tls-handshake, Hannes Reinecke

kTLS ->sendpage() doesn't support the MSG_EOR flag, so blank
it out if kTLS is enabled.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/nvme/target/tcp.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c
index ebec882120fd..5daf16138471 100644
--- a/drivers/nvme/target/tcp.c
+++ b/drivers/nvme/target/tcp.c
@@ -650,7 +650,7 @@ static int nvmet_try_send_response(struct nvmet_tcp_cmd *cmd,
 
 	if (!last_in_batch && cmd->queue->send_list_len)
 		flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST;
-	else
+	else if (!cmd->queue->tls_psk)
 		flags |= MSG_EOR;
 
 	ret = kernel_sendpage(cmd->queue->sock, virt_to_page(cmd->rsp_pdu),
@@ -678,7 +678,7 @@ static int nvmet_try_send_r2t(struct nvmet_tcp_cmd *cmd, bool last_in_batch)
 
 	if (!last_in_batch && cmd->queue->send_list_len)
 		flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST;
-	else
+	else if (!cmd->queue->tls_psk)
 		flags |= MSG_EOR;
 
 	ret = kernel_sendpage(cmd->queue->sock, virt_to_page(cmd->r2t_pdu),
@@ -708,7 +708,7 @@ static int nvmet_try_send_ddgst(struct nvmet_tcp_cmd *cmd, bool last_in_batch)
 
 	if (!last_in_batch && cmd->queue->send_list_len)
 		msg.msg_flags |= MSG_MORE;
-	else
+	else if (!cmd->queue->tls_psk)
 		msg.msg_flags |= MSG_EOR;
 
 	ret = kernel_sendmsg(queue->sock, &msg, &iov, 1, iov.iov_len);
-- 
2.35.3


^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [PATCH 17/18] nvmet-tcp: control messages for recvmsg()
  2023-03-29 13:59 [PATCHv2 00/18] nvme: In-kernel TLS support for TCP Hannes Reinecke
                   ` (15 preceding siblings ...)
  2023-03-29 13:59 ` [PATCH 16/18] nvmet-tcp: rework sendpage for kTLS Hannes Reinecke
@ 2023-03-29 13:59 ` Hannes Reinecke
  2023-04-03 12:59   ` Sagi Grimberg
  2023-03-29 13:59 ` [PATCH 18/18] nvmet-tcp: add configfs attribute 'param_keyring' Hannes Reinecke
  17 siblings, 1 reply; 74+ messages in thread
From: Hannes Reinecke @ 2023-03-29 13:59 UTC (permalink / raw)
  To: Christoph Hellwig
  Cc: Sagi Grimberg, Keith Busch, linux-nvme, Chuck Lever,
	kernel-tls-handshake, Hannes Reinecke

kTLS requires control messages for recvmsg() to relay any out-of-band
TLS messages (eg TLS alerts) to the caller.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/nvme/target/tcp.c | 71 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 71 insertions(+)

diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c
index 5daf16138471..75017fd5da9f 100644
--- a/drivers/nvme/target/tcp.c
+++ b/drivers/nvme/target/tcp.c
@@ -12,6 +12,7 @@
 #include <net/sock.h>
 #include <net/tcp.h>
 #ifdef CONFIG_NVME_TLS
+#include <net/tls.h>
 #include <net/handshake.h>
 #include <linux/nvme-keyring.h>
 #endif
@@ -92,6 +93,7 @@ struct nvmet_tcp_cmd {
 	u32				pdu_len;
 	u32				pdu_recv;
 	int				sg_idx;
+	char				recv_cbuf[CMSG_LEN(sizeof(char))];
 	struct msghdr			recv_msg;
 	struct bio_vec			*iov;
 	u32				flags;
@@ -1086,7 +1088,18 @@ static int nvmet_tcp_try_recv_pdu(struct nvmet_tcp_queue *queue)
 	struct nvme_tcp_hdr *hdr = &queue->pdu.cmd.hdr;
 	int len;
 	struct kvec iov;
+#ifdef CONFIG_NVME_TLS
+	char cbuf[CMSG_LEN(sizeof(char))] = {};
+	unsigned char ctype;
+	struct cmsghdr *cmsg;
+	struct msghdr msg = {
+		.msg_control = cbuf,
+		.msg_controllen = sizeof(cbuf),
+		.msg_flags = MSG_DONTWAIT
+	};
+#else
 	struct msghdr msg = { .msg_flags = MSG_DONTWAIT };
+#endif
 
 recv:
 	iov.iov_base = (void *)&queue->pdu + queue->offset;
@@ -1095,6 +1108,19 @@ static int nvmet_tcp_try_recv_pdu(struct nvmet_tcp_queue *queue)
 			iov.iov_len, msg.msg_flags);
 	if (unlikely(len < 0))
 		return len;
+#ifdef CONFIG_NVME_TLS
+	cmsg = (struct cmsghdr *)cbuf;
+	if (CMSG_OK(&msg, cmsg) &&
+	    cmsg->cmsg_level == SOL_TLS &&
+	    cmsg->cmsg_type == TLS_GET_RECORD_TYPE) {
+		ctype = *((unsigned char *)CMSG_DATA(cmsg));
+		if (ctype != TLS_RECORD_TYPE_DATA) {
+			pr_err("queue %d unhandled TLS record %d\n",
+				queue->idx, ctype);
+			return -ENOTCONN;
+		}
+	}
+#endif
 
 	queue->offset += len;
 	queue->left -= len;
@@ -1150,10 +1176,27 @@ static int nvmet_tcp_try_recv_data(struct nvmet_tcp_queue *queue)
 	int ret;
 
 	while (msg_data_left(&cmd->recv_msg)) {
+#ifdef CONFIG_NVME_TLS
+		struct cmsghdr *cmsg;
+		unsigned char ctype;
+#endif
 		ret = sock_recvmsg(cmd->queue->sock, &cmd->recv_msg,
 			cmd->recv_msg.msg_flags);
 		if (ret <= 0)
 			return ret;
+#ifdef CONFIG_NVME_TLS
+		cmsg = (struct cmsghdr *)cmd->recv_cbuf;
+		if (CMSG_OK(&cmd->recv_msg, cmsg) &&
+		    cmsg->cmsg_level == SOL_TLS &&
+		    cmsg->cmsg_type == TLS_GET_RECORD_TYPE) {
+			ctype = *((unsigned char *)CMSG_DATA(cmsg));
+			if (ctype != TLS_RECORD_TYPE_DATA) {
+				pr_err("queue %d unhandled TLS record %d\n",
+				       queue->idx, ctype);
+				return -ENOTCONN;
+			}
+		}
+#endif
 
 		cmd->pdu_recv += ret;
 		cmd->rbytes_done += ret;
@@ -1175,7 +1218,18 @@ static int nvmet_tcp_try_recv_ddgst(struct nvmet_tcp_queue *queue)
 {
 	struct nvmet_tcp_cmd *cmd = queue->cmd;
 	int ret;
+#ifdef CONFIG_NVME_TLS
+	char cbuf[CMSG_LEN(sizeof(char))] = {};
+	unsigned char ctype;
+	struct cmsghdr *cmsg;
+	struct msghdr msg = {
+		.msg_control = cbuf,
+		.msg_controllen = sizeof(cbuf),
+		.msg_flags = MSG_DONTWAIT
+	};
+#else
 	struct msghdr msg = { .msg_flags = MSG_DONTWAIT };
+#endif
 	struct kvec iov = {
 		.iov_base = (void *)&cmd->recv_ddgst + queue->offset,
 		.iov_len = queue->left
@@ -1185,6 +1239,19 @@ static int nvmet_tcp_try_recv_ddgst(struct nvmet_tcp_queue *queue)
 			iov.iov_len, msg.msg_flags);
 	if (unlikely(ret < 0))
 		return ret;
+#ifdef CONFIG_NVME_TLS
+	cmsg = (struct cmsghdr *)cbuf;
+	if (CMSG_OK(&msg, cmsg) &&
+	    cmsg->cmsg_level == SOL_TLS &&
+	    cmsg->cmsg_type == TLS_GET_RECORD_TYPE) {
+		ctype = *((unsigned char *)CMSG_DATA(cmsg));
+		if (ctype != TLS_RECORD_TYPE_DATA) {
+			pr_err("queue %d unhandled TLS record %d\n",
+				queue->idx, ctype);
+			return -ENOTCONN;
+		}
+	}
+#endif
 
 	queue->offset += ret;
 	queue->left -= ret;
@@ -1354,6 +1421,10 @@ static int nvmet_tcp_alloc_cmd(struct nvmet_tcp_queue *queue,
 	if (!c->r2t_pdu)
 		goto out_free_data;
 
+#ifdef CONFIG_NVME_TLS
+	c->recv_msg.msg_control = c->recv_cbuf;
+	c->recv_msg.msg_controllen = sizeof(c->recv_cbuf);
+#endif
 	c->recv_msg.msg_flags = MSG_DONTWAIT | MSG_NOSIGNAL;
 
 	list_add_tail(&c->entry, &queue->free_list);
-- 
2.35.3


^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [PATCH 18/18] nvmet-tcp: add configfs attribute 'param_keyring'
  2023-03-29 13:59 [PATCHv2 00/18] nvme: In-kernel TLS support for TCP Hannes Reinecke
                   ` (16 preceding siblings ...)
  2023-03-29 13:59 ` [PATCH 17/18] nvmet-tcp: control messages for recvmsg() Hannes Reinecke
@ 2023-03-29 13:59 ` Hannes Reinecke
  2023-04-03 13:03   ` Sagi Grimberg
  17 siblings, 1 reply; 74+ messages in thread
From: Hannes Reinecke @ 2023-03-29 13:59 UTC (permalink / raw)
  To: Christoph Hellwig
  Cc: Sagi Grimberg, Keith Busch, linux-nvme, Chuck Lever,
	kernel-tls-handshake, Hannes Reinecke

Add a configfs attribute to list and change the default keyring.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/nvme/target/configfs.c | 68 +++++++++++++++++++++++++++++++++-
 drivers/nvme/target/nvmet.h    |  1 +
 drivers/nvme/target/tcp.c      |  3 +-
 3 files changed, 70 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/target/configfs.c b/drivers/nvme/target/configfs.c
index 36fbf6a22d09..0a92bc8b25e5 100644
--- a/drivers/nvme/target/configfs.c
+++ b/drivers/nvme/target/configfs.c
@@ -15,6 +15,10 @@
 #ifdef CONFIG_NVME_TARGET_AUTH
 #include <linux/nvme-auth.h>
 #endif
+#include <linux/key.h>
+#ifdef CONFIG_NVME_TLS
+#include <linux/nvme-keyring.h>
+#endif
 #include <crypto/hash.h>
 #include <crypto/kpp.h>
 
@@ -294,6 +298,48 @@ static ssize_t nvmet_param_pi_enable_store(struct config_item *item,
 CONFIGFS_ATTR(nvmet_, param_pi_enable);
 #endif
 
+#ifdef CONFIG_NVME_TLS
+static ssize_t nvmet_param_keyring_show(struct config_item *item,
+		char *page)
+{
+	struct nvmet_port *port = to_nvmet_port(item);
+
+	if (!port->keyring)
+		return sprintf(page, "\n");
+	return snprintf(page, PAGE_SIZE, "%s\n",
+			port->keyring->description);
+}
+
+static ssize_t nvmet_param_keyring_store(struct config_item *item,
+		const char *page, size_t count)
+{
+	struct nvmet_port *port = to_nvmet_port(item);
+	struct key *keyring;
+	unsigned int keyring_id;
+	int ret;
+
+	if (nvmet_is_port_enabled(port, __func__))
+		return -EACCES;
+
+	ret = kstrtou32(page, 0, &keyring_id);
+	if (ret) {
+		pr_err("Invalid keyring id '%s'\n", page);
+		return ret;
+	}
+	keyring = key_lookup(keyring_id);
+	if (IS_ERR(keyring)) {
+		pr_err("Invalid keyring '%08x'\n", keyring_id);
+		return PTR_ERR(keyring);
+	}
+	if (port->keyring)
+		key_put(port->keyring);
+	port->keyring = keyring;
+	return count;
+}
+
+CONFIGFS_ATTR(nvmet_, param_keyring);
+#endif
+
 static ssize_t nvmet_addr_trtype_show(struct config_item *item,
 		char *page)
 {
@@ -404,9 +450,12 @@ static ssize_t nvmet_addr_tsas_store(struct config_item *item,
 	return -EINVAL;
 
 found:
-	nvmet_port_init_tsas_tcp(port, nvmet_addr_tsas_tcp[i].type);
 	if (nvmet_addr_tsas_tcp[i].type == NVMF_TCP_SECTYPE_TLS13) {
 #ifdef CONFIG_NVME_TLS
+		if (!port->keyring) {
+			pr_err("NVMe keyring not available, cannot enable TLS 1.3\n");
+			return -ENOTSUPP;
+		}
 		if (NVMET_PORT_TREQ(port) == NVMF_TREQ_NOT_SPECIFIED)
 			treq |= NVMF_TREQ_REQUIRED;
 		else
@@ -419,6 +468,8 @@ static ssize_t nvmet_addr_tsas_store(struct config_item *item,
 		/* Set to 'not specified' if TLS is not enabled */
 		treq |= NVMF_TREQ_NOT_SPECIFIED;
 	}
+	port->disc_addr.treq = treq;
+	nvmet_port_init_tsas_tcp(port, nvmet_addr_tsas_tcp[i].type);
 	return count;
 }
 
@@ -1825,6 +1876,8 @@ static void nvmet_port_release(struct config_item *item)
 	flush_workqueue(nvmet_wq);
 	list_del(&port->global_entry);
 
+	if (port->keyring)
+		key_put(port->keyring);
 	kfree(port->ana_state);
 	kfree(port);
 }
@@ -1839,6 +1892,9 @@ static struct configfs_attribute *nvmet_port_attrs[] = {
 	&nvmet_attr_param_inline_data_size,
 #ifdef CONFIG_BLK_DEV_INTEGRITY
 	&nvmet_attr_param_pi_enable,
+#endif
+#ifdef CONFIG_NVME_TLS
+	&nvmet_attr_param_keyring,
 #endif
 	NULL,
 };
@@ -1857,6 +1913,7 @@ static struct config_group *nvmet_ports_make(struct config_group *group,
 		const char *name)
 {
 	struct nvmet_port *port;
+	struct key *keyring = NULL;
 	u16 portid;
 	u32 i;
 
@@ -1874,6 +1931,15 @@ static struct config_group *nvmet_ports_make(struct config_group *group,
 		return ERR_PTR(-ENOMEM);
 	}
 
+#ifdef CONFIG_NVME_TLS
+	keyring = key_lookup(nvme_keyring_id());
+	if (IS_ERR(keyring)) {
+		pr_warn("NVMe keyring not available, disabling TLS\n");
+		keyring = NULL;
+	}
+#endif
+	port->keyring = keyring;
+
 	for (i = 1; i <= NVMET_MAX_ANAGRPS; i++) {
 		if (i == NVMET_DEFAULT_ANA_GRPID)
 			port->ana_state[1] = NVME_ANA_OPTIMIZED;
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index ed3786140965..c31e310d269a 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -158,6 +158,7 @@ struct nvmet_port {
 	struct config_group		ana_groups_group;
 	struct nvmet_ana_group		ana_default_group;
 	enum nvme_ana_state		*ana_state;
+	struct key			*keyring;
 	void				*priv;
 	bool				enabled;
 	int				inline_data_size;
diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c
index 75017fd5da9f..5b1069854f15 100644
--- a/drivers/nvme/target/tcp.c
+++ b/drivers/nvme/target/tcp.c
@@ -8,6 +8,7 @@
 #include <linux/init.h>
 #include <linux/slab.h>
 #include <linux/err.h>
+#include <linux/key.h>
 #include <linux/nvme-tcp.h>
 #include <net/sock.h>
 #include <net/tcp.h>
@@ -1763,7 +1764,7 @@ static int nvmet_tcp_tls_handshake(struct nvmet_tcp_queue *queue)
 	args.ta_sock = queue->sock;
 	args.ta_done = nvmet_tcp_tls_handshake_done;
 	args.ta_data = queue;
-	args.ta_keyring = nvme_keyring_id();
+	args.ta_keyring = key_serial(queue->port->nport->keyring);
 	args.ta_timeout_ms = tls_handshake_timeout * 2 * 1024;
 
 	ret = tls_server_hello_psk(&args, GFP_KERNEL);
-- 
2.35.3


^ permalink raw reply related	[flat|nested] 74+ messages in thread

* Re: [PATCH 01/18] nvme-keyring: register '.nvme' keyring and add CONFIG_NVME_TLS
  2023-03-29 13:59 ` [PATCH 01/18] nvme-keyring: register '.nvme' keyring and add CONFIG_NVME_TLS Hannes Reinecke
@ 2023-03-29 14:49   ` Sagi Grimberg
  2023-03-29 15:24     ` Hannes Reinecke
  2023-03-29 15:04   ` Sagi Grimberg
  2023-03-30  8:53   ` Daniel Wagner
  2 siblings, 1 reply; 74+ messages in thread
From: Sagi Grimberg @ 2023-03-29 14:49 UTC (permalink / raw)
  To: Hannes Reinecke, Christoph Hellwig
  Cc: Keith Busch, linux-nvme, Chuck Lever, kernel-tls-handshake


> Register a '.nvme' keyring to hold keys for TLS and DH-HMAC-CHAP and
> add a new config option NVME_TLS to enable support for NVMe-TCP/TLS.
> We need a separate keyring for NVMe as the configuration is done
> via individual commands (eg for configfs), and the usual per-session
> or per-process keyrings can't be used.
> 
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>   drivers/nvme/common/Kconfig   |  9 +++++++++
>   drivers/nvme/common/Makefile  |  1 +
>   drivers/nvme/common/keyring.c | 36 +++++++++++++++++++++++++++++++++++
>   drivers/nvme/host/core.c      | 19 +++++++++++++++---
>   include/linux/nvme-keyring.h  | 12 ++++++++++++
>   5 files changed, 74 insertions(+), 3 deletions(-)
>   create mode 100644 drivers/nvme/common/keyring.c
>   create mode 100644 include/linux/nvme-keyring.h
> 
> diff --git a/drivers/nvme/common/Kconfig b/drivers/nvme/common/Kconfig
> index 4514f44362dd..b6fff16da1fb 100644
> --- a/drivers/nvme/common/Kconfig
> +++ b/drivers/nvme/common/Kconfig
> @@ -2,3 +2,12 @@
>   
>   config NVME_COMMON
>          tristate
> +
> +config NVME_TLS
> +	bool "NVMe/TCP TLS encryption support"
> +	depends on NVME_COMMON

depends on TLS as well? Or maybe select would be more appropriate?

> +	select KEYS
> +	help
> +	  Enables TLS encryption for NVMe/TCP using the netlink handshake API.
> +
> +	  If unsure, say N.

Would it not make sense to default it to y?

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 01/18] nvme-keyring: register '.nvme' keyring and add CONFIG_NVME_TLS
  2023-03-29 13:59 ` [PATCH 01/18] nvme-keyring: register '.nvme' keyring and add CONFIG_NVME_TLS Hannes Reinecke
  2023-03-29 14:49   ` Sagi Grimberg
@ 2023-03-29 15:04   ` Sagi Grimberg
  2023-03-29 15:26     ` Hannes Reinecke
  2023-03-30  8:53   ` Daniel Wagner
  2 siblings, 1 reply; 74+ messages in thread
From: Sagi Grimberg @ 2023-03-29 15:04 UTC (permalink / raw)
  To: Hannes Reinecke, Christoph Hellwig
  Cc: Keith Busch, linux-nvme, Chuck Lever, kernel-tls-handshake


> Register a '.nvme' keyring to hold keys for TLS and DH-HMAC-CHAP and
> add a new config option NVME_TLS to enable support for NVMe-TCP/TLS.
> We need a separate keyring for NVMe as the configuration is done
> via individual commands (eg for configfs), and the usual per-session
> or per-process keyrings can't be used.
> 
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>   drivers/nvme/common/Kconfig   |  9 +++++++++
>   drivers/nvme/common/Makefile  |  1 +
>   drivers/nvme/common/keyring.c | 36 +++++++++++++++++++++++++++++++++++
>   drivers/nvme/host/core.c      | 19 +++++++++++++++---
>   include/linux/nvme-keyring.h  | 12 ++++++++++++
>   5 files changed, 74 insertions(+), 3 deletions(-)
>   create mode 100644 drivers/nvme/common/keyring.c
>   create mode 100644 include/linux/nvme-keyring.h
> 
> diff --git a/drivers/nvme/common/Kconfig b/drivers/nvme/common/Kconfig
> index 4514f44362dd..b6fff16da1fb 100644
> --- a/drivers/nvme/common/Kconfig
> +++ b/drivers/nvme/common/Kconfig
> @@ -2,3 +2,12 @@
>   
>   config NVME_COMMON
>          tristate
> +
> +config NVME_TLS

Better to call it NVME_TCP_TLS... although not a biggie, should be
well understood this is tcp

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 01/18] nvme-keyring: register '.nvme' keyring and add CONFIG_NVME_TLS
  2023-03-29 14:49   ` Sagi Grimberg
@ 2023-03-29 15:24     ` Hannes Reinecke
  0 siblings, 0 replies; 74+ messages in thread
From: Hannes Reinecke @ 2023-03-29 15:24 UTC (permalink / raw)
  To: Sagi Grimberg, Christoph Hellwig
  Cc: Keith Busch, linux-nvme, Chuck Lever, kernel-tls-handshake

On 3/29/23 16:49, Sagi Grimberg wrote:
> 
>> Register a '.nvme' keyring to hold keys for TLS and DH-HMAC-CHAP and
>> add a new config option NVME_TLS to enable support for NVMe-TCP/TLS.
>> We need a separate keyring for NVMe as the configuration is done
>> via individual commands (eg for configfs), and the usual per-session
>> or per-process keyrings can't be used.
>>
>> Signed-off-by: Hannes Reinecke <hare@suse.de>
>> ---
>>   drivers/nvme/common/Kconfig   |  9 +++++++++
>>   drivers/nvme/common/Makefile  |  1 +
>>   drivers/nvme/common/keyring.c | 36 +++++++++++++++++++++++++++++++++++
>>   drivers/nvme/host/core.c      | 19 +++++++++++++++---
>>   include/linux/nvme-keyring.h  | 12 ++++++++++++
>>   5 files changed, 74 insertions(+), 3 deletions(-)
>>   create mode 100644 drivers/nvme/common/keyring.c
>>   create mode 100644 include/linux/nvme-keyring.h
>>
>> diff --git a/drivers/nvme/common/Kconfig b/drivers/nvme/common/Kconfig
>> index 4514f44362dd..b6fff16da1fb 100644
>> --- a/drivers/nvme/common/Kconfig
>> +++ b/drivers/nvme/common/Kconfig
>> @@ -2,3 +2,12 @@
>>   config NVME_COMMON
>>          tristate
>> +
>> +config NVME_TLS
>> +    bool "NVMe/TCP TLS encryption support"
>> +    depends on NVME_COMMON
> 
> depends on TLS as well? Or maybe select would be more appropriate?
> 
I would argue 'depends', as we cannot use the standard 
session/process/thread keyrings, as they'll vanish after the initial 
'connect', and then there's no keyring to lookup keys for subsection 
reconnects.

>> +    select KEYS
>> +    help
>> +      Enables TLS encryption for NVMe/TCP using the netlink handshake 
>> API.
>> +
>> +      If unsure, say N.
> 
> Would it not make sense to default it to y?

Yeah.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman


^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 01/18] nvme-keyring: register '.nvme' keyring and add CONFIG_NVME_TLS
  2023-03-29 15:04   ` Sagi Grimberg
@ 2023-03-29 15:26     ` Hannes Reinecke
  0 siblings, 0 replies; 74+ messages in thread
From: Hannes Reinecke @ 2023-03-29 15:26 UTC (permalink / raw)
  To: Sagi Grimberg, Christoph Hellwig
  Cc: Keith Busch, linux-nvme, Chuck Lever, kernel-tls-handshake

On 3/29/23 17:04, Sagi Grimberg wrote:
> 
>> Register a '.nvme' keyring to hold keys for TLS and DH-HMAC-CHAP and
>> add a new config option NVME_TLS to enable support for NVMe-TCP/TLS.
>> We need a separate keyring for NVMe as the configuration is done
>> via individual commands (eg for configfs), and the usual per-session
>> or per-process keyrings can't be used.
>>
>> Signed-off-by: Hannes Reinecke <hare@suse.de>
>> ---
>>   drivers/nvme/common/Kconfig   |  9 +++++++++
>>   drivers/nvme/common/Makefile  |  1 +
>>   drivers/nvme/common/keyring.c | 36 +++++++++++++++++++++++++++++++++++
>>   drivers/nvme/host/core.c      | 19 +++++++++++++++---
>>   include/linux/nvme-keyring.h  | 12 ++++++++++++
>>   5 files changed, 74 insertions(+), 3 deletions(-)
>>   create mode 100644 drivers/nvme/common/keyring.c
>>   create mode 100644 include/linux/nvme-keyring.h
>>
>> diff --git a/drivers/nvme/common/Kconfig b/drivers/nvme/common/Kconfig
>> index 4514f44362dd..b6fff16da1fb 100644
>> --- a/drivers/nvme/common/Kconfig
>> +++ b/drivers/nvme/common/Kconfig
>> @@ -2,3 +2,12 @@
>>   config NVME_COMMON
>>          tristate
>> +
>> +config NVME_TLS
> 
> Better to call it NVME_TCP_TLS... although not a biggie, should be
> well understood this is tcp
> 
Sure.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman


^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 07/18] nvme-keyring: implement nvme_tls_psk_default()
  2023-03-29 13:59 ` [PATCH 07/18] nvme-keyring: implement nvme_tls_psk_default() Hannes Reinecke
@ 2023-03-29 15:35   ` Sagi Grimberg
  0 siblings, 0 replies; 74+ messages in thread
From: Sagi Grimberg @ 2023-03-29 15:35 UTC (permalink / raw)
  To: Hannes Reinecke, Christoph Hellwig
  Cc: Keith Busch, linux-nvme, Chuck Lever, kernel-tls-handshake



On 3/29/23 16:59, Hannes Reinecke wrote:
> Implement a function to select the 'best' PSK for TLS.
> 
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>   drivers/nvme/common/keyring.c | 47 +++++++++++++++++++++++++++++++++++
>   include/linux/nvme-keyring.h  |  2 ++
>   2 files changed, 49 insertions(+)
> 
> diff --git a/drivers/nvme/common/keyring.c b/drivers/nvme/common/keyring.c
> index 4ac33538f839..ca36a061bd48 100644
> --- a/drivers/nvme/common/keyring.c
> +++ b/drivers/nvme/common/keyring.c
> @@ -103,6 +103,53 @@ struct key *nvme_tls_psk_lookup(struct key *keyring,
>   }
>   EXPORT_SYMBOL_GPL(nvme_tls_psk_lookup);
>   
> +/*
> + * NVMe PSK priority list
> + *
> + * 'Retained' PSKs (ie 'generated == false')
> + * should be preferred to 'generated' PSKs,
> + * and SHA-384 should be preferred to SHA-256.
> + */
> +struct nvme_psk_priority_list {

nvme_tls_psk_priority_list

> +	bool generated;
> +	enum nvme_tcp_tls_cipher cipher;
> +} nvme_psk_prio[] = {

nvme_tls_psk_prio

> +	{ .generated = false,
> +	  .cipher = NVME_TCP_TLS_CIPHER_SHA384, },
> +	{ .generated = false,
> +	  .cipher = NVME_TCP_TLS_CIPHER_SHA256, },
> +	{ .generated = true,
> +	  .cipher = NVME_TCP_TLS_CIPHER_SHA384, },
> +	{ .generated = true,
> +	  .cipher = NVME_TCP_TLS_CIPHER_SHA256, },
> +};
> +
> +/*
> + * nvme_tls_psk_default - Return 'best' PSK to use for TLS ClientHello

s/'best'/the preferred/...

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 05/18] net/tls: implement ->read_sock()
  2023-03-29 13:59 ` [PATCH 05/18] net/tls: implement ->read_sock() Hannes Reinecke
@ 2023-03-29 15:37   ` Sagi Grimberg
  2023-03-29 15:41     ` Hannes Reinecke
  2023-03-29 15:44   ` Sagi Grimberg
  1 sibling, 1 reply; 74+ messages in thread
From: Sagi Grimberg @ 2023-03-29 15:37 UTC (permalink / raw)
  To: Hannes Reinecke, Christoph Hellwig
  Cc: Keith Busch, linux-nvme, Chuck Lever, kernel-tls-handshake,
	Boris Pismenny


> Implement ->read_sock() function for use with nvme-tcp.

Nice.

> Signed-off-by: Hannes Reinecke <hare@suse.de>
> Cc: Boris Pismenny <boris.pismenny@gmail.com>

Would be great to see a tls_device implementation as well.

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 05/18] net/tls: implement ->read_sock()
  2023-03-29 15:37   ` Sagi Grimberg
@ 2023-03-29 15:41     ` Hannes Reinecke
  2023-03-29 15:43       ` Sagi Grimberg
  0 siblings, 1 reply; 74+ messages in thread
From: Hannes Reinecke @ 2023-03-29 15:41 UTC (permalink / raw)
  To: Sagi Grimberg, Christoph Hellwig
  Cc: Keith Busch, linux-nvme, Chuck Lever, kernel-tls-handshake,
	Boris Pismenny

On 3/29/23 17:37, Sagi Grimberg wrote:
> 
>> Implement ->read_sock() function for use with nvme-tcp.
> 
> Nice.
> 
>> Signed-off-by: Hannes Reinecke <hare@suse.de>
>> Cc: Boris Pismenny <boris.pismenny@gmail.com>
> 
> Would be great to see a tls_device implementation as well.

Sheesh. But first we would need to update the tls_device implementation 
to TLS 1.3:

int tls_set_device_offload(struct sock *sk, struct tls_context *ctx)
[ .. ]
         crypto_info = &ctx->crypto_send.info;
         if (crypto_info->version != TLS_1_2_VERSION) {
                 rc = -EOPNOTSUPP;
                 goto release_netdev;
         }

which I'd rather defer to Boris or whoever maintains the hardware.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman


^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 05/18] net/tls: implement ->read_sock()
  2023-03-29 15:41     ` Hannes Reinecke
@ 2023-03-29 15:43       ` Sagi Grimberg
  0 siblings, 0 replies; 74+ messages in thread
From: Sagi Grimberg @ 2023-03-29 15:43 UTC (permalink / raw)
  To: Hannes Reinecke, Christoph Hellwig
  Cc: Keith Busch, linux-nvme, Chuck Lever, kernel-tls-handshake,
	Boris Pismenny


>>> Implement ->read_sock() function for use with nvme-tcp.
>>
>> Nice.
>>
>>> Signed-off-by: Hannes Reinecke <hare@suse.de>
>>> Cc: Boris Pismenny <boris.pismenny@gmail.com>
>>
>> Would be great to see a tls_device implementation as well.
> 
> Sheesh. But first we would need to update the tls_device implementation 
> to TLS 1.3:

;)

The comment was for the nvidia folks..

> int tls_set_device_offload(struct sock *sk, struct tls_context *ctx)
> [ .. ]
>          crypto_info = &ctx->crypto_send.info;
>          if (crypto_info->version != TLS_1_2_VERSION) {
>                  rc = -EOPNOTSUPP;
>                  goto release_netdev;
>          }
> 
> which I'd rather defer to Boris or whoever maintains the hardware

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 05/18] net/tls: implement ->read_sock()
  2023-03-29 13:59 ` [PATCH 05/18] net/tls: implement ->read_sock() Hannes Reinecke
  2023-03-29 15:37   ` Sagi Grimberg
@ 2023-03-29 15:44   ` Sagi Grimberg
  1 sibling, 0 replies; 74+ messages in thread
From: Sagi Grimberg @ 2023-03-29 15:44 UTC (permalink / raw)
  To: Hannes Reinecke, Christoph Hellwig
  Cc: Keith Busch, linux-nvme, Chuck Lever, kernel-tls-handshake,
	Boris Pismenny


> Implement ->read_sock() function for use with nvme-tcp.
> 
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> Cc: Boris Pismenny <boris.pismenny@gmail.com>
> ---
>   net/tls/tls.h      |  2 ++
>   net/tls/tls_main.c |  2 ++
>   net/tls/tls_sw.c   | 71 ++++++++++++++++++++++++++++++++++++++++++++++
>   3 files changed, 75 insertions(+)
> 
> diff --git a/net/tls/tls.h b/net/tls/tls.h
> index 804c3880d028..a5bf3a9ce142 100644
> --- a/net/tls/tls.h
> +++ b/net/tls/tls.h
> @@ -113,6 +113,8 @@ bool tls_sw_sock_is_readable(struct sock *sk);
>   ssize_t tls_sw_splice_read(struct socket *sock, loff_t *ppos,
>   			   struct pipe_inode_info *pipe,
>   			   size_t len, unsigned int flags);
> +int tls_sw_read_sock(struct sock *sk, read_descriptor_t *desc,
> +		     sk_read_actor_t read_actor);
>   
>   int tls_device_sendmsg(struct sock *sk, struct msghdr *msg, size_t size);
>   int tls_device_sendpage(struct sock *sk, struct page *page,
> diff --git a/net/tls/tls_main.c b/net/tls/tls_main.c
> index 3735cb00905d..9b097f2902f6 100644
> --- a/net/tls/tls_main.c
> +++ b/net/tls/tls_main.c
> @@ -934,9 +934,11 @@ static void build_proto_ops(struct proto_ops ops[TLS_NUM_CONFIG][TLS_NUM_CONFIG]
>   
>   	ops[TLS_BASE][TLS_SW  ] = ops[TLS_BASE][TLS_BASE];
>   	ops[TLS_BASE][TLS_SW  ].splice_read	= tls_sw_splice_read;
> +	ops[TLS_BASE][TLS_SW  ].read_sock	= tls_sw_read_sock;
>   
>   	ops[TLS_SW  ][TLS_SW  ] = ops[TLS_SW  ][TLS_BASE];
>   	ops[TLS_SW  ][TLS_SW  ].splice_read	= tls_sw_splice_read;
> +	ops[TLS_SW  ][TLS_SW  ].read_sock	= tls_sw_read_sock;
>   
>   #ifdef CONFIG_TLS_DEVICE
>   	ops[TLS_HW  ][TLS_BASE] = ops[TLS_BASE][TLS_BASE];
> diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c
> index 782d3701b86f..77d57066daf4 100644
> --- a/net/tls/tls_sw.c
> +++ b/net/tls/tls_sw.c
> @@ -2210,6 +2210,77 @@ ssize_t tls_sw_splice_read(struct socket *sock,  loff_t *ppos,
>   	goto splice_read_end;
>   }
>   
> +int tls_sw_read_sock(struct sock *sk, read_descriptor_t *desc,
> +		     sk_read_actor_t read_actor)
> +{
> +	struct tls_context *tls_ctx = tls_get_ctx(sk);
> +	struct tls_sw_context_rx *ctx = tls_sw_ctx_rx(tls_ctx);
> +	struct strp_msg *rxm = NULL;
> +	struct tls_msg *tlm;
> +	struct sk_buff *skb;
> +	ssize_t copied = 0;
> +	int err, used;
> +
> +	if (!skb_queue_empty(&ctx->rx_list)) {
> +		skb = __skb_dequeue(&ctx->rx_list);
> +	} else {
> +		struct tls_decrypt_arg darg;
> +
> +		err = tls_rx_rec_wait(sk, NULL, true, true);
> +		if (err <= 0)
> +			return err;
> +
> +		memset(&darg.inargs, 0, sizeof(darg.inargs));
> +
> +		err = tls_rx_one_record(sk, NULL, &darg);
> +		if (err < 0) {
> +			tls_err_abort(sk, -EBADMSG);
> +			return err;
> +		}
> +
> +		tls_rx_rec_done(ctx);
> +		skb = darg.skb;
> +	}
> +
> +	do {
> +		rxm = strp_msg(skb);
> +		tlm = tls_msg(skb);
> +
> +		/* read_sock does not support reading control messages */
> +		if (tlm->control != TLS_RECORD_TYPE_DATA) {
> +			err = -EINVAL;
> +			goto read_sock_requeue;

In the general case, this could just pass in the skb to recv_actor
and have it access the tlm and decide what to do with it...
(will require moving tls_msg() to include/net/strparser.h)

I agree that for nvme-tcp this would not have anything useful
other than breaking the connection and re-establish it later...

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 06/18] nvme/tcp: allocate socket file
  2023-03-29 13:59 ` [PATCH 06/18] nvme/tcp: allocate socket file Hannes Reinecke
@ 2023-03-29 15:57   ` Sagi Grimberg
  0 siblings, 0 replies; 74+ messages in thread
From: Sagi Grimberg @ 2023-03-29 15:57 UTC (permalink / raw)
  To: Hannes Reinecke, Christoph Hellwig
  Cc: Keith Busch, linux-nvme, Chuck Lever, kernel-tls-handshake

> When using the TLS upcall we need to allocate a socket file such
> that the userspace daemon is able to use the socket.
> 
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>   drivers/nvme/host/tcp.c | 17 +++++++++++++++--
>   1 file changed, 15 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
> index 42c0598c31f2..fddf785aba74 100644
> --- a/drivers/nvme/host/tcp.c
> +++ b/drivers/nvme/host/tcp.c
> @@ -115,6 +115,7 @@ enum nvme_tcp_recv_state {
>   struct nvme_tcp_ctrl;
>   struct nvme_tcp_queue {
>   	struct socket		*sock;
> +	struct file		*sock_file;

I think that sock->file is storing the file, no need to store it as
well.

>   	struct work_struct	io_work;
>   	int			io_cpu;
>   
> @@ -1330,7 +1331,10 @@ static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid)
>   	}
>   
>   	noreclaim_flag = memalloc_noreclaim_save();
> -	sock_release(queue->sock);
> +	/* ->sock will be released by fput() */
> +	fput(queue->sock_file);
> +	queue->sock_file = NULL;
> +	queue->sock = NULL;

Is the queue->sock = NULL here required because we allocate a sock_file?

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 01/18] nvme-keyring: register '.nvme' keyring and add CONFIG_NVME_TLS
  2023-03-29 13:59 ` [PATCH 01/18] nvme-keyring: register '.nvme' keyring and add CONFIG_NVME_TLS Hannes Reinecke
  2023-03-29 14:49   ` Sagi Grimberg
  2023-03-29 15:04   ` Sagi Grimberg
@ 2023-03-30  8:53   ` Daniel Wagner
  2023-03-30 14:38     ` Sagi Grimberg
  2 siblings, 1 reply; 74+ messages in thread
From: Daniel Wagner @ 2023-03-30  8:53 UTC (permalink / raw)
  To: Hannes Reinecke
  Cc: Christoph Hellwig, Sagi Grimberg, Keith Busch, linux-nvme,
	Chuck Lever, kernel-tls-handshake

On Wed, Mar 29, 2023 at 03:59:21PM +0200, Hannes Reinecke wrote:
> +++ b/drivers/nvme/host/core.c
> @@ -25,6 +25,9 @@
>  #include "nvme.h"
>  #include "fabrics.h"
>  #include <linux/nvme-auth.h>
> +#ifdef CONFIG_NVME_TLS
> +#include <linux/nvme-keyring.h>
> +#endif
>  
>  #define CREATE_TRACE_POINTS
>  #include "trace.h"
> @@ -3954,7 +3957,6 @@ static umode_t nvme_dev_attrs_are_visible(struct kobject *kobj,
>  	if (a == &dev_attr_dhchap_ctrl_secret.attr && !ctrl->opts)
>  		return 0;
>  #endif
> -
>  	return a->mode;
>  }
>  
> @@ -5414,13 +5416,21 @@ static int __init nvme_core_init(void)
>  		result = PTR_ERR(nvme_ns_chr_class);
>  		goto unregister_generic_ns;
>  	}
> -
> -	result = nvme_init_auth();
> +#ifdef CONFIG_NVME_TLS
> +	result = nvme_keyring_init();
>  	if (result)
>  		goto destroy_ns_chr;
> +#endif

Would it make sense to introduce empty nvme_keyring_init() function for
!CONFIG_NVME_TLS and avoid this ifdefery here?

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 09/18] nvme-tcp: enable TLS handshake upcall
  2023-03-29 13:59 ` [PATCH 09/18] nvme-tcp: enable TLS handshake upcall Hannes Reinecke
@ 2023-03-30 12:54   ` Daniel Wagner
  2023-03-30 12:59     ` Hannes Reinecke
  2023-03-30 15:03   ` Sagi Grimberg
  1 sibling, 1 reply; 74+ messages in thread
From: Daniel Wagner @ 2023-03-30 12:54 UTC (permalink / raw)
  To: Hannes Reinecke
  Cc: Christoph Hellwig, Sagi Grimberg, Keith Busch, linux-nvme,
	Chuck Lever, kernel-tls-handshake

On Wed, Mar 29, 2023 at 03:59:29PM +0200, Hannes Reinecke wrote:
 --- a/drivers/nvme/host/fabrics.c
> +++ b/drivers/nvme/host/fabrics.c
> @@ -609,6 +609,7 @@ static const match_table_t opt_tokens = {
>  	{ NVMF_OPT_DISCOVERY,		"discovery"		},
>  	{ NVMF_OPT_DHCHAP_SECRET,	"dhchap_secret=%s"	},
>  	{ NVMF_OPT_DHCHAP_CTRL_SECRET,	"dhchap_ctrl_secret=%s"	},
> +	{ NVMF_OPT_TLS,			"tls"			},
>  	{ NVMF_OPT_ERR,			NULL			}
>  };

This should be ifdef out for !CONFIG_NVME_TLS

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 09/18] nvme-tcp: enable TLS handshake upcall
  2023-03-30 12:54   ` Daniel Wagner
@ 2023-03-30 12:59     ` Hannes Reinecke
  0 siblings, 0 replies; 74+ messages in thread
From: Hannes Reinecke @ 2023-03-30 12:59 UTC (permalink / raw)
  To: Daniel Wagner
  Cc: Christoph Hellwig, Sagi Grimberg, Keith Busch, linux-nvme,
	Chuck Lever, kernel-tls-handshake

On 3/30/23 14:54, Daniel Wagner wrote:
> On Wed, Mar 29, 2023 at 03:59:29PM +0200, Hannes Reinecke wrote:
>   --- a/drivers/nvme/host/fabrics.c
>> +++ b/drivers/nvme/host/fabrics.c
>> @@ -609,6 +609,7 @@ static const match_table_t opt_tokens = {
>>   	{ NVMF_OPT_DISCOVERY,		"discovery"		},
>>   	{ NVMF_OPT_DHCHAP_SECRET,	"dhchap_secret=%s"	},
>>   	{ NVMF_OPT_DHCHAP_CTRL_SECRET,	"dhchap_ctrl_secret=%s"	},
>> +	{ NVMF_OPT_TLS,			"tls"			},
>>   	{ NVMF_OPT_ERR,			NULL			}
>>   };
> 
> This should be ifdef out for !CONFIG_NVME_TLS

No, I did that on purpose.
Error handling in the fabrics option parsing is notoriously bad, and if 
we disable that nvme-cli will fail with annoying interfaces.
As it stands, the 'tls' option will be accepted, but you'll get a kernel 
message telling you it's not compiled in, and the function returns -EINVAL.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		           Kernel Storage Architect
hare@suse.de			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Frankenstr. 146, 90461 Nürnberg
Managing Directors: I. Totev, A. Myers, A. McDonald, M. B. Moerman
(HRB 36809, AG Nürnberg)


^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 01/18] nvme-keyring: register '.nvme' keyring and add CONFIG_NVME_TLS
  2023-03-30  8:53   ` Daniel Wagner
@ 2023-03-30 14:38     ` Sagi Grimberg
  0 siblings, 0 replies; 74+ messages in thread
From: Sagi Grimberg @ 2023-03-30 14:38 UTC (permalink / raw)
  To: Daniel Wagner, Hannes Reinecke
  Cc: Christoph Hellwig, Keith Busch, linux-nvme, Chuck Lever,
	kernel-tls-handshake



On 3/30/23 11:53, Daniel Wagner wrote:
> On Wed, Mar 29, 2023 at 03:59:21PM +0200, Hannes Reinecke wrote:
>> +++ b/drivers/nvme/host/core.c
>> @@ -25,6 +25,9 @@
>>   #include "nvme.h"
>>   #include "fabrics.h"
>>   #include <linux/nvme-auth.h>
>> +#ifdef CONFIG_NVME_TLS
>> +#include <linux/nvme-keyring.h>
>> +#endif
>>   
>>   #define CREATE_TRACE_POINTS
>>   #include "trace.h"
>> @@ -3954,7 +3957,6 @@ static umode_t nvme_dev_attrs_are_visible(struct kobject *kobj,
>>   	if (a == &dev_attr_dhchap_ctrl_secret.attr && !ctrl->opts)
>>   		return 0;
>>   #endif
>> -
>>   	return a->mode;
>>   }
>>   
>> @@ -5414,13 +5416,21 @@ static int __init nvme_core_init(void)
>>   		result = PTR_ERR(nvme_ns_chr_class);
>>   		goto unregister_generic_ns;
>>   	}
>> -
>> -	result = nvme_init_auth();
>> +#ifdef CONFIG_NVME_TLS
>> +	result = nvme_keyring_init();
>>   	if (result)
>>   		goto destroy_ns_chr;
>> +#endif
> 
> Would it make sense to introduce empty nvme_keyring_init() function for
> !CONFIG_NVME_TLS and avoid this ifdefery here?

Yes.

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 09/18] nvme-tcp: enable TLS handshake upcall
  2023-03-29 13:59 ` [PATCH 09/18] nvme-tcp: enable TLS handshake upcall Hannes Reinecke
  2023-03-30 12:54   ` Daniel Wagner
@ 2023-03-30 15:03   ` Sagi Grimberg
  2023-03-30 17:16     ` Hannes Reinecke
  1 sibling, 1 reply; 74+ messages in thread
From: Sagi Grimberg @ 2023-03-30 15:03 UTC (permalink / raw)
  To: Hannes Reinecke, Christoph Hellwig
  Cc: Keith Busch, linux-nvme, Chuck Lever, kernel-tls-handshake



On 3/29/23 16:59, Hannes Reinecke wrote:
> Add a fabrics option 'tls' and start the TLS handshake upcall
> with the default PSK. When TLS is started the PSK key serial
> number is displayed in the sysfs attribute 'tls_key'
> 
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>   drivers/nvme/host/core.c    |  25 ++++++-
>   drivers/nvme/host/fabrics.c |  11 +++
>   drivers/nvme/host/fabrics.h |   3 +
>   drivers/nvme/host/nvme.h    |   4 +-
>   drivers/nvme/host/tcp.c     | 141 ++++++++++++++++++++++++++++++++++--
>   5 files changed, 174 insertions(+), 10 deletions(-)
> 
> diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
> index 416d0a898f56..869396058e2e 100644
> --- a/drivers/nvme/host/core.c
> +++ b/drivers/nvme/host/core.c
> @@ -3901,6 +3901,19 @@ static DEVICE_ATTR(dhchap_ctrl_secret, S_IRUGO | S_IWUSR,
>   	nvme_ctrl_dhchap_ctrl_secret_show, nvme_ctrl_dhchap_ctrl_secret_store);
>   #endif
>   
> +#ifdef CONFIG_NVME_TLS
> +static ssize_t tls_key_show(struct device *dev,
> +			    struct device_attribute *attr, char *buf)
> +{
> +	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
> +
> +	if (!ctrl->tls_key)
> +		return 0;
> +	return sysfs_emit(buf, "%08x", key_serial(ctrl->tls_key));
> +}
> +static DEVICE_ATTR_RO(tls_key);
> +#endif
> +
>   static struct attribute *nvme_dev_attrs[] = {
>   	&dev_attr_reset_controller.attr,
>   	&dev_attr_rescan_controller.attr,
> @@ -3927,6 +3940,9 @@ static struct attribute *nvme_dev_attrs[] = {
>   #ifdef CONFIG_NVME_AUTH
>   	&dev_attr_dhchap_secret.attr,
>   	&dev_attr_dhchap_ctrl_secret.attr,
> +#endif
> +#ifdef CONFIG_NVME_TLS
> +	&dev_attr_tls_key.attr,
>   #endif
>   	NULL
>   };
> @@ -3956,6 +3972,10 @@ static umode_t nvme_dev_attrs_are_visible(struct kobject *kobj,
>   		return 0;
>   	if (a == &dev_attr_dhchap_ctrl_secret.attr && !ctrl->opts)
>   		return 0;
> +#endif
> +#ifdef CONFIG_NVME_TLS
> +	if (a == &dev_attr_tls_key.attr && !ctrl->opts)
> +		return 0;
>   #endif
>   	return a->mode;
>   }
> @@ -5094,7 +5114,10 @@ static void nvme_free_ctrl(struct device *dev)
>   
>   	if (!subsys || ctrl->instance != subsys->instance)
>   		ida_free(&nvme_instance_ida, ctrl->instance);
> -
> +#ifdef CONFIG_NVME_TLS
> +	if (ctrl->tls_key)
> +		key_put(ctrl->tls_key);
> +#endif
>   	nvme_free_cels(ctrl);
>   	nvme_mpath_uninit(ctrl);
>   	nvme_auth_stop(ctrl);
> diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
> index bbaa04a0c502..3e4f0e45b58f 100644
> --- a/drivers/nvme/host/fabrics.c
> +++ b/drivers/nvme/host/fabrics.c
> @@ -609,6 +609,7 @@ static const match_table_t opt_tokens = {
>   	{ NVMF_OPT_DISCOVERY,		"discovery"		},
>   	{ NVMF_OPT_DHCHAP_SECRET,	"dhchap_secret=%s"	},
>   	{ NVMF_OPT_DHCHAP_CTRL_SECRET,	"dhchap_ctrl_secret=%s"	},
> +	{ NVMF_OPT_TLS,			"tls"			},
>   	{ NVMF_OPT_ERR,			NULL			}
>   };
>   
> @@ -632,6 +633,7 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
>   	opts->hdr_digest = false;
>   	opts->data_digest = false;
>   	opts->tos = -1; /* < 0 == use transport default */
> +	opts->tls = false;
>   
>   	options = o = kstrdup(buf, GFP_KERNEL);
>   	if (!options)
> @@ -918,6 +920,15 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
>   			kfree(opts->dhchap_ctrl_secret);
>   			opts->dhchap_ctrl_secret = p;
>   			break;
> +		case NVMF_OPT_TLS:
> +#ifdef CONFIG_NVME_TLS
> +			opts->tls = true;
> +			break;
> +#else
> +			pr_err("TLS is not supported\n");
> +			ret = -EINVAL;
> +			goto out;
> +#endif

This can become:
			if (!IS_ENABLED(CONFIG_NVME_TLS)) {
				pr_err("TLS is not supported\n");
				ret = -EINVAL;
				goto out;
			}
			opts->tls = true;
			break;

>   		default:
>   			pr_warn("unknown parameter or missing value '%s' in ctrl creation request\n",
>   				p);
> diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h
> index dcac3df8a5f7..5db36e250e7a 100644
> --- a/drivers/nvme/host/fabrics.h
> +++ b/drivers/nvme/host/fabrics.h
> @@ -70,6 +70,7 @@ enum {
>   	NVMF_OPT_DISCOVERY	= 1 << 22,
>   	NVMF_OPT_DHCHAP_SECRET	= 1 << 23,
>   	NVMF_OPT_DHCHAP_CTRL_SECRET = 1 << 24,
> +	NVMF_OPT_TLS		= 1 << 25,
>   };
>   
>   /**
> @@ -102,6 +103,7 @@ enum {
>    * @dhchap_secret: DH-HMAC-CHAP secret
>    * @dhchap_ctrl_secret: DH-HMAC-CHAP controller secret for bi-directional
>    *              authentication
> + * @tls:        Start TLS encrypted connections (TCP)
>    * @disable_sqflow: disable controller sq flow control
>    * @hdr_digest: generate/verify header digest (TCP)
>    * @data_digest: generate/verify data digest (TCP)
> @@ -128,6 +130,7 @@ struct nvmf_ctrl_options {
>   	int			max_reconnects;
>   	char			*dhchap_secret;
>   	char			*dhchap_ctrl_secret;
> +	bool			tls;
>   	bool			disable_sqflow;
>   	bool			hdr_digest;
>   	bool			data_digest;
> diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
> index bf46f122e9e1..421d01f849d1 100644
> --- a/drivers/nvme/host/nvme.h
> +++ b/drivers/nvme/host/nvme.h
> @@ -347,7 +347,9 @@ struct nvme_ctrl {
>   	struct nvme_dhchap_key *ctrl_key;
>   	u16 transaction;
>   #endif
> -
> +#ifdef CONFIG_NVME_TLS
> +	struct key *tls_key;
> +#endif

I'd add a comment that this is nvme_tcp specific.

>   	/* Power saving configuration */
>   	u64 ps_max_latency_us;
>   	bool apst_enabled;
> diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
> index fddf785aba74..fdb564d0b9f4 100644
> --- a/drivers/nvme/host/tcp.c
> +++ b/drivers/nvme/host/tcp.c
> @@ -8,9 +8,15 @@
>   #include <linux/init.h>
>   #include <linux/slab.h>
>   #include <linux/err.h>
> +#include <linux/key.h>
>   #include <linux/nvme-tcp.h>
>   #include <net/sock.h>
>   #include <net/tcp.h>
> +#ifdef CONFIG_NVME_TLS
> +#include <linux/nvme-keyring.h>
> +#include <net/tls.h>
> +#include <net/handshake.h>
> +#endif
>   #include <linux/blk-mq.h>
>   #include <crypto/hash.h>
>   #include <net/busy_poll.h>
> @@ -31,6 +37,16 @@ static int so_priority;
>   module_param(so_priority, int, 0644);
>   MODULE_PARM_DESC(so_priority, "nvme tcp socket optimize priority");
>   
> +#ifdef CONFIG_NVME_TLS
> +/*
> + * TLS handshake timeout
> + */
> +static int tls_handshake_timeout = 10;
> +module_param(tls_handshake_timeout, int, 0644);
> +MODULE_PARM_DESC(tls_handshake_timeout,
> +		 "nvme TLS handshake timeout in seconds (default 10)");
> +#endif
> +
>   #ifdef CONFIG_DEBUG_LOCK_ALLOC
>   /* lockdep can detect a circular dependency of the form
>    *   sk_lock -> mmap_lock (page fault) -> fs locks -> sk_lock
> @@ -147,7 +163,10 @@ struct nvme_tcp_queue {
>   	struct ahash_request	*snd_hash;
>   	__le32			exp_ddgst;
>   	__le32			recv_ddgst;
> -
> +#ifdef CONFIG_NVME_TLS
> +	struct completion       *tls_complete;

Why is this a pointer?

> +	int                     tls_err;
> +#endif
>   	struct page_frag_cache	pf_cache;
>   
>   	void (*state_change)(struct sock *);
> @@ -1503,7 +1522,80 @@ static void nvme_tcp_set_queue_io_cpu(struct nvme_tcp_queue *queue)
>   	queue->io_cpu = cpumask_next_wrap(n - 1, cpu_online_mask, -1, false);
>   }
>   
> -static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid)
> +#ifdef CONFIG_NVME_TLS
> +static void nvme_tcp_tls_done(void *data, int status, key_serial_t peerid)
> +{
> +	struct nvme_tcp_queue *queue = data;
> +	struct nvme_tcp_ctrl *ctrl = queue->ctrl;
> +	int qid = nvme_tcp_queue_id(queue);
> +
> +	dev_dbg(ctrl->ctrl.device, "queue %d: TLS handshake done, key %x, status %d\n",
> +		qid, peerid, status);

Can we avoid naming the key peerid? pskid perhaps.

> +
> +	if (status)
> +		queue->tls_err = -status;

Maybe just return or goto err, instead of the else case?

> +	else {
> +		struct key *tls_key = key_lookup(peerid);
> +		if (IS_ERR(tls_key)) {
> +			dev_warn(ctrl->ctrl.device, "queue %d: Invalid key %x\n",
> +				 qid, peerid);
> +			queue->tls_err = -ENOKEY;
> +		} else {
> +			ctrl->ctrl.tls_key = tls_key;
> +			queue->tls_err = 0;
> +		}
> +	}
> +
> +	if (queue->tls_complete)
> +		complete(queue->tls_complete);
> +}
> +
> +static int nvme_tcp_start_tls(struct nvme_ctrl *nctrl,
> +			      struct nvme_tcp_queue *queue,
> +			      key_serial_t peerid)
> +{
> +	int qid = nvme_tcp_queue_id(queue);
> +	int ret;
> +	struct tls_handshake_args args;
> +	unsigned long tmo = tls_handshake_timeout * HZ;
> +	DECLARE_COMPLETION_ONSTACK(tls_complete);

I think we can have this completion a queue member.

> +	key_serial_t keyring = nvme_keyring_id();
> +
> +	dev_dbg(nctrl->device, "queue %d: start TLS with key %x\n",
> +		qid, peerid);
> +	args.ta_sock = queue->sock;
> +	args.ta_done = nvme_tcp_tls_done;
> +	args.ta_data = queue;
> +	args.ta_my_peerids[0] = peerid;
> +	args.ta_num_peerids = 1;
> +	args.ta_keyring = keyring;
> +	args.ta_timeout_ms = tls_handshake_timeout * 2 * 1000;

remove the 2x.

> +	queue->tls_err = -EOPNOTSUPP;
> +	queue->tls_complete = &tls_complete;
> +	ret = tls_client_hello_psk(&args, GFP_KERNEL);
> +	if (ret) {
> +		dev_dbg(nctrl->device, "queue %d: failed to start TLS: %d\n",
> +			qid, ret);

dev_err

> +		return ret;
> +	}
> +	if (wait_for_completion_timeout(queue->tls_complete, tmo) == 0) {
> +		dev_dbg(nctrl->device,
> +			"queue %d: TLS handshake timeout\n", qid);

dev_err

> +		queue->tls_complete = NULL;
> +		ret = -ETIMEDOUT;
> +	} else {
> +		dev_dbg(nctrl->device,
> +			"queue %d: TLS handshake complete, error %d\n",
> +			qid, queue->tls_err);
> +		ret = queue->tls_err;
> +	}
> +	queue->tls_complete = NULL;

as mentioned, lets make the completion a queue member and not a pointer
that we need to pay attention to.

> +	return ret;
> +}
> +#endif
> +
> +static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid,
> +				key_serial_t peerid)
>   {
>   	struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(nctrl);
>   	struct nvme_tcp_queue *queue = &ctrl->queues[qid];
> @@ -1627,6 +1719,14 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid)
>   		goto err_rcv_pdu;
>   	}
>   
> +#ifdef CONFIG_NVME_TLS
> +	/* If PSKs are configured try to start TLS */
> +	if (peerid) {
> +		ret = nvme_tcp_start_tls(nctrl, queue, peerid);
> +		if (ret)
> +			goto err_init_connect;
> +	}
> +#endif
>   	ret = nvme_tcp_init_connection(queue);
>   	if (ret)
>   		goto err_init_connect;
> @@ -1771,10 +1871,23 @@ static int nvme_tcp_start_io_queues(struct nvme_ctrl *ctrl,
>   static int nvme_tcp_alloc_admin_queue(struct nvme_ctrl *ctrl)
>   {
>   	int ret;
> -
> -	ret = nvme_tcp_alloc_queue(ctrl, 0);
> +	key_serial_t psk_id = 0;
> +
> +#ifdef CONFIG_NVME_TLS
> +	if (ctrl->opts->tls) {
> +		psk_id = nvme_tls_psk_default(NULL,
> +					      ctrl->opts->host->nqn,
> +					      ctrl->opts->subsysnqn);
> +		if (!psk_id) {
> +			dev_err(ctrl->device, "no valid PSK found\n");
> +			ret = -ENOKEY;
> +			goto out_free_queue;
> +		}
> +	}
> +#endif

ctrl->opts->tls == true guarantes that CONFIG_NVME_TLS is enabled.
No need for it, as long as you give a nvme_tls_psk_default stub when
it is not.

> +	ret = nvme_tcp_alloc_queue(ctrl, 0, psk_id);
>   	if (ret)
> -		return ret;
> +		goto out_free_queue;
>   
>   	ret = nvme_tcp_alloc_async_req(to_tcp_ctrl(ctrl));
>   	if (ret)
> @@ -1790,9 +1903,21 @@ static int nvme_tcp_alloc_admin_queue(struct nvme_ctrl *ctrl)
>   static int __nvme_tcp_alloc_io_queues(struct nvme_ctrl *ctrl)
>   {
>   	int i, ret;
> -
> +	key_serial_t psk_id = 0;
> +
> +#ifdef CONFIG_NVME_TLS
> +	if (ctrl->opts->tls) {
> +		psk_id = nvme_tls_psk_default(NULL,
> +					      ctrl->opts->host->nqn,
> +					      ctrl->opts->subsysnqn);
> +		if (!psk_id) {
> +			dev_err(ctrl->device, "no valid PSK found\n");
> +			return -ENOKEY;
> +		}
> +	}
> +#endif

Do we need this for io queues? why not just store it on nvme_tcp_ctrl
and just use it?

>   	for (i = 1; i < ctrl->queue_count; i++) {
> -		ret = nvme_tcp_alloc_queue(ctrl, i);
> +		ret = nvme_tcp_alloc_queue(ctrl, i, psk_id);
>   		if (ret)
>   			goto out_free_queues;
>   	}
> @@ -2701,7 +2826,7 @@ static struct nvmf_transport_ops nvme_tcp_transport = {
>   			  NVMF_OPT_HOST_TRADDR | NVMF_OPT_CTRL_LOSS_TMO |
>   			  NVMF_OPT_HDR_DIGEST | NVMF_OPT_DATA_DIGEST |
>   			  NVMF_OPT_NR_WRITE_QUEUES | NVMF_OPT_NR_POLL_QUEUES |
> -			  NVMF_OPT_TOS | NVMF_OPT_HOST_IFACE,
> +			  NVMF_OPT_TOS | NVMF_OPT_HOST_IFACE | NVMF_OPT_TLS,
>   	.create_ctrl	= nvme_tcp_create_ctrl,
>   };
>   


Overall this looks much better Hannes!

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 10/18] nvme-tcp: fixup send workflow for kTLS
  2023-03-29 13:59 ` [PATCH 10/18] nvme-tcp: fixup send workflow for kTLS Hannes Reinecke
@ 2023-03-30 15:24   ` Sagi Grimberg
  2023-03-30 17:26     ` Hannes Reinecke
  2023-03-31  5:49     ` Jakub Kicinski
  0 siblings, 2 replies; 74+ messages in thread
From: Sagi Grimberg @ 2023-03-30 15:24 UTC (permalink / raw)
  To: Hannes Reinecke, Christoph Hellwig, Boris Pismenny,
	john.fastabend, kuba, Paolo Abeni
  Cc: Keith Busch, linux-nvme, Chuck Lever, kernel-tls-handshake, netdev


> kTLS does not support MSG_EOR flag for sendmsg(), and in general
> is really picky about invalid MSG_XXX flags.

CC'ing TLS folks.

Can't tls simply ignore MSG_EOR instead of consumers having to be 
careful over it?

> So ensure that the MSG_EOR flags is blanked out for TLS, and that
> the MSG_SENDPAGE_LAST is only set if we actually do sendpage().

You mean MSG_SENDPAGE_NOTLAST.

It is also a bit annoying that a tls socket dictates different behavior
than a normal socket.

The current logic is rather simple:
if more data comming:
	flags = MSG_MORE | MSG_SENDPAGE_NOTLAST
else:
	flags = MSG_EOR

Would like to keep it that way for tls as well. Can someone
explain why this is a problem with tls?

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 11/18] nvme-tcp: control message handling for recvmsg()
  2023-03-29 13:59 ` [PATCH 11/18] nvme-tcp: control message handling for recvmsg() Hannes Reinecke
@ 2023-03-30 15:25   ` Sagi Grimberg
  0 siblings, 0 replies; 74+ messages in thread
From: Sagi Grimberg @ 2023-03-30 15:25 UTC (permalink / raw)
  To: Hannes Reinecke, Christoph Hellwig
  Cc: Keith Busch, linux-nvme, Chuck Lever, kernel-tls-handshake

This looks fine to me,

Reviewed-by: Sagi Grimberg <sagi@grimberg.me>

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 12/18] nvme-fabrics: parse options 'keyring' and 'tls_key'
  2023-03-29 13:59 ` [PATCH 12/18] nvme-fabrics: parse options 'keyring' and 'tls_key' Hannes Reinecke
@ 2023-03-30 15:33   ` Sagi Grimberg
  2023-03-30 17:34     ` Hannes Reinecke
  0 siblings, 1 reply; 74+ messages in thread
From: Sagi Grimberg @ 2023-03-30 15:33 UTC (permalink / raw)
  To: Hannes Reinecke, Christoph Hellwig
  Cc: Keith Busch, linux-nvme, Chuck Lever, kernel-tls-handshake

> Parse the fabrics options 'keyring' and 'tls_key' and store the
> referenced keys in the options structure.

Can you explain the reasoning to why a user need to pass a keyring
given that we already set up one?

> 
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>   drivers/nvme/host/fabrics.c | 79 ++++++++++++++++++++++++++++++++++++-
>   drivers/nvme/host/fabrics.h |  6 +++
>   drivers/nvme/host/tcp.c     | 20 +++++++---
>   3 files changed, 98 insertions(+), 7 deletions(-)
> 
> diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
> index 3e4f0e45b58f..5f5e487d498c 100644
> --- a/drivers/nvme/host/fabrics.c
> +++ b/drivers/nvme/host/fabrics.c
> @@ -605,6 +605,8 @@ static const match_table_t opt_tokens = {
>   	{ NVMF_OPT_NR_WRITE_QUEUES,	"nr_write_queues=%d"	},
>   	{ NVMF_OPT_NR_POLL_QUEUES,	"nr_poll_queues=%d"	},
>   	{ NVMF_OPT_TOS,			"tos=%d"		},
> +	{ NVMF_OPT_KEYRING,		"keyring=%d"		},
> +	{ NVMF_OPT_TLS_KEY,		"tls_key=%d"		},
>   	{ NVMF_OPT_FAIL_FAST_TMO,	"fast_io_fail_tmo=%d"	},
>   	{ NVMF_OPT_DISCOVERY,		"discovery"		},
>   	{ NVMF_OPT_DHCHAP_SECRET,	"dhchap_secret=%s"	},
> @@ -620,8 +622,9 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
>   	char *options, *o, *p;
>   	int token, ret = 0;
>   	size_t nqnlen  = 0;
> -	int ctrl_loss_tmo = NVMF_DEF_CTRL_LOSS_TMO;
> +	int ctrl_loss_tmo = NVMF_DEF_CTRL_LOSS_TMO, key_id;
>   	uuid_t hostid;
> +	struct key *key = NULL;
>   
>   	/* Set defaults */
>   	opts->queue_size = NVMF_DEF_QUEUE_SIZE;
> @@ -889,6 +892,74 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
>   			}
>   			opts->tos = token;
>   			break;
> +		case NVMF_OPT_KEYRING:
> +#ifdef CONFIG_NVME_TLS

Same comment as before:
			if (!IS_ENABLED(CONFIG_NVME_TLS)) {
				pr_err("TLS is not supported\n");
				ret = -EINVAL;
				goto out;
			}

> +			if (match_int(args, &key_id)) {
> +				ret = -EINVAL;
> +				goto out;
> +			}
> +			if (key_id < 0) {
> +				pr_err("Invalid keyring id %d\n", key_id);
> +				ret = -EINVAL;
> +				goto out;
> +			}
> +			if (!key_id) {
> +				pr_debug("Using default keyring\n");
> +				if (opts->keyring) {
> +					key_put(opts->keyring);
> +					opts->keyring = NULL;
> +				}
> +				break;
> +			}
> +			key = key_lookup(key_id);
> +			if (!key) {
> +				pr_err("Keyring id %08x not found\n", key_id);
> +				ret = -ENOKEY;
> +				goto out;
> +			}
> +			if (opts->keyring)
> +				key_put(opts->keyring);
> +			opts->keyring = key;
> +			break;
> +#else
> +			pr_err("TLS is not supported\n");
> +			ret = -EINVAL;
> +			goto out;
> +#endif
> +		case NVMF_OPT_TLS_KEY:
> +#ifdef CONFIG_NVME_TLS
> +			if (match_int(args, &key_id)) {
> +				ret = -EINVAL;
> +				goto out;
> +			}
> +			if (key_id < 0) {
> +				pr_err("Invalid key id %d\n", key_id);
> +				ret = -EINVAL;
> +				goto out;
> +			}
> +			if (!key_id) {
> +				pr_debug("Using 'best' PSK\n");
> +				if (opts->tls_key) {
> +					key_put(opts->tls_key);
> +					opts->tls_key = NULL;
> +				}
> +				break;
> +			}
> +			key = key_lookup(key_id);
> +			if (!key) {
> +				pr_err("Key id %08x not found\n", key_id);
> +				ret = -ENOKEY;
> +				goto out;
> +			}
> +			if (opts->tls_key)
> +				key_put(opts->tls_key);
> +			opts->tls_key = key;
> +#else
> +			pr_err("TLS is not supported\n");
> +			ret = -EINVAL;
> +			goto out;
> +#endif
> +			break;
>   		case NVMF_OPT_DISCOVERY:
>   			opts->discovery_nqn = true;
>   			break;
> @@ -1054,6 +1125,12 @@ static int nvmf_check_allowed_opts(struct nvmf_ctrl_options *opts,
>   void nvmf_free_options(struct nvmf_ctrl_options *opts)
>   {
>   	nvmf_host_put(opts->host);
> +#ifdef CONFIG_NVME_TLS
> +	if (opts->keyring)
> +		key_put(opts->keyring);
> +	if (opts->tls_key)
> +		key_put(opts->tls_key);

I think key_put is null safe.

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 13/18] nvmet: make TCP sectype settable via configfs
  2023-03-29 13:59 ` [PATCH 13/18] nvmet: make TCP sectype settable via configfs Hannes Reinecke
@ 2023-03-30 16:07   ` Sagi Grimberg
  2023-03-30 17:37     ` Hannes Reinecke
  0 siblings, 1 reply; 74+ messages in thread
From: Sagi Grimberg @ 2023-03-30 16:07 UTC (permalink / raw)
  To: Hannes Reinecke, Christoph Hellwig
  Cc: Keith Busch, linux-nvme, Chuck Lever, kernel-tls-handshake


> Add a new configfs attribute 'addr_tsas' to make the TCP sectype
> settable via configfs.
> 
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>   drivers/nvme/target/configfs.c | 67 ++++++++++++++++++++++++++++++++++
>   1 file changed, 67 insertions(+)
> 
> diff --git a/drivers/nvme/target/configfs.c b/drivers/nvme/target/configfs.c
> index 907143870da5..ca66ee6dc153 100644
> --- a/drivers/nvme/target/configfs.c
> +++ b/drivers/nvme/target/configfs.c
> @@ -303,6 +303,11 @@ static void nvmet_port_init_tsas_rdma(struct nvmet_port *port)
>   	port->disc_addr.tsas.rdma.cms = NVMF_RDMA_CMS_RDMA_CM;
>   }
>   
> +static void nvmet_port_init_tsas_tcp(struct nvmet_port *port, int sectype)
> +{
> +	port->disc_addr.tsas.tcp.sectype = sectype;
> +}
> +
>   static ssize_t nvmet_addr_trtype_store(struct config_item *item,
>   		const char *page, size_t count)
>   {
> @@ -325,11 +330,72 @@ static ssize_t nvmet_addr_trtype_store(struct config_item *item,
>   	port->disc_addr.trtype = nvmet_transport[i].type;
>   	if (port->disc_addr.trtype == NVMF_TRTYPE_RDMA)
>   		nvmet_port_init_tsas_rdma(port);
> +	else if (port->disc_addr.trtype == NVMF_TRTYPE_TCP)
> +		nvmet_port_init_tsas_tcp(port, NVMF_TCP_SECTYPE_NONE);
>   	return count;
>   }
>   
>   CONFIGFS_ATTR(nvmet_, addr_trtype);
>   
> +static const struct nvmet_type_name_map nvmet_addr_tsas_tcp[] = {
> +	{ NVMF_TCP_SECTYPE_NONE,	"none" },
> +	{ NVMF_TCP_SECTYPE_TLS13,	"tls1.3" },
> +};
> +
> +static const struct nvmet_type_name_map nvmet_addr_tsas_rdma[] = {
> +	{ NVMF_RDMA_QPTYPE_CONNECTED,	"connected" },
> +	{ NVMF_RDMA_QPTYPE_DATAGRAM,	"datagram"  },
> +};
> +
> +static ssize_t nvmet_addr_tsas_show(struct config_item *item,
> +		char *page)
> +{
> +	struct nvmet_port *port = to_nvmet_port(item);
> +	int i;
> +
> +	if (port->disc_addr.trtype == NVMF_TRTYPE_TCP) {
> +		for (i = 0; i < ARRAY_SIZE(nvmet_addr_tsas_tcp); i++) {
> +			if (port->disc_addr.tsas.tcp.sectype == nvmet_addr_tsas_tcp[i].type)
> +				return sprintf(page, "%s\n", nvmet_addr_tsas_tcp[i].name);
> +		}
> +		return sprintf(page, "reserved\n");

What is reserved?

> +	} else if (port->disc_addr.trtype == NVMF_TRTYPE_RDMA) {
> +		for (i = 0; i < ARRAY_SIZE(nvmet_addr_tsas_rdma); i++) {
> +			if (port->disc_addr.tsas.rdma.qptype == nvmet_addr_tsas_rdma[i].type)
> +				return sprintf(page, "%s\n", nvmet_addr_tsas_rdma[i].name);
> +		}
> +		return sprintf(page, "reserved\n");
> +	}
> +	return sprintf(page, "not required\n");
> +}
> +
> +static ssize_t nvmet_addr_tsas_store(struct config_item *item,
> +		const char *page, size_t count)
> +{
> +	struct nvmet_port *port = to_nvmet_port(item);
> +	int i;
> +
> +	if (nvmet_is_port_enabled(port, __func__))
> +		return -EACCES;
> +
> +	if (port->disc_addr.trtype != NVMF_TRTYPE_TCP)
> +		return -EINVAL;
> +
> +	for (i = 0; i < ARRAY_SIZE(nvmet_addr_tsas_tcp); i++) {
> +		if (sysfs_streq(page, nvmet_addr_tsas_tcp[i].name))
> +			goto found;
> +	}
> +
> +	pr_err("Invalid value '%s' for tsas\n", page);
> +	return -EINVAL;
> +
> +found:
> +	nvmet_port_init_tsas_tcp(port, nvmet_addr_tsas_tcp[i].type);
> +	return count;
> +}
> +
> +CONFIGFS_ATTR(nvmet_, addr_tsas);
> +
>   /*
>    * Namespace structures & file operation functions below
>    */
> @@ -1741,6 +1807,7 @@ static struct configfs_attribute *nvmet_port_attrs[] = {
>   	&nvmet_attr_addr_traddr,
>   	&nvmet_attr_addr_trsvcid,
>   	&nvmet_attr_addr_trtype,
> +	&nvmet_attr_addr_tsas,
>   	&nvmet_attr_param_inline_data_size,
>   #ifdef CONFIG_BLK_DEV_INTEGRITY
>   	&nvmet_attr_param_pi_enable,

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 14/18] nvmet-tcp: allocate socket file
  2023-03-29 13:59 ` [PATCH 14/18] nvmet-tcp: allocate socket file Hannes Reinecke
@ 2023-03-30 16:08   ` Sagi Grimberg
  2023-03-30 17:37     ` Hannes Reinecke
  0 siblings, 1 reply; 74+ messages in thread
From: Sagi Grimberg @ 2023-03-30 16:08 UTC (permalink / raw)
  To: Hannes Reinecke, Christoph Hellwig
  Cc: Keith Busch, linux-nvme, Chuck Lever, kernel-tls-handshake



On 3/29/23 16:59, Hannes Reinecke wrote:
> When using the TLS upcall we need to allocate a socket file such
> that the userspace daemon is able to use the socket.
> 
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>   drivers/nvme/target/tcp.c | 51 +++++++++++++++++++++++++++++----------
>   1 file changed, 38 insertions(+), 13 deletions(-)
> 
> diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c
> index 66e8f9fd0ca7..5931971d715f 100644
> --- a/drivers/nvme/target/tcp.c
> +++ b/drivers/nvme/target/tcp.c
> @@ -96,12 +96,14 @@ struct nvmet_tcp_cmd {
>   
>   enum nvmet_tcp_queue_state {
>   	NVMET_TCP_Q_CONNECTING,
> +	NVMET_TCP_Q_TLS_HANDSHAKE,
>   	NVMET_TCP_Q_LIVE,
>   	NVMET_TCP_Q_DISCONNECTING,
>   };
>   
>   struct nvmet_tcp_queue {
>   	struct socket		*sock;
> +	struct file		*sock_file;
>   	struct nvmet_tcp_port	*port;
>   	struct work_struct	io_work;
>   	struct nvmet_cq		nvme_cq;
> @@ -1406,6 +1408,19 @@ static void nvmet_tcp_restore_socket_callbacks(struct nvmet_tcp_queue *queue)
>   	write_unlock_bh(&sock->sk->sk_callback_lock);
>   }
>   
> +static void nvmet_tcp_close_sock(struct nvmet_tcp_queue *queue)
> +{
> +	if (queue->sock_file) {
> +		fput(queue->sock_file);
> +		queue->sock_file = NULL;
> +		queue->sock = NULL;
> +	} else {
> +		WARN_ON(!queue->sock->ops);
> +		sock_release(queue->sock);
> +		queue->sock = NULL;
> +	}
> +}
> +
>   static void nvmet_tcp_uninit_data_in_cmds(struct nvmet_tcp_queue *queue)
>   {
>   	struct nvmet_tcp_cmd *cmd = queue->cmds;
> @@ -1455,12 +1470,11 @@ static void nvmet_tcp_release_queue_work(struct work_struct *w)
>   	nvmet_sq_destroy(&queue->nvme_sq);
>   	cancel_work_sync(&queue->io_work);
>   	nvmet_tcp_free_cmd_data_in_buffers(queue);
> -	sock_release(queue->sock);
> +	nvmet_tcp_close_sock(queue);
>   	nvmet_tcp_free_cmds(queue);
>   	if (queue->hdr_digest || queue->data_digest)
>   		nvmet_tcp_free_crypto(queue);
>   	ida_free(&nvmet_tcp_queue_ida, queue->idx);
> -
>   	page = virt_to_head_page(queue->pf_cache.va);
>   	__page_frag_cache_drain(page, queue->pf_cache.pagecnt_bias);
>   	kfree(queue);
> @@ -1583,7 +1597,7 @@ static int nvmet_tcp_set_queue_sock(struct nvmet_tcp_queue *queue)
>   	return ret;
>   }
>   
> -static int nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port,
> +static void nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port,
>   		struct socket *newsock)
>   {
>   	struct nvmet_tcp_queue *queue;
> @@ -1591,7 +1605,7 @@ static int nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port,
>   
>   	queue = kzalloc(sizeof(*queue), GFP_KERNEL);
>   	if (!queue)
> -		return -ENOMEM;
> +		return;
>   
>   	INIT_WORK(&queue->release_work, nvmet_tcp_release_queue_work);
>   	INIT_WORK(&queue->io_work, nvmet_tcp_io_work);
> @@ -1599,15 +1613,28 @@ static int nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port,
>   	queue->port = port;
>   	queue->nr_cmds = 0;
>   	spin_lock_init(&queue->state_lock);
> -	queue->state = NVMET_TCP_Q_CONNECTING;
> +	if (queue->port->nport->disc_addr.tsas.tcp.sectype ==
> +	    NVMF_TCP_SECTYPE_TLS13)
> +		queue->state = NVMET_TCP_Q_TLS_HANDSHAKE;
> +	else
> +		queue->state = NVMET_TCP_Q_CONNECTING;
>   	INIT_LIST_HEAD(&queue->free_list);
>   	init_llist_head(&queue->resp_list);
>   	INIT_LIST_HEAD(&queue->resp_send_list);
>   
> +	if (queue->state == NVMET_TCP_Q_TLS_HANDSHAKE) {
> +		queue->sock_file = sock_alloc_file(queue->sock, O_CLOEXEC, NULL);
> +		if (IS_ERR(queue->sock_file)) {
> +			ret = PTR_ERR(queue->sock_file);
> +			queue->sock_file = NULL;
> +			goto out_free_queue;
> +		}
> +	}

Why not always allocate a sock_file? Like in the host?

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 09/18] nvme-tcp: enable TLS handshake upcall
  2023-03-30 15:03   ` Sagi Grimberg
@ 2023-03-30 17:16     ` Hannes Reinecke
  0 siblings, 0 replies; 74+ messages in thread
From: Hannes Reinecke @ 2023-03-30 17:16 UTC (permalink / raw)
  To: Sagi Grimberg, Christoph Hellwig
  Cc: Keith Busch, linux-nvme, Chuck Lever, kernel-tls-handshake

On 3/30/23 17:03, Sagi Grimberg wrote:
> 
> 
> On 3/29/23 16:59, Hannes Reinecke wrote:
>> Add a fabrics option 'tls' and start the TLS handshake upcall
>> with the default PSK. When TLS is started the PSK key serial
>> number is displayed in the sysfs attribute 'tls_key'
>>
>> Signed-off-by: Hannes Reinecke <hare@suse.de>
>> ---
>>   drivers/nvme/host/core.c    |  25 ++++++-
>>   drivers/nvme/host/fabrics.c |  11 +++
>>   drivers/nvme/host/fabrics.h |   3 +
>>   drivers/nvme/host/nvme.h    |   4 +-
>>   drivers/nvme/host/tcp.c     | 141 ++++++++++++++++++++++++++++++++++--
>>   5 files changed, 174 insertions(+), 10 deletions(-)
>>
>> diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
>> index 416d0a898f56..869396058e2e 100644
>> --- a/drivers/nvme/host/core.c
>> +++ b/drivers/nvme/host/core.c
>> @@ -3901,6 +3901,19 @@ static DEVICE_ATTR(dhchap_ctrl_secret, S_IRUGO 
>> | S_IWUSR,
>>       nvme_ctrl_dhchap_ctrl_secret_show, 
>> nvme_ctrl_dhchap_ctrl_secret_store);
>>   #endif
>> +#ifdef CONFIG_NVME_TLS
>> +static ssize_t tls_key_show(struct device *dev,
>> +                struct device_attribute *attr, char *buf)
>> +{
>> +    struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
>> +
>> +    if (!ctrl->tls_key)
>> +        return 0;
>> +    return sysfs_emit(buf, "%08x", key_serial(ctrl->tls_key));
>> +}
>> +static DEVICE_ATTR_RO(tls_key);
>> +#endif
>> +
>>   static struct attribute *nvme_dev_attrs[] = {
>>       &dev_attr_reset_controller.attr,
>>       &dev_attr_rescan_controller.attr,
>> @@ -3927,6 +3940,9 @@ static struct attribute *nvme_dev_attrs[] = {
>>   #ifdef CONFIG_NVME_AUTH
>>       &dev_attr_dhchap_secret.attr,
>>       &dev_attr_dhchap_ctrl_secret.attr,
>> +#endif
>> +#ifdef CONFIG_NVME_TLS
>> +    &dev_attr_tls_key.attr,
>>   #endif
>>       NULL
>>   };
>> @@ -3956,6 +3972,10 @@ static umode_t 
>> nvme_dev_attrs_are_visible(struct kobject *kobj,
>>           return 0;
>>       if (a == &dev_attr_dhchap_ctrl_secret.attr && !ctrl->opts)
>>           return 0;
>> +#endif
>> +#ifdef CONFIG_NVME_TLS
>> +    if (a == &dev_attr_tls_key.attr && !ctrl->opts)
>> +        return 0;
>>   #endif
>>       return a->mode;
>>   }
>> @@ -5094,7 +5114,10 @@ static void nvme_free_ctrl(struct device *dev)
>>       if (!subsys || ctrl->instance != subsys->instance)
>>           ida_free(&nvme_instance_ida, ctrl->instance);
>> -
>> +#ifdef CONFIG_NVME_TLS
>> +    if (ctrl->tls_key)
>> +        key_put(ctrl->tls_key);
>> +#endif
>>       nvme_free_cels(ctrl);
>>       nvme_mpath_uninit(ctrl);
>>       nvme_auth_stop(ctrl);
>> diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
>> index bbaa04a0c502..3e4f0e45b58f 100644
>> --- a/drivers/nvme/host/fabrics.c
>> +++ b/drivers/nvme/host/fabrics.c
>> @@ -609,6 +609,7 @@ static const match_table_t opt_tokens = {
>>       { NVMF_OPT_DISCOVERY,        "discovery"        },
>>       { NVMF_OPT_DHCHAP_SECRET,    "dhchap_secret=%s"    },
>>       { NVMF_OPT_DHCHAP_CTRL_SECRET,    "dhchap_ctrl_secret=%s"    },
>> +    { NVMF_OPT_TLS,            "tls"            },
>>       { NVMF_OPT_ERR,            NULL            }
>>   };
>> @@ -632,6 +633,7 @@ static int nvmf_parse_options(struct 
>> nvmf_ctrl_options *opts,
>>       opts->hdr_digest = false;
>>       opts->data_digest = false;
>>       opts->tos = -1; /* < 0 == use transport default */
>> +    opts->tls = false;
>>       options = o = kstrdup(buf, GFP_KERNEL);
>>       if (!options)
>> @@ -918,6 +920,15 @@ static int nvmf_parse_options(struct 
>> nvmf_ctrl_options *opts,
>>               kfree(opts->dhchap_ctrl_secret);
>>               opts->dhchap_ctrl_secret = p;
>>               break;
>> +        case NVMF_OPT_TLS:
>> +#ifdef CONFIG_NVME_TLS
>> +            opts->tls = true;
>> +            break;
>> +#else
>> +            pr_err("TLS is not supported\n");
>> +            ret = -EINVAL;
>> +            goto out;
>> +#endif
> 
> This can become:
>              if (!IS_ENABLED(CONFIG_NVME_TLS)) {
>                  pr_err("TLS is not supported\n");
>                  ret = -EINVAL;
>                  goto out;
>              }
>              opts->tls = true;
>              break;
> 
Yes.

>>           default:
>>               pr_warn("unknown parameter or missing value '%s' in ctrl 
>> creation request\n",
>>                   p);
>> diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h
>> index dcac3df8a5f7..5db36e250e7a 100644
>> --- a/drivers/nvme/host/fabrics.h
>> +++ b/drivers/nvme/host/fabrics.h
>> @@ -70,6 +70,7 @@ enum {
>>       NVMF_OPT_DISCOVERY    = 1 << 22,
>>       NVMF_OPT_DHCHAP_SECRET    = 1 << 23,
>>       NVMF_OPT_DHCHAP_CTRL_SECRET = 1 << 24,
>> +    NVMF_OPT_TLS        = 1 << 25,
>>   };
>>   /**
>> @@ -102,6 +103,7 @@ enum {
>>    * @dhchap_secret: DH-HMAC-CHAP secret
>>    * @dhchap_ctrl_secret: DH-HMAC-CHAP controller secret for 
>> bi-directional
>>    *              authentication
>> + * @tls:        Start TLS encrypted connections (TCP)
>>    * @disable_sqflow: disable controller sq flow control
>>    * @hdr_digest: generate/verify header digest (TCP)
>>    * @data_digest: generate/verify data digest (TCP)
>> @@ -128,6 +130,7 @@ struct nvmf_ctrl_options {
>>       int            max_reconnects;
>>       char            *dhchap_secret;
>>       char            *dhchap_ctrl_secret;
>> +    bool            tls;
>>       bool            disable_sqflow;
>>       bool            hdr_digest;
>>       bool            data_digest;
>> diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
>> index bf46f122e9e1..421d01f849d1 100644
>> --- a/drivers/nvme/host/nvme.h
>> +++ b/drivers/nvme/host/nvme.h
>> @@ -347,7 +347,9 @@ struct nvme_ctrl {
>>       struct nvme_dhchap_key *ctrl_key;
>>       u16 transaction;
>>   #endif
>> -
>> +#ifdef CONFIG_NVME_TLS
>> +    struct key *tls_key;
>> +#endif
> 
> I'd add a comment that this is nvme_tcp specific.
> 
Okay.

>>       /* Power saving configuration */
>>       u64 ps_max_latency_us;
>>       bool apst_enabled;
>> diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
>> index fddf785aba74..fdb564d0b9f4 100644
>> --- a/drivers/nvme/host/tcp.c
>> +++ b/drivers/nvme/host/tcp.c
>> @@ -8,9 +8,15 @@
>>   #include <linux/init.h>
>>   #include <linux/slab.h>
>>   #include <linux/err.h>
>> +#include <linux/key.h>
>>   #include <linux/nvme-tcp.h>
>>   #include <net/sock.h>
>>   #include <net/tcp.h>
>> +#ifdef CONFIG_NVME_TLS
>> +#include <linux/nvme-keyring.h>
>> +#include <net/tls.h>
>> +#include <net/handshake.h>
>> +#endif
>>   #include <linux/blk-mq.h>
>>   #include <crypto/hash.h>
>>   #include <net/busy_poll.h>
>> @@ -31,6 +37,16 @@ static int so_priority;
>>   module_param(so_priority, int, 0644);
>>   MODULE_PARM_DESC(so_priority, "nvme tcp socket optimize priority");
>> +#ifdef CONFIG_NVME_TLS
>> +/*
>> + * TLS handshake timeout
>> + */
>> +static int tls_handshake_timeout = 10;
>> +module_param(tls_handshake_timeout, int, 0644);
>> +MODULE_PARM_DESC(tls_handshake_timeout,
>> +         "nvme TLS handshake timeout in seconds (default 10)");
>> +#endif
>> +
>>   #ifdef CONFIG_DEBUG_LOCK_ALLOC
>>   /* lockdep can detect a circular dependency of the form
>>    *   sk_lock -> mmap_lock (page fault) -> fs locks -> sk_lock
>> @@ -147,7 +163,10 @@ struct nvme_tcp_queue {
>>       struct ahash_request    *snd_hash;
>>       __le32            exp_ddgst;
>>       __le32            recv_ddgst;
>> -
>> +#ifdef CONFIG_NVME_TLS
>> +    struct completion       *tls_complete;
> 
> Why is this a pointer?
> 
Hmm. I'll change it.

>> +    int                     tls_err;
>> +#endif
>>       struct page_frag_cache    pf_cache;
>>       void (*state_change)(struct sock *);
>> @@ -1503,7 +1522,80 @@ static void nvme_tcp_set_queue_io_cpu(struct 
>> nvme_tcp_queue *queue)
>>       queue->io_cpu = cpumask_next_wrap(n - 1, cpu_online_mask, -1, 
>> false);
>>   }
>> -static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid)
>> +#ifdef CONFIG_NVME_TLS
>> +static void nvme_tcp_tls_done(void *data, int status, key_serial_t 
>> peerid)
>> +{
>> +    struct nvme_tcp_queue *queue = data;
>> +    struct nvme_tcp_ctrl *ctrl = queue->ctrl;
>> +    int qid = nvme_tcp_queue_id(queue);
>> +
>> +    dev_dbg(ctrl->ctrl.device, "queue %d: TLS handshake done, key %x, 
>> status %d\n",
>> +        qid, peerid, status);
> 
> Can we avoid naming the key peerid? pskid perhaps.
> 
Okay.

>> +
>> +    if (status)
>> +        queue->tls_err = -status;
> 
> Maybe just return or goto err, instead of the else case?
> 
Okay.

>> +    else {
>> +        struct key *tls_key = key_lookup(peerid);
>> +        if (IS_ERR(tls_key)) {
>> +            dev_warn(ctrl->ctrl.device, "queue %d: Invalid key %x\n",
>> +                 qid, peerid);
>> +            queue->tls_err = -ENOKEY;
>> +        } else {
>> +            ctrl->ctrl.tls_key = tls_key;
>> +            queue->tls_err = 0;
>> +        }
>> +    }
>> +
>> +    if (queue->tls_complete)
>> +        complete(queue->tls_complete);
>> +}
>> +
>> +static int nvme_tcp_start_tls(struct nvme_ctrl *nctrl,
>> +                  struct nvme_tcp_queue *queue,
>> +                  key_serial_t peerid)
>> +{
>> +    int qid = nvme_tcp_queue_id(queue);
>> +    int ret;
>> +    struct tls_handshake_args args;
>> +    unsigned long tmo = tls_handshake_timeout * HZ;
>> +    DECLARE_COMPLETION_ONSTACK(tls_complete);
> 
> I think we can have this completion a queue member.
> 
Let's see.

>> +    key_serial_t keyring = nvme_keyring_id();
>> +
>> +    dev_dbg(nctrl->device, "queue %d: start TLS with key %x\n",
>> +        qid, peerid);
>> +    args.ta_sock = queue->sock;
>> +    args.ta_done = nvme_tcp_tls_done;
>> +    args.ta_data = queue;
>> +    args.ta_my_peerids[0] = peerid;
>> +    args.ta_num_peerids = 1;
>> +    args.ta_keyring = keyring;
>> +    args.ta_timeout_ms = tls_handshake_timeout * 2 * 1000;
> 
> remove the 2x.
> 
Leftover, indeed.

>> +    queue->tls_err = -EOPNOTSUPP;
>> +    queue->tls_complete = &tls_complete;
>> +    ret = tls_client_hello_psk(&args, GFP_KERNEL);
>> +    if (ret) {
>> +        dev_dbg(nctrl->device, "queue %d: failed to start TLS: %d\n",
>> +            qid, ret);
> 
> dev_err
> 
>> +        return ret;
>> +    }
>> +    if (wait_for_completion_timeout(queue->tls_complete, tmo) == 0) {
>> +        dev_dbg(nctrl->device,
>> +            "queue %d: TLS handshake timeout\n", qid);
> 
> dev_err
> 
>> +        queue->tls_complete = NULL;
>> +        ret = -ETIMEDOUT;
>> +    } else {
>> +        dev_dbg(nctrl->device,
>> +            "queue %d: TLS handshake complete, error %d\n",
>> +            qid, queue->tls_err);
>> +        ret = queue->tls_err;
>> +    }
>> +    queue->tls_complete = NULL;
> 
> as mentioned, lets make the completion a queue member and not a pointer
> that we need to pay attention to.
> 
Okay.

>> +    return ret;
>> +}
>> +#endif
>> +
>> +static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid,
>> +                key_serial_t peerid)
>>   {
>>       struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(nctrl);
>>       struct nvme_tcp_queue *queue = &ctrl->queues[qid];
>> @@ -1627,6 +1719,14 @@ static int nvme_tcp_alloc_queue(struct 
>> nvme_ctrl *nctrl, int qid)
>>           goto err_rcv_pdu;
>>       }
>> +#ifdef CONFIG_NVME_TLS
>> +    /* If PSKs are configured try to start TLS */
>> +    if (peerid) {
>> +        ret = nvme_tcp_start_tls(nctrl, queue, peerid);
>> +        if (ret)
>> +            goto err_init_connect;
>> +    }
>> +#endif
>>       ret = nvme_tcp_init_connection(queue);
>>       if (ret)
>>           goto err_init_connect;
>> @@ -1771,10 +1871,23 @@ static int nvme_tcp_start_io_queues(struct 
>> nvme_ctrl *ctrl,
>>   static int nvme_tcp_alloc_admin_queue(struct nvme_ctrl *ctrl)
>>   {
>>       int ret;
>> -
>> -    ret = nvme_tcp_alloc_queue(ctrl, 0);
>> +    key_serial_t psk_id = 0;
>> +
>> +#ifdef CONFIG_NVME_TLS
>> +    if (ctrl->opts->tls) {
>> +        psk_id = nvme_tls_psk_default(NULL,
>> +                          ctrl->opts->host->nqn,
>> +                          ctrl->opts->subsysnqn);
>> +        if (!psk_id) {
>> +            dev_err(ctrl->device, "no valid PSK found\n");
>> +            ret = -ENOKEY;
>> +            goto out_free_queue;
>> +        }
>> +    }
>> +#endif
> 
> ctrl->opts->tls == true guarantes that CONFIG_NVME_TLS is enabled.
> No need for it, as long as you give a nvme_tls_psk_default stub when
> it is not.
> 
Hmm. I deliberately had the 'tls' option always enabled to avoid parsing 
errors. But sure, if we make that a compile condition this can be 
simplified.

>> +    ret = nvme_tcp_alloc_queue(ctrl, 0, psk_id);
>>       if (ret)
>> -        return ret;
>> +        goto out_free_queue;
>>       ret = nvme_tcp_alloc_async_req(to_tcp_ctrl(ctrl));
>>       if (ret)
>> @@ -1790,9 +1903,21 @@ static int nvme_tcp_alloc_admin_queue(struct 
>> nvme_ctrl *ctrl)
>>   static int __nvme_tcp_alloc_io_queues(struct nvme_ctrl *ctrl)
>>   {
>>       int i, ret;
>> -
>> +    key_serial_t psk_id = 0;
>> +
>> +#ifdef CONFIG_NVME_TLS
>> +    if (ctrl->opts->tls) {
>> +        psk_id = nvme_tls_psk_default(NULL,
>> +                          ctrl->opts->host->nqn,
>> +                          ctrl->opts->subsysnqn);
>> +        if (!psk_id) {
>> +            dev_err(ctrl->device, "no valid PSK found\n");
>> +            return -ENOKEY;
>> +        }
>> +    }
>> +#endif
> 
> Do we need this for io queues? why not just store it on nvme_tcp_ctrl
> and just use it?
> 
Guess we can; current debate at fmds is heading to having just one PSK
for both admin and I/O queues anyway.

>>       for (i = 1; i < ctrl->queue_count; i++) {
>> -        ret = nvme_tcp_alloc_queue(ctrl, i);
>> +        ret = nvme_tcp_alloc_queue(ctrl, i, psk_id);
>>           if (ret)
>>               goto out_free_queues;
>>       }
>> @@ -2701,7 +2826,7 @@ static struct nvmf_transport_ops 
>> nvme_tcp_transport = {
>>                 NVMF_OPT_HOST_TRADDR | NVMF_OPT_CTRL_LOSS_TMO |
>>                 NVMF_OPT_HDR_DIGEST | NVMF_OPT_DATA_DIGEST |
>>                 NVMF_OPT_NR_WRITE_QUEUES | NVMF_OPT_NR_POLL_QUEUES |
>> -              NVMF_OPT_TOS | NVMF_OPT_HOST_IFACE,
>> +              NVMF_OPT_TOS | NVMF_OPT_HOST_IFACE | NVMF_OPT_TLS,
>>       .create_ctrl    = nvme_tcp_create_ctrl,
>>   };
> 
> 
> Overall this looks much better Hannes!

Thanks.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman


^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 10/18] nvme-tcp: fixup send workflow for kTLS
  2023-03-30 15:24   ` Sagi Grimberg
@ 2023-03-30 17:26     ` Hannes Reinecke
  2023-03-31  5:49     ` Jakub Kicinski
  1 sibling, 0 replies; 74+ messages in thread
From: Hannes Reinecke @ 2023-03-30 17:26 UTC (permalink / raw)
  To: Sagi Grimberg, Christoph Hellwig, Boris Pismenny, john.fastabend,
	kuba, Paolo Abeni
  Cc: Keith Busch, linux-nvme, Chuck Lever, kernel-tls-handshake, netdev

On 3/30/23 17:24, Sagi Grimberg wrote:
> 
>> kTLS does not support MSG_EOR flag for sendmsg(), and in general
>> is really picky about invalid MSG_XXX flags.
> 
> CC'ing TLS folks.
> 
> Can't tls simply ignore MSG_EOR instead of consumers having to be 
> careful over it?
> 
>> So ensure that the MSG_EOR flags is blanked out for TLS, and that
>> the MSG_SENDPAGE_LAST is only set if we actually do sendpage().
> 
> You mean MSG_SENDPAGE_NOTLAST.
> 
> It is also a bit annoying that a tls socket dictates different behavior
> than a normal socket.
> 
> The current logic is rather simple:
> if more data comming:
>      flags = MSG_MORE | MSG_SENDPAGE_NOTLAST
> else:
>      flags = MSG_EOR
> 
> Would like to keep it that way for tls as well. Can someone
> explain why this is a problem with tls?

TLS is using MSG_EOR internally (to control the flow of the underlying 
tcp stream), so it's meaningless for the ULP.
I've no idea about SENDPAGE_NOTLAST; that one is particularly annoying
as we have to check whether we do sendpage() or sendmsg().

But TLS really should blank out invalid flags, as it has different rules 
for them than anyone else. And the caller really shouldn't be burdened
with checking whether TLS is enabled or not.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman


^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 12/18] nvme-fabrics: parse options 'keyring' and 'tls_key'
  2023-03-30 15:33   ` Sagi Grimberg
@ 2023-03-30 17:34     ` Hannes Reinecke
  2023-04-03 12:24       ` Sagi Grimberg
  0 siblings, 1 reply; 74+ messages in thread
From: Hannes Reinecke @ 2023-03-30 17:34 UTC (permalink / raw)
  To: Sagi Grimberg, Christoph Hellwig
  Cc: Keith Busch, linux-nvme, Chuck Lever, kernel-tls-handshake

On 3/30/23 17:33, Sagi Grimberg wrote:
>> Parse the fabrics options 'keyring' and 'tls_key' and store the
>> referenced keys in the options structure.
> 
> Can you explain the reasoning to why a user need to pass a keyring
> given that we already set up one?
> 
Choice.
With a single keyring we can only have a single identity.
But there might be configurations where we want to have different PSKs
for the same identity (eg for key rotation).
With this option we can prepare a new keyring, and use that instead of 
the old one.

(And it really doesn't add much complexity...)

>>
>> Signed-off-by: Hannes Reinecke <hare@suse.de>
>> ---
>>   drivers/nvme/host/fabrics.c | 79 ++++++++++++++++++++++++++++++++++++-
>>   drivers/nvme/host/fabrics.h |  6 +++
>>   drivers/nvme/host/tcp.c     | 20 +++++++---
>>   3 files changed, 98 insertions(+), 7 deletions(-)
>>
>> diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
>> index 3e4f0e45b58f..5f5e487d498c 100644
>> --- a/drivers/nvme/host/fabrics.c
>> +++ b/drivers/nvme/host/fabrics.c
>> @@ -605,6 +605,8 @@ static const match_table_t opt_tokens = {
>>       { NVMF_OPT_NR_WRITE_QUEUES,    "nr_write_queues=%d"    },
>>       { NVMF_OPT_NR_POLL_QUEUES,    "nr_poll_queues=%d"    },
>>       { NVMF_OPT_TOS,            "tos=%d"        },
>> +    { NVMF_OPT_KEYRING,        "keyring=%d"        },
>> +    { NVMF_OPT_TLS_KEY,        "tls_key=%d"        },
>>       { NVMF_OPT_FAIL_FAST_TMO,    "fast_io_fail_tmo=%d"    },
>>       { NVMF_OPT_DISCOVERY,        "discovery"        },
>>       { NVMF_OPT_DHCHAP_SECRET,    "dhchap_secret=%s"    },
>> @@ -620,8 +622,9 @@ static int nvmf_parse_options(struct 
>> nvmf_ctrl_options *opts,
>>       char *options, *o, *p;
>>       int token, ret = 0;
>>       size_t nqnlen  = 0;
>> -    int ctrl_loss_tmo = NVMF_DEF_CTRL_LOSS_TMO;
>> +    int ctrl_loss_tmo = NVMF_DEF_CTRL_LOSS_TMO, key_id;
>>       uuid_t hostid;
>> +    struct key *key = NULL;
>>       /* Set defaults */
>>       opts->queue_size = NVMF_DEF_QUEUE_SIZE;
>> @@ -889,6 +892,74 @@ static int nvmf_parse_options(struct 
>> nvmf_ctrl_options *opts,
>>               }
>>               opts->tos = token;
>>               break;
>> +        case NVMF_OPT_KEYRING:
>> +#ifdef CONFIG_NVME_TLS
> 
> Same comment as before:
>              if (!IS_ENABLED(CONFIG_NVME_TLS)) {
>                  pr_err("TLS is not supported\n");
>                  ret = -EINVAL;
>                  goto out;
>              }
> 
OK.

>> +            if (match_int(args, &key_id)) {
>> +                ret = -EINVAL;
>> +                goto out;
>> +            }
>> +            if (key_id < 0) {
>> +                pr_err("Invalid keyring id %d\n", key_id);
>> +                ret = -EINVAL;
>> +                goto out;
>> +            }
>> +            if (!key_id) {
>> +                pr_debug("Using default keyring\n");
>> +                if (opts->keyring) {
>> +                    key_put(opts->keyring);
>> +                    opts->keyring = NULL;
>> +                }
>> +                break;
>> +            }
>> +            key = key_lookup(key_id);
>> +            if (!key) {
>> +                pr_err("Keyring id %08x not found\n", key_id);
>> +                ret = -ENOKEY;
>> +                goto out;
>> +            }
>> +            if (opts->keyring)
>> +                key_put(opts->keyring);
>> +            opts->keyring = key;
>> +            break;
>> +#else
>> +            pr_err("TLS is not supported\n");
>> +            ret = -EINVAL;
>> +            goto out;
>> +#endif
>> +        case NVMF_OPT_TLS_KEY:
>> +#ifdef CONFIG_NVME_TLS
>> +            if (match_int(args, &key_id)) {
>> +                ret = -EINVAL;
>> +                goto out;
>> +            }
>> +            if (key_id < 0) {
>> +                pr_err("Invalid key id %d\n", key_id);
>> +                ret = -EINVAL;
>> +                goto out;
>> +            }
>> +            if (!key_id) {
>> +                pr_debug("Using 'best' PSK\n");
>> +                if (opts->tls_key) {
>> +                    key_put(opts->tls_key);
>> +                    opts->tls_key = NULL;
>> +                }
>> +                break;
>> +            }
>> +            key = key_lookup(key_id);
>> +            if (!key) {
>> +                pr_err("Key id %08x not found\n", key_id);
>> +                ret = -ENOKEY;
>> +                goto out;
>> +            }
>> +            if (opts->tls_key)
>> +                key_put(opts->tls_key);
>> +            opts->tls_key = key;
>> +#else
>> +            pr_err("TLS is not supported\n");
>> +            ret = -EINVAL;
>> +            goto out;
>> +#endif
>> +            break;
>>           case NVMF_OPT_DISCOVERY:
>>               opts->discovery_nqn = true;
>>               break;
>> @@ -1054,6 +1125,12 @@ static int nvmf_check_allowed_opts(struct 
>> nvmf_ctrl_options *opts,
>>   void nvmf_free_options(struct nvmf_ctrl_options *opts)
>>   {
>>       nvmf_host_put(opts->host);
>> +#ifdef CONFIG_NVME_TLS
>> +    if (opts->keyring)
>> +        key_put(opts->keyring);
>> +    if (opts->tls_key)
>> +        key_put(opts->tls_key);
> 
> I think key_put is null safe.

Possibly. I'll check.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman


^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 13/18] nvmet: make TCP sectype settable via configfs
  2023-03-30 16:07   ` Sagi Grimberg
@ 2023-03-30 17:37     ` Hannes Reinecke
  2023-04-03 12:31       ` Sagi Grimberg
  0 siblings, 1 reply; 74+ messages in thread
From: Hannes Reinecke @ 2023-03-30 17:37 UTC (permalink / raw)
  To: Sagi Grimberg, Christoph Hellwig
  Cc: Keith Busch, linux-nvme, Chuck Lever, kernel-tls-handshake

On 3/30/23 18:07, Sagi Grimberg wrote:
> 
>> Add a new configfs attribute 'addr_tsas' to make the TCP sectype
>> settable via configfs.
>>
>> Signed-off-by: Hannes Reinecke <hare@suse.de>
>> ---
>>   drivers/nvme/target/configfs.c | 67 ++++++++++++++++++++++++++++++++++
>>   1 file changed, 67 insertions(+)
>>
>> diff --git a/drivers/nvme/target/configfs.c 
>> b/drivers/nvme/target/configfs.c
>> index 907143870da5..ca66ee6dc153 100644
>> --- a/drivers/nvme/target/configfs.c
>> +++ b/drivers/nvme/target/configfs.c
>> @@ -303,6 +303,11 @@ static void nvmet_port_init_tsas_rdma(struct 
>> nvmet_port *port)
>>       port->disc_addr.tsas.rdma.cms = NVMF_RDMA_CMS_RDMA_CM;
>>   }
>> +static void nvmet_port_init_tsas_tcp(struct nvmet_port *port, int 
>> sectype)
>> +{
>> +    port->disc_addr.tsas.tcp.sectype = sectype;
>> +}
>> +
>>   static ssize_t nvmet_addr_trtype_store(struct config_item *item,
>>           const char *page, size_t count)
>>   {
>> @@ -325,11 +330,72 @@ static ssize_t nvmet_addr_trtype_store(struct 
>> config_item *item,
>>       port->disc_addr.trtype = nvmet_transport[i].type;
>>       if (port->disc_addr.trtype == NVMF_TRTYPE_RDMA)
>>           nvmet_port_init_tsas_rdma(port);
>> +    else if (port->disc_addr.trtype == NVMF_TRTYPE_TCP)
>> +        nvmet_port_init_tsas_tcp(port, NVMF_TCP_SECTYPE_NONE);
>>       return count;
>>   }
>>   CONFIGFS_ATTR(nvmet_, addr_trtype);
>> +static const struct nvmet_type_name_map nvmet_addr_tsas_tcp[] = {
>> +    { NVMF_TCP_SECTYPE_NONE,    "none" },
>> +    { NVMF_TCP_SECTYPE_TLS13,    "tls1.3" },
>> +};
>> +
>> +static const struct nvmet_type_name_map nvmet_addr_tsas_rdma[] = {
>> +    { NVMF_RDMA_QPTYPE_CONNECTED,    "connected" },
>> +    { NVMF_RDMA_QPTYPE_DATAGRAM,    "datagram"  },
>> +};
>> +
>> +static ssize_t nvmet_addr_tsas_show(struct config_item *item,
>> +        char *page)
>> +{
>> +    struct nvmet_port *port = to_nvmet_port(item);
>> +    int i;
>> +
>> +    if (port->disc_addr.trtype == NVMF_TRTYPE_TCP) {
>> +        for (i = 0; i < ARRAY_SIZE(nvmet_addr_tsas_tcp); i++) {
>> +            if (port->disc_addr.tsas.tcp.sectype == 
>> nvmet_addr_tsas_tcp[i].type)
>> +                return sprintf(page, "%s\n", 
>> nvmet_addr_tsas_tcp[i].name);
>> +        }
>> +        return sprintf(page, "reserved\n");
> 
> What is reserved?
> 
All other values?

>> +    } else if (port->disc_addr.trtype == NVMF_TRTYPE_RDMA) {
>> +        for (i = 0; i < ARRAY_SIZE(nvmet_addr_tsas_rdma); i++) {
>> +            if (port->disc_addr.tsas.rdma.qptype == 
>> nvmet_addr_tsas_rdma[i].type)
>> +                return sprintf(page, "%s\n", 
>> nvmet_addr_tsas_rdma[i].name);
>> +        }
>> +        return sprintf(page, "reserved\n");
>> +    }
>> +    return sprintf(page, "not required\n");
>> +}
>> +
>> +static ssize_t nvmet_addr_tsas_store(struct config_item *item,
>> +        const char *page, size_t count)
>> +{
>> +    struct nvmet_port *port = to_nvmet_port(item);
>> +    int i;
>> +
>> +    if (nvmet_is_port_enabled(port, __func__))
>> +        return -EACCES;
>> +
>> +    if (port->disc_addr.trtype != NVMF_TRTYPE_TCP)
>> +        return -EINVAL;
>> +
>> +    for (i = 0; i < ARRAY_SIZE(nvmet_addr_tsas_tcp); i++) {
>> +        if (sysfs_streq(page, nvmet_addr_tsas_tcp[i].name))
>> +            goto found;
>> +    }
>> +
>> +    pr_err("Invalid value '%s' for tsas\n", page);
>> +    return -EINVAL;
>> +
>> +found:
>> +    nvmet_port_init_tsas_tcp(port, nvmet_addr_tsas_tcp[i].type);
>> +    return count;
>> +}
>> +
>> +CONFIGFS_ATTR(nvmet_, addr_tsas);
>> +
>>   /*
>>    * Namespace structures & file operation functions below
>>    */
>> @@ -1741,6 +1807,7 @@ static struct configfs_attribute 
>> *nvmet_port_attrs[] = {
>>       &nvmet_attr_addr_traddr,
>>       &nvmet_attr_addr_trsvcid,
>>       &nvmet_attr_addr_trtype,
>> +    &nvmet_attr_addr_tsas,
>>       &nvmet_attr_param_inline_data_size,
>>   #ifdef CONFIG_BLK_DEV_INTEGRITY
>>       &nvmet_attr_param_pi_enable,

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman


^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 14/18] nvmet-tcp: allocate socket file
  2023-03-30 16:08   ` Sagi Grimberg
@ 2023-03-30 17:37     ` Hannes Reinecke
  0 siblings, 0 replies; 74+ messages in thread
From: Hannes Reinecke @ 2023-03-30 17:37 UTC (permalink / raw)
  To: Sagi Grimberg, Christoph Hellwig
  Cc: Keith Busch, linux-nvme, Chuck Lever, kernel-tls-handshake

On 3/30/23 18:08, Sagi Grimberg wrote:
> 
> 
> On 3/29/23 16:59, Hannes Reinecke wrote:
>> When using the TLS upcall we need to allocate a socket file such
>> that the userspace daemon is able to use the socket.
>>
>> Signed-off-by: Hannes Reinecke <hare@suse.de>
>> ---
>>   drivers/nvme/target/tcp.c | 51 +++++++++++++++++++++++++++++----------
>>   1 file changed, 38 insertions(+), 13 deletions(-)
>>
>> diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c
>> index 66e8f9fd0ca7..5931971d715f 100644
>> --- a/drivers/nvme/target/tcp.c
>> +++ b/drivers/nvme/target/tcp.c
>> @@ -96,12 +96,14 @@ struct nvmet_tcp_cmd {
>>   enum nvmet_tcp_queue_state {
>>       NVMET_TCP_Q_CONNECTING,
>> +    NVMET_TCP_Q_TLS_HANDSHAKE,
>>       NVMET_TCP_Q_LIVE,
>>       NVMET_TCP_Q_DISCONNECTING,
>>   };
>>   struct nvmet_tcp_queue {
>>       struct socket        *sock;
>> +    struct file        *sock_file;
>>       struct nvmet_tcp_port    *port;
>>       struct work_struct    io_work;
>>       struct nvmet_cq        nvme_cq;
>> @@ -1406,6 +1408,19 @@ static void 
>> nvmet_tcp_restore_socket_callbacks(struct nvmet_tcp_queue *queue)
>>       write_unlock_bh(&sock->sk->sk_callback_lock);
>>   }
>> +static void nvmet_tcp_close_sock(struct nvmet_tcp_queue *queue)
>> +{
>> +    if (queue->sock_file) {
>> +        fput(queue->sock_file);
>> +        queue->sock_file = NULL;
>> +        queue->sock = NULL;
>> +    } else {
>> +        WARN_ON(!queue->sock->ops);
>> +        sock_release(queue->sock);
>> +        queue->sock = NULL;
>> +    }
>> +}
>> +
>>   static void nvmet_tcp_uninit_data_in_cmds(struct nvmet_tcp_queue 
>> *queue)
>>   {
>>       struct nvmet_tcp_cmd *cmd = queue->cmds;
>> @@ -1455,12 +1470,11 @@ static void 
>> nvmet_tcp_release_queue_work(struct work_struct *w)
>>       nvmet_sq_destroy(&queue->nvme_sq);
>>       cancel_work_sync(&queue->io_work);
>>       nvmet_tcp_free_cmd_data_in_buffers(queue);
>> -    sock_release(queue->sock);
>> +    nvmet_tcp_close_sock(queue);
>>       nvmet_tcp_free_cmds(queue);
>>       if (queue->hdr_digest || queue->data_digest)
>>           nvmet_tcp_free_crypto(queue);
>>       ida_free(&nvmet_tcp_queue_ida, queue->idx);
>> -
>>       page = virt_to_head_page(queue->pf_cache.va);
>>       __page_frag_cache_drain(page, queue->pf_cache.pagecnt_bias);
>>       kfree(queue);
>> @@ -1583,7 +1597,7 @@ static int nvmet_tcp_set_queue_sock(struct 
>> nvmet_tcp_queue *queue)
>>       return ret;
>>   }
>> -static int nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port,
>> +static void nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port,
>>           struct socket *newsock)
>>   {
>>       struct nvmet_tcp_queue *queue;
>> @@ -1591,7 +1605,7 @@ static int nvmet_tcp_alloc_queue(struct 
>> nvmet_tcp_port *port,
>>       queue = kzalloc(sizeof(*queue), GFP_KERNEL);
>>       if (!queue)
>> -        return -ENOMEM;
>> +        return;
>>       INIT_WORK(&queue->release_work, nvmet_tcp_release_queue_work);
>>       INIT_WORK(&queue->io_work, nvmet_tcp_io_work);
>> @@ -1599,15 +1613,28 @@ static int nvmet_tcp_alloc_queue(struct 
>> nvmet_tcp_port *port,
>>       queue->port = port;
>>       queue->nr_cmds = 0;
>>       spin_lock_init(&queue->state_lock);
>> -    queue->state = NVMET_TCP_Q_CONNECTING;
>> +    if (queue->port->nport->disc_addr.tsas.tcp.sectype ==
>> +        NVMF_TCP_SECTYPE_TLS13)
>> +        queue->state = NVMET_TCP_Q_TLS_HANDSHAKE;
>> +    else
>> +        queue->state = NVMET_TCP_Q_CONNECTING;
>>       INIT_LIST_HEAD(&queue->free_list);
>>       init_llist_head(&queue->resp_list);
>>       INIT_LIST_HEAD(&queue->resp_send_list);
>> +    if (queue->state == NVMET_TCP_Q_TLS_HANDSHAKE) {
>> +        queue->sock_file = sock_alloc_file(queue->sock, O_CLOEXEC, 
>> NULL);
>> +        if (IS_ERR(queue->sock_file)) {
>> +            ret = PTR_ERR(queue->sock_file);
>> +            queue->sock_file = NULL;
>> +            goto out_free_queue;
>> +        }
>> +    }
> 
> Why not always allocate a sock_file? Like in the host?

Wasn't sure if you'd go with it.
But sure.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman


^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 10/18] nvme-tcp: fixup send workflow for kTLS
  2023-03-30 15:24   ` Sagi Grimberg
  2023-03-30 17:26     ` Hannes Reinecke
@ 2023-03-31  5:49     ` Jakub Kicinski
  2023-03-31  6:03       ` Hannes Reinecke
  1 sibling, 1 reply; 74+ messages in thread
From: Jakub Kicinski @ 2023-03-31  5:49 UTC (permalink / raw)
  To: Sagi Grimberg
  Cc: Hannes Reinecke, Christoph Hellwig, Boris Pismenny,
	john.fastabend, Paolo Abeni, Keith Busch, linux-nvme,
	Chuck Lever, kernel-tls-handshake, netdev

On Thu, 30 Mar 2023 18:24:04 +0300 Sagi Grimberg wrote:
> > kTLS does not support MSG_EOR flag for sendmsg(), and in general
> > is really picky about invalid MSG_XXX flags.  
> 
> CC'ing TLS folks.
>
> Can't tls simply ignore MSG_EOR instead of consumers having to be 
> careful over it?

I think we can support EOR, I don't see any fundamental problem there.

> > So ensure that the MSG_EOR flags is blanked out for TLS, and that
> > the MSG_SENDPAGE_LAST is only set if we actually do sendpage().  
> 
> You mean MSG_SENDPAGE_NOTLAST.
> 
> It is also a bit annoying that a tls socket dictates different behavior
> than a normal socket.
> 
> The current logic is rather simple:
> if more data comming:
> 	flags = MSG_MORE | MSG_SENDPAGE_NOTLAST
> else:
> 	flags = MSG_EOR
> 
> Would like to keep it that way for tls as well. Can someone
> explain why this is a problem with tls?

Some of the flags are call specific, others may be internal to the
networking stack (e.g. the DECRYPTED flag). Old protocols didn't do 
any validation because people coded more haphazardly in the 90s.
This lack of validation is a major source of technical debt :(

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 10/18] nvme-tcp: fixup send workflow for kTLS
  2023-03-31  5:49     ` Jakub Kicinski
@ 2023-03-31  6:03       ` Hannes Reinecke
  2023-04-03 12:20         ` Sagi Grimberg
  0 siblings, 1 reply; 74+ messages in thread
From: Hannes Reinecke @ 2023-03-31  6:03 UTC (permalink / raw)
  To: Jakub Kicinski, Sagi Grimberg
  Cc: Christoph Hellwig, Boris Pismenny, john.fastabend, Paolo Abeni,
	Keith Busch, linux-nvme, Chuck Lever, kernel-tls-handshake,
	netdev

On 3/31/23 07:49, Jakub Kicinski wrote:
> On Thu, 30 Mar 2023 18:24:04 +0300 Sagi Grimberg wrote:
>>> kTLS does not support MSG_EOR flag for sendmsg(), and in general
>>> is really picky about invalid MSG_XXX flags.
>>
>> CC'ing TLS folks.
>>
>> Can't tls simply ignore MSG_EOR instead of consumers having to be
>> careful over it?
> 
> I think we can support EOR, I don't see any fundamental problem there.
> 
>>> So ensure that the MSG_EOR flags is blanked out for TLS, and that
>>> the MSG_SENDPAGE_LAST is only set if we actually do sendpage().
>>
>> You mean MSG_SENDPAGE_NOTLAST.
>>
>> It is also a bit annoying that a tls socket dictates different behavior
>> than a normal socket.
>>
>> The current logic is rather simple:
>> if more data comming:
>> 	flags = MSG_MORE | MSG_SENDPAGE_NOTLAST
>> else:
>> 	flags = MSG_EOR
>>
>> Would like to keep it that way for tls as well. Can someone
>> explain why this is a problem with tls?
> 
> Some of the flags are call specific, others may be internal to the
> networking stack (e.g. the DECRYPTED flag). Old protocols didn't do
> any validation because people coded more haphazardly in the 90s.
> This lack of validation is a major source of technical debt :(

A-ha. So what is the plan?
Should the stack validate flags?
And should the rules for validating be the same for all protocols?

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman


^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 10/18] nvme-tcp: fixup send workflow for kTLS
  2023-03-31  6:03       ` Hannes Reinecke
@ 2023-04-03 12:20         ` Sagi Grimberg
  2023-04-03 14:59           ` Jakub Kicinski
  0 siblings, 1 reply; 74+ messages in thread
From: Sagi Grimberg @ 2023-04-03 12:20 UTC (permalink / raw)
  To: Hannes Reinecke, Jakub Kicinski
  Cc: Christoph Hellwig, Boris Pismenny, john.fastabend, Paolo Abeni,
	Keith Busch, linux-nvme, Chuck Lever, kernel-tls-handshake,
	netdev

Hey Jakub, Hannes.

>>>> kTLS does not support MSG_EOR flag for sendmsg(), and in general
>>>> is really picky about invalid MSG_XXX flags.
>>>
>>> CC'ing TLS folks.
>>>
>>> Can't tls simply ignore MSG_EOR instead of consumers having to be
>>> careful over it?
>>
>> I think we can support EOR, I don't see any fundamental problem there.

It would help at least one consumer (nvme-tcp) to not change its
behavior between tls and non-tls. At the very minimum don't fail
the send operation (just do the same as if it wasn't passed).

>>>> So ensure that the MSG_EOR flags is blanked out for TLS, and that
>>>> the MSG_SENDPAGE_LAST is only set if we actually do sendpage().
>>>
>>> You mean MSG_SENDPAGE_NOTLAST.
>>>
>>> It is also a bit annoying that a tls socket dictates different behavior
>>> than a normal socket.
>>>
>>> The current logic is rather simple:
>>> if more data comming:
>>>     flags = MSG_MORE | MSG_SENDPAGE_NOTLAST
>>> else:
>>>     flags = MSG_EOR
>>>
>>> Would like to keep it that way for tls as well. Can someone
>>> explain why this is a problem with tls?
>>
>> Some of the flags are call specific, others may be internal to the
>> networking stack (e.g. the DECRYPTED flag). Old protocols didn't do
>> any validation because people coded more haphazardly in the 90s.
>> This lack of validation is a major source of technical debt :(
> 
> A-ha. So what is the plan?
> Should the stack validate flags?
> And should the rules for validating be the same for all protocols?

MSG_SENDPAGE_NOTLAST is not an internal flag, I thought it was
essentially similar semantics to MSG_MORE but for sendpage. It'd
be great if this can be allowed in tls (again, at the very least
don't fail but continue as if it wasn't passed).

If this turns out to be a big project, I would prefer to change
nvme-tcp for now in order not to block nvme tls support (although it is
a hidden capability interface, which is always bad).

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 12/18] nvme-fabrics: parse options 'keyring' and 'tls_key'
  2023-03-30 17:34     ` Hannes Reinecke
@ 2023-04-03 12:24       ` Sagi Grimberg
  2023-04-03 12:36         ` Hannes Reinecke
  0 siblings, 1 reply; 74+ messages in thread
From: Sagi Grimberg @ 2023-04-03 12:24 UTC (permalink / raw)
  To: Hannes Reinecke, Christoph Hellwig
  Cc: Keith Busch, linux-nvme, Chuck Lever, kernel-tls-handshake

>>> Parse the fabrics options 'keyring' and 'tls_key' and store the
>>> referenced keys in the options structure.
>>
>> Can you explain the reasoning to why a user need to pass a keyring
>> given that we already set up one?
>>
> Choice.
> With a single keyring we can only have a single identity.
> But there might be configurations where we want to have different PSKs
> for the same identity (eg for key rotation).

How do you expect that rotation would work with this?

How does nvmet handle a non-nvme keyring?

> With this option we can prepare a new keyring, and use that instead of 
> the old one.

On an existing controller?

> (And it really doesn't add much complexity...)

I know, it just adds one more argument, and I want to understand if it
is really needed.

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 13/18] nvmet: make TCP sectype settable via configfs
  2023-03-30 17:37     ` Hannes Reinecke
@ 2023-04-03 12:31       ` Sagi Grimberg
  2023-04-03 12:43         ` Hannes Reinecke
  0 siblings, 1 reply; 74+ messages in thread
From: Sagi Grimberg @ 2023-04-03 12:31 UTC (permalink / raw)
  To: Hannes Reinecke, Christoph Hellwig
  Cc: Keith Busch, linux-nvme, Chuck Lever, kernel-tls-handshake


>>> Add a new configfs attribute 'addr_tsas' to make the TCP sectype
>>> settable via configfs.
>>>
>>> Signed-off-by: Hannes Reinecke <hare@suse.de>
>>> ---
>>>   drivers/nvme/target/configfs.c | 67 ++++++++++++++++++++++++++++++++++
>>>   1 file changed, 67 insertions(+)
>>>
>>> diff --git a/drivers/nvme/target/configfs.c 
>>> b/drivers/nvme/target/configfs.c
>>> index 907143870da5..ca66ee6dc153 100644
>>> --- a/drivers/nvme/target/configfs.c
>>> +++ b/drivers/nvme/target/configfs.c
>>> @@ -303,6 +303,11 @@ static void nvmet_port_init_tsas_rdma(struct 
>>> nvmet_port *port)
>>>       port->disc_addr.tsas.rdma.cms = NVMF_RDMA_CMS_RDMA_CM;
>>>   }
>>> +static void nvmet_port_init_tsas_tcp(struct nvmet_port *port, int 
>>> sectype)
>>> +{
>>> +    port->disc_addr.tsas.tcp.sectype = sectype;
>>> +}
>>> +
>>>   static ssize_t nvmet_addr_trtype_store(struct config_item *item,
>>>           const char *page, size_t count)
>>>   {
>>> @@ -325,11 +330,72 @@ static ssize_t nvmet_addr_trtype_store(struct 
>>> config_item *item,
>>>       port->disc_addr.trtype = nvmet_transport[i].type;
>>>       if (port->disc_addr.trtype == NVMF_TRTYPE_RDMA)
>>>           nvmet_port_init_tsas_rdma(port);
>>> +    else if (port->disc_addr.trtype == NVMF_TRTYPE_TCP)
>>> +        nvmet_port_init_tsas_tcp(port, NVMF_TCP_SECTYPE_NONE);
>>>       return count;
>>>   }
>>>   CONFIGFS_ATTR(nvmet_, addr_trtype);
>>> +static const struct nvmet_type_name_map nvmet_addr_tsas_tcp[] = {
>>> +    { NVMF_TCP_SECTYPE_NONE,    "none" },
>>> +    { NVMF_TCP_SECTYPE_TLS13,    "tls1.3" },
>>> +};
>>> +
>>> +static const struct nvmet_type_name_map nvmet_addr_tsas_rdma[] = {
>>> +    { NVMF_RDMA_QPTYPE_CONNECTED,    "connected" },
>>> +    { NVMF_RDMA_QPTYPE_DATAGRAM,    "datagram"  },
>>> +};
>>> +
>>> +static ssize_t nvmet_addr_tsas_show(struct config_item *item,
>>> +        char *page)
>>> +{
>>> +    struct nvmet_port *port = to_nvmet_port(item);
>>> +    int i;
>>> +
>>> +    if (port->disc_addr.trtype == NVMF_TRTYPE_TCP) {
>>> +        for (i = 0; i < ARRAY_SIZE(nvmet_addr_tsas_tcp); i++) {
>>> +            if (port->disc_addr.tsas.tcp.sectype == 
>>> nvmet_addr_tsas_tcp[i].type)
>>> +                return sprintf(page, "%s\n", 
>>> nvmet_addr_tsas_tcp[i].name);
>>> +        }
>>> +        return sprintf(page, "reserved\n");
>>
>> What is reserved?
>>
> All other values?

Yes, you should just consolidate it outside the if-else instead of "not
required" (which is a treq value iirc).

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 12/18] nvme-fabrics: parse options 'keyring' and 'tls_key'
  2023-04-03 12:24       ` Sagi Grimberg
@ 2023-04-03 12:36         ` Hannes Reinecke
  2023-04-03 13:07           ` Sagi Grimberg
  0 siblings, 1 reply; 74+ messages in thread
From: Hannes Reinecke @ 2023-04-03 12:36 UTC (permalink / raw)
  To: Sagi Grimberg, Christoph Hellwig
  Cc: Keith Busch, linux-nvme, Chuck Lever, kernel-tls-handshake

On 4/3/23 14:24, Sagi Grimberg wrote:
>>>> Parse the fabrics options 'keyring' and 'tls_key' and store the
>>>> referenced keys in the options structure.
>>>
>>> Can you explain the reasoning to why a user need to pass a keyring
>>> given that we already set up one?
>>>
>> Choice.
>> With a single keyring we can only have a single identity.
>> But there might be configurations where we want to have different PSKs
>> for the same identity (eg for key rotation).
> 
> How do you expect that rotation would work with this?
> 
The user creates a new keyring, adds the necessary keys to it, and then
either
- updates the keyring for nvmet (via configfs)
or
- reconnects using the new keyring

> How does nvmet handle a non-nvme keyring?
> 
There is a configfs attribute 'param_keyring', allowing you to use a 
different keyring (the nvme one is just the default).

>> With this option we can prepare a new keyring, and use that instead of 
>> the old one.
> 
> On an existing controller?
> 
Sure; when we are updating the keyring and the key id the controller 
will use that after the next reset.
Much like we do for DH-CHAP nowadays.

>> (And it really doesn't add much complexity...)
> 
> I know, it just adds one more argument, and I want to understand if it
> is really needed.

I do agree that the keyring argument would not necessarily required if 
we pass in the key id, but I'll be needing the keyring for secure 
concatenation. And plan is to move DH-HMAC-CHAP over to keyrings, too.

So I'd prefer to keep it.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		           Kernel Storage Architect
hare@suse.de			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Frankenstr. 146, 90461 Nürnberg
Managing Directors: I. Totev, A. Myers, A. McDonald, M. B. Moerman
(HRB 36809, AG Nürnberg)


^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 13/18] nvmet: make TCP sectype settable via configfs
  2023-04-03 12:31       ` Sagi Grimberg
@ 2023-04-03 12:43         ` Hannes Reinecke
  0 siblings, 0 replies; 74+ messages in thread
From: Hannes Reinecke @ 2023-04-03 12:43 UTC (permalink / raw)
  To: Sagi Grimberg, Christoph Hellwig
  Cc: Keith Busch, linux-nvme, Chuck Lever, kernel-tls-handshake

On 4/3/23 14:31, Sagi Grimberg wrote:
> 
>>>> Add a new configfs attribute 'addr_tsas' to make the TCP sectype
>>>> settable via configfs.
>>>>
>>>> Signed-off-by: Hannes Reinecke <hare@suse.de>
>>>> ---
>>>>   drivers/nvme/target/configfs.c | 67 
>>>> ++++++++++++++++++++++++++++++++++
>>>>   1 file changed, 67 insertions(+)
>>>>
>>>> diff --git a/drivers/nvme/target/configfs.c 
>>>> b/drivers/nvme/target/configfs.c
>>>> index 907143870da5..ca66ee6dc153 100644
>>>> --- a/drivers/nvme/target/configfs.c
>>>> +++ b/drivers/nvme/target/configfs.c
>>>> @@ -303,6 +303,11 @@ static void nvmet_port_init_tsas_rdma(struct 
>>>> nvmet_port *port)
>>>>       port->disc_addr.tsas.rdma.cms = NVMF_RDMA_CMS_RDMA_CM;
>>>>   }
>>>> +static void nvmet_port_init_tsas_tcp(struct nvmet_port *port, int 
>>>> sectype)
>>>> +{
>>>> +    port->disc_addr.tsas.tcp.sectype = sectype;
>>>> +}
>>>> +
>>>>   static ssize_t nvmet_addr_trtype_store(struct config_item *item,
>>>>           const char *page, size_t count)
>>>>   {
>>>> @@ -325,11 +330,72 @@ static ssize_t nvmet_addr_trtype_store(struct 
>>>> config_item *item,
>>>>       port->disc_addr.trtype = nvmet_transport[i].type;
>>>>       if (port->disc_addr.trtype == NVMF_TRTYPE_RDMA)
>>>>           nvmet_port_init_tsas_rdma(port);
>>>> +    else if (port->disc_addr.trtype == NVMF_TRTYPE_TCP)
>>>> +        nvmet_port_init_tsas_tcp(port, NVMF_TCP_SECTYPE_NONE);
>>>>       return count;
>>>>   }
>>>>   CONFIGFS_ATTR(nvmet_, addr_trtype);
>>>> +static const struct nvmet_type_name_map nvmet_addr_tsas_tcp[] = {
>>>> +    { NVMF_TCP_SECTYPE_NONE,    "none" },
>>>> +    { NVMF_TCP_SECTYPE_TLS13,    "tls1.3" },
>>>> +};
>>>> +
>>>> +static const struct nvmet_type_name_map nvmet_addr_tsas_rdma[] = {
>>>> +    { NVMF_RDMA_QPTYPE_CONNECTED,    "connected" },
>>>> +    { NVMF_RDMA_QPTYPE_DATAGRAM,    "datagram"  },
>>>> +};
>>>> +
>>>> +static ssize_t nvmet_addr_tsas_show(struct config_item *item,
>>>> +        char *page)
>>>> +{
>>>> +    struct nvmet_port *port = to_nvmet_port(item);
>>>> +    int i;
>>>> +
>>>> +    if (port->disc_addr.trtype == NVMF_TRTYPE_TCP) {
>>>> +        for (i = 0; i < ARRAY_SIZE(nvmet_addr_tsas_tcp); i++) {
>>>> +            if (port->disc_addr.tsas.tcp.sectype == 
>>>> nvmet_addr_tsas_tcp[i].type)
>>>> +                return sprintf(page, "%s\n", 
>>>> nvmet_addr_tsas_tcp[i].name);
>>>> +        }
>>>> +        return sprintf(page, "reserved\n");
>>>
>>> What is reserved?
>>>
>> All other values?
> 
> Yes, you should just consolidate it outside the if-else instead of "not
> required" (which is a treq value iirc).
> 
Okay, will do.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		           Kernel Storage Architect
hare@suse.de			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Frankenstr. 146, 90461 Nürnberg
Managing Directors: I. Totev, A. Myers, A. McDonald, M. B. Moerman
(HRB 36809, AG Nürnberg)


^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 15/18] nvmet-tcp: enable TLS handshake upcall
  2023-03-29 13:59 ` [PATCH 15/18] nvmet-tcp: enable TLS handshake upcall Hannes Reinecke
@ 2023-04-03 12:51   ` Sagi Grimberg
  2023-04-03 14:05     ` Hannes Reinecke
  0 siblings, 1 reply; 74+ messages in thread
From: Sagi Grimberg @ 2023-04-03 12:51 UTC (permalink / raw)
  To: Hannes Reinecke, Christoph Hellwig
  Cc: Keith Busch, linux-nvme, Chuck Lever, kernel-tls-handshake


> Add functions to start the TLS handshake upcall when
> the TCP RSAS sectype is set to 'tls1.3'.

TSAS

> 
> Signed-off-by: Hannes Reincke <hare@suse.de>
> ---
>   drivers/nvme/target/configfs.c |  32 +++++++-
>   drivers/nvme/target/tcp.c      | 135 ++++++++++++++++++++++++++++++++-
>   2 files changed, 163 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/nvme/target/configfs.c b/drivers/nvme/target/configfs.c
> index ca66ee6dc153..36fbf6a22d09 100644
> --- a/drivers/nvme/target/configfs.c
> +++ b/drivers/nvme/target/configfs.c
> @@ -159,10 +159,12 @@ static const struct nvmet_type_name_map nvmet_addr_treq[] = {
>   	{ NVMF_TREQ_NOT_REQUIRED,	"not required" },
>   };
>   
> +#define NVMET_PORT_TREQ(port) ((port)->disc_addr.treq & NVME_TREQ_SECURE_CHANNEL_MASK)

Can you make it a static inline?

> +
>   static ssize_t nvmet_addr_treq_show(struct config_item *item, char *page)
>   {
> -	u8 treq = to_nvmet_port(item)->disc_addr.treq &
> -		NVME_TREQ_SECURE_CHANNEL_MASK;
> +	struct nvmet_port *port = to_nvmet_port(item);
> +	u8 treq = NVMET_PORT_TREQ(port);
>   	int i;
>   
>   	for (i = 0; i < ARRAY_SIZE(nvmet_addr_treq); i++) {
> @@ -193,6 +195,17 @@ static ssize_t nvmet_addr_treq_store(struct config_item *item,
>   	return -EINVAL;
>   
>   found:
> +#ifdef CONFIG_NVME_TLS
> +	if (port->disc_addr.trtype == NVMF_TRTYPE_TCP) {
> +		if (port->disc_addr.tsas.tcp.sectype != NVMF_TCP_SECTYPE_TLS13) {
> +			pr_warn("cannot change TREQ when TLS is not enabled\n");
> +			return -EINVAL;
> +		} else if (nvmet_addr_treq[i].type == NVMF_TREQ_NOT_SPECIFIED) {
> +			pr_warn("cannot set TREQ to 'not specified' when TLS is enabled\n");
> +			return -EINVAL;
> +		}
> +	}

Is this code wrong if CONFIG_NVME_TLS is not enabled?

> +#endif
>   	treq |= nvmet_addr_treq[i].type;
>   	port->disc_addr.treq = treq;
>   	return count;
> @@ -373,6 +386,7 @@ static ssize_t nvmet_addr_tsas_store(struct config_item *item,
>   		const char *page, size_t count)
>   {
>   	struct nvmet_port *port = to_nvmet_port(item);
> +	u8 treq = port->disc_addr.treq & ~NVME_TREQ_SECURE_CHANNEL_MASK;
>   	int i;
>   
>   	if (nvmet_is_port_enabled(port, __func__))
> @@ -391,6 +405,20 @@ static ssize_t nvmet_addr_tsas_store(struct config_item *item,
>   
>   found:
>   	nvmet_port_init_tsas_tcp(port, nvmet_addr_tsas_tcp[i].type);
> +	if (nvmet_addr_tsas_tcp[i].type == NVMF_TCP_SECTYPE_TLS13) {
> +#ifdef CONFIG_NVME_TLS

Maybe in the start of the function just do:

	if (!IS_ENABLED(CONFIG_NVME_TLS)) {
		pr_err("TLS not supported\n");
		return -EINVAL;
	}

Instead of incorporating it here.

> +		if (NVMET_PORT_TREQ(port) == NVMF_TREQ_NOT_SPECIFIED)
> +			treq |= NVMF_TREQ_REQUIRED;
> +		else
> +			treq |= NVMET_PORT_TREQ(port);
> +#else
> +		pr_err("TLS not supported\n");
> +		return -EINVAL;
> +#endif
> +	} else {
> +		/* Set to 'not specified' if TLS is not enabled */
> +		treq |= NVMF_TREQ_NOT_SPECIFIED;
> +	}
>   	return count;
>   }
>   
> diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c
> index 5931971d715f..ebec882120fd 100644
> --- a/drivers/nvme/target/tcp.c
> +++ b/drivers/nvme/target/tcp.c
> @@ -11,6 +11,10 @@
>   #include <linux/nvme-tcp.h>
>   #include <net/sock.h>
>   #include <net/tcp.h>
> +#ifdef CONFIG_NVME_TLS
> +#include <net/handshake.h>

Is net/handshake.h under an ifdef? If so, CONFIG_NVME_TLS should
select it.

> +#include <linux/nvme-keyring.h>

will this include not work if CONFIG_NVME_TLS not work?
<linux/nvme-auth.h> is not under CONFIG_NVME_AUTH for example.

> +#endif
>   #include <linux/inet.h>
>   #include <linux/llist.h>
>   #include <crypto/hash.h>
> @@ -40,6 +44,16 @@ module_param(idle_poll_period_usecs, int, 0644);
>   MODULE_PARM_DESC(idle_poll_period_usecs,
>   		"nvmet tcp io_work poll till idle time period in usecs");
>   
> +#ifdef CONFIG_NVME_TLS
> +/*
> + * TLS handshake timeout
> + */
> +static int tls_handshake_timeout = 30;
> +module_param(tls_handshake_timeout, int, 0644);
> +MODULE_PARM_DESC(tls_handshake_timeout,
> +		 "nvme TLS handshake timeout in seconds (default 30)");
> +#endif
> +
>   #define NVMET_TCP_RECV_BUDGET		8
>   #define NVMET_TCP_SEND_BUDGET		8
>   #define NVMET_TCP_IO_WORK_BUDGET	64
> @@ -130,6 +144,10 @@ struct nvmet_tcp_queue {
>   	bool			data_digest;
>   	struct ahash_request	*snd_hash;
>   	struct ahash_request	*rcv_hash;
> +#ifdef CONFIG_NVME_TLS
> +	struct key		*tls_psk;
> +	struct delayed_work	tls_handshake_work;
> +#endif

If these won't be under CONFIG_NVME_TLS will it save a lot of the other
ifdefs in the code?

>   
>   	unsigned long           poll_end;
>   
> @@ -1474,6 +1492,10 @@ static void nvmet_tcp_release_queue_work(struct work_struct *w)
>   	nvmet_tcp_free_cmds(queue);
>   	if (queue->hdr_digest || queue->data_digest)
>   		nvmet_tcp_free_crypto(queue);
> +#ifdef CONFIG_NVME_TLS
> +	if (queue->tls_psk)
> +		key_put(queue->tls_psk);

key_put is NULL safe.

> +#endif
>   	ida_free(&nvmet_tcp_queue_ida, queue->idx);
>   	page = virt_to_head_page(queue->pf_cache.va);
>   	__page_frag_cache_drain(page, queue->pf_cache.pagecnt_bias);
> @@ -1488,8 +1510,12 @@ static void nvmet_tcp_data_ready(struct sock *sk)
>   
>   	read_lock_bh(&sk->sk_callback_lock);
>   	queue = sk->sk_user_data;
> -	if (likely(queue))
> -		queue_work_on(queue_cpu(queue), nvmet_tcp_wq, &queue->io_work);
> +	if (queue->data_ready)
> +		queue->data_ready(sk);
> +	if (likely(queue) &&
> +	    queue->state != NVMET_TCP_Q_TLS_HANDSHAKE)
> +		queue_work_on(queue_cpu(queue), nvmet_tcp_wq,
> +			      &queue->io_work);
>   	read_unlock_bh(&sk->sk_callback_lock);
>   }
>   
> @@ -1597,6 +1623,89 @@ static int nvmet_tcp_set_queue_sock(struct nvmet_tcp_queue *queue)
>   	return ret;
>   }
>   
> +#ifdef CONFIG_NVME_TLS
> +static void nvmet_tcp_tls_queue_restart(struct nvmet_tcp_queue *queue)
> +{
> +	spin_lock(&queue->state_lock);
> +	if (queue->state != NVMET_TCP_Q_TLS_HANDSHAKE) {
> +		pr_warn("queue %d: TLS handshake already completed\n",
> +			queue->idx);
> +		spin_unlock(&queue->state_lock);
> +		return;
> +	}
> +	queue->state = NVMET_TCP_Q_CONNECTING;
> +	spin_unlock(&queue->state_lock);
> +
> +	pr_debug("queue %d: restarting queue after TLS handshake\n",
> +		 queue->idx);
> +	/*
> +	 * Set callbacks after handshake; TLS implementation
> +	 * might have changed the socket callbacks.
> +	 */
> +	nvmet_tcp_set_queue_sock(queue);

Maybe fold it into the caller? The name is confusing anyways.
The queue is not restarted, it is post-configured for lack of
a better term.

> +}
> +
> +static void nvmet_tcp_tls_handshake_done(void *data, int status,
> +					 key_serial_t peerid)

					pskid.

> +{
> +	struct nvmet_tcp_queue *queue = data;
> +
> +	pr_debug("queue %d: TLS handshake done, key %x, status %d\n",
> +		 queue->idx, peerid, status);
> +	if (!status) {
> +		spin_lock(&queue->state_lock);
> +		queue->tls_psk = key_lookup(peerid);
> +		if (IS_ERR(queue->tls_psk)) {
> +			pr_warn("queue %d: TLS key %x not found\n",
> +				queue->idx, peerid);
> +			queue->tls_psk = NULL;

Here you let the timeout take care of it later?

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 16/18] nvmet-tcp: rework sendpage for kTLS
  2023-03-29 13:59 ` [PATCH 16/18] nvmet-tcp: rework sendpage for kTLS Hannes Reinecke
@ 2023-04-03 12:52   ` Sagi Grimberg
  0 siblings, 0 replies; 74+ messages in thread
From: Sagi Grimberg @ 2023-04-03 12:52 UTC (permalink / raw)
  To: Hannes Reinecke, Christoph Hellwig
  Cc: Keith Busch, linux-nvme, Chuck Lever, kernel-tls-handshake

> kTLS ->sendpage() doesn't support the MSG_EOR flag, so blank
> it out if kTLS is enabled.

I'd prefer we don't do this, but lets wait for the verdict from
Jakub.

> 
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>   drivers/nvme/target/tcp.c | 6 +++---
>   1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c
> index ebec882120fd..5daf16138471 100644
> --- a/drivers/nvme/target/tcp.c
> +++ b/drivers/nvme/target/tcp.c
> @@ -650,7 +650,7 @@ static int nvmet_try_send_response(struct nvmet_tcp_cmd *cmd,
>   
>   	if (!last_in_batch && cmd->queue->send_list_len)
>   		flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST;
> -	else
> +	else if (!cmd->queue->tls_psk)
>   		flags |= MSG_EOR;
>   
>   	ret = kernel_sendpage(cmd->queue->sock, virt_to_page(cmd->rsp_pdu),
> @@ -678,7 +678,7 @@ static int nvmet_try_send_r2t(struct nvmet_tcp_cmd *cmd, bool last_in_batch)
>   
>   	if (!last_in_batch && cmd->queue->send_list_len)
>   		flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST;
> -	else
> +	else if (!cmd->queue->tls_psk)
>   		flags |= MSG_EOR;
>   
>   	ret = kernel_sendpage(cmd->queue->sock, virt_to_page(cmd->r2t_pdu),
> @@ -708,7 +708,7 @@ static int nvmet_try_send_ddgst(struct nvmet_tcp_cmd *cmd, bool last_in_batch)
>   
>   	if (!last_in_batch && cmd->queue->send_list_len)
>   		msg.msg_flags |= MSG_MORE;
> -	else
> +	else if (!cmd->queue->tls_psk)
>   		msg.msg_flags |= MSG_EOR;
>   
>   	ret = kernel_sendmsg(queue->sock, &msg, &iov, 1, iov.iov_len);

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 17/18] nvmet-tcp: control messages for recvmsg()
  2023-03-29 13:59 ` [PATCH 17/18] nvmet-tcp: control messages for recvmsg() Hannes Reinecke
@ 2023-04-03 12:59   ` Sagi Grimberg
  0 siblings, 0 replies; 74+ messages in thread
From: Sagi Grimberg @ 2023-04-03 12:59 UTC (permalink / raw)
  To: Hannes Reinecke, Christoph Hellwig
  Cc: Keith Busch, linux-nvme, Chuck Lever, kernel-tls-handshake



On 3/29/23 16:59, Hannes Reinecke wrote:
> kTLS requires control messages for recvmsg() to relay any out-of-band
> TLS messages (eg TLS alerts) to the caller.
> 
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>   drivers/nvme/target/tcp.c | 71 +++++++++++++++++++++++++++++++++++++++
>   1 file changed, 71 insertions(+)
> 
> diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c
> index 5daf16138471..75017fd5da9f 100644
> --- a/drivers/nvme/target/tcp.c
> +++ b/drivers/nvme/target/tcp.c
> @@ -12,6 +12,7 @@
>   #include <net/sock.h>
>   #include <net/tcp.h>
>   #ifdef CONFIG_NVME_TLS
> +#include <net/tls.h>
>   #include <net/handshake.h>
>   #include <linux/nvme-keyring.h>
>   #endif
> @@ -92,6 +93,7 @@ struct nvmet_tcp_cmd {
>   	u32				pdu_len;
>   	u32				pdu_recv;
>   	int				sg_idx;
> +	char				recv_cbuf[CMSG_LEN(sizeof(char))];
>   	struct msghdr			recv_msg;
>   	struct bio_vec			*iov;
>   	u32				flags;
> @@ -1086,7 +1088,18 @@ static int nvmet_tcp_try_recv_pdu(struct nvmet_tcp_queue *queue)
>   	struct nvme_tcp_hdr *hdr = &queue->pdu.cmd.hdr;
>   	int len;
>   	struct kvec iov;
> +#ifdef CONFIG_NVME_TLS

Do you need this ifdef? The recv_cbuf in the cmd is not under and ifdef.

> +	char cbuf[CMSG_LEN(sizeof(char))] = {};
> +	unsigned char ctype;
> +	struct cmsghdr *cmsg;
> +	struct msghdr msg = {
> +		.msg_control = cbuf,
> +		.msg_controllen = sizeof(cbuf),
> +		.msg_flags = MSG_DONTWAIT
> +	};
> +#else
>   	struct msghdr msg = { .msg_flags = MSG_DONTWAIT };
> +#endif
>   
>   recv:
>   	iov.iov_base = (void *)&queue->pdu + queue->offset;
> @@ -1095,6 +1108,19 @@ static int nvmet_tcp_try_recv_pdu(struct nvmet_tcp_queue *queue)
>   			iov.iov_len, msg.msg_flags);
>   	if (unlikely(len < 0))
>   		return len;
> +#ifdef CONFIG_NVME_TLS
> +	cmsg = (struct cmsghdr *)cbuf;
> +	if (CMSG_OK(&msg, cmsg) &&
> +	    cmsg->cmsg_level == SOL_TLS &&
> +	    cmsg->cmsg_type == TLS_GET_RECORD_TYPE) {
> +		ctype = *((unsigned char *)CMSG_DATA(cmsg));
> +		if (ctype != TLS_RECORD_TYPE_DATA) {
> +			pr_err("queue %d unhandled TLS record %d\n",
> +				queue->idx, ctype);
> +			return -ENOTCONN;
> +		}
> +	}
> +#endif

Can we consolidate this to a helper?

Something like:
static int nvmet_tcp_check_cmsg(struct nvmet_tcp_queue *queue, struct 
cmsghdr *cmsg)
{
	if (!IS_ENABLED(CONFIG_NVME_TLS))
		return 0;

	if (CMSG_OK(&msg, cmsg) &&
	    cmsg->cmsg_level == SOL_TLS &&
	    cmsg->cmsg_type == TLS_GET_RECORD_TYPE) {
		ctype = *((unsigned char *)CMSG_DATA(cmsg));
		if (ctype != TLS_RECORD_TYPE_DATA) {
			pr_err("queue %d unhandled TLS record %d\n",
				queue->idx, ctype);
			return -ENOTCONN;
		}
	}
	return 0;
}

>   
>   	queue->offset += len;
>   	queue->left -= len;
> @@ -1150,10 +1176,27 @@ static int nvmet_tcp_try_recv_data(struct nvmet_tcp_queue *queue)
>   	int ret;
>   
>   	while (msg_data_left(&cmd->recv_msg)) {
> +#ifdef CONFIG_NVME_TLS
> +		struct cmsghdr *cmsg;
> +		unsigned char ctype;
> +#endif
>   		ret = sock_recvmsg(cmd->queue->sock, &cmd->recv_msg,
>   			cmd->recv_msg.msg_flags);
>   		if (ret <= 0)
>   			return ret;
> +#ifdef CONFIG_NVME_TLS
> +		cmsg = (struct cmsghdr *)cmd->recv_cbuf;
> +		if (CMSG_OK(&cmd->recv_msg, cmsg) &&
> +		    cmsg->cmsg_level == SOL_TLS &&
> +		    cmsg->cmsg_type == TLS_GET_RECORD_TYPE) {
> +			ctype = *((unsigned char *)CMSG_DATA(cmsg));
> +			if (ctype != TLS_RECORD_TYPE_DATA) {
> +				pr_err("queue %d unhandled TLS record %d\n",
> +				       queue->idx, ctype);
> +				return -ENOTCONN;
> +			}
> +		}
> +#endif
>   
>   		cmd->pdu_recv += ret;
>   		cmd->rbytes_done += ret;
> @@ -1175,7 +1218,18 @@ static int nvmet_tcp_try_recv_ddgst(struct nvmet_tcp_queue *queue)
>   {
>   	struct nvmet_tcp_cmd *cmd = queue->cmd;
>   	int ret;
> +#ifdef CONFIG_NVME_TLS
> +	char cbuf[CMSG_LEN(sizeof(char))] = {};
> +	unsigned char ctype;
> +	struct cmsghdr *cmsg;
> +	struct msghdr msg = {
> +		.msg_control = cbuf,
> +		.msg_controllen = sizeof(cbuf),
> +		.msg_flags = MSG_DONTWAIT
> +	};
> +#else
>   	struct msghdr msg = { .msg_flags = MSG_DONTWAIT };
> +#endif
>   	struct kvec iov = {
>   		.iov_base = (void *)&cmd->recv_ddgst + queue->offset,
>   		.iov_len = queue->left
> @@ -1185,6 +1239,19 @@ static int nvmet_tcp_try_recv_ddgst(struct nvmet_tcp_queue *queue)
>   			iov.iov_len, msg.msg_flags);
>   	if (unlikely(ret < 0))
>   		return ret;
> +#ifdef CONFIG_NVME_TLS
> +	cmsg = (struct cmsghdr *)cbuf;
> +	if (CMSG_OK(&msg, cmsg) &&
> +	    cmsg->cmsg_level == SOL_TLS &&
> +	    cmsg->cmsg_type == TLS_GET_RECORD_TYPE) {
> +		ctype = *((unsigned char *)CMSG_DATA(cmsg));
> +		if (ctype != TLS_RECORD_TYPE_DATA) {
> +			pr_err("queue %d unhandled TLS record %d\n",
> +				queue->idx, ctype);
> +			return -ENOTCONN;
> +		}
> +	}
> +#endif
>   
>   	queue->offset += ret;
>   	queue->left -= ret;
> @@ -1354,6 +1421,10 @@ static int nvmet_tcp_alloc_cmd(struct nvmet_tcp_queue *queue,
>   	if (!c->r2t_pdu)
>   		goto out_free_data;
>   
> +#ifdef CONFIG_NVME_TLS
> +	c->recv_msg.msg_control = c->recv_cbuf;
> +	c->recv_msg.msg_controllen = sizeof(c->recv_cbuf);
> +#endif
>   	c->recv_msg.msg_flags = MSG_DONTWAIT | MSG_NOSIGNAL;
>   
>   	list_add_tail(&c->entry, &queue->free_list);

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 18/18] nvmet-tcp: add configfs attribute 'param_keyring'
  2023-03-29 13:59 ` [PATCH 18/18] nvmet-tcp: add configfs attribute 'param_keyring' Hannes Reinecke
@ 2023-04-03 13:03   ` Sagi Grimberg
  2023-04-03 14:13     ` Hannes Reinecke
  0 siblings, 1 reply; 74+ messages in thread
From: Sagi Grimberg @ 2023-04-03 13:03 UTC (permalink / raw)
  To: Hannes Reinecke, Christoph Hellwig
  Cc: Keith Busch, linux-nvme, Chuck Lever, kernel-tls-handshake


> Add a configfs attribute to list and change the default keyring.

is the keyring an attribute of a port? or a subsystem? Why the choice
of making it per port?

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 12/18] nvme-fabrics: parse options 'keyring' and 'tls_key'
  2023-04-03 12:36         ` Hannes Reinecke
@ 2023-04-03 13:07           ` Sagi Grimberg
  2023-04-03 14:11             ` Hannes Reinecke
  0 siblings, 1 reply; 74+ messages in thread
From: Sagi Grimberg @ 2023-04-03 13:07 UTC (permalink / raw)
  To: Hannes Reinecke, Christoph Hellwig
  Cc: Keith Busch, linux-nvme, Chuck Lever, kernel-tls-handshake


>>>>> Parse the fabrics options 'keyring' and 'tls_key' and store the
>>>>> referenced keys in the options structure.
>>>>
>>>> Can you explain the reasoning to why a user need to pass a keyring
>>>> given that we already set up one?
>>>>
>>> Choice.
>>> With a single keyring we can only have a single identity.
>>> But there might be configurations where we want to have different PSKs
>>> for the same identity (eg for key rotation).
>>
>> How do you expect that rotation would work with this?
>>
> The user creates a new keyring, adds the necessary keys to it, and then
> either
> - updates the keyring for nvmet (via configfs)
> or
> - reconnects using the new keyring

And the user cannot update the nvme keyring?

>> How does nvmet handle a non-nvme keyring?
>>
> There is a configfs attribute 'param_keyring', allowing you to use a 
> different keyring (the nvme one is just the default).

OK.

>>> With this option we can prepare a new keyring, and use that instead 
>>> of the old one.
>>
>> On an existing controller?
>>
> Sure; when we are updating the keyring and the key id the controller 
> will use that after the next reset.
> Much like we do for DH-CHAP nowadays.
> 
>>> (And it really doesn't add much complexity...)
>>
>> I know, it just adds one more argument, and I want to understand if it
>> is really needed.
> 
> I do agree that the keyring argument would not necessarily required if 
> we pass in the key id, but I'll be needing the keyring for secure 
> concatenation. And plan is to move DH-HMAC-CHAP over to keyrings, too.

I'm not sure I understand why authentication move to keyrings requires
that the user is able to bring its own keyring.

> So I'd prefer to keep it.

I'm not against it, but I failing to understand in what situation a user
_must_ send a different keyring to the driver (and not use the nvme
keyring).

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 15/18] nvmet-tcp: enable TLS handshake upcall
  2023-04-03 12:51   ` Sagi Grimberg
@ 2023-04-03 14:05     ` Hannes Reinecke
  0 siblings, 0 replies; 74+ messages in thread
From: Hannes Reinecke @ 2023-04-03 14:05 UTC (permalink / raw)
  To: Sagi Grimberg, Christoph Hellwig
  Cc: Keith Busch, linux-nvme, Chuck Lever, kernel-tls-handshake

On 4/3/23 14:51, Sagi Grimberg wrote:
> 
>> Add functions to start the TLS handshake upcall when
>> the TCP RSAS sectype is set to 'tls1.3'.
> 
> TSAS
> 
Ok.

>>
>> Signed-off-by: Hannes Reincke <hare@suse.de>
>> ---
>>   drivers/nvme/target/configfs.c |  32 +++++++-
>>   drivers/nvme/target/tcp.c      | 135 ++++++++++++++++++++++++++++++++-
>>   2 files changed, 163 insertions(+), 4 deletions(-)
>>
>> diff --git a/drivers/nvme/target/configfs.c 
>> b/drivers/nvme/target/configfs.c
>> index ca66ee6dc153..36fbf6a22d09 100644
>> --- a/drivers/nvme/target/configfs.c
>> +++ b/drivers/nvme/target/configfs.c
>> @@ -159,10 +159,12 @@ static const struct nvmet_type_name_map 
>> nvmet_addr_treq[] = {
>>       { NVMF_TREQ_NOT_REQUIRED,    "not required" },
>>   };
>> +#define NVMET_PORT_TREQ(port) ((port)->disc_addr.treq & 
>> NVME_TREQ_SECURE_CHANNEL_MASK)
> 
> Can you make it a static inline?
> 
Sure.

>> +
>>   static ssize_t nvmet_addr_treq_show(struct config_item *item, char 
>> *page)
>>   {
>> -    u8 treq = to_nvmet_port(item)->disc_addr.treq &
>> -        NVME_TREQ_SECURE_CHANNEL_MASK;
>> +    struct nvmet_port *port = to_nvmet_port(item);
>> +    u8 treq = NVMET_PORT_TREQ(port);
>>       int i;
>>       for (i = 0; i < ARRAY_SIZE(nvmet_addr_treq); i++) {
>> @@ -193,6 +195,17 @@ static ssize_t nvmet_addr_treq_store(struct 
>> config_item *item,
>>       return -EINVAL;
>>   found:
>> +#ifdef CONFIG_NVME_TLS
>> +    if (port->disc_addr.trtype == NVMF_TRTYPE_TCP) {
>> +        if (port->disc_addr.tsas.tcp.sectype != 
>> NVMF_TCP_SECTYPE_TLS13) {
>> +            pr_warn("cannot change TREQ when TLS is not enabled\n");
>> +            return -EINVAL;
>> +        } else if (nvmet_addr_treq[i].type == NVMF_TREQ_NOT_SPECIFIED) {
>> +            pr_warn("cannot set TREQ to 'not specified' when TLS is 
>> enabled\n");
>> +            return -EINVAL;
>> +        }
>> +    }
> 
> Is this code wrong if CONFIG_NVME_TLS is not enabled?
> 
Strictly speaking, no; it just won't do anything except from having a 
different value in the discovery log page.

>> +#endif
>>       treq |= nvmet_addr_treq[i].type;
>>       port->disc_addr.treq = treq;
>>       return count;
>> @@ -373,6 +386,7 @@ static ssize_t nvmet_addr_tsas_store(struct 
>> config_item *item,
>>           const char *page, size_t count)
>>   {
>>       struct nvmet_port *port = to_nvmet_port(item);
>> +    u8 treq = port->disc_addr.treq & ~NVME_TREQ_SECURE_CHANNEL_MASK;
>>       int i;
>>       if (nvmet_is_port_enabled(port, __func__))
>> @@ -391,6 +405,20 @@ static ssize_t nvmet_addr_tsas_store(struct 
>> config_item *item,
>>   found:
>>       nvmet_port_init_tsas_tcp(port, nvmet_addr_tsas_tcp[i].type);
>> +    if (nvmet_addr_tsas_tcp[i].type == NVMF_TCP_SECTYPE_TLS13) {
>> +#ifdef CONFIG_NVME_TLS
> 
> Maybe in the start of the function just do:
> 
>      if (!IS_ENABLED(CONFIG_NVME_TLS)) {
>          pr_err("TLS not supported\n");
>          return -EINVAL;
>      }
> 
> Instead of incorporating it here.
> 
Ok.

>> +        if (NVMET_PORT_TREQ(port) == NVMF_TREQ_NOT_SPECIFIED)
>> +            treq |= NVMF_TREQ_REQUIRED;
>> +        else
>> +            treq |= NVMET_PORT_TREQ(port);
>> +#else
>> +        pr_err("TLS not supported\n");
>> +        return -EINVAL;
>> +#endif
>> +    } else {
>> +        /* Set to 'not specified' if TLS is not enabled */
>> +        treq |= NVMF_TREQ_NOT_SPECIFIED;
>> +    }
>>       return count;
>>   }
>> diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c
>> index 5931971d715f..ebec882120fd 100644
>> --- a/drivers/nvme/target/tcp.c
>> +++ b/drivers/nvme/target/tcp.c
>> @@ -11,6 +11,10 @@
>>   #include <linux/nvme-tcp.h>
>>   #include <net/sock.h>
>>   #include <net/tcp.h>
>> +#ifdef CONFIG_NVME_TLS
>> +#include <net/handshake.h>
> 
> Is net/handshake.h under an ifdef? If so, CONFIG_NVME_TLS should
> select it.
> 
>> +#include <linux/nvme-keyring.h>
> 
> will this include not work if CONFIG_NVME_TLS not work?
> <linux/nvme-auth.h> is not under CONFIG_NVME_AUTH for example.
> 
Hmm. Should. I can remove the ifdefs

>> +#endif
>>   #include <linux/inet.h>
>>   #include <linux/llist.h>
>>   #include <crypto/hash.h>
>> @@ -40,6 +44,16 @@ module_param(idle_poll_period_usecs, int, 0644);
>>   MODULE_PARM_DESC(idle_poll_period_usecs,
>>           "nvmet tcp io_work poll till idle time period in usecs");
>> +#ifdef CONFIG_NVME_TLS
>> +/*
>> + * TLS handshake timeout
>> + */
>> +static int tls_handshake_timeout = 30;
>> +module_param(tls_handshake_timeout, int, 0644);
>> +MODULE_PARM_DESC(tls_handshake_timeout,
>> +         "nvme TLS handshake timeout in seconds (default 30)");
>> +#endif
>> +
>>   #define NVMET_TCP_RECV_BUDGET        8
>>   #define NVMET_TCP_SEND_BUDGET        8
>>   #define NVMET_TCP_IO_WORK_BUDGET    64
>> @@ -130,6 +144,10 @@ struct nvmet_tcp_queue {
>>       bool            data_digest;
>>       struct ahash_request    *snd_hash;
>>       struct ahash_request    *rcv_hash;
>> +#ifdef CONFIG_NVME_TLS
>> +    struct key        *tls_psk;
>> +    struct delayed_work    tls_handshake_work;
>> +#endif
> 
> If these won't be under CONFIG_NVME_TLS will it save a lot of the other
> ifdefs in the code?
> 
Wasn't sure if we always want to have it.
But if we do, sure, things will be easier.

>>       unsigned long           poll_end;
>> @@ -1474,6 +1492,10 @@ static void nvmet_tcp_release_queue_work(struct 
>> work_struct *w)
>>       nvmet_tcp_free_cmds(queue);
>>       if (queue->hdr_digest || queue->data_digest)
>>           nvmet_tcp_free_crypto(queue);
>> +#ifdef CONFIG_NVME_TLS
>> +    if (queue->tls_psk)
>> +        key_put(queue->tls_psk);
> 
> key_put is NULL safe.
> 
>> +#endif
>>       ida_free(&nvmet_tcp_queue_ida, queue->idx);
>>       page = virt_to_head_page(queue->pf_cache.va);
>>       __page_frag_cache_drain(page, queue->pf_cache.pagecnt_bias);
>> @@ -1488,8 +1510,12 @@ static void nvmet_tcp_data_ready(struct sock *sk)
>>       read_lock_bh(&sk->sk_callback_lock);
>>       queue = sk->sk_user_data;
>> -    if (likely(queue))
>> -        queue_work_on(queue_cpu(queue), nvmet_tcp_wq, &queue->io_work);
>> +    if (queue->data_ready)
>> +        queue->data_ready(sk);
>> +    if (likely(queue) &&
>> +        queue->state != NVMET_TCP_Q_TLS_HANDSHAKE)
>> +        queue_work_on(queue_cpu(queue), nvmet_tcp_wq,
>> +                  &queue->io_work);
>>       read_unlock_bh(&sk->sk_callback_lock);
>>   }
>> @@ -1597,6 +1623,89 @@ static int nvmet_tcp_set_queue_sock(struct 
>> nvmet_tcp_queue *queue)
>>       return ret;
>>   }
>> +#ifdef CONFIG_NVME_TLS
>> +static void nvmet_tcp_tls_queue_restart(struct nvmet_tcp_queue *queue)
>> +{
>> +    spin_lock(&queue->state_lock);
>> +    if (queue->state != NVMET_TCP_Q_TLS_HANDSHAKE) {
>> +        pr_warn("queue %d: TLS handshake already completed\n",
>> +            queue->idx);
>> +        spin_unlock(&queue->state_lock);
>> +        return;
>> +    }
>> +    queue->state = NVMET_TCP_Q_CONNECTING;
>> +    spin_unlock(&queue->state_lock);
>> +
>> +    pr_debug("queue %d: restarting queue after TLS handshake\n",
>> +         queue->idx);
>> +    /*
>> +     * Set callbacks after handshake; TLS implementation
>> +     * might have changed the socket callbacks.
>> +     */
>> +    nvmet_tcp_set_queue_sock(queue);
> 
> Maybe fold it into the caller? The name is confusing anyways.
> The queue is not restarted, it is post-configured for lack of
> a better term.
> 
I see what I can do.

>> +}
>> +
>> +static void nvmet_tcp_tls_handshake_done(void *data, int status,
>> +                     key_serial_t peerid)
> 
>                      pskid.
> 
>> +{
>> +    struct nvmet_tcp_queue *queue = data;
>> +
>> +    pr_debug("queue %d: TLS handshake done, key %x, status %d\n",
>> +         queue->idx, peerid, status);
>> +    if (!status) {
>> +        spin_lock(&queue->state_lock);
>> +        queue->tls_psk = key_lookup(peerid);
>> +        if (IS_ERR(queue->tls_psk)) {
>> +            pr_warn("queue %d: TLS key %x not found\n",
>> +                queue->idx, peerid);
>> +            queue->tls_psk = NULL;
> 
> Here you let the timeout take care of it later?

Well, this is a slightly odd case; we get a '0' status but failed
to lookup the key.
_Technically_ we should be able to continue, but wasn't sure if I should.

But in the light of the key rotation discussion I probably should; new 
keys (and keyrings) might be provided at any time, so I might hit that 
case here.

Will be updating the code.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		           Kernel Storage Architect
hare@suse.de			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Frankenstr. 146, 90461 Nürnberg
Managing Directors: I. Totev, A. Myers, A. McDonald, M. B. Moerman
(HRB 36809, AG Nürnberg)


^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 12/18] nvme-fabrics: parse options 'keyring' and 'tls_key'
  2023-04-03 13:07           ` Sagi Grimberg
@ 2023-04-03 14:11             ` Hannes Reinecke
  2023-04-03 16:13               ` Sagi Grimberg
  0 siblings, 1 reply; 74+ messages in thread
From: Hannes Reinecke @ 2023-04-03 14:11 UTC (permalink / raw)
  To: Sagi Grimberg, Christoph Hellwig
  Cc: Keith Busch, linux-nvme, Chuck Lever, kernel-tls-handshake

On 4/3/23 15:07, Sagi Grimberg wrote:
> 
>>>>>> Parse the fabrics options 'keyring' and 'tls_key' and store the
>>>>>> referenced keys in the options structure.
>>>>>
>>>>> Can you explain the reasoning to why a user need to pass a keyring
>>>>> given that we already set up one?
>>>>>
>>>> Choice.
>>>> With a single keyring we can only have a single identity.
>>>> But there might be configurations where we want to have different PSKs
>>>> for the same identity (eg for key rotation).
>>>
>>> How do you expect that rotation would work with this?
>>>
>> The user creates a new keyring, adds the necessary keys to it, and then
>> either
>> - updates the keyring for nvmet (via configfs)
>> or
>> - reconnects using the new keyring
> 
> And the user cannot update the nvme keyring?
> 
Sure, he can. All I'm saying is that he might choose to create a 
different one (Different tenants, setups, what do I know).

>>> How does nvmet handle a non-nvme keyring?
>>>
>> There is a configfs attribute 'param_keyring', allowing you to use a 
>> different keyring (the nvme one is just the default).
> 
> OK.
> 
>>>> With this option we can prepare a new keyring, and use that instead 
>>>> of the old one.
>>>
>>> On an existing controller?
>>>
>> Sure; when we are updating the keyring and the key id the controller 
>> will use that after the next reset.
>> Much like we do for DH-CHAP nowadays.
>>
>>>> (And it really doesn't add much complexity...)
>>>
>>> I know, it just adds one more argument, and I want to understand if it
>>> is really needed.
>>
>> I do agree that the keyring argument would not necessarily required if 
>> we pass in the key id, but I'll be needing the keyring for secure 
>> concatenation. And plan is to move DH-HMAC-CHAP over to keyrings, too.
> 
> I'm not sure I understand why authentication move to keyrings requires
> that the user is able to bring its own keyring.
> 
>> So I'd prefer to keep it.
> 
> I'm not against it, but I failing to understand in what situation a user
> _must_ send a different keyring to the driver (and not use the nvme
> keyring).

Oh, there is no 'must'. He _can_. And under certain circumstances
(in a multi-tenant environment, say) there are reasons to keep the keys 
separate into several keyrings.
Bit like the '--key' parameter; the code will try to select the 
'default' key, but if the user wants to use a different one by all 
means, do.
Same for the keyring; we do have the default keyring, but if the user 
for whatever reason doesn't want to use it we're giving him the 
possibility for that.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		           Kernel Storage Architect
hare@suse.de			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Frankenstr. 146, 90461 Nürnberg
Managing Directors: I. Totev, A. Myers, A. McDonald, M. B. Moerman
(HRB 36809, AG Nürnberg)


^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 18/18] nvmet-tcp: add configfs attribute 'param_keyring'
  2023-04-03 13:03   ` Sagi Grimberg
@ 2023-04-03 14:13     ` Hannes Reinecke
  2023-04-03 15:53       ` Sagi Grimberg
  0 siblings, 1 reply; 74+ messages in thread
From: Hannes Reinecke @ 2023-04-03 14:13 UTC (permalink / raw)
  To: Sagi Grimberg, Christoph Hellwig
  Cc: Keith Busch, linux-nvme, Chuck Lever, kernel-tls-handshake

On 4/3/23 15:03, Sagi Grimberg wrote:
> 
>> Add a configfs attribute to list and change the default keyring.
> 
> is the keyring an attribute of a port? or a subsystem? Why the choice
> of making it per port?
> 
As per discussion at fmds each port might want to use different credentials.
We can move it to subsystems if we decide we don't want to support that 
use-case.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		           Kernel Storage Architect
hare@suse.de			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Frankenstr. 146, 90461 Nürnberg
Managing Directors: I. Totev, A. Myers, A. McDonald, M. B. Moerman
(HRB 36809, AG Nürnberg)


^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 10/18] nvme-tcp: fixup send workflow for kTLS
  2023-04-03 12:20         ` Sagi Grimberg
@ 2023-04-03 14:59           ` Jakub Kicinski
  2023-04-03 15:51             ` Sagi Grimberg
  0 siblings, 1 reply; 74+ messages in thread
From: Jakub Kicinski @ 2023-04-03 14:59 UTC (permalink / raw)
  To: Sagi Grimberg
  Cc: Hannes Reinecke, Christoph Hellwig, Boris Pismenny,
	john.fastabend, Paolo Abeni, Keith Busch, linux-nvme,
	Chuck Lever, kernel-tls-handshake, netdev

On Mon, 3 Apr 2023 15:20:13 +0300 Sagi Grimberg wrote:
> >> Some of the flags are call specific, others may be internal to the
> >> networking stack (e.g. the DECRYPTED flag). Old protocols didn't do
> >> any validation because people coded more haphazardly in the 90s.
> >> This lack of validation is a major source of technical debt :(  
> > 
> > A-ha. So what is the plan?
> > Should the stack validate flags?
> > And should the rules for validating be the same for all protocols?  
> 
> MSG_SENDPAGE_NOTLAST is not an internal flag, I thought it was
> essentially similar semantics to MSG_MORE but for sendpage. It'd
> be great if this can be allowed in tls (again, at the very least
> don't fail but continue as if it wasn't passed).

.. but.. MSG_SENDPAGE_NOTLAST is supported in TLS, isn't it?
Why are we talking about it?

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 10/18] nvme-tcp: fixup send workflow for kTLS
  2023-04-03 14:59           ` Jakub Kicinski
@ 2023-04-03 15:51             ` Sagi Grimberg
  2023-04-03 18:48               ` Jakub Kicinski
  0 siblings, 1 reply; 74+ messages in thread
From: Sagi Grimberg @ 2023-04-03 15:51 UTC (permalink / raw)
  To: Jakub Kicinski
  Cc: Hannes Reinecke, Christoph Hellwig, Boris Pismenny,
	john.fastabend, Paolo Abeni, Keith Busch, linux-nvme,
	Chuck Lever, kernel-tls-handshake, netdev


>>>> Some of the flags are call specific, others may be internal to the
>>>> networking stack (e.g. the DECRYPTED flag). Old protocols didn't do
>>>> any validation because people coded more haphazardly in the 90s.
>>>> This lack of validation is a major source of technical debt :(
>>>
>>> A-ha. So what is the plan?
>>> Should the stack validate flags?
>>> And should the rules for validating be the same for all protocols?
>>
>> MSG_SENDPAGE_NOTLAST is not an internal flag, I thought it was
>> essentially similar semantics to MSG_MORE but for sendpage. It'd
>> be great if this can be allowed in tls (again, at the very least
>> don't fail but continue as if it wasn't passed).
> 
> .. but.. MSG_SENDPAGE_NOTLAST is supported in TLS, isn't it?
> Why are we talking about it?

Ah, right.

What I'm assuming that Hannes is tripping on is that tls does
not accept when this flag is sent to sock_no_sendpage, which
is simply calling sendmsg. TLS will not accept this flag when
passed to sendmsg IIUC.

Today the rough logic in nvme send path is:

	if (more_coming(queue)) {
		flags = MSG_MORE | MSG_SENDPAGE_NOTLAST;
	} else {
		flags = MSG_EOR;
	}

	if (!sendpage_ok(page)) {
		kernel_sendpage();
	} else {
		sock_no_sendpage();
	}

This pattern (note that sock_no_sednpage was added later following bug
reports where nvme attempted to sendpage a slab allocated page), is
perfectly acceptable with normal sockets, but not with TLS.

So there are two options:
1. have tls accept MSG_SENDPAGE_NOTLAST in sendmsg (called from
    sock_no_sendpage)
2. Make nvme set MSG_SENDPAGE_NOTLAST only when calling
    kernel_sendpage and clear it when calling sock_no_sendpage

If you say that MSG_SENDPAGE_NOTLAST must be cleared when calling
sock_no_sendpage and it is a bug that it isn't enforced for normal tcp
sockets, then we need to change nvme, but I did not find
any documentation that indicates it, and right now, normal sockets
behave differently than tls sockets (wrt this flag in particular).

Hope this clarifies.

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 18/18] nvmet-tcp: add configfs attribute 'param_keyring'
  2023-04-03 14:13     ` Hannes Reinecke
@ 2023-04-03 15:53       ` Sagi Grimberg
  2023-04-14 10:30         ` Hannes Reinecke
  0 siblings, 1 reply; 74+ messages in thread
From: Sagi Grimberg @ 2023-04-03 15:53 UTC (permalink / raw)
  To: Hannes Reinecke, Christoph Hellwig
  Cc: Keith Busch, linux-nvme, Chuck Lever, kernel-tls-handshake


>>> Add a configfs attribute to list and change the default keyring.
>>
>> is the keyring an attribute of a port? or a subsystem? Why the choice
>> of making it per port?
>>
> As per discussion at fmds each port might want to use different 
> credentials.
> We can move it to subsystems if we decide we don't want to support that 
> use-case.

I think we should move it to the subsystem in Linux.

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 12/18] nvme-fabrics: parse options 'keyring' and 'tls_key'
  2023-04-03 14:11             ` Hannes Reinecke
@ 2023-04-03 16:13               ` Sagi Grimberg
  0 siblings, 0 replies; 74+ messages in thread
From: Sagi Grimberg @ 2023-04-03 16:13 UTC (permalink / raw)
  To: Hannes Reinecke, Christoph Hellwig
  Cc: Keith Busch, linux-nvme, Chuck Lever, kernel-tls-handshake



On 4/3/23 17:11, Hannes Reinecke wrote:
> On 4/3/23 15:07, Sagi Grimberg wrote:
>>
>>>>>>> Parse the fabrics options 'keyring' and 'tls_key' and store the
>>>>>>> referenced keys in the options structure.
>>>>>>
>>>>>> Can you explain the reasoning to why a user need to pass a keyring
>>>>>> given that we already set up one?
>>>>>>
>>>>> Choice.
>>>>> With a single keyring we can only have a single identity.
>>>>> But there might be configurations where we want to have different PSKs
>>>>> for the same identity (eg for key rotation).
>>>>
>>>> How do you expect that rotation would work with this?
>>>>
>>> The user creates a new keyring, adds the necessary keys to it, and then
>>> either
>>> - updates the keyring for nvmet (via configfs)
>>> or
>>> - reconnects using the new keyring
>>
>> And the user cannot update the nvme keyring?
>>
> Sure, he can. All I'm saying is that he might choose to create a 
> different one (Different tenants, setups, what do I know).
> 
>>>> How does nvmet handle a non-nvme keyring?
>>>>
>>> There is a configfs attribute 'param_keyring', allowing you to use a 
>>> different keyring (the nvme one is just the default).
>>
>> OK.
>>
>>>>> With this option we can prepare a new keyring, and use that instead 
>>>>> of the old one.
>>>>
>>>> On an existing controller?
>>>>
>>> Sure; when we are updating the keyring and the key id the controller 
>>> will use that after the next reset.
>>> Much like we do for DH-CHAP nowadays.
>>>
>>>>> (And it really doesn't add much complexity...)
>>>>
>>>> I know, it just adds one more argument, and I want to understand if it
>>>> is really needed.
>>>
>>> I do agree that the keyring argument would not necessarily required 
>>> if we pass in the key id, but I'll be needing the keyring for secure 
>>> concatenation. And plan is to move DH-HMAC-CHAP over to keyrings, too.
>>
>> I'm not sure I understand why authentication move to keyrings requires
>> that the user is able to bring its own keyring.
>>
>>> So I'd prefer to keep it.
>>
>> I'm not against it, but I failing to understand in what situation a user
>> _must_ send a different keyring to the driver (and not use the nvme
>> keyring).
> 
> Oh, there is no 'must'. He _can_. And under certain circumstances
> (in a multi-tenant environment, say) there are reasons to keep the keys 
> separate into several keyrings.
> Bit like the '--key' parameter; the code will try to select the 
> 'default' key, but if the user wants to use a different one by all 
> means, do.
> Same for the keyring; we do have the default keyring, but if the user 
> for whatever reason doesn't want to use it we're giving him the 
> possibility for that.

I'm not overly keen to overload the argument space without proper
real-life justification. But maybe it does make sense to allow users
to store their secrets in separate keyrings...

So I'm not really opposed to this one.


^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 10/18] nvme-tcp: fixup send workflow for kTLS
  2023-04-03 15:51             ` Sagi Grimberg
@ 2023-04-03 18:48               ` Jakub Kicinski
  2023-04-03 22:36                 ` Sagi Grimberg
  0 siblings, 1 reply; 74+ messages in thread
From: Jakub Kicinski @ 2023-04-03 18:48 UTC (permalink / raw)
  To: Sagi Grimberg
  Cc: Hannes Reinecke, Christoph Hellwig, Boris Pismenny,
	john.fastabend, Paolo Abeni, Keith Busch, linux-nvme,
	Chuck Lever, kernel-tls-handshake, netdev

On Mon, 3 Apr 2023 18:51:09 +0300 Sagi Grimberg wrote:
> What I'm assuming that Hannes is tripping on is that tls does
> not accept when this flag is sent to sock_no_sendpage, which
> is simply calling sendmsg. TLS will not accept this flag when
> passed to sendmsg IIUC.
> 
> Today the rough logic in nvme send path is:
> 
> 	if (more_coming(queue)) {
> 		flags = MSG_MORE | MSG_SENDPAGE_NOTLAST;
> 	} else {
> 		flags = MSG_EOR;
> 	}
> 
> 	if (!sendpage_ok(page)) {
> 		kernel_sendpage();
> 	} else {
> 		sock_no_sendpage();
> 	}
> 
> This pattern (note that sock_no_sednpage was added later following bug
> reports where nvme attempted to sendpage a slab allocated page), is
> perfectly acceptable with normal sockets, but not with TLS.
> 
> So there are two options:
> 1. have tls accept MSG_SENDPAGE_NOTLAST in sendmsg (called from
>     sock_no_sendpage)
> 2. Make nvme set MSG_SENDPAGE_NOTLAST only when calling
>     kernel_sendpage and clear it when calling sock_no_sendpage
> 
> If you say that MSG_SENDPAGE_NOTLAST must be cleared when calling
> sock_no_sendpage and it is a bug that it isn't enforced for normal tcp
> sockets, then we need to change nvme, but I did not find
> any documentation that indicates it, and right now, normal sockets
> behave differently than tls sockets (wrt this flag in particular).
> 
> Hope this clarifies.

Oh right, it does, the context evaporated from my head over the weekend.

IMHO it's best if the caller passes the right flags. The semantics of
MSG_MORE vs NOTLAST are quite murky and had already caused bugs in the
past :(

See commit d452d48b9f8b ("tls: prevent oversized sendfile() hangs by
ignoring MSG_MORE")

Alternatively we could have sock_no_sendpage drop NOTLAST to help
all protos. But if we consider sendfile behavior as the standard
simply clearing it isn't right, it should be a:

	more = (flags & (MORE | NOTLAST)) == MORE | NOTLAST
	flags &= ~(MORE | NOTLAST)
	if (more)
		flags |= MORE

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 10/18] nvme-tcp: fixup send workflow for kTLS
  2023-04-03 18:48               ` Jakub Kicinski
@ 2023-04-03 22:36                 ` Sagi Grimberg
  0 siblings, 0 replies; 74+ messages in thread
From: Sagi Grimberg @ 2023-04-03 22:36 UTC (permalink / raw)
  To: Jakub Kicinski
  Cc: Hannes Reinecke, Christoph Hellwig, Boris Pismenny,
	john.fastabend, Paolo Abeni, Keith Busch, linux-nvme,
	Chuck Lever, kernel-tls-handshake, netdev



On 4/3/23 21:48, Jakub Kicinski wrote:
> On Mon, 3 Apr 2023 18:51:09 +0300 Sagi Grimberg wrote:
>> What I'm assuming that Hannes is tripping on is that tls does
>> not accept when this flag is sent to sock_no_sendpage, which
>> is simply calling sendmsg. TLS will not accept this flag when
>> passed to sendmsg IIUC.
>>
>> Today the rough logic in nvme send path is:
>>
>> 	if (more_coming(queue)) {
>> 		flags = MSG_MORE | MSG_SENDPAGE_NOTLAST;
>> 	} else {
>> 		flags = MSG_EOR;
>> 	}
>>
>> 	if (!sendpage_ok(page)) {
>> 		kernel_sendpage();
>> 	} else {
>> 		sock_no_sendpage();
>> 	}
>>
>> This pattern (note that sock_no_sednpage was added later following bug
>> reports where nvme attempted to sendpage a slab allocated page), is
>> perfectly acceptable with normal sockets, but not with TLS.
>>
>> So there are two options:
>> 1. have tls accept MSG_SENDPAGE_NOTLAST in sendmsg (called from
>>      sock_no_sendpage)
>> 2. Make nvme set MSG_SENDPAGE_NOTLAST only when calling
>>      kernel_sendpage and clear it when calling sock_no_sendpage
>>
>> If you say that MSG_SENDPAGE_NOTLAST must be cleared when calling
>> sock_no_sendpage and it is a bug that it isn't enforced for normal tcp
>> sockets, then we need to change nvme, but I did not find
>> any documentation that indicates it, and right now, normal sockets
>> behave differently than tls sockets (wrt this flag in particular).
>>
>> Hope this clarifies.
> 
> Oh right, it does, the context evaporated from my head over the weekend.
> 
> IMHO it's best if the caller passes the right flags. The semantics of
> MSG_MORE vs NOTLAST are quite murky and had already caused bugs in the
> past :(
> 
> See commit d452d48b9f8b ("tls: prevent oversized sendfile() hangs by
> ignoring MSG_MORE")

Well, that is fine with me. This may change anyways with
the new MSG_SPLICE_PAGES from David.

> Alternatively we could have sock_no_sendpage drop NOTLAST to help
> all protos. But if we consider sendfile behavior as the standard
> simply clearing it isn't right, it should be a:
> 
> 	more = (flags & (MORE | NOTLAST)) == MORE | NOTLAST
> 	flags &= ~(MORE | NOTLAST)
> 	if (more)
> 		flags |= MORE

I don't think this would be the best option. Requiring callers
to clear NOTLAST if not calling sendpages is reasonable, but we
need to have this consistent. And also fix this pattern for the
rest of the kernel socket consumers that use this flag.

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 18/18] nvmet-tcp: add configfs attribute 'param_keyring'
  2023-04-03 15:53       ` Sagi Grimberg
@ 2023-04-14 10:30         ` Hannes Reinecke
  2023-04-17 13:50           ` Sagi Grimberg
  0 siblings, 1 reply; 74+ messages in thread
From: Hannes Reinecke @ 2023-04-14 10:30 UTC (permalink / raw)
  To: Sagi Grimberg, Christoph Hellwig
  Cc: Keith Busch, linux-nvme, Chuck Lever, kernel-tls-handshake

On 4/3/23 17:53, Sagi Grimberg wrote:
> 
>>>> Add a configfs attribute to list and change the default keyring.
>>>
>>> is the keyring an attribute of a port? or a subsystem? Why the choice
>>> of making it per port?
>>>
>> As per discussion at fmds each port might want to use different 
>> credentials.
>> We can move it to subsystems if we decide we don't want to support 
>> that use-case.
> 
> I think we should move it to the subsystem in Linux.
> 
Easier said than done.

We can only fetch the subsystem once we have the NQN, which we will
know after the connect command.
But out of necessity the connect command will only be transferred
after TLS handshake completed, so there is not subsystem NQN allowing
us to figure out the correct subsystem.

So we'll have to keep it at the port level, I'm afraid.

Cheers,

Hannes


^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 18/18] nvmet-tcp: add configfs attribute 'param_keyring'
  2023-04-14 10:30         ` Hannes Reinecke
@ 2023-04-17 13:50           ` Sagi Grimberg
  2023-04-17 14:01             ` Hannes Reinecke
  0 siblings, 1 reply; 74+ messages in thread
From: Sagi Grimberg @ 2023-04-17 13:50 UTC (permalink / raw)
  To: Hannes Reinecke, Christoph Hellwig
  Cc: Keith Busch, linux-nvme, Chuck Lever, kernel-tls-handshake


>>>>> Add a configfs attribute to list and change the default keyring.
>>>>
>>>> is the keyring an attribute of a port? or a subsystem? Why the choice
>>>> of making it per port?
>>>>
>>> As per discussion at fmds each port might want to use different 
>>> credentials.
>>> We can move it to subsystems if we decide we don't want to support 
>>> that use-case.
>>
>> I think we should move it to the subsystem in Linux.
>>
> Easier said than done.
> 
> We can only fetch the subsystem once we have the NQN, which we will
> know after the connect command.
> But out of necessity the connect command will only be transferred
> after TLS handshake completed, so there is not subsystem NQN allowing
> us to figure out the correct subsystem.
> 
> So we'll have to keep it at the port level, I'm afraid.

Question, why does nvmet needs to support user defined keyrings?
I understood your argument for the host, but why should the target
support it?

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 18/18] nvmet-tcp: add configfs attribute 'param_keyring'
  2023-04-17 13:50           ` Sagi Grimberg
@ 2023-04-17 14:01             ` Hannes Reinecke
  2023-04-17 15:12               ` Sagi Grimberg
  0 siblings, 1 reply; 74+ messages in thread
From: Hannes Reinecke @ 2023-04-17 14:01 UTC (permalink / raw)
  To: Sagi Grimberg, Christoph Hellwig
  Cc: Keith Busch, linux-nvme, Chuck Lever, kernel-tls-handshake

On 4/17/23 15:50, Sagi Grimberg wrote:
> 
>>>>>> Add a configfs attribute to list and change the default keyring.
>>>>>
>>>>> is the keyring an attribute of a port? or a subsystem? Why the choice
>>>>> of making it per port?
>>>>>
>>>> As per discussion at fmds each port might want to use different 
>>>> credentials.
>>>> We can move it to subsystems if we decide we don't want to support 
>>>> that use-case.
>>>
>>> I think we should move it to the subsystem in Linux.
>>>
>> Easier said than done.
>>
>> We can only fetch the subsystem once we have the NQN, which we will
>> know after the connect command.
>> But out of necessity the connect command will only be transferred
>> after TLS handshake completed, so there is not subsystem NQN allowing
>> us to figure out the correct subsystem.
>>
>> So we'll have to keep it at the port level, I'm afraid.
> 
> Question, why does nvmet needs to support user defined keyrings?
> I understood your argument for the host, but why should the target
> support it?

Hmm. Originally I implemented it to be similar to the host 
configuration, but from the implementation we don't _actually_ need it 
as we select the key from the entire key store.
Having it is just a nice way of avoiding having to have a daemon 
configuration.
So if you insist we can drop it for the initial implementation.

Cheers,

Hannes



^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 18/18] nvmet-tcp: add configfs attribute 'param_keyring'
  2023-04-17 14:01             ` Hannes Reinecke
@ 2023-04-17 15:12               ` Sagi Grimberg
  0 siblings, 0 replies; 74+ messages in thread
From: Sagi Grimberg @ 2023-04-17 15:12 UTC (permalink / raw)
  To: Hannes Reinecke, Christoph Hellwig
  Cc: Keith Busch, linux-nvme, Chuck Lever, kernel-tls-handshake


>>>>>>> Add a configfs attribute to list and change the default keyring.
>>>>>>
>>>>>> is the keyring an attribute of a port? or a subsystem? Why the choice
>>>>>> of making it per port?
>>>>>>
>>>>> As per discussion at fmds each port might want to use different 
>>>>> credentials.
>>>>> We can move it to subsystems if we decide we don't want to support 
>>>>> that use-case.
>>>>
>>>> I think we should move it to the subsystem in Linux.
>>>>
>>> Easier said than done.
>>>
>>> We can only fetch the subsystem once we have the NQN, which we will
>>> know after the connect command.
>>> But out of necessity the connect command will only be transferred
>>> after TLS handshake completed, so there is not subsystem NQN allowing
>>> us to figure out the correct subsystem.
>>>
>>> So we'll have to keep it at the port level, I'm afraid.
>>
>> Question, why does nvmet needs to support user defined keyrings?
>> I understood your argument for the host, but why should the target
>> support it?
> 
> Hmm. Originally I implemented it to be similar to the host 
> configuration, but from the implementation we don't _actually_ need it 
> as we select the key from the entire key store.

Thanks for confirming.

> Having it is just a nice way of avoiding having to have a daemon 
> configuration.

I'm sorry, but I don't understand this statement. Can you clarify?

> So if you insist we can drop it for the initial implementation.

If there is a use-case, we can add it. But I'm trying to understand what
is it.

^ permalink raw reply	[flat|nested] 74+ messages in thread

* [PATCH 02/18] nvme-keyring: define a 'psk' keytype
  2023-04-17 13:02 [PATCHv3 00/18] nvme: In-kernel TLS support for TCP Hannes Reinecke
@ 2023-04-17 13:02 ` Hannes Reinecke
  0 siblings, 0 replies; 74+ messages in thread
From: Hannes Reinecke @ 2023-04-17 13:02 UTC (permalink / raw)
  To: Sagi Grimberg
  Cc: Christoph Hellwig, Keith Busch, linux-nvme, Chuck Lever,
	kernel-tls-handshake, Hannes Reinecke

Define a 'psk' keytype to hold the NVMe TLS PSKs.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/nvme/common/keyring.c | 94 +++++++++++++++++++++++++++++++++++
 1 file changed, 94 insertions(+)

diff --git a/drivers/nvme/common/keyring.c b/drivers/nvme/common/keyring.c
index 5cf64b278119..494dd365052e 100644
--- a/drivers/nvme/common/keyring.c
+++ b/drivers/nvme/common/keyring.c
@@ -8,6 +8,8 @@
 #include <linux/key-type.h>
 #include <keys/user-type.h>
 #include <linux/nvme.h>
+#include <linux/nvme-tcp.h>
+#include <linux/nvme-keyring.h>
 
 static struct key *nvme_keyring;
 
@@ -17,8 +19,94 @@ key_serial_t nvme_keyring_id(void)
 }
 EXPORT_SYMBOL_GPL(nvme_keyring_id);
 
+static void nvme_tls_psk_describe(const struct key *key, struct seq_file *m)
+{
+	seq_puts(m, key->description);
+	seq_printf(m, ": %u", key->datalen);
+}
+
+static bool nvme_tls_psk_match(const struct key *key,
+			       const struct key_match_data *match_data)
+{
+	const char *match_id;
+	size_t match_len;
+
+	if (!key->description) {
+		pr_debug("%s: no key description\n", __func__);
+		return false;
+	}
+	match_len = strlen(key->description);
+	pr_debug("%s: id %s len %zd\n", __func__, key->description, match_len);
+
+	if (!match_data->raw_data) {
+		pr_debug("%s: no match data\n", __func__);
+		return false;
+	}
+	match_id = match_data->raw_data;
+	pr_debug("%s: match '%s' '%s' len %zd\n",
+		 __func__, match_id, key->description, match_len);
+	return !memcmp(key->description, match_id, match_len);
+}
+
+static int nvme_tls_psk_match_preparse(struct key_match_data *match_data)
+{
+	match_data->lookup_type = KEYRING_SEARCH_LOOKUP_ITERATE;
+	match_data->cmp = nvme_tls_psk_match;
+	return 0;
+}
+
+static struct key_type nvme_tls_psk_key_type = {
+	.name           = "psk",
+	.flags          = KEY_TYPE_NET_DOMAIN,
+	.preparse       = user_preparse,
+	.free_preparse  = user_free_preparse,
+	.match_preparse = nvme_tls_psk_match_preparse,
+	.instantiate    = generic_key_instantiate,
+	.revoke         = user_revoke,
+	.destroy        = user_destroy,
+	.describe       = nvme_tls_psk_describe,
+	.read           = user_read,
+};
+
+static struct key *nvme_tls_psk_lookup(struct key *keyring,
+		const char *hostnqn, const char *subnqn,
+		int hmac, bool generated)
+{
+	char *identity;
+	size_t identity_len = (NVMF_NQN_SIZE) * 2 + 11;
+	key_ref_t keyref;
+	key_serial_t keyring_id;
+
+	identity = kzalloc(identity_len, GFP_KERNEL);
+	if (!identity)
+		return ERR_PTR(-ENOMEM);
+
+	snprintf(identity, identity_len, "NVMe0%c%02d %s %s",
+		 generated ? 'G' : 'R', hmac, hostnqn, subnqn);
+
+	if (!keyring)
+		keyring = nvme_keyring;
+	keyring_id = key_serial(keyring);
+	pr_debug("keyring %x lookup tls psk '%s'\n",
+		 keyring_id, identity);
+	keyref = keyring_search(make_key_ref(keyring, true),
+				&nvme_tls_psk_key_type,
+				identity, false);
+	if (IS_ERR(keyref)) {
+		pr_debug("lookup tls psk '%s' failed, error %ld\n",
+			 identity, PTR_ERR(keyref));
+		kfree(identity);
+		return ERR_PTR(-ENOKEY);
+	}
+	kfree(identity);
+
+	return key_ref_to_ptr(keyref);
+}
+
 int nvme_keyring_init(void)
 {
+	int err;
+
 	nvme_keyring = keyring_alloc(".nvme",
 				     GLOBAL_ROOT_UID, GLOBAL_ROOT_GID,
 				     current_cred(),
@@ -28,12 +116,18 @@ int nvme_keyring_init(void)
 	if (IS_ERR(nvme_keyring))
 		return PTR_ERR(nvme_keyring);
 
+	err = register_key_type(&nvme_tls_psk_key_type);
+	if (err) {
+		key_put(nvme_keyring);
+		return err;
+	}
 	return 0;
 }
 EXPORT_SYMBOL_GPL(nvme_keyring_init);
 
 void nvme_keyring_exit(void)
 {
+	unregister_key_type(&nvme_tls_psk_key_type);
 	key_revoke(nvme_keyring);
 	key_put(nvme_keyring);
 }
-- 
2.35.3


^ permalink raw reply related	[flat|nested] 74+ messages in thread

* Re: [PATCH 02/18] nvme-keyring: define a 'psk' keytype
  2023-03-22  8:38     ` Hannes Reinecke
@ 2023-03-22  8:49       ` Sagi Grimberg
  0 siblings, 0 replies; 74+ messages in thread
From: Sagi Grimberg @ 2023-03-22  8:49 UTC (permalink / raw)
  To: Hannes Reinecke, Christoph Hellwig
  Cc: Keith Busch, linux-nvme, Chuck Lever, kernel-tls-handshake


>> On 3/21/23 14:43, Hannes Reinecke wrote:
>>> Define a 'psk' keytype to hold the NVMe TLS PSKs.
>>>
>>> Signed-off-by: Hannes Reinecke <hare@suse.de>
>>> ---
>>>   drivers/nvme/common/keyring.c | 96 +++++++++++++++++++++++++++++++++++
>>>   include/linux/nvme-keyring.h  |  8 +++
>>>   2 files changed, 104 insertions(+)
>>>
>>> diff --git a/drivers/nvme/common/keyring.c 
>>> b/drivers/nvme/common/keyring.c
>>> index 3a6e8a0b38e2..6cbb9d66e0f6 100644
>>> --- a/drivers/nvme/common/keyring.c
>>> +++ b/drivers/nvme/common/keyring.c
>>> @@ -11,6 +11,96 @@
>>>   static struct key *nvme_keyring;
>>> +key_serial_t nvme_keyring_id(void)
>>> +{
>>> +    return nvme_keyring->serial;
>>> +}
>>> +EXPORT_SYMBOL_GPL(nvme_keyring_id);
>>> +
>>> +static void nvme_tls_psk_describe(const struct key *key, struct 
>>> seq_file *m)
>>> +{
>>> +    seq_puts(m, key->description);
>>> +    seq_printf(m, ": %u", key->datalen);
>>> +}
>>> +
>>> +static bool nvme_tls_psk_match(const struct key *key,
>>> +                   const struct key_match_data *match_data)
>>> +{
>>> +    const char *match_id;
>>> +    size_t match_len;
>>> +
>>> +    if (!key->description) {
>>> +        pr_debug("%s: no key description\n", __func__);
>>> +        return false;
>>> +    }
>>> +    match_len = strlen(key->description);
>>> +    pr_debug("%s: id %s len %zd\n", __func__, key->description, 
>>> match_len);
>>> +
>>> +    if (!match_data->raw_data) {
>>> +        pr_debug("%s: no match data\n", __func__);
>>> +        return false;
>>> +    }
>>> +    match_id = match_data->raw_data;
>>> +    pr_debug("%s: match '%s' '%s' len %lu\n",
>>> +         __func__, match_id, key->description, match_len);
>>> +    return !memcmp(key->description, match_id, match_len);
>>> +}
>>> +
>>> +static int nvme_tls_psk_match_preparse(struct key_match_data 
>>> *match_data)
>>> +{
>>> +    match_data->lookup_type = KEYRING_SEARCH_LOOKUP_ITERATE;
>>> +    match_data->cmp = nvme_tls_psk_match;
>>> +    return 0;
>>> +}
>>> +
>>> +static struct key_type nvme_tls_psk_key_type = {
>>> +    .name           = "psk",
>>> +    .flags          = KEY_TYPE_NET_DOMAIN,
>>> +    .preparse       = user_preparse,
>>> +    .free_preparse  = user_free_preparse,
>>> +    .match_preparse = nvme_tls_psk_match_preparse,
>>> +    .instantiate    = generic_key_instantiate,
>>> +    .revoke         = user_revoke,
>>> +    .destroy        = user_destroy,
>>> +    .describe       = nvme_tls_psk_describe,
>>> +    .read           = user_read,
>>> +};
>>> +
>>
>> Hannes, can you please provide a documentation section
>> to this function? most importantly 'generated' argument.
>>
> Sure. There are two types of PSK types specified in the NVMe TCP spec;
> a 'retained' one which has to be provided by the admin, and a 
> 'generated' one which is derived from the shared key material from 
> DH-HMAC-CHAP when doing a secure channel concatenation.
> And each type has two possible hash algorithms (SHA-256 and SHA-384),
> resulting in 4 possible PSKs.
> 
> And indeed, the secure channel concatenation bits are missing as we're 
> still hashing out details at the fmds group.
> I do have a patchset for that, though, but decided not to include it in 
> this submission as it'll increase the patchset even more.
> Can do if you like ...

I'd prefer to split the secure channel concatenation out of this.
But it would be good to give some context in the code.

Also the bool is confusing, can you make this an enumeration that
translates to 'G' or 'R' and pass that in?

> 
> But will be updating the documentation.
> 
>>> +struct key *nvme_tls_psk_lookup(key_ref_t keyring,
>>> +        const char *hostnqn, const char *subnqn,
>>> +        int hmac, bool generated)
>>> +{
>>> +    char *identity;
>>> +    size_t identity_len = (NVMF_NQN_SIZE) * 2 + 11;
>>> +    key_ref_t keyref;
>>> +    key_serial_t keyring_id;
>>> +
>>> +    identity = kzalloc(identity_len, GFP_KERNEL);
>>> +    if (!identity)
>>> +        return ERR_PTR(-ENOMEM);
>>> +
>>> +    snprintf(identity, identity_len, "NVMe0%c%02d %s %s",
>>> +         generated ? 'G' : 'R', hmac, hostnqn, subnqn);
>>
>> Is that a format that is expected from userspace to produce?
>>
> Yes. See
> NVMe-TCP 1.0a section 3.6.1.3 "TLS PSK and PSK Identity Derivation"
> for details.
> 

Yes, I see it now. Thanks.

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 02/18] nvme-keyring: define a 'psk' keytype
  2023-03-22  8:29   ` Sagi Grimberg
@ 2023-03-22  8:38     ` Hannes Reinecke
  2023-03-22  8:49       ` Sagi Grimberg
  0 siblings, 1 reply; 74+ messages in thread
From: Hannes Reinecke @ 2023-03-22  8:38 UTC (permalink / raw)
  To: Sagi Grimberg, Christoph Hellwig
  Cc: Keith Busch, linux-nvme, Chuck Lever, kernel-tls-handshake

On 3/22/23 09:29, Sagi Grimberg wrote:
> 
> 
> On 3/21/23 14:43, Hannes Reinecke wrote:
>> Define a 'psk' keytype to hold the NVMe TLS PSKs.
>>
>> Signed-off-by: Hannes Reinecke <hare@suse.de>
>> ---
>>   drivers/nvme/common/keyring.c | 96 +++++++++++++++++++++++++++++++++++
>>   include/linux/nvme-keyring.h  |  8 +++
>>   2 files changed, 104 insertions(+)
>>
>> diff --git a/drivers/nvme/common/keyring.c 
>> b/drivers/nvme/common/keyring.c
>> index 3a6e8a0b38e2..6cbb9d66e0f6 100644
>> --- a/drivers/nvme/common/keyring.c
>> +++ b/drivers/nvme/common/keyring.c
>> @@ -11,6 +11,96 @@
>>   static struct key *nvme_keyring;
>> +key_serial_t nvme_keyring_id(void)
>> +{
>> +    return nvme_keyring->serial;
>> +}
>> +EXPORT_SYMBOL_GPL(nvme_keyring_id);
>> +
>> +static void nvme_tls_psk_describe(const struct key *key, struct 
>> seq_file *m)
>> +{
>> +    seq_puts(m, key->description);
>> +    seq_printf(m, ": %u", key->datalen);
>> +}
>> +
>> +static bool nvme_tls_psk_match(const struct key *key,
>> +                   const struct key_match_data *match_data)
>> +{
>> +    const char *match_id;
>> +    size_t match_len;
>> +
>> +    if (!key->description) {
>> +        pr_debug("%s: no key description\n", __func__);
>> +        return false;
>> +    }
>> +    match_len = strlen(key->description);
>> +    pr_debug("%s: id %s len %zd\n", __func__, key->description, 
>> match_len);
>> +
>> +    if (!match_data->raw_data) {
>> +        pr_debug("%s: no match data\n", __func__);
>> +        return false;
>> +    }
>> +    match_id = match_data->raw_data;
>> +    pr_debug("%s: match '%s' '%s' len %lu\n",
>> +         __func__, match_id, key->description, match_len);
>> +    return !memcmp(key->description, match_id, match_len);
>> +}
>> +
>> +static int nvme_tls_psk_match_preparse(struct key_match_data 
>> *match_data)
>> +{
>> +    match_data->lookup_type = KEYRING_SEARCH_LOOKUP_ITERATE;
>> +    match_data->cmp = nvme_tls_psk_match;
>> +    return 0;
>> +}
>> +
>> +static struct key_type nvme_tls_psk_key_type = {
>> +    .name           = "psk",
>> +    .flags          = KEY_TYPE_NET_DOMAIN,
>> +    .preparse       = user_preparse,
>> +    .free_preparse  = user_free_preparse,
>> +    .match_preparse = nvme_tls_psk_match_preparse,
>> +    .instantiate    = generic_key_instantiate,
>> +    .revoke         = user_revoke,
>> +    .destroy        = user_destroy,
>> +    .describe       = nvme_tls_psk_describe,
>> +    .read           = user_read,
>> +};
>> +
> 
> Hannes, can you please provide a documentation section
> to this function? most importantly 'generated' argument.
> 
Sure. There are two types of PSK types specified in the NVMe TCP spec;
a 'retained' one which has to be provided by the admin, and a 
'generated' one which is derived from the shared key material from 
DH-HMAC-CHAP when doing a secure channel concatenation.
And each type has two possible hash algorithms (SHA-256 and SHA-384),
resulting in 4 possible PSKs.

And indeed, the secure channel concatenation bits are missing as we're 
still hashing out details at the fmds group.
I do have a patchset for that, though, but decided not to include it in 
this submission as it'll increase the patchset even more.
Can do if you like ...

But will be updating the documentation.

>> +struct key *nvme_tls_psk_lookup(key_ref_t keyring,
>> +        const char *hostnqn, const char *subnqn,
>> +        int hmac, bool generated)
>> +{
>> +    char *identity;
>> +    size_t identity_len = (NVMF_NQN_SIZE) * 2 + 11;
>> +    key_ref_t keyref;
>> +    key_serial_t keyring_id;
>> +
>> +    identity = kzalloc(identity_len, GFP_KERNEL);
>> +    if (!identity)
>> +        return ERR_PTR(-ENOMEM);
>> +
>> +    snprintf(identity, identity_len, "NVMe0%c%02d %s %s",
>> +         generated ? 'G' : 'R', hmac, hostnqn, subnqn);
> 
> Is that a format that is expected from userspace to produce?
> 
Yes. See
NVMe-TCP 1.0a section 3.6.1.3 "TLS PSK and PSK Identity Derivation"
for details.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman


^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [PATCH 02/18] nvme-keyring: define a 'psk' keytype
  2023-03-21 12:43 ` [PATCH 02/18] nvme-keyring: define a 'psk' keytype Hannes Reinecke
@ 2023-03-22  8:29   ` Sagi Grimberg
  2023-03-22  8:38     ` Hannes Reinecke
  0 siblings, 1 reply; 74+ messages in thread
From: Sagi Grimberg @ 2023-03-22  8:29 UTC (permalink / raw)
  To: Hannes Reinecke, Christoph Hellwig
  Cc: Keith Busch, linux-nvme, Chuck Lever, kernel-tls-handshake



On 3/21/23 14:43, Hannes Reinecke wrote:
> Define a 'psk' keytype to hold the NVMe TLS PSKs.
> 
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>   drivers/nvme/common/keyring.c | 96 +++++++++++++++++++++++++++++++++++
>   include/linux/nvme-keyring.h  |  8 +++
>   2 files changed, 104 insertions(+)
> 
> diff --git a/drivers/nvme/common/keyring.c b/drivers/nvme/common/keyring.c
> index 3a6e8a0b38e2..6cbb9d66e0f6 100644
> --- a/drivers/nvme/common/keyring.c
> +++ b/drivers/nvme/common/keyring.c
> @@ -11,6 +11,96 @@
>   
>   static struct key *nvme_keyring;
>   
> +key_serial_t nvme_keyring_id(void)
> +{
> +	return nvme_keyring->serial;
> +}
> +EXPORT_SYMBOL_GPL(nvme_keyring_id);
> +
> +static void nvme_tls_psk_describe(const struct key *key, struct seq_file *m)
> +{
> +	seq_puts(m, key->description);
> +	seq_printf(m, ": %u", key->datalen);
> +}
> +
> +static bool nvme_tls_psk_match(const struct key *key,
> +			       const struct key_match_data *match_data)
> +{
> +	const char *match_id;
> +	size_t match_len;
> +
> +	if (!key->description) {
> +		pr_debug("%s: no key description\n", __func__);
> +		return false;
> +	}
> +	match_len = strlen(key->description);
> +	pr_debug("%s: id %s len %zd\n", __func__, key->description, match_len);
> +
> +	if (!match_data->raw_data) {
> +		pr_debug("%s: no match data\n", __func__);
> +		return false;
> +	}
> +	match_id = match_data->raw_data;
> +	pr_debug("%s: match '%s' '%s' len %lu\n",
> +		 __func__, match_id, key->description, match_len);
> +	return !memcmp(key->description, match_id, match_len);
> +}
> +
> +static int nvme_tls_psk_match_preparse(struct key_match_data *match_data)
> +{
> +	match_data->lookup_type = KEYRING_SEARCH_LOOKUP_ITERATE;
> +	match_data->cmp = nvme_tls_psk_match;
> +	return 0;
> +}
> +
> +static struct key_type nvme_tls_psk_key_type = {
> +	.name           = "psk",
> +	.flags          = KEY_TYPE_NET_DOMAIN,
> +	.preparse       = user_preparse,
> +	.free_preparse  = user_free_preparse,
> +	.match_preparse = nvme_tls_psk_match_preparse,
> +	.instantiate    = generic_key_instantiate,
> +	.revoke         = user_revoke,
> +	.destroy        = user_destroy,
> +	.describe       = nvme_tls_psk_describe,
> +	.read           = user_read,
> +};
> +

Hannes, can you please provide a documentation section
to this function? most importantly 'generated' argument.

> +struct key *nvme_tls_psk_lookup(key_ref_t keyring,
> +		const char *hostnqn, const char *subnqn,
> +		int hmac, bool generated)
> +{
> +	char *identity;
> +	size_t identity_len = (NVMF_NQN_SIZE) * 2 + 11;
> +	key_ref_t keyref;
> +	key_serial_t keyring_id;
> +
> +	identity = kzalloc(identity_len, GFP_KERNEL);
> +	if (!identity)
> +		return ERR_PTR(-ENOMEM);
> +
> +	snprintf(identity, identity_len, "NVMe0%c%02d %s %s",
> +		 generated ? 'G' : 'R', hmac, hostnqn, subnqn);

Is that a format that is expected from userspace to produce?


> +
> +	if (!keyring)
> +		keyring = make_key_ref(nvme_keyring, true);
> +	keyring_id = key_serial(key_ref_to_ptr(keyring));
> +	pr_debug("keyring %x lookup tls psk '%s'\n",
> +		 keyring_id, identity);
> +	keyref = keyring_search(keyring, &nvme_tls_psk_key_type,
> +				identity, false);
> +	if (IS_ERR(keyref)) {
> +		pr_debug("lookup tls psk '%s' failed, error %ld\n",
> +			 identity, PTR_ERR(keyref));
> +		kfree(identity);
> +		return ERR_PTR(-ENOKEY);
> +	}
> +	kfree(identity);
> +
> +	return key_ref_to_ptr(keyref);
> +}
> +EXPORT_SYMBOL_GPL(nvme_tls_psk_lookup);
> +
>   int nvme_keyring_init(void)
>   {
>   	int err;
> @@ -24,12 +114,18 @@ int nvme_keyring_init(void)
>   	if (IS_ERR(nvme_keyring))
>   		return PTR_ERR(nvme_keyring);
>   
> +	err = register_key_type(&nvme_tls_psk_key_type);
> +	if (err) {
> +		key_put(nvme_keyring);
> +		return err;
> +	}
>   	return 0;
>   }
>   EXPORT_SYMBOL_GPL(nvme_keyring_init);
>   
>   void nvme_keyring_exit(void)
>   {
> +	unregister_key_type(&nvme_tls_psk_key_type);
>   	key_revoke(nvme_keyring);
>   	key_put(nvme_keyring);
>   }
> diff --git a/include/linux/nvme-keyring.h b/include/linux/nvme-keyring.h
> index a875c06cc922..c0c3d934f474 100644
> --- a/include/linux/nvme-keyring.h
> +++ b/include/linux/nvme-keyring.h
> @@ -6,6 +6,14 @@
>   #ifndef _NVME_KEYRING_H
>   #define _NVME_KEYRING_H
>   
> +#include <linux/key.h>
> +
> +struct key *nvme_tls_psk_lookup(key_ref_t keyring,
> +				const char *hostnqn, const char *subnqn,
> +				int hmac, bool generated);
> +
> +key_serial_t nvme_keyring_id(void);
> +
>   int nvme_keyring_init(void);
>   void nvme_keyring_exit(void);
>   

^ permalink raw reply	[flat|nested] 74+ messages in thread

* [PATCH 02/18] nvme-keyring: define a 'psk' keytype
  2023-03-21 12:43 [RFC PATCH 00/18] nvme: In-kernel TLS support for TCP Hannes Reinecke
@ 2023-03-21 12:43 ` Hannes Reinecke
  2023-03-22  8:29   ` Sagi Grimberg
  0 siblings, 1 reply; 74+ messages in thread
From: Hannes Reinecke @ 2023-03-21 12:43 UTC (permalink / raw)
  To: Christoph Hellwig
  Cc: Sagi Grimberg, Keith Busch, linux-nvme, Chuck Lever,
	kernel-tls-handshake, Hannes Reinecke

Define a 'psk' keytype to hold the NVMe TLS PSKs.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/nvme/common/keyring.c | 96 +++++++++++++++++++++++++++++++++++
 include/linux/nvme-keyring.h  |  8 +++
 2 files changed, 104 insertions(+)

diff --git a/drivers/nvme/common/keyring.c b/drivers/nvme/common/keyring.c
index 3a6e8a0b38e2..6cbb9d66e0f6 100644
--- a/drivers/nvme/common/keyring.c
+++ b/drivers/nvme/common/keyring.c
@@ -11,6 +11,96 @@
 
 static struct key *nvme_keyring;
 
+key_serial_t nvme_keyring_id(void)
+{
+	return nvme_keyring->serial;
+}
+EXPORT_SYMBOL_GPL(nvme_keyring_id);
+
+static void nvme_tls_psk_describe(const struct key *key, struct seq_file *m)
+{
+	seq_puts(m, key->description);
+	seq_printf(m, ": %u", key->datalen);
+}
+
+static bool nvme_tls_psk_match(const struct key *key,
+			       const struct key_match_data *match_data)
+{
+	const char *match_id;
+	size_t match_len;
+
+	if (!key->description) {
+		pr_debug("%s: no key description\n", __func__);
+		return false;
+	}
+	match_len = strlen(key->description);
+	pr_debug("%s: id %s len %zd\n", __func__, key->description, match_len);
+
+	if (!match_data->raw_data) {
+		pr_debug("%s: no match data\n", __func__);
+		return false;
+	}
+	match_id = match_data->raw_data;
+	pr_debug("%s: match '%s' '%s' len %lu\n",
+		 __func__, match_id, key->description, match_len);
+	return !memcmp(key->description, match_id, match_len);
+}
+
+static int nvme_tls_psk_match_preparse(struct key_match_data *match_data)
+{
+	match_data->lookup_type = KEYRING_SEARCH_LOOKUP_ITERATE;
+	match_data->cmp = nvme_tls_psk_match;
+	return 0;
+}
+
+static struct key_type nvme_tls_psk_key_type = {
+	.name           = "psk",
+	.flags          = KEY_TYPE_NET_DOMAIN,
+	.preparse       = user_preparse,
+	.free_preparse  = user_free_preparse,
+	.match_preparse = nvme_tls_psk_match_preparse,
+	.instantiate    = generic_key_instantiate,
+	.revoke         = user_revoke,
+	.destroy        = user_destroy,
+	.describe       = nvme_tls_psk_describe,
+	.read           = user_read,
+};
+
+struct key *nvme_tls_psk_lookup(key_ref_t keyring,
+		const char *hostnqn, const char *subnqn,
+		int hmac, bool generated)
+{
+	char *identity;
+	size_t identity_len = (NVMF_NQN_SIZE) * 2 + 11;
+	key_ref_t keyref;
+	key_serial_t keyring_id;
+
+	identity = kzalloc(identity_len, GFP_KERNEL);
+	if (!identity)
+		return ERR_PTR(-ENOMEM);
+
+	snprintf(identity, identity_len, "NVMe0%c%02d %s %s",
+		 generated ? 'G' : 'R', hmac, hostnqn, subnqn);
+
+	if (!keyring)
+		keyring = make_key_ref(nvme_keyring, true);
+	keyring_id = key_serial(key_ref_to_ptr(keyring));
+	pr_debug("keyring %x lookup tls psk '%s'\n",
+		 keyring_id, identity);
+	keyref = keyring_search(keyring, &nvme_tls_psk_key_type,
+				identity, false);
+	if (IS_ERR(keyref)) {
+		pr_debug("lookup tls psk '%s' failed, error %ld\n",
+			 identity, PTR_ERR(keyref));
+		kfree(identity);
+		return ERR_PTR(-ENOKEY);
+	}
+	kfree(identity);
+
+	return key_ref_to_ptr(keyref);
+}
+EXPORT_SYMBOL_GPL(nvme_tls_psk_lookup);
+
 int nvme_keyring_init(void)
 {
 	int err;
@@ -24,12 +114,18 @@ int nvme_keyring_init(void)
 	if (IS_ERR(nvme_keyring))
 		return PTR_ERR(nvme_keyring);
 
+	err = register_key_type(&nvme_tls_psk_key_type);
+	if (err) {
+		key_put(nvme_keyring);
+		return err;
+	}
 	return 0;
 }
 EXPORT_SYMBOL_GPL(nvme_keyring_init);
 
 void nvme_keyring_exit(void)
 {
+	unregister_key_type(&nvme_tls_psk_key_type);
 	key_revoke(nvme_keyring);
 	key_put(nvme_keyring);
 }
diff --git a/include/linux/nvme-keyring.h b/include/linux/nvme-keyring.h
index a875c06cc922..c0c3d934f474 100644
--- a/include/linux/nvme-keyring.h
+++ b/include/linux/nvme-keyring.h
@@ -6,6 +6,14 @@
 #ifndef _NVME_KEYRING_H
 #define _NVME_KEYRING_H
 
+#include <linux/key.h>
+
+struct key *nvme_tls_psk_lookup(key_ref_t keyring,
+				const char *hostnqn, const char *subnqn,
+				int hmac, bool generated);
+
+key_serial_t nvme_keyring_id(void);
+
 int nvme_keyring_init(void);
 void nvme_keyring_exit(void);
 
-- 
2.35.3


^ permalink raw reply related	[flat|nested] 74+ messages in thread

end of thread, other threads:[~2023-04-17 15:12 UTC | newest]

Thread overview: 74+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-03-29 13:59 [PATCHv2 00/18] nvme: In-kernel TLS support for TCP Hannes Reinecke
2023-03-29 13:59 ` [PATCH 01/18] nvme-keyring: register '.nvme' keyring and add CONFIG_NVME_TLS Hannes Reinecke
2023-03-29 14:49   ` Sagi Grimberg
2023-03-29 15:24     ` Hannes Reinecke
2023-03-29 15:04   ` Sagi Grimberg
2023-03-29 15:26     ` Hannes Reinecke
2023-03-30  8:53   ` Daniel Wagner
2023-03-30 14:38     ` Sagi Grimberg
2023-03-29 13:59 ` [PATCH 02/18] nvme-keyring: define a 'psk' keytype Hannes Reinecke
2023-03-29 13:59 ` [PATCH 03/18] nvme: add TCP TSAS definitions Hannes Reinecke
2023-03-29 13:59 ` [PATCH 04/18] nvme-tcp: add definitions for TLS cipher suites Hannes Reinecke
2023-03-29 13:59 ` [PATCH 05/18] net/tls: implement ->read_sock() Hannes Reinecke
2023-03-29 15:37   ` Sagi Grimberg
2023-03-29 15:41     ` Hannes Reinecke
2023-03-29 15:43       ` Sagi Grimberg
2023-03-29 15:44   ` Sagi Grimberg
2023-03-29 13:59 ` [PATCH 06/18] nvme/tcp: allocate socket file Hannes Reinecke
2023-03-29 15:57   ` Sagi Grimberg
2023-03-29 13:59 ` [PATCH 07/18] nvme-keyring: implement nvme_tls_psk_default() Hannes Reinecke
2023-03-29 15:35   ` Sagi Grimberg
2023-03-29 13:59 ` [PATCH 08/18] security/keys: export key_lookup() Hannes Reinecke
2023-03-29 13:59 ` [PATCH 09/18] nvme-tcp: enable TLS handshake upcall Hannes Reinecke
2023-03-30 12:54   ` Daniel Wagner
2023-03-30 12:59     ` Hannes Reinecke
2023-03-30 15:03   ` Sagi Grimberg
2023-03-30 17:16     ` Hannes Reinecke
2023-03-29 13:59 ` [PATCH 10/18] nvme-tcp: fixup send workflow for kTLS Hannes Reinecke
2023-03-30 15:24   ` Sagi Grimberg
2023-03-30 17:26     ` Hannes Reinecke
2023-03-31  5:49     ` Jakub Kicinski
2023-03-31  6:03       ` Hannes Reinecke
2023-04-03 12:20         ` Sagi Grimberg
2023-04-03 14:59           ` Jakub Kicinski
2023-04-03 15:51             ` Sagi Grimberg
2023-04-03 18:48               ` Jakub Kicinski
2023-04-03 22:36                 ` Sagi Grimberg
2023-03-29 13:59 ` [PATCH 11/18] nvme-tcp: control message handling for recvmsg() Hannes Reinecke
2023-03-30 15:25   ` Sagi Grimberg
2023-03-29 13:59 ` [PATCH 12/18] nvme-fabrics: parse options 'keyring' and 'tls_key' Hannes Reinecke
2023-03-30 15:33   ` Sagi Grimberg
2023-03-30 17:34     ` Hannes Reinecke
2023-04-03 12:24       ` Sagi Grimberg
2023-04-03 12:36         ` Hannes Reinecke
2023-04-03 13:07           ` Sagi Grimberg
2023-04-03 14:11             ` Hannes Reinecke
2023-04-03 16:13               ` Sagi Grimberg
2023-03-29 13:59 ` [PATCH 13/18] nvmet: make TCP sectype settable via configfs Hannes Reinecke
2023-03-30 16:07   ` Sagi Grimberg
2023-03-30 17:37     ` Hannes Reinecke
2023-04-03 12:31       ` Sagi Grimberg
2023-04-03 12:43         ` Hannes Reinecke
2023-03-29 13:59 ` [PATCH 14/18] nvmet-tcp: allocate socket file Hannes Reinecke
2023-03-30 16:08   ` Sagi Grimberg
2023-03-30 17:37     ` Hannes Reinecke
2023-03-29 13:59 ` [PATCH 15/18] nvmet-tcp: enable TLS handshake upcall Hannes Reinecke
2023-04-03 12:51   ` Sagi Grimberg
2023-04-03 14:05     ` Hannes Reinecke
2023-03-29 13:59 ` [PATCH 16/18] nvmet-tcp: rework sendpage for kTLS Hannes Reinecke
2023-04-03 12:52   ` Sagi Grimberg
2023-03-29 13:59 ` [PATCH 17/18] nvmet-tcp: control messages for recvmsg() Hannes Reinecke
2023-04-03 12:59   ` Sagi Grimberg
2023-03-29 13:59 ` [PATCH 18/18] nvmet-tcp: add configfs attribute 'param_keyring' Hannes Reinecke
2023-04-03 13:03   ` Sagi Grimberg
2023-04-03 14:13     ` Hannes Reinecke
2023-04-03 15:53       ` Sagi Grimberg
2023-04-14 10:30         ` Hannes Reinecke
2023-04-17 13:50           ` Sagi Grimberg
2023-04-17 14:01             ` Hannes Reinecke
2023-04-17 15:12               ` Sagi Grimberg
  -- strict thread matches above, loose matches on Subject: below --
2023-04-17 13:02 [PATCHv3 00/18] nvme: In-kernel TLS support for TCP Hannes Reinecke
2023-04-17 13:02 ` [PATCH 02/18] nvme-keyring: define a 'psk' keytype Hannes Reinecke
2023-03-21 12:43 [RFC PATCH 00/18] nvme: In-kernel TLS support for TCP Hannes Reinecke
2023-03-21 12:43 ` [PATCH 02/18] nvme-keyring: define a 'psk' keytype Hannes Reinecke
2023-03-22  8:29   ` Sagi Grimberg
2023-03-22  8:38     ` Hannes Reinecke
2023-03-22  8:49       ` Sagi Grimberg

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).