[PATCHv16 00/11] nvme: In-band authentication support

[PATCHv16 00/11] nvme: In-band authentication support

[Hannes Reinecke]

Hi all,

recent updates to the NVMe spec have added definitions for in-band
authentication, and seeing that it provides some real benefit
especially for NVMe-TCP here's an attempt to implement it.

Thanks to Nicolai Stange the crypto DH framework has been upgraded
to provide us with a FFDHE implementation; I've updated the patchset
to use the ephemeral key generation provided there.

Note that this is just for in-band authentication. Secure
concatenation (ie starting TLS with the negotiated parameters)
requires a TLS handshake, which the in-kernel TLS implementation
does not provide. This is being worked on with a different patchset
which is still WIP.

The nvme-cli support has already been merged; please use the latest
nvme-cli git repository to build the most recent version.

A copy of this patchset can be found at
git://git.kernel.org/pub/scm/linux/kernel/git/hare/scsi-devel
branch auth.v15

The patchset is being cut against nvme-5.20.

As usual, comments and reviews are welcome.

Changes to v15:
- Rebase to nvme-5.20

Changes to v14:
- Rebase to nvme-5.19
- Fixup sparse warnings as noted by Christoph
- Fixup Kconfig dependencies
- Add NVME_CONNECT_AUTHREQ definitions

Changes to v13:
- Fixup sysfs attribute visibility (reported by Chaitanya)
- Fixup SUCCESS1 handling
- Re-introduce crypto_has_shash() and crypto_has_kpp() helper
- Unse 'authenticated' flag on failure

Changes to v12:
- Replaces crypto_has_shash() and crypto_has_kpp() helper
  with CONFIG_ checks
- Fixed kbuild robot warning in pr_fmt()

Changes to v11:
- Fixup type for FAILURE2 message (Prashant Nayak)
- Do not sent SUCCESS2 if bi-directional authentication is not
  requested (Martin George)

Changes to v10:
- Fixup error return value when authentication failed

Changes to v9:
- Include review from Chaitanya
- Use sparse array for dhgroup and hash lookup
- Common function for auth_send and auth_receive

Changes to v8:
- Rebased to Nicolais crypto DH rework
- Fixed oops on non-fabrics devices

Changes to v7:
- Space out hash list and dhgroup list in nvme negotiate data
  to be conformant with the spec
- Update sequence number handling to start with a random value and
  ignore '0' as mandated by the spec
- Update nvme_auth_generate_key to return the key as
  suggested by Sagi
- Add nvmet_parse_fabrics_io_cmd() as suggested by hch

Changes to v6:
- Use 'u8' for DH group id and hash id
- Use 'struct nvme_dhchap_key'
- Rename variables to drop 'DHCHAP'
- Include reviews from Chaitanya

Changes to v5:
- Unify nvme_auth_generate_key()
- Unify nvme_auth_extract_key()
- Fixed bug where re-authentication with wrong controller key
  would not fail
- Include reviews from Sagi

Changes to v4:
- Validate against blktest suite
- Fixup base64 decoding
- Transform secret with correct hmac algorithm

Changes to v3:
- Renamed parameter to 'dhchap_ctrl_key'
- Fixed bi-directional authentication
- Included reviews from Sagi
- Fixed base64 algorithm for transport encoding

Changes to v2:
- Dropped non-standard algorithms
- Reworked base64 based on fs/crypto/fname.c
- Fixup crash with no keys

Changes to the original submission:
- Included reviews from Vladislav
- Included reviews from Sagi
- Implemented re-authentication support
- Fixed up key handling

Hannes Reinecke (11):
  crypto: add crypto_has_shash()
  crypto: add crypto_has_kpp()
  lib/base64: RFC4648-compliant base64 encoding
  nvme: add definitions for NVMe In-Band authentication
  nvme-fabrics: decode 'authentication required' connect error
  nvme: Implement In-Band authentication
  nvme-auth: Diffie-Hellman key exchange support
  nvmet: parse fabrics commands on io queues
  nvmet: Implement basic In-Band Authentication
  nvmet-auth: Diffie-Hellman key exchange support
  nvmet-auth: expire authentication sessions

 crypto/kpp.c                           |    6 +
 crypto/shash.c                         |    6 +
 drivers/nvme/Kconfig                   |    1 +
 drivers/nvme/Makefile                  |    1 +
 drivers/nvme/common/Kconfig            |    4 +
 drivers/nvme/common/Makefile           |    7 +
 drivers/nvme/common/auth.c             |  481 ++++++++++++
 drivers/nvme/host/Kconfig              |   15 +
 drivers/nvme/host/Makefile             |    1 +
 drivers/nvme/host/auth.c               | 1000 ++++++++++++++++++++++++
 drivers/nvme/host/core.c               |  143 +++-
 drivers/nvme/host/fabrics.c            |   83 +-
 drivers/nvme/host/fabrics.h            |    7 +
 drivers/nvme/host/nvme.h               |   30 +
 drivers/nvme/host/rdma.c               |    1 +
 drivers/nvme/host/tcp.c                |    1 +
 drivers/nvme/host/trace.c              |   32 +
 drivers/nvme/target/Kconfig            |   15 +
 drivers/nvme/target/Makefile           |    1 +
 drivers/nvme/target/admin-cmd.c        |    4 +-
 drivers/nvme/target/auth.c             |  525 +++++++++++++
 drivers/nvme/target/configfs.c         |  138 +++-
 drivers/nvme/target/core.c             |   15 +
 drivers/nvme/target/fabrics-cmd-auth.c |  545 +++++++++++++
 drivers/nvme/target/fabrics-cmd.c      |   55 +-
 drivers/nvme/target/nvmet.h            |   75 +-
 include/crypto/hash.h                  |    2 +
 include/crypto/kpp.h                   |    2 +
 include/linux/base64.h                 |   16 +
 include/linux/nvme-auth.h              |   41 +
 include/linux/nvme.h                   |  209 ++++-
 lib/Makefile                           |    2 +-
 lib/base64.c                           |  103 +++
 33 files changed, 3552 insertions(+), 15 deletions(-)
 create mode 100644 drivers/nvme/common/Kconfig
 create mode 100644 drivers/nvme/common/Makefile
 create mode 100644 drivers/nvme/common/auth.c
 create mode 100644 drivers/nvme/host/auth.c
 create mode 100644 drivers/nvme/target/auth.c
 create mode 100644 drivers/nvme/target/fabrics-cmd-auth.c
 create mode 100644 include/linux/base64.h
 create mode 100644 include/linux/nvme-auth.h
 create mode 100644 lib/base64.c

-- 
2.29.2

[PATCH 01/11] crypto: add crypto_has_shash()

[Hannes Reinecke]

Add helper function to determine if a given synchronous hash is supported.

Signed-off-by: Hannes Reinecke <hare@suse.de>
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
Reviewed-by: Chaitanya Kulkarni <kch@nvidia.com>
Reviewed-by: Himanshu Madhani <himanshu.madhani@oracle.com>
Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
---
 crypto/shash.c        | 6 ++++++
 include/crypto/hash.h | 2 ++
 2 files changed, 8 insertions(+)

diff --git a/crypto/shash.c b/crypto/shash.c
index 0a0a50cb694f..4c88e63b3350 100644
--- a/crypto/shash.c
+++ b/crypto/shash.c
@@ -521,6 +521,12 @@ struct crypto_shash *crypto_alloc_shash(const char *alg_name, u32 type,
 }
 EXPORT_SYMBOL_GPL(crypto_alloc_shash);
 
+int crypto_has_shash(const char *alg_name, u32 type, u32 mask)
+{
+	return crypto_type_has_alg(alg_name, &crypto_shash_type, type, mask);
+}
+EXPORT_SYMBOL_GPL(crypto_has_shash);
+
 static int shash_prepare_alg(struct shash_alg *alg)
 {
 	struct crypto_alg *base = &alg->base;
diff --git a/include/crypto/hash.h b/include/crypto/hash.h
index f140e4643949..f5841992dc9b 100644
--- a/include/crypto/hash.h
+++ b/include/crypto/hash.h
@@ -718,6 +718,8 @@ static inline void ahash_request_set_crypt(struct ahash_request *req,
 struct crypto_shash *crypto_alloc_shash(const char *alg_name, u32 type,
 					u32 mask);
 
+int crypto_has_shash(const char *alg_name, u32 type, u32 mask);
+
 static inline struct crypto_tfm *crypto_shash_tfm(struct crypto_shash *tfm)
 {
 	return &tfm->base;
-- 
2.29.2

[PATCH 02/11] crypto: add crypto_has_kpp()

[Hannes Reinecke]

Add helper function to determine if a given key-agreement protocol
primitive is supported.

Signed-off-by: Hannes Reinecke <hare@suse.de>
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
Reviewed-by: Chaitanya Kulkarni <kch@nvidia.com>
Reviewed-by: Himanshu Madhani <himanshu.madhani@oracle.com>
Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
---
 crypto/kpp.c         | 6 ++++++
 include/crypto/kpp.h | 2 ++
 2 files changed, 8 insertions(+)

diff --git a/crypto/kpp.c b/crypto/kpp.c
index 7aa6ba4b60a4..678e871ce418 100644
--- a/crypto/kpp.c
+++ b/crypto/kpp.c
@@ -104,6 +104,12 @@ int crypto_grab_kpp(struct crypto_kpp_spawn *spawn,
 }
 EXPORT_SYMBOL_GPL(crypto_grab_kpp);
 
+int crypto_has_kpp(const char *alg_name, u32 type, u32 mask)
+{
+	return crypto_type_has_alg(alg_name, &crypto_kpp_type, type, mask);
+}
+EXPORT_SYMBOL_GPL(crypto_has_kpp);
+
 static void kpp_prepare_alg(struct kpp_alg *alg)
 {
 	struct crypto_alg *base = &alg->base;
diff --git a/include/crypto/kpp.h b/include/crypto/kpp.h
index cccceadc164b..24d01e9877c1 100644
--- a/include/crypto/kpp.h
+++ b/include/crypto/kpp.h
@@ -104,6 +104,8 @@ struct kpp_alg {
  */
 struct crypto_kpp *crypto_alloc_kpp(const char *alg_name, u32 type, u32 mask);
 
+int crypto_has_kpp(const char *alg_name, u32 type, u32 mask);
+
 static inline struct crypto_tfm *crypto_kpp_tfm(struct crypto_kpp *tfm)
 {
 	return &tfm->base;
-- 
2.29.2

[PATCH 03/11] lib/base64: RFC4648-compliant base64 encoding

[Hannes Reinecke]

Add RFC4648-compliant base64 encoding and decoding routines, based on
the base64url encoding in fs/crypto/fname.c.

Signed-off-by: Hannes Reinecke <hare@suse.de>
Reviewed-by: Himanshu Madhani <himanshu.madhani@oracle.com>
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
Cc: Eric Biggers <ebiggers@kernel.org>
---
 include/linux/base64.h |  16 +++++++
 lib/Makefile           |   2 +-
 lib/base64.c           | 103 +++++++++++++++++++++++++++++++++++++++++
 3 files changed, 120 insertions(+), 1 deletion(-)
 create mode 100644 include/linux/base64.h
 create mode 100644 lib/base64.c

diff --git a/include/linux/base64.h b/include/linux/base64.h
new file mode 100644
index 000000000000..660d4cb1ef31
--- /dev/null
+++ b/include/linux/base64.h
@@ -0,0 +1,16 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * base64 encoding, lifted from fs/crypto/fname.c.
+ */
+
+#ifndef _LINUX_BASE64_H
+#define _LINUX_BASE64_H
+
+#include <linux/types.h>
+
+#define BASE64_CHARS(nbytes)   DIV_ROUND_UP((nbytes) * 4, 3)
+
+int base64_encode(const u8 *src, int len, char *dst);
+int base64_decode(const char *src, int len, u8 *dst);
+
+#endif /* _LINUX_BASE64_H */
diff --git a/lib/Makefile b/lib/Makefile
index ea54294d73bf..b276684beab3 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -46,7 +46,7 @@ obj-y += bcd.o sort.o parser.o debug_locks.o random32.o \
 	 bust_spinlocks.o kasprintf.o bitmap.o scatterlist.o \
 	 list_sort.o uuid.o iov_iter.o clz_ctz.o \
 	 bsearch.o find_bit.o llist.o memweight.o kfifo.o \
-	 percpu-refcount.o rhashtable.o \
+	 percpu-refcount.o rhashtable.o base64.o \
 	 once.o refcount.o usercopy.o errseq.o bucket_locks.o \
 	 generic-radix-tree.o
 obj-$(CONFIG_STRING_SELFTEST) += test_string.o
diff --git a/lib/base64.c b/lib/base64.c
new file mode 100644
index 000000000000..b736a7a431c5
--- /dev/null
+++ b/lib/base64.c
@@ -0,0 +1,103 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * base64.c - RFC4648-compliant base64 encoding
+ *
+ * Copyright (c) 2020 Hannes Reinecke, SUSE
+ *
+ * Based on the base64url routines from fs/crypto/fname.c
+ * (which are using the URL-safe base64 encoding),
+ * modified to use the standard coding table from RFC4648 section 4.
+ */
+
+#include <linux/kernel.h>
+#include <linux/types.h>
+#include <linux/export.h>
+#include <linux/string.h>
+#include <linux/base64.h>
+
+static const char base64_table[65] =
+	"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
+
+/**
+ * base64_encode() - base64-encode some binary data
+ * @src: the binary data to encode
+ * @srclen: the length of @src in bytes
+ * @dst: (output) the base64-encoded string.  Not NUL-terminated.
+ *
+ * Encodes data using base64 encoding, i.e. the "Base 64 Encoding" specified
+ * by RFC 4648, including the  '='-padding.
+ *
+ * Return: the length of the resulting base64-encoded string in bytes.
+ */
+int base64_encode(const u8 *src, int srclen, char *dst)
+{
+	u32 ac = 0;
+	int bits = 0;
+	int i;
+	char *cp = dst;
+
+	for (i = 0; i < srclen; i++) {
+		ac = (ac << 8) | src[i];
+		bits += 8;
+		do {
+			bits -= 6;
+			*cp++ = base64_table[(ac >> bits) & 0x3f];
+		} while (bits >= 6);
+	}
+	if (bits) {
+		*cp++ = base64_table[(ac << (6 - bits)) & 0x3f];
+		bits -= 6;
+	}
+	while (bits < 0) {
+		*cp++ = '=';
+		bits += 2;
+	}
+	return cp - dst;
+}
+EXPORT_SYMBOL_GPL(base64_encode);
+
+/**
+ * base64_decode() - base64-decode a string
+ * @src: the string to decode.  Doesn't need to be NUL-terminated.
+ * @srclen: the length of @src in bytes
+ * @dst: (output) the decoded binary data
+ *
+ * Decodes a string using base64 encoding, i.e. the "Base 64 Encoding"
+ * specified by RFC 4648, including the  '='-padding.
+ *
+ * This implementation hasn't been optimized for performance.
+ *
+ * Return: the length of the resulting decoded binary data in bytes,
+ *	   or -1 if the string isn't a valid base64 string.
+ */
+int base64_decode(const char *src, int srclen, u8 *dst)
+{
+	u32 ac = 0;
+	int bits = 0;
+	int i;
+	u8 *bp = dst;
+
+	for (i = 0; i < srclen; i++) {
+		const char *p = strchr(base64_table, src[i]);
+
+		if (src[i] == '=') {
+			ac = (ac << 6);
+			bits += 6;
+			if (bits >= 8)
+				bits -= 8;
+			continue;
+		}
+		if (p == NULL || src[i] == 0)
+			return -1;
+		ac = (ac << 6) | (p - base64_table);
+		bits += 6;
+		if (bits >= 8) {
+			bits -= 8;
+			*bp++ = (u8)(ac >> bits);
+		}
+	}
+	if (ac & ((1 << bits) - 1))
+		return -1;
+	return bp - dst;
+}
+EXPORT_SYMBOL_GPL(base64_decode);
-- 
2.29.2

[PATCH 04/11] nvme: add definitions for NVMe In-Band authentication

[Hannes Reinecke]

Add new definitions for NVMe In-band authentication as defined in
the NVMe Base Specification v2.0.

Signed-off-by: Hannes Reinecke <hare@suse.de>
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
Reviewed-by: Himanshu Madhani <himanshu.madhani@oracle.com>
Reviewed-by: Chaitanya Kulkarni <kch@nvidia.com>
---
 include/linux/nvme.h | 209 ++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 208 insertions(+), 1 deletion(-)

diff --git a/include/linux/nvme.h b/include/linux/nvme.h
index 8ced2439f1f3..27996548f270 100644
--- a/include/linux/nvme.h
+++ b/include/linux/nvme.h
@@ -19,6 +19,7 @@
 #define NVMF_TRSVCID_SIZE	32
 #define NVMF_TRADDR_SIZE	256
 #define NVMF_TSAS_SIZE		256
+#define NVMF_AUTH_HASH_LEN	64
 
 #define NVME_DISC_SUBSYS_NAME	"nqn.2014-08.org.nvmexpress.discovery"
 
@@ -1371,6 +1372,8 @@ enum nvmf_capsule_command {
 	nvme_fabrics_type_property_set	= 0x00,
 	nvme_fabrics_type_connect	= 0x01,
 	nvme_fabrics_type_property_get	= 0x04,
+	nvme_fabrics_type_auth_send	= 0x05,
+	nvme_fabrics_type_auth_receive	= 0x06,
 };
 
 #define nvme_fabrics_type_name(type)   { type, #type }
@@ -1378,7 +1381,9 @@ enum nvmf_capsule_command {
 	__print_symbolic(type,						\
 		nvme_fabrics_type_name(nvme_fabrics_type_property_set),	\
 		nvme_fabrics_type_name(nvme_fabrics_type_connect),	\
-		nvme_fabrics_type_name(nvme_fabrics_type_property_get))
+		nvme_fabrics_type_name(nvme_fabrics_type_property_get), \
+		nvme_fabrics_type_name(nvme_fabrics_type_auth_send),	\
+		nvme_fabrics_type_name(nvme_fabrics_type_auth_receive))
 
 /*
  * If not fabrics command, fctype will be ignored.
@@ -1474,6 +1479,11 @@ struct nvmf_connect_command {
 	__u8		resv4[12];
 };
 
+enum {
+	NVME_CONNECT_AUTHREQ_ASCR	= (1 << 2),
+	NVME_CONNECT_AUTHREQ_ATR	= (1 << 1),
+};
+
 struct nvmf_connect_data {
 	uuid_t		hostid;
 	__le16		cntlid;
@@ -1508,6 +1518,200 @@ struct nvmf_property_get_command {
 	__u8		resv4[16];
 };
 
+struct nvmf_auth_common_command {
+	__u8		opcode;
+	__u8		resv1;
+	__u16		command_id;
+	__u8		fctype;
+	__u8		resv2[19];
+	union nvme_data_ptr dptr;
+	__u8		resv3;
+	__u8		spsp0;
+	__u8		spsp1;
+	__u8		secp;
+	__le32		al_tl;
+	__u8		resv4[16];
+};
+
+struct nvmf_auth_send_command {
+	__u8		opcode;
+	__u8		resv1;
+	__u16		command_id;
+	__u8		fctype;
+	__u8		resv2[19];
+	union nvme_data_ptr dptr;
+	__u8		resv3;
+	__u8		spsp0;
+	__u8		spsp1;
+	__u8		secp;
+	__le32		tl;
+	__u8		resv4[16];
+};
+
+struct nvmf_auth_receive_command {
+	__u8		opcode;
+	__u8		resv1;
+	__u16		command_id;
+	__u8		fctype;
+	__u8		resv2[19];
+	union nvme_data_ptr dptr;
+	__u8		resv3;
+	__u8		spsp0;
+	__u8		spsp1;
+	__u8		secp;
+	__le32		al;
+	__u8		resv4[16];
+};
+
+/* Value for secp */
+enum {
+	NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER	= 0xe9,
+};
+
+/* Defined value for auth_type */
+enum {
+	NVME_AUTH_COMMON_MESSAGES	= 0x00,
+	NVME_AUTH_DHCHAP_MESSAGES	= 0x01,
+};
+
+/* Defined messages for auth_id */
+enum {
+	NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE	= 0x00,
+	NVME_AUTH_DHCHAP_MESSAGE_CHALLENGE	= 0x01,
+	NVME_AUTH_DHCHAP_MESSAGE_REPLY		= 0x02,
+	NVME_AUTH_DHCHAP_MESSAGE_SUCCESS1	= 0x03,
+	NVME_AUTH_DHCHAP_MESSAGE_SUCCESS2	= 0x04,
+	NVME_AUTH_DHCHAP_MESSAGE_FAILURE2	= 0xf0,
+	NVME_AUTH_DHCHAP_MESSAGE_FAILURE1	= 0xf1,
+};
+
+struct nvmf_auth_dhchap_protocol_descriptor {
+	__u8		authid;
+	__u8		rsvd;
+	__u8		halen;
+	__u8		dhlen;
+	__u8		idlist[60];
+};
+
+enum {
+	NVME_AUTH_DHCHAP_AUTH_ID	= 0x01,
+};
+
+/* Defined hash functions for DH-HMAC-CHAP authentication */
+enum {
+	NVME_AUTH_HASH_SHA256	= 0x01,
+	NVME_AUTH_HASH_SHA384	= 0x02,
+	NVME_AUTH_HASH_SHA512	= 0x03,
+	NVME_AUTH_HASH_INVALID	= 0xff,
+};
+
+/* Defined Diffie-Hellman group identifiers for DH-HMAC-CHAP authentication */
+enum {
+	NVME_AUTH_DHGROUP_NULL		= 0x00,
+	NVME_AUTH_DHGROUP_2048		= 0x01,
+	NVME_AUTH_DHGROUP_3072		= 0x02,
+	NVME_AUTH_DHGROUP_4096		= 0x03,
+	NVME_AUTH_DHGROUP_6144		= 0x04,
+	NVME_AUTH_DHGROUP_8192		= 0x05,
+	NVME_AUTH_DHGROUP_INVALID	= 0xff,
+};
+
+union nvmf_auth_protocol {
+	struct nvmf_auth_dhchap_protocol_descriptor dhchap;
+};
+
+struct nvmf_auth_dhchap_negotiate_data {
+	__u8		auth_type;
+	__u8		auth_id;
+	__le16		rsvd;
+	__le16		t_id;
+	__u8		sc_c;
+	__u8		napd;
+	union nvmf_auth_protocol auth_protocol[];
+};
+
+struct nvmf_auth_dhchap_challenge_data {
+	__u8		auth_type;
+	__u8		auth_id;
+	__u16		rsvd1;
+	__le16		t_id;
+	__u8		hl;
+	__u8		rsvd2;
+	__u8		hashid;
+	__u8		dhgid;
+	__le16		dhvlen;
+	__le32		seqnum;
+	/* 'hl' bytes of challenge value */
+	__u8		cval[];
+	/* followed by 'dhvlen' bytes of DH value */
+};
+
+struct nvmf_auth_dhchap_reply_data {
+	__u8		auth_type;
+	__u8		auth_id;
+	__le16		rsvd1;
+	__le16		t_id;
+	__u8		hl;
+	__u8		rsvd2;
+	__u8		cvalid;
+	__u8		rsvd3;
+	__le16		dhvlen;
+	__le32		seqnum;
+	/* 'hl' bytes of response data */
+	__u8		rval[];
+	/* followed by 'hl' bytes of Challenge value */
+	/* followed by 'dhvlen' bytes of DH value */
+};
+
+enum {
+	NVME_AUTH_DHCHAP_RESPONSE_VALID	= (1 << 0),
+};
+
+struct nvmf_auth_dhchap_success1_data {
+	__u8		auth_type;
+	__u8		auth_id;
+	__le16		rsvd1;
+	__le16		t_id;
+	__u8		hl;
+	__u8		rsvd2;
+	__u8		rvalid;
+	__u8		rsvd3[7];
+	/* 'hl' bytes of response value if 'rvalid' is set */
+	__u8		rval[];
+};
+
+struct nvmf_auth_dhchap_success2_data {
+	__u8		auth_type;
+	__u8		auth_id;
+	__le16		rsvd1;
+	__le16		t_id;
+	__u8		rsvd2[10];
+};
+
+struct nvmf_auth_dhchap_failure_data {
+	__u8		auth_type;
+	__u8		auth_id;
+	__le16		rsvd1;
+	__le16		t_id;
+	__u8		rescode;
+	__u8		rescode_exp;
+};
+
+enum {
+	NVME_AUTH_DHCHAP_FAILURE_REASON_FAILED	= 0x01,
+};
+
+enum {
+	NVME_AUTH_DHCHAP_FAILURE_FAILED			= 0x01,
+	NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE		= 0x02,
+	NVME_AUTH_DHCHAP_FAILURE_CONCAT_MISMATCH	= 0x03,
+	NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE		= 0x04,
+	NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE	= 0x05,
+	NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD	= 0x06,
+	NVME_AUTH_DHCHAP_FAILURE_INCORRECT_MESSAGE	= 0x07,
+};
+
+
 struct nvme_dbbuf {
 	__u8			opcode;
 	__u8			flags;
@@ -1551,6 +1755,9 @@ struct nvme_command {
 		struct nvmf_connect_command connect;
 		struct nvmf_property_set_command prop_set;
 		struct nvmf_property_get_command prop_get;
+		struct nvmf_auth_common_command auth_common;
+		struct nvmf_auth_send_command auth_send;
+		struct nvmf_auth_receive_command auth_receive;
 		struct nvme_dbbuf dbbuf;
 		struct nvme_directive_cmd directive;
 	};
-- 
2.29.2

[PATCH 05/11] nvme-fabrics: decode 'authentication required' connect error

[Hannes Reinecke]

The 'connect' command might fail with NVME_SC_AUTH_REQUIRED, so we
should be decoding this error, too.

Signed-off-by: Hannes Reinecke <hare@suse.de>
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
Reviewed-by: Chaitanya Kulkarni <kch@nvidia.com>
Reviewed-by: Himanshu Madhani <himanshu.madhani@oracle.com>
---
 drivers/nvme/host/fabrics.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index 0a0512300f1b..e4b1520862d8 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -331,6 +331,10 @@ static void nvmf_log_connect_error(struct nvme_ctrl *ctrl,
 		dev_err(ctrl->device,
 			"Connect command failed: host path error\n");
 		break;
+	case NVME_SC_AUTH_REQUIRED:
+		dev_err(ctrl->device,
+			"Connect command failed: authentication required\n");
+		break;
 	default:
 		dev_err(ctrl->device,
 			"Connect command failed, error wo/DNR bit: %d\n",
-- 
2.29.2

[PATCH 06/11] nvme: Implement In-Band authentication

[Hannes Reinecke]

Implement NVMe-oF In-Band authentication according to NVMe TPAR 8006.
This patch adds two new fabric options 'dhchap_secret' to specify the
pre-shared key (in ASCII respresentation according to NVMe 2.0 section
8.13.5.8 'Secret representation') and 'dhchap_ctrl_secret' to specify
the pre-shared controller key for bi-directional authentication of both
the host and the controller.
Re-authentication can be triggered by writing the PSK into the new
controller sysfs attribute 'dhchap_secret' or 'dhchap_ctrl_secret'.

Signed-off-by: Hannes Reinecke <hare@suse.de>
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
---
 drivers/nvme/Kconfig         |   1 +
 drivers/nvme/Makefile        |   1 +
 drivers/nvme/common/Kconfig  |   4 +
 drivers/nvme/common/Makefile |   7 +
 drivers/nvme/common/auth.c   | 328 ++++++++++++++
 drivers/nvme/host/Kconfig    |  13 +
 drivers/nvme/host/Makefile   |   1 +
 drivers/nvme/host/auth.c     | 813 +++++++++++++++++++++++++++++++++++
 drivers/nvme/host/core.c     | 143 +++++-
 drivers/nvme/host/fabrics.c  |  79 +++-
 drivers/nvme/host/fabrics.h  |   7 +
 drivers/nvme/host/nvme.h     |  30 ++
 drivers/nvme/host/rdma.c     |   1 +
 drivers/nvme/host/tcp.c      |   1 +
 drivers/nvme/host/trace.c    |  32 ++
 include/linux/nvme-auth.h    |  33 ++
 16 files changed, 1487 insertions(+), 7 deletions(-)
 create mode 100644 drivers/nvme/common/Kconfig
 create mode 100644 drivers/nvme/common/Makefile
 create mode 100644 drivers/nvme/common/auth.c
 create mode 100644 drivers/nvme/host/auth.c
 create mode 100644 include/linux/nvme-auth.h

diff --git a/drivers/nvme/Kconfig b/drivers/nvme/Kconfig
index 87ae409a32b9..656e46d938da 100644
--- a/drivers/nvme/Kconfig
+++ b/drivers/nvme/Kconfig
@@ -1,6 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0-only
 menu "NVME Support"
 
+source "drivers/nvme/common/Kconfig"
 source "drivers/nvme/host/Kconfig"
 source "drivers/nvme/target/Kconfig"
 
diff --git a/drivers/nvme/Makefile b/drivers/nvme/Makefile
index fb42c44609a8..eedca8c72098 100644
--- a/drivers/nvme/Makefile
+++ b/drivers/nvme/Makefile
@@ -1,4 +1,5 @@
 # SPDX-License-Identifier: GPL-2.0-only
 
+obj-$(CONFIG_NVME_COMMON)		+= common/
 obj-y		+= host/
 obj-y		+= target/
diff --git a/drivers/nvme/common/Kconfig b/drivers/nvme/common/Kconfig
new file mode 100644
index 000000000000..4514f44362dd
--- /dev/null
+++ b/drivers/nvme/common/Kconfig
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: GPL-2.0-only
+
+config NVME_COMMON
+       tristate
diff --git a/drivers/nvme/common/Makefile b/drivers/nvme/common/Makefile
new file mode 100644
index 000000000000..720c625b8a52
--- /dev/null
+++ b/drivers/nvme/common/Makefile
@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: GPL-2.0
+
+ccflags-y			+= -I$(src)
+
+obj-$(CONFIG_NVME_COMMON)	+= nvme-common.o
+
+nvme-common-y			+= auth.o
diff --git a/drivers/nvme/common/auth.c b/drivers/nvme/common/auth.c
new file mode 100644
index 000000000000..1cccaf6afdd2
--- /dev/null
+++ b/drivers/nvme/common/auth.c
@@ -0,0 +1,328 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2020 Hannes Reinecke, SUSE Linux
+ */
+
+#include <linux/module.h>
+#include <linux/crc32.h>
+#include <linux/base64.h>
+#include <linux/prandom.h>
+#include <linux/scatterlist.h>
+#include <asm/unaligned.h>
+#include <crypto/hash.h>
+#include <crypto/dh.h>
+#include <linux/nvme.h>
+#include <linux/nvme-auth.h>
+
+static u32 nvme_dhchap_seqnum;
+static DEFINE_MUTEX(nvme_dhchap_mutex);
+
+u32 nvme_auth_get_seqnum(void)
+{
+	u32 seqnum;
+
+	mutex_lock(&nvme_dhchap_mutex);
+	if (!nvme_dhchap_seqnum)
+		nvme_dhchap_seqnum = prandom_u32();
+	else {
+		nvme_dhchap_seqnum++;
+		if (!nvme_dhchap_seqnum)
+			nvme_dhchap_seqnum++;
+	}
+	seqnum = nvme_dhchap_seqnum;
+	mutex_unlock(&nvme_dhchap_mutex);
+	return seqnum;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_get_seqnum);
+
+static struct nvme_auth_dhgroup_map {
+	const char name[16];
+	const char kpp[16];
+} dhgroup_map[] = {
+	[NVME_AUTH_DHGROUP_NULL] = {
+		.name = "null", .kpp = "null" },
+	[NVME_AUTH_DHGROUP_2048] = {
+		.name = "ffdhe2048", .kpp = "ffdhe2048(dh)" },
+	[NVME_AUTH_DHGROUP_3072] = {
+		.name = "ffdhe3072", .kpp = "ffdhe3072(dh)" },
+	[NVME_AUTH_DHGROUP_4096] = {
+		.name = "ffdhe4096", .kpp = "ffdhe4096(dh)" },
+	[NVME_AUTH_DHGROUP_6144] = {
+		.name = "ffdhe6144", .kpp = "ffdhe6144(dh)" },
+	[NVME_AUTH_DHGROUP_8192] = {
+		.name = "ffdhe8192", .kpp = "ffdhe8192(dh)" },
+};
+
+const char *nvme_auth_dhgroup_name(u8 dhgroup_id)
+{
+	if ((dhgroup_id > ARRAY_SIZE(dhgroup_map)) ||
+	    !dhgroup_map[dhgroup_id].name ||
+	    !strlen(dhgroup_map[dhgroup_id].name))
+		return NULL;
+	return dhgroup_map[dhgroup_id].name;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_dhgroup_name);
+
+const char *nvme_auth_dhgroup_kpp(u8 dhgroup_id)
+{
+	if ((dhgroup_id > ARRAY_SIZE(dhgroup_map)) ||
+	    !dhgroup_map[dhgroup_id].kpp ||
+	    !strlen(dhgroup_map[dhgroup_id].kpp))
+		return NULL;
+	return dhgroup_map[dhgroup_id].kpp;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_dhgroup_kpp);
+
+u8 nvme_auth_dhgroup_id(const char *dhgroup_name)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(dhgroup_map); i++) {
+		if (!dhgroup_map[i].name ||
+		    !strlen(dhgroup_map[i].name))
+			continue;
+		if (!strncmp(dhgroup_map[i].name, dhgroup_name,
+			     strlen(dhgroup_map[i].name)))
+			return i;
+	}
+	return NVME_AUTH_DHGROUP_INVALID;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_dhgroup_id);
+
+static struct nvme_dhchap_hash_map {
+	int len;
+	const char hmac[15];
+	const char digest[8];
+} hash_map[] = {
+	[NVME_AUTH_HASH_SHA256] = {
+		.len = 32,
+		.hmac = "hmac(sha256)",
+		.digest = "sha256",
+	},
+	[NVME_AUTH_HASH_SHA384] = {
+		.len = 48,
+		.hmac = "hmac(sha384)",
+		.digest = "sha384",
+	},
+	[NVME_AUTH_HASH_SHA512] = {
+		.len = 64,
+		.hmac = "hmac(sha512)",
+		.digest = "sha512",
+	},
+};
+
+const char *nvme_auth_hmac_name(u8 hmac_id)
+{
+	if ((hmac_id > ARRAY_SIZE(hash_map)) ||
+	    !hash_map[hmac_id].hmac ||
+	    !strlen(hash_map[hmac_id].hmac))
+		return NULL;
+	return hash_map[hmac_id].hmac;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_hmac_name);
+
+const char *nvme_auth_digest_name(u8 hmac_id)
+{
+	if ((hmac_id > ARRAY_SIZE(hash_map)) ||
+	    !hash_map[hmac_id].digest ||
+	    !strlen(hash_map[hmac_id].digest))
+		return NULL;
+	return hash_map[hmac_id].digest;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_digest_name);
+
+u8 nvme_auth_hmac_id(const char *hmac_name)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
+		if (!hash_map[i].hmac || !strlen(hash_map[i].hmac))
+			continue;
+		if (!strncmp(hash_map[i].hmac, hmac_name,
+			     strlen(hash_map[i].hmac)))
+			return i;
+	}
+	return NVME_AUTH_HASH_INVALID;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_hmac_id);
+
+size_t nvme_auth_hmac_hash_len(u8 hmac_id)
+{
+	if ((hmac_id > ARRAY_SIZE(hash_map)) ||
+	    !hash_map[hmac_id].hmac ||
+	    !strlen(hash_map[hmac_id].hmac))
+		return 0;
+	return hash_map[hmac_id].len;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_hmac_hash_len);
+
+struct nvme_dhchap_key *nvme_auth_extract_key(unsigned char *secret,
+					      u8 key_hash)
+{
+	struct nvme_dhchap_key *key;
+	unsigned char *p;
+	u32 crc;
+	int ret, key_len;
+	size_t allocated_len = strlen(secret);
+
+	/* Secret might be affixed with a ':' */
+	p = strrchr(secret, ':');
+	if (p)
+		allocated_len = p - secret;
+	key = kzalloc(sizeof(*key), GFP_KERNEL);
+	if (!key)
+		return ERR_PTR(-ENOMEM);
+	key->key = kzalloc(allocated_len, GFP_KERNEL);
+	if (!key->key) {
+		ret = -ENOMEM;
+		goto out_free_key;
+	}
+
+	key_len = base64_decode(secret, allocated_len, key->key);
+	if (key_len < 0) {
+		pr_debug("base64 key decoding error %d\n",
+			 key_len);
+		ret = key_len;
+		goto out_free_secret;
+	}
+
+	if (key_len != 36 && key_len != 52 &&
+	    key_len != 68) {
+		pr_err("Invalid key len %d\n", key_len);
+		ret = -EINVAL;
+		goto out_free_secret;
+	}
+
+	if (key_hash > 0 &&
+	    (key_len - 4) != nvme_auth_hmac_hash_len(key_hash)) {
+		pr_err("Mismatched key len %d for %s\n", key_len,
+		       nvme_auth_hmac_name(key_hash));
+		ret = -EINVAL;
+		goto out_free_secret;
+	}
+
+	/* The last four bytes is the CRC in little-endian format */
+	key_len -= 4;
+	/*
+	 * The linux implementation doesn't do pre- and post-increments,
+	 * so we have to do it manually.
+	 */
+	crc = ~crc32(~0, key->key, key_len);
+
+	if (get_unaligned_le32(key->key + key_len) != crc) {
+		pr_err("key crc mismatch (key %08x, crc %08x)\n",
+		       get_unaligned_le32(key->key + key_len), crc);
+		ret = -EKEYREJECTED;
+		goto out_free_secret;
+	}
+	key->len = key_len;
+	key->hash = key_hash;
+	return key;
+out_free_secret:
+	kfree_sensitive(key->key);
+out_free_key:
+	kfree(key);
+	return ERR_PTR(ret);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_extract_key);
+
+void nvme_auth_free_key(struct nvme_dhchap_key *key)
+{
+	if (!key)
+		return;
+	kfree_sensitive(key->key);
+	kfree(key);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_free_key);
+
+u8 *nvme_auth_transform_key(struct nvme_dhchap_key *key, char *nqn)
+{
+	const char *hmac_name = nvme_auth_hmac_name(key->hash);
+	struct crypto_shash *key_tfm;
+	struct shash_desc *shash;
+	u8 *transformed_key;
+	int ret;
+
+	if (key->hash == 0) {
+		transformed_key = kmemdup(key->key, key->len, GFP_KERNEL);
+		return transformed_key ? transformed_key : ERR_PTR(-ENOMEM);
+	}
+
+	if (!key || !key->key) {
+		pr_warn("No key specified\n");
+		return ERR_PTR(-ENOKEY);
+	}
+	if (!hmac_name) {
+		pr_warn("Invalid key hash id %d\n", key->hash);
+		return ERR_PTR(-EINVAL);
+	}
+
+	key_tfm = crypto_alloc_shash(hmac_name, 0, 0);
+	if (IS_ERR(key_tfm))
+		return (u8 *)key_tfm;
+
+	shash = kmalloc(sizeof(struct shash_desc) +
+			crypto_shash_descsize(key_tfm),
+			GFP_KERNEL);
+	if (!shash) {
+		ret = -ENOMEM;
+		goto out_free_key;
+	}
+
+	transformed_key = kzalloc(crypto_shash_digestsize(key_tfm), GFP_KERNEL);
+	if (!transformed_key) {
+		ret = -ENOMEM;
+		goto out_free_shash;
+	}
+
+	shash->tfm = key_tfm;
+	ret = crypto_shash_setkey(key_tfm, key->key, key->len);
+	if (ret < 0)
+		goto out_free_shash;
+	ret = crypto_shash_init(shash);
+	if (ret < 0)
+		goto out_free_shash;
+	ret = crypto_shash_update(shash, nqn, strlen(nqn));
+	if (ret < 0)
+		goto out_free_shash;
+	ret = crypto_shash_update(shash, "NVMe-over-Fabrics", 17);
+	if (ret < 0)
+		goto out_free_shash;
+	ret = crypto_shash_final(shash, transformed_key);
+out_free_shash:
+	kfree(shash);
+out_free_key:
+	crypto_free_shash(key_tfm);
+	if (ret < 0) {
+		kfree_sensitive(transformed_key);
+		return ERR_PTR(ret);
+	}
+	return transformed_key;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_transform_key);
+
+int nvme_auth_generate_key(u8 *secret, struct nvme_dhchap_key **ret_key)
+{
+	struct nvme_dhchap_key *key;
+	u8 key_hash;
+
+	if (!secret) {
+		*ret_key = NULL;
+		return 0;
+	}
+
+	if (sscanf(secret, "DHHC-1:%hhd:%*s:", &key_hash) != 1)
+		return -EINVAL;
+
+	/* Pass in the secret without the 'DHHC-1:XX:' prefix */
+	key = nvme_auth_extract_key(secret + 10, key_hash);
+	if (IS_ERR(key)) {
+		return PTR_ERR(key);
+	}
+
+	*ret_key = key;
+	return 0;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_generate_key);
+
+MODULE_LICENSE("GPL v2");
diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig
index 877d2ec4ea9f..6c503f42f3c6 100644
--- a/drivers/nvme/host/Kconfig
+++ b/drivers/nvme/host/Kconfig
@@ -92,6 +92,19 @@ config NVME_TCP
 
 	  If unsure, say N.
 
+config NVME_AUTH
+	bool "NVM Express over Fabrics In-Band Authentication"
+	depends on NVME_CORE
+	select NVME_COMMON
+	select CRYPTO
+	select CRYPTO_HMAC
+	select CRYPTO_SHA256
+	select CRYPTO_SHA512
+	help
+	  This provides support for NVMe over Fabrics In-Band Authentication.
+
+	  If unsure, say N.
+
 config NVME_APPLE
 	tristate "Apple ANS2 NVM Express host driver"
 	depends on OF && BLOCK
diff --git a/drivers/nvme/host/Makefile b/drivers/nvme/host/Makefile
index a36ae1612059..a3e88f32f560 100644
--- a/drivers/nvme/host/Makefile
+++ b/drivers/nvme/host/Makefile
@@ -16,6 +16,7 @@ nvme-core-$(CONFIG_NVME_MULTIPATH)	+= multipath.o
 nvme-core-$(CONFIG_BLK_DEV_ZONED)	+= zns.o
 nvme-core-$(CONFIG_FAULT_INJECTION_DEBUG_FS)	+= fault_inject.o
 nvme-core-$(CONFIG_NVME_HWMON)		+= hwmon.o
+nvme-core-$(CONFIG_NVME_AUTH)		+= auth.o
 
 nvme-y					+= pci.o
 
diff --git a/drivers/nvme/host/auth.c b/drivers/nvme/host/auth.c
new file mode 100644
index 000000000000..c0353783fe27
--- /dev/null
+++ b/drivers/nvme/host/auth.c
@@ -0,0 +1,813 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2020 Hannes Reinecke, SUSE Linux
+ */
+
+#include <linux/crc32.h>
+#include <linux/base64.h>
+#include <linux/prandom.h>
+#include <asm/unaligned.h>
+#include <crypto/hash.h>
+#include <crypto/dh.h>
+#include "nvme.h"
+#include "fabrics.h"
+#include <linux/nvme-auth.h>
+
+struct nvme_dhchap_queue_context {
+	struct list_head entry;
+	struct work_struct auth_work;
+	struct nvme_ctrl *ctrl;
+	struct crypto_shash *shash_tfm;
+	void *buf;
+	size_t buf_size;
+	int qid;
+	int error;
+	u32 s1;
+	u32 s2;
+	u16 transaction;
+	u8 status;
+	u8 hash_id;
+	size_t hash_len;
+	u8 dhgroup_id;
+	u8 c1[64];
+	u8 c2[64];
+	u8 response[64];
+	u8 *host_response;
+};
+
+#define nvme_auth_flags_from_qid(qid) \
+	(qid == 0) ? 0 : BLK_MQ_REQ_NOWAIT | BLK_MQ_REQ_RESERVED
+#define nvme_auth_queue_from_qid(ctrl, qid) \
+	(qid == 0) ? (ctrl)->fabrics_q : (ctrl)->connect_q
+
+static int nvme_auth_submit(struct nvme_ctrl *ctrl, int qid,
+			    void *data, size_t data_len, bool auth_send)
+{
+	struct nvme_command cmd = {};
+	blk_mq_req_flags_t flags = nvme_auth_flags_from_qid(qid);
+	struct request_queue *q = nvme_auth_queue_from_qid(ctrl, qid);
+	int ret;
+
+	cmd.auth_common.opcode = nvme_fabrics_command;
+	cmd.auth_common.secp = NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER;
+	cmd.auth_common.spsp0 = 0x01;
+	cmd.auth_common.spsp1 = 0x01;
+	if (auth_send) {
+		cmd.auth_send.fctype = nvme_fabrics_type_auth_send;
+		cmd.auth_send.tl = cpu_to_le32(data_len);
+	} else {
+		cmd.auth_receive.fctype = nvme_fabrics_type_auth_receive;
+		cmd.auth_receive.al = cpu_to_le32(data_len);
+	}
+
+	ret = __nvme_submit_sync_cmd(q, &cmd, NULL, data, data_len,
+				     qid == 0 ? NVME_QID_ANY : qid,
+				     0, flags);
+	if (ret > 0)
+		dev_warn(ctrl->device,
+			"qid %d auth_send failed with status %d\n", qid, ret);
+	else if (ret < 0)
+		dev_err(ctrl->device,
+			"qid %d auth_send failed with error %d\n", qid, ret);
+	return ret;
+}
+
+static int nvme_auth_receive_validate(struct nvme_ctrl *ctrl, int qid,
+		struct nvmf_auth_dhchap_failure_data *data,
+		u16 transaction, u8 expected_msg)
+{
+	dev_dbg(ctrl->device, "%s: qid %d auth_type %d auth_id %x\n",
+		__func__, qid, data->auth_type, data->auth_id);
+
+	if (data->auth_type == NVME_AUTH_COMMON_MESSAGES &&
+	    data->auth_id == NVME_AUTH_DHCHAP_MESSAGE_FAILURE1) {
+		return data->rescode_exp;
+	}
+	if (data->auth_type != NVME_AUTH_DHCHAP_MESSAGES ||
+	    data->auth_id != expected_msg) {
+		dev_warn(ctrl->device,
+			 "qid %d invalid message %02x/%02x\n",
+			 qid, data->auth_type, data->auth_id);
+		return NVME_AUTH_DHCHAP_FAILURE_INCORRECT_MESSAGE;
+	}
+	if (le16_to_cpu(data->t_id) != transaction) {
+		dev_warn(ctrl->device,
+			 "qid %d invalid transaction ID %d\n",
+			 qid, le16_to_cpu(data->t_id));
+		return NVME_AUTH_DHCHAP_FAILURE_INCORRECT_MESSAGE;
+	}
+	return 0;
+}
+
+static int nvme_auth_set_dhchap_negotiate_data(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_negotiate_data *data = chap->buf;
+	size_t size = sizeof(*data) + sizeof(union nvmf_auth_protocol);
+
+	if (chap->buf_size < size) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return -EINVAL;
+	}
+	memset((u8 *)chap->buf, 0, size);
+	data->auth_type = NVME_AUTH_COMMON_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE;
+	data->t_id = cpu_to_le16(chap->transaction);
+	data->sc_c = 0; /* No secure channel concatenation */
+	data->napd = 1;
+	data->auth_protocol[0].dhchap.authid = NVME_AUTH_DHCHAP_AUTH_ID;
+	data->auth_protocol[0].dhchap.halen = 3;
+	data->auth_protocol[0].dhchap.dhlen = 6;
+	data->auth_protocol[0].dhchap.idlist[0] = NVME_AUTH_HASH_SHA256;
+	data->auth_protocol[0].dhchap.idlist[1] = NVME_AUTH_HASH_SHA384;
+	data->auth_protocol[0].dhchap.idlist[2] = NVME_AUTH_HASH_SHA512;
+	data->auth_protocol[0].dhchap.idlist[30] = NVME_AUTH_DHGROUP_NULL;
+	data->auth_protocol[0].dhchap.idlist[31] = NVME_AUTH_DHGROUP_2048;
+	data->auth_protocol[0].dhchap.idlist[32] = NVME_AUTH_DHGROUP_3072;
+	data->auth_protocol[0].dhchap.idlist[33] = NVME_AUTH_DHGROUP_4096;
+	data->auth_protocol[0].dhchap.idlist[34] = NVME_AUTH_DHGROUP_6144;
+	data->auth_protocol[0].dhchap.idlist[35] = NVME_AUTH_DHGROUP_8192;
+
+	return size;
+}
+
+static int nvme_auth_process_dhchap_challenge(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_challenge_data *data = chap->buf;
+	u16 dhvlen = le16_to_cpu(data->dhvlen);
+	size_t size = sizeof(*data) + data->hl + dhvlen;
+	const char *hmac_name, *kpp_name;
+
+	if (chap->buf_size < size) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return NVME_SC_INVALID_FIELD;
+	}
+
+	hmac_name = nvme_auth_hmac_name(data->hashid);
+	if (!hmac_name) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid HASH ID %d\n",
+			 chap->qid, data->hashid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
+		return NVME_SC_INVALID_FIELD;
+	}
+
+	if (chap->hash_id == data->hashid && chap->shash_tfm &&
+	    !strcmp(crypto_shash_alg_name(chap->shash_tfm), hmac_name) &&
+	    crypto_shash_digestsize(chap->shash_tfm) == data->hl) {
+		dev_dbg(ctrl->device,
+			"qid %d: reuse existing hash %s\n",
+			chap->qid, hmac_name);
+		goto select_kpp;
+	}
+
+	/* Reset if hash cannot be reused */
+	if (chap->shash_tfm) {
+		crypto_free_shash(chap->shash_tfm);
+		chap->hash_id = 0;
+		chap->hash_len = 0;
+	}
+	chap->shash_tfm = crypto_alloc_shash(hmac_name, 0,
+					     CRYPTO_ALG_ALLOCATES_MEMORY);
+	if (IS_ERR(chap->shash_tfm)) {
+		dev_warn(ctrl->device,
+			 "qid %d: failed to allocate hash %s, error %ld\n",
+			 chap->qid, hmac_name, PTR_ERR(chap->shash_tfm));
+		chap->shash_tfm = NULL;
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_FAILED;
+		return NVME_SC_AUTH_REQUIRED;
+	}
+
+	if (crypto_shash_digestsize(chap->shash_tfm) != data->hl) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid hash length %d\n",
+			 chap->qid, data->hl);
+		crypto_free_shash(chap->shash_tfm);
+		chap->shash_tfm = NULL;
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
+		return NVME_SC_AUTH_REQUIRED;
+	}
+
+	/* Reset host response if the hash had been changed */
+	if (chap->hash_id != data->hashid) {
+		kfree(chap->host_response);
+		chap->host_response = NULL;
+	}
+
+	chap->hash_id = data->hashid;
+	chap->hash_len = data->hl;
+	dev_dbg(ctrl->device, "qid %d: selected hash %s\n",
+		chap->qid, hmac_name);
+
+select_kpp:
+	kpp_name = nvme_auth_dhgroup_kpp(data->dhgid);
+	if (!kpp_name) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid DH group id %d\n",
+			 chap->qid, data->dhgid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
+		return NVME_SC_AUTH_REQUIRED;
+	}
+
+	if (data->dhgid != NVME_AUTH_DHGROUP_NULL) {
+		dev_warn(ctrl->device,
+			 "qid %d: unsupported DH group %s\n",
+			 chap->qid, kpp_name);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
+		return NVME_SC_AUTH_REQUIRED;
+	} else if (dhvlen != 0) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid DH value for NULL DH\n",
+			 chap->qid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return NVME_SC_INVALID_FIELD;
+	}
+	chap->dhgroup_id = data->dhgid;
+
+	chap->s1 = le32_to_cpu(data->seqnum);
+	memcpy(chap->c1, data->cval, chap->hash_len);
+
+	return 0;
+}
+
+static int nvme_auth_set_dhchap_reply_data(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_reply_data *data = chap->buf;
+	size_t size = sizeof(*data);
+
+	size += 2 * chap->hash_len;
+
+	if (chap->buf_size < size) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return -EINVAL;
+	}
+
+	memset(chap->buf, 0, size);
+	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_REPLY;
+	data->t_id = cpu_to_le16(chap->transaction);
+	data->hl = chap->hash_len;
+	data->dhvlen = 0;
+	memcpy(data->rval, chap->response, chap->hash_len);
+	if (ctrl->opts->dhchap_ctrl_secret) {
+		get_random_bytes(chap->c2, chap->hash_len);
+		data->cvalid = 1;
+		chap->s2 = nvme_auth_get_seqnum();
+		memcpy(data->rval + chap->hash_len, chap->c2,
+		       chap->hash_len);
+		dev_dbg(ctrl->device, "%s: qid %d ctrl challenge %*ph\n",
+			__func__, chap->qid, (int)chap->hash_len, chap->c2);
+	} else {
+		memset(chap->c2, 0, chap->hash_len);
+		chap->s2 = 0;
+	}
+	data->seqnum = cpu_to_le32(chap->s2);
+	return size;
+}
+
+static int nvme_auth_process_dhchap_success1(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_success1_data *data = chap->buf;
+	size_t size = sizeof(*data);
+
+	if (ctrl->opts->dhchap_ctrl_secret)
+		size += chap->hash_len;
+
+	if (chap->buf_size < size) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return NVME_SC_INVALID_FIELD;
+	}
+
+	if (data->hl != chap->hash_len) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid hash length %u\n",
+			 chap->qid, data->hl);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
+		return NVME_SC_INVALID_FIELD;
+	}
+
+	/* Just print out information for the admin queue */
+	if (chap->qid == 0)
+		dev_info(ctrl->device,
+			 "qid 0: authenticated with hash %s dhgroup %s\n",
+			 nvme_auth_hmac_name(chap->hash_id),
+			 nvme_auth_dhgroup_name(chap->dhgroup_id));
+
+	if (!data->rvalid)
+		return 0;
+
+	/* Validate controller response */
+	if (memcmp(chap->response, data->rval, data->hl)) {
+		dev_dbg(ctrl->device, "%s: qid %d ctrl response %*ph\n",
+			__func__, chap->qid, (int)chap->hash_len, data->rval);
+		dev_dbg(ctrl->device, "%s: qid %d host response %*ph\n",
+			__func__, chap->qid, (int)chap->hash_len,
+			chap->response);
+		dev_warn(ctrl->device,
+			 "qid %d: controller authentication failed\n",
+			 chap->qid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_FAILED;
+		return NVME_SC_AUTH_REQUIRED;
+	}
+
+	/* Just print out information for the admin queue */
+	if (chap->qid == 0)
+		dev_info(ctrl->device,
+			 "qid 0: controller authenticated\n");
+	return 0;
+}
+
+static int nvme_auth_set_dhchap_success2_data(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_success2_data *data = chap->buf;
+	size_t size = sizeof(*data);
+
+	memset(chap->buf, 0, size);
+	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_SUCCESS2;
+	data->t_id = cpu_to_le16(chap->transaction);
+
+	return size;
+}
+
+static int nvme_auth_set_dhchap_failure2_data(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_failure_data *data = chap->buf;
+	size_t size = sizeof(*data);
+
+	memset(chap->buf, 0, size);
+	data->auth_type = NVME_AUTH_COMMON_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_FAILURE2;
+	data->t_id = cpu_to_le16(chap->transaction);
+	data->rescode = NVME_AUTH_DHCHAP_FAILURE_REASON_FAILED;
+	data->rescode_exp = chap->status;
+
+	return size;
+}
+
+static int nvme_auth_dhchap_setup_host_response(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
+	u8 buf[4], *challenge = chap->c1;
+	int ret;
+
+	dev_dbg(ctrl->device, "%s: qid %d host response seq %u transaction %d\n",
+		__func__, chap->qid, chap->s1, chap->transaction);
+
+	if (!chap->host_response) {
+		chap->host_response = nvme_auth_transform_key(ctrl->host_key,
+						ctrl->opts->host->nqn);
+		if (IS_ERR(chap->host_response)) {
+			ret = PTR_ERR(chap->host_response);
+			chap->host_response = NULL;
+			return ret;
+		}
+	} else {
+		dev_dbg(ctrl->device, "%s: qid %d re-using host response\n",
+			__func__, chap->qid);
+	}
+
+	ret = crypto_shash_setkey(chap->shash_tfm,
+			chap->host_response, ctrl->host_key->len);
+	if (ret) {
+		dev_warn(ctrl->device, "qid %d: failed to set key, error %d\n",
+			 chap->qid, ret);
+		goto out;
+	}
+
+	shash->tfm = chap->shash_tfm;
+	ret = crypto_shash_init(shash);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, challenge, chap->hash_len);
+	if (ret)
+		goto out;
+	put_unaligned_le32(chap->s1, buf);
+	ret = crypto_shash_update(shash, buf, 4);
+	if (ret)
+		goto out;
+	put_unaligned_le16(chap->transaction, buf);
+	ret = crypto_shash_update(shash, buf, 2);
+	if (ret)
+		goto out;
+	memset(buf, 0, sizeof(buf));
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, "HostHost", 8);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
+				  strlen(ctrl->opts->host->nqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
+			    strlen(ctrl->opts->subsysnqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_final(shash, chap->response);
+out:
+	if (challenge != chap->c1)
+		kfree(challenge);
+	return ret;
+}
+
+static int nvme_auth_dhchap_setup_ctrl_response(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
+	u8 *ctrl_response;
+	u8 buf[4], *challenge = chap->c2;
+	int ret;
+
+	ctrl_response = nvme_auth_transform_key(ctrl->ctrl_key,
+				ctrl->opts->subsysnqn);
+	if (IS_ERR(ctrl_response)) {
+		ret = PTR_ERR(ctrl_response);
+		return ret;
+	}
+	ret = crypto_shash_setkey(chap->shash_tfm,
+			ctrl_response, ctrl->ctrl_key->len);
+	if (ret) {
+		dev_warn(ctrl->device, "qid %d: failed to set key, error %d\n",
+			 chap->qid, ret);
+		goto out;
+	}
+
+	dev_dbg(ctrl->device, "%s: qid %d ctrl response seq %u transaction %d\n",
+		__func__, chap->qid, chap->s2, chap->transaction);
+	dev_dbg(ctrl->device, "%s: qid %d challenge %*ph\n",
+		__func__, chap->qid, (int)chap->hash_len, challenge);
+	dev_dbg(ctrl->device, "%s: qid %d subsysnqn %s\n",
+		__func__, chap->qid, ctrl->opts->subsysnqn);
+	dev_dbg(ctrl->device, "%s: qid %d hostnqn %s\n",
+		__func__, chap->qid, ctrl->opts->host->nqn);
+	shash->tfm = chap->shash_tfm;
+	ret = crypto_shash_init(shash);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, challenge, chap->hash_len);
+	if (ret)
+		goto out;
+	put_unaligned_le32(chap->s2, buf);
+	ret = crypto_shash_update(shash, buf, 4);
+	if (ret)
+		goto out;
+	put_unaligned_le16(chap->transaction, buf);
+	ret = crypto_shash_update(shash, buf, 2);
+	if (ret)
+		goto out;
+	memset(buf, 0, 4);
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, "Controller", 10);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
+				  strlen(ctrl->opts->subsysnqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
+				  strlen(ctrl->opts->host->nqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_final(shash, chap->response);
+out:
+	if (challenge != chap->c2)
+		kfree(challenge);
+	return ret;
+}
+
+static void __nvme_auth_reset(struct nvme_dhchap_queue_context *chap)
+{
+	chap->status = 0;
+	chap->error = 0;
+	chap->s1 = 0;
+	chap->s2 = 0;
+	chap->transaction = 0;
+	memset(chap->c1, 0, sizeof(chap->c1));
+	memset(chap->c2, 0, sizeof(chap->c2));
+}
+
+static void __nvme_auth_free(struct nvme_dhchap_queue_context *chap)
+{
+	__nvme_auth_reset(chap);
+	if (chap->shash_tfm)
+		crypto_free_shash(chap->shash_tfm);
+	kfree_sensitive(chap->host_response);
+	kfree(chap->buf);
+	kfree(chap);
+}
+
+static void __nvme_auth_work(struct work_struct *work)
+{
+	struct nvme_dhchap_queue_context *chap =
+		container_of(work, struct nvme_dhchap_queue_context, auth_work);
+	struct nvme_ctrl *ctrl = chap->ctrl;
+	size_t tl;
+	int ret = 0;
+
+	chap->transaction = ctrl->transaction++;
+
+	/* DH-HMAC-CHAP Step 1: send negotiate */
+	dev_dbg(ctrl->device, "%s: qid %d send negotiate\n",
+		__func__, chap->qid);
+	ret = nvme_auth_set_dhchap_negotiate_data(ctrl, chap);
+	if (ret < 0) {
+		chap->error = ret;
+		return;
+	}
+	tl = ret;
+	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, tl, true);
+	if (ret) {
+		chap->error = ret;
+		return;
+	}
+
+	/* DH-HMAC-CHAP Step 2: receive challenge */
+	dev_dbg(ctrl->device, "%s: qid %d receive challenge\n",
+		__func__, chap->qid);
+
+	memset(chap->buf, 0, chap->buf_size);
+	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, chap->buf_size, false);
+	if (ret) {
+		dev_warn(ctrl->device,
+			 "qid %d failed to receive challenge, %s %d\n",
+			 chap->qid, ret < 0 ? "error" : "nvme status", ret);
+		chap->error = ret;
+		return;
+	}
+	ret = nvme_auth_receive_validate(ctrl, chap->qid, chap->buf, chap->transaction,
+					 NVME_AUTH_DHCHAP_MESSAGE_CHALLENGE);
+	if (ret) {
+		chap->status = ret;
+		chap->error = NVME_SC_AUTH_REQUIRED;
+		return;
+	}
+
+	ret = nvme_auth_process_dhchap_challenge(ctrl, chap);
+	if (ret) {
+		/* Invalid challenge parameters */
+		goto fail2;
+	}
+
+	dev_dbg(ctrl->device, "%s: qid %d host response\n",
+		__func__, chap->qid);
+	ret = nvme_auth_dhchap_setup_host_response(ctrl, chap);
+	if (ret)
+		goto fail2;
+
+	/* DH-HMAC-CHAP Step 3: send reply */
+	dev_dbg(ctrl->device, "%s: qid %d send reply\n",
+		__func__, chap->qid);
+	ret = nvme_auth_set_dhchap_reply_data(ctrl, chap);
+	if (ret < 0)
+		goto fail2;
+
+	tl = ret;
+	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, tl, true);
+	if (ret)
+		goto fail2;
+
+	/* DH-HMAC-CHAP Step 4: receive success1 */
+	dev_dbg(ctrl->device, "%s: qid %d receive success1\n",
+		__func__, chap->qid);
+
+	memset(chap->buf, 0, chap->buf_size);
+	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, chap->buf_size, false);
+	if (ret) {
+		dev_warn(ctrl->device,
+			 "qid %d failed to receive success1, %s %d\n",
+			 chap->qid, ret < 0 ? "error" : "nvme status", ret);
+		chap->error = ret;
+		return;
+	}
+	ret = nvme_auth_receive_validate(ctrl, chap->qid,
+					 chap->buf, chap->transaction,
+					 NVME_AUTH_DHCHAP_MESSAGE_SUCCESS1);
+	if (ret) {
+		chap->status = ret;
+		chap->error = NVME_SC_AUTH_REQUIRED;
+		return;
+	}
+
+	if (ctrl->opts->dhchap_ctrl_secret) {
+		dev_dbg(ctrl->device,
+			"%s: qid %d controller response\n",
+			__func__, chap->qid);
+		ret = nvme_auth_dhchap_setup_ctrl_response(ctrl, chap);
+		if (ret) {
+			chap->error = ret;
+			goto fail2;
+		}
+	}
+
+	ret = nvme_auth_process_dhchap_success1(ctrl, chap);
+	if (ret) {
+		/* Controller authentication failed */
+		chap->error = NVME_SC_AUTH_REQUIRED;
+		goto fail2;
+	}
+
+	if (ctrl->opts->dhchap_ctrl_secret) {
+		/* DH-HMAC-CHAP Step 5: send success2 */
+		dev_dbg(ctrl->device, "%s: qid %d send success2\n",
+			__func__, chap->qid);
+		tl = nvme_auth_set_dhchap_success2_data(ctrl, chap);
+		ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, tl, true);
+	}
+	if (!ret) {
+		chap->error = 0;
+		return;
+	}
+
+fail2:
+	dev_dbg(ctrl->device, "%s: qid %d send failure2, status %x\n",
+		__func__, chap->qid, chap->status);
+	tl = nvme_auth_set_dhchap_failure2_data(ctrl, chap);
+	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, tl, true);
+	/*
+	 * only update error if send failure2 failed and no other
+	 * error had been set during authentication.
+	 */
+	if (ret && !chap->error)
+		chap->error = ret;
+}
+
+int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid)
+{
+	struct nvme_dhchap_queue_context *chap;
+
+	if (!ctrl->host_key) {
+		dev_warn(ctrl->device, "qid %d: no key\n", qid);
+		return -ENOKEY;
+	}
+
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	/* Check if the context is already queued */
+	list_for_each_entry(chap, &ctrl->dhchap_auth_list, entry) {
+		WARN_ON(!chap->buf);
+		if (chap->qid == qid) {
+			dev_dbg(ctrl->device, "qid %d: re-using context\n", qid);
+			mutex_unlock(&ctrl->dhchap_auth_mutex);
+			flush_work(&chap->auth_work);
+			__nvme_auth_reset(chap);
+			queue_work(nvme_wq, &chap->auth_work);
+			return 0;
+		}
+	}
+	chap = kzalloc(sizeof(*chap), GFP_KERNEL);
+	if (!chap) {
+		mutex_unlock(&ctrl->dhchap_auth_mutex);
+		return -ENOMEM;
+	}
+	chap->qid = (qid == NVME_QID_ANY) ? 0 : qid;
+	chap->ctrl = ctrl;
+
+	/*
+	 * Allocate a large enough buffer for the entire negotiation:
+	 * 4k should be enough to ffdhe8192.
+	 */
+	chap->buf_size = 4096;
+	chap->buf = kzalloc(chap->buf_size, GFP_KERNEL);
+	if (!chap->buf) {
+		mutex_unlock(&ctrl->dhchap_auth_mutex);
+		kfree(chap);
+		return -ENOMEM;
+	}
+
+	INIT_WORK(&chap->auth_work, __nvme_auth_work);
+	list_add(&chap->entry, &ctrl->dhchap_auth_list);
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+	queue_work(nvme_wq, &chap->auth_work);
+	return 0;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_negotiate);
+
+int nvme_auth_wait(struct nvme_ctrl *ctrl, int qid)
+{
+	struct nvme_dhchap_queue_context *chap;
+	int ret;
+
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	list_for_each_entry(chap, &ctrl->dhchap_auth_list, entry) {
+		if (chap->qid != qid)
+			continue;
+		mutex_unlock(&ctrl->dhchap_auth_mutex);
+		flush_work(&chap->auth_work);
+		ret = chap->error;
+		return ret;
+	}
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+	return -ENXIO;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_wait);
+
+void nvme_auth_reset(struct nvme_ctrl *ctrl)
+{
+	struct nvme_dhchap_queue_context *chap;
+
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	list_for_each_entry(chap, &ctrl->dhchap_auth_list, entry) {
+		mutex_unlock(&ctrl->dhchap_auth_mutex);
+		flush_work(&chap->auth_work);
+		__nvme_auth_reset(chap);
+	}
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_reset);
+
+static void nvme_dhchap_auth_work(struct work_struct *work)
+{
+	struct nvme_ctrl *ctrl =
+		container_of(work, struct nvme_ctrl, dhchap_auth_work);
+	int ret, q;
+
+	/* Authenticate admin queue first */
+	ret = nvme_auth_negotiate(ctrl, 0);
+	if (ret) {
+		dev_warn(ctrl->device,
+			 "qid 0: error %d setting up authentication\n", ret);
+		return;
+	}
+	ret = nvme_auth_wait(ctrl, 0);
+	if (ret) {
+		dev_warn(ctrl->device,
+			 "qid 0: authentication failed\n");
+		return;
+	}
+
+	for (q = 1; q < ctrl->queue_count; q++) {
+		ret = nvme_auth_negotiate(ctrl, q);
+		if (ret) {
+			dev_warn(ctrl->device,
+				 "qid %d: error %d setting up authentication\n",
+				 q, ret);
+			break;
+		}
+	}
+
+	/*
+	 * Failure is a soft-state; credentials remain valid until
+	 * the controller terminates the connection.
+	 */
+}
+
+void nvme_auth_init_ctrl(struct nvme_ctrl *ctrl)
+{
+	INIT_LIST_HEAD(&ctrl->dhchap_auth_list);
+	INIT_WORK(&ctrl->dhchap_auth_work, nvme_dhchap_auth_work);
+	mutex_init(&ctrl->dhchap_auth_mutex);
+	if (!ctrl->opts)
+		return;
+	nvme_auth_generate_key(ctrl->opts->dhchap_secret, &ctrl->host_key);
+	nvme_auth_generate_key(ctrl->opts->dhchap_ctrl_secret, &ctrl->ctrl_key);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_init_ctrl);
+
+void nvme_auth_stop(struct nvme_ctrl *ctrl)
+{
+	struct nvme_dhchap_queue_context *chap = NULL, *tmp;
+
+	cancel_work_sync(&ctrl->dhchap_auth_work);
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	list_for_each_entry_safe(chap, tmp, &ctrl->dhchap_auth_list, entry)
+		cancel_work_sync(&chap->auth_work);
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_stop);
+
+void nvme_auth_free(struct nvme_ctrl *ctrl)
+{
+	struct nvme_dhchap_queue_context *chap = NULL, *tmp;
+
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	list_for_each_entry_safe(chap, tmp, &ctrl->dhchap_auth_list, entry) {
+		list_del_init(&chap->entry);
+		flush_work(&chap->auth_work);
+		__nvme_auth_free(chap);
+	}
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+	if (ctrl->host_key) {
+		nvme_auth_free_key(ctrl->host_key);
+		ctrl->host_key = NULL;
+	}
+	if (ctrl->ctrl_key) {
+		nvme_auth_free_key(ctrl->ctrl_key);
+		ctrl->ctrl_key = NULL;
+	}
+}
+EXPORT_SYMBOL_GPL(nvme_auth_free);
diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 57480075550c..9071a9798996 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -24,6 +24,7 @@
 
 #include "nvme.h"
 #include "fabrics.h"
+#include <linux/nvme-auth.h>
 
 #define CREATE_TRACE_POINTS
 #include "trace.h"
@@ -330,6 +331,7 @@ enum nvme_disposition {
 	COMPLETE,
 	RETRY,
 	FAILOVER,
+	AUTHENTICATE,
 };
 
 static inline enum nvme_disposition nvme_decide_disposition(struct request *req)
@@ -337,6 +339,9 @@ static inline enum nvme_disposition nvme_decide_disposition(struct request *req)
 	if (likely(nvme_req(req)->status == 0))
 		return COMPLETE;
 
+	if ((nvme_req(req)->status & 0x7ff) == NVME_SC_AUTH_REQUIRED)
+		return AUTHENTICATE;
+
 	if (blk_noretry_request(req) ||
 	    (nvme_req(req)->status & NVME_SC_DNR) ||
 	    nvme_req(req)->retries >= nvme_max_retries)
@@ -375,11 +380,13 @@ static inline void nvme_end_req(struct request *req)
 
 void nvme_complete_rq(struct request *req)
 {
+	struct nvme_ctrl *ctrl = nvme_req(req)->ctrl;
+
 	trace_nvme_complete_rq(req);
 	nvme_cleanup_cmd(req);
 
-	if (nvme_req(req)->ctrl->kas)
-		nvme_req(req)->ctrl->comp_seen = true;
+	if (ctrl->kas)
+		ctrl->comp_seen = true;
 
 	switch (nvme_decide_disposition(req)) {
 	case COMPLETE:
@@ -391,6 +398,14 @@ void nvme_complete_rq(struct request *req)
 	case FAILOVER:
 		nvme_failover_req(req);
 		return;
+	case AUTHENTICATE:
+#ifdef CONFIG_NVME_AUTH
+		queue_work(nvme_wq, &ctrl->dhchap_auth_work);
+		nvme_retry_req(req);
+#else
+		nvme_end_req(req);
+#endif
+		return;
 	}
 }
 EXPORT_SYMBOL_GPL(nvme_complete_rq);
@@ -702,7 +717,9 @@ bool __nvme_check_ready(struct nvme_ctrl *ctrl, struct request *rq,
 		switch (ctrl->state) {
 		case NVME_CTRL_CONNECTING:
 			if (blk_rq_is_passthrough(rq) && nvme_is_fabrics(req->cmd) &&
-			    req->cmd->fabrics.fctype == nvme_fabrics_type_connect)
+			    (req->cmd->fabrics.fctype == nvme_fabrics_type_connect ||
+			     req->cmd->fabrics.fctype == nvme_fabrics_type_auth_send ||
+			     req->cmd->fabrics.fctype == nvme_fabrics_type_auth_receive))
 				return true;
 			break;
 		default:
@@ -3595,6 +3612,108 @@ static ssize_t dctype_show(struct device *dev,
 }
 static DEVICE_ATTR_RO(dctype);
 
+#ifdef CONFIG_NVME_AUTH
+static ssize_t nvme_ctrl_dhchap_secret_show(struct device *dev,
+		struct device_attribute *attr, char *buf)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	struct nvmf_ctrl_options *opts = ctrl->opts;
+
+	if (!opts->dhchap_secret)
+		return sysfs_emit(buf, "none\n");
+	return sysfs_emit(buf, "%s\n", opts->dhchap_secret);
+}
+
+static ssize_t nvme_ctrl_dhchap_secret_store(struct device *dev,
+		struct device_attribute *attr, const char *buf, size_t count)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	struct nvmf_ctrl_options *opts = ctrl->opts;
+	char *dhchap_secret;
+
+	if (!ctrl->opts->dhchap_secret)
+		return -EINVAL;
+	if (count < 7)
+		return -EINVAL;
+	if (memcmp(buf, "DHHC-1:", 7))
+		return -EINVAL;
+
+	dhchap_secret = kzalloc(count + 1, GFP_KERNEL);
+	if (!dhchap_secret)
+		return -ENOMEM;
+	memcpy(dhchap_secret, buf, count);
+	nvme_auth_stop(ctrl);
+	if (strcmp(dhchap_secret, opts->dhchap_secret)) {
+		int ret;
+
+		ret = nvme_auth_generate_key(dhchap_secret, &ctrl->host_key);
+		if (ret)
+			return ret;
+		kfree(opts->dhchap_secret);
+		opts->dhchap_secret = dhchap_secret;
+		/* Key has changed; re-authentication with new key */
+		nvme_auth_reset(ctrl);
+	}
+	/* Start re-authentication */
+	dev_info(ctrl->device, "re-authenticating controller\n");
+	queue_work(nvme_wq, &ctrl->dhchap_auth_work);
+
+	return count;
+}
+static DEVICE_ATTR(dhchap_secret, S_IRUGO | S_IWUSR,
+	nvme_ctrl_dhchap_secret_show, nvme_ctrl_dhchap_secret_store);
+
+static ssize_t nvme_ctrl_dhchap_ctrl_secret_show(struct device *dev,
+		struct device_attribute *attr, char *buf)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	struct nvmf_ctrl_options *opts = ctrl->opts;
+
+	if (!opts->dhchap_ctrl_secret)
+		return sysfs_emit(buf, "none\n");
+	return sysfs_emit(buf, "%s\n", opts->dhchap_ctrl_secret);
+}
+
+static ssize_t nvme_ctrl_dhchap_ctrl_secret_store(struct device *dev,
+		struct device_attribute *attr, const char *buf, size_t count)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	struct nvmf_ctrl_options *opts = ctrl->opts;
+	char *dhchap_secret;
+
+	if (!ctrl->opts->dhchap_ctrl_secret)
+		return -EINVAL;
+	if (count < 7)
+		return -EINVAL;
+	if (memcmp(buf, "DHHC-1:", 7))
+		return -EINVAL;
+
+	dhchap_secret = kzalloc(count + 1, GFP_KERNEL);
+	if (!dhchap_secret)
+		return -ENOMEM;
+	memcpy(dhchap_secret, buf, count);
+	nvme_auth_stop(ctrl);
+	if (strcmp(dhchap_secret, opts->dhchap_ctrl_secret)) {
+		int ret;
+
+		ret = nvme_auth_generate_key(dhchap_secret, &ctrl->ctrl_key);
+		if (ret)
+			return ret;
+		kfree(opts->dhchap_ctrl_secret);
+		opts->dhchap_ctrl_secret = dhchap_secret;
+		/* Key has changed; re-authentication with new key */
+		nvme_auth_reset(ctrl);
+	}
+	/* Start re-authentication */
+	dev_info(ctrl->device, "re-authenticating controller\n");
+	queue_work(nvme_wq, &ctrl->dhchap_auth_work);
+
+	return count;
+}
+static DEVICE_ATTR(dhchap_ctrl_secret, S_IRUGO | S_IWUSR,
+	nvme_ctrl_dhchap_ctrl_secret_show, nvme_ctrl_dhchap_ctrl_secret_store);
+#endif
+
 static struct attribute *nvme_dev_attrs[] = {
 	&dev_attr_reset_controller.attr,
 	&dev_attr_rescan_controller.attr,
@@ -3618,6 +3737,10 @@ static struct attribute *nvme_dev_attrs[] = {
 	&dev_attr_kato.attr,
 	&dev_attr_cntrltype.attr,
 	&dev_attr_dctype.attr,
+#ifdef CONFIG_NVME_AUTH
+	&dev_attr_dhchap_secret.attr,
+	&dev_attr_dhchap_ctrl_secret.attr,
+#endif
 	NULL
 };
 
@@ -3641,6 +3764,12 @@ static umode_t nvme_dev_attrs_are_visible(struct kobject *kobj,
 		return 0;
 	if (a == &dev_attr_fast_io_fail_tmo.attr && !ctrl->opts)
 		return 0;
+#ifdef CONFIG_NVME_AUTH
+	if (a == &dev_attr_dhchap_secret.attr && !ctrl->opts)
+		return 0;
+	if (a == &dev_attr_dhchap_ctrl_secret.attr && !ctrl->opts)
+		return 0;
+#endif
 
 	return a->mode;
 }
@@ -4535,8 +4664,10 @@ static void nvme_handle_aen_notice(struct nvme_ctrl *ctrl, u32 result)
 		 * recovery actions from interfering with the controller's
 		 * firmware activation.
 		 */
-		if (nvme_change_ctrl_state(ctrl, NVME_CTRL_RESETTING))
+		if (nvme_change_ctrl_state(ctrl, NVME_CTRL_RESETTING)) {
+			nvme_auth_stop(ctrl);
 			queue_work(nvme_wq, &ctrl->fw_act_work);
+		}
 		break;
 #ifdef CONFIG_NVME_MULTIPATH
 	case NVME_AER_NOTICE_ANA:
@@ -4600,6 +4731,7 @@ EXPORT_SYMBOL_GPL(nvme_complete_async_event);
 void nvme_stop_ctrl(struct nvme_ctrl *ctrl)
 {
 	nvme_mpath_stop(ctrl);
+	nvme_auth_stop(ctrl);
 	nvme_stop_keep_alive(ctrl);
 	nvme_stop_failfast_work(ctrl);
 	flush_work(&ctrl->async_event_work);
@@ -4657,6 +4789,8 @@ static void nvme_free_ctrl(struct device *dev)
 
 	nvme_free_cels(ctrl);
 	nvme_mpath_uninit(ctrl);
+	nvme_auth_stop(ctrl);
+	nvme_auth_free(ctrl);
 	__free_page(ctrl->discard_page);
 
 	if (subsys) {
@@ -4747,6 +4881,7 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev,
 
 	nvme_fault_inject_init(&ctrl->fault_inject, dev_name(ctrl->device));
 	nvme_mpath_init_ctrl(ctrl);
+	nvme_auth_init_ctrl(ctrl);
 
 	return 0;
 out_free_name:
diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index e4b1520862d8..fba777ba719e 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -369,6 +369,7 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)
 	union nvme_result res;
 	struct nvmf_connect_data *data;
 	int ret;
+	u32 result;
 
 	cmd.connect.opcode = nvme_fabrics_command;
 	cmd.connect.fctype = nvme_fabrics_type_connect;
@@ -401,8 +402,25 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)
 		goto out_free_data;
 	}
 
-	ctrl->cntlid = le16_to_cpu(res.u16);
-
+	result = le32_to_cpu(res.u32);
+	ctrl->cntlid = result & 0xFFFF;
+	if ((result >> 16) & 0x3) {
+		/* Authentication required */
+		ret = nvme_auth_negotiate(ctrl, 0);
+		if (ret) {
+			dev_warn(ctrl->device,
+				 "qid 0: authentication setup failed\n");
+			ret = NVME_SC_AUTH_REQUIRED;
+			goto out_free_data;
+		}
+		ret = nvme_auth_wait(ctrl, 0);
+		if (ret)
+			dev_warn(ctrl->device,
+				 "qid 0: authentication failed\n");
+		else
+			dev_info(ctrl->device,
+				 "qid 0: authenticated\n");
+	}
 out_free_data:
 	kfree(data);
 	return ret;
@@ -435,6 +453,7 @@ int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, u16 qid)
 	struct nvmf_connect_data *data;
 	union nvme_result res;
 	int ret;
+	u32 result;
 
 	cmd.connect.opcode = nvme_fabrics_command;
 	cmd.connect.fctype = nvme_fabrics_type_connect;
@@ -460,6 +479,21 @@ int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, u16 qid)
 		nvmf_log_connect_error(ctrl, ret, le32_to_cpu(res.u32),
 				       &cmd, data);
 	}
+	result = le32_to_cpu(res.u32);
+	if ((result >> 16) & 2) {
+		/* Authentication required */
+		ret = nvme_auth_negotiate(ctrl, qid);
+		if (ret) {
+			dev_warn(ctrl->device,
+				 "qid %d: authentication setup failed\n", qid);
+			ret = NVME_SC_AUTH_REQUIRED;
+		} else {
+			ret = nvme_auth_wait(ctrl, qid);
+			if (ret)
+				dev_warn(ctrl->device,
+					 "qid %u: authentication failed\n", qid);
+		}
+	}
 	kfree(data);
 	return ret;
 }
@@ -552,6 +586,8 @@ static const match_table_t opt_tokens = {
 	{ NVMF_OPT_TOS,			"tos=%d"		},
 	{ NVMF_OPT_FAIL_FAST_TMO,	"fast_io_fail_tmo=%d"	},
 	{ NVMF_OPT_DISCOVERY,		"discovery"		},
+	{ NVMF_OPT_DHCHAP_SECRET,	"dhchap_secret=%s"	},
+	{ NVMF_OPT_DHCHAP_CTRL_SECRET,	"dhchap_ctrl_secret=%s"	},
 	{ NVMF_OPT_ERR,			NULL			}
 };
 
@@ -833,6 +869,34 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
 		case NVMF_OPT_DISCOVERY:
 			opts->discovery_nqn = true;
 			break;
+		case NVMF_OPT_DHCHAP_SECRET:
+			p = match_strdup(args);
+			if (!p) {
+				ret = -ENOMEM;
+				goto out;
+			}
+			if (strlen(p) < 11 || strncmp(p, "DHHC-1:", 7)) {
+				pr_err("Invalid DH-CHAP secret %s\n", p);
+				ret = -EINVAL;
+				goto out;
+			}
+			kfree(opts->dhchap_secret);
+			opts->dhchap_secret = p;
+			break;
+		case NVMF_OPT_DHCHAP_CTRL_SECRET:
+			p = match_strdup(args);
+			if (!p) {
+				ret = -ENOMEM;
+				goto out;
+			}
+			if (strlen(p) < 11 || strncmp(p, "DHHC-1:", 7)) {
+				pr_err("Invalid DH-CHAP secret %s\n", p);
+				ret = -EINVAL;
+				goto out;
+			}
+			kfree(opts->dhchap_ctrl_secret);
+			opts->dhchap_ctrl_secret = p;
+			break;
 		default:
 			pr_warn("unknown parameter or missing value '%s' in ctrl creation request\n",
 				p);
@@ -951,6 +1015,7 @@ void nvmf_free_options(struct nvmf_ctrl_options *opts)
 	kfree(opts->subsysnqn);
 	kfree(opts->host_traddr);
 	kfree(opts->host_iface);
+	kfree(opts->dhchap_secret);
 	kfree(opts);
 }
 EXPORT_SYMBOL_GPL(nvmf_free_options);
@@ -960,7 +1025,8 @@ EXPORT_SYMBOL_GPL(nvmf_free_options);
 				 NVMF_OPT_KATO | NVMF_OPT_HOSTNQN | \
 				 NVMF_OPT_HOST_ID | NVMF_OPT_DUP_CONNECT |\
 				 NVMF_OPT_DISABLE_SQFLOW | NVMF_OPT_DISCOVERY |\
-				 NVMF_OPT_FAIL_FAST_TMO)
+				 NVMF_OPT_FAIL_FAST_TMO | NVMF_OPT_DHCHAP_SECRET |\
+				 NVMF_OPT_DHCHAP_CTRL_SECRET)
 
 static struct nvme_ctrl *
 nvmf_create_ctrl(struct device *dev, const char *buf)
@@ -1196,7 +1262,14 @@ static void __exit nvmf_exit(void)
 	BUILD_BUG_ON(sizeof(struct nvmf_connect_command) != 64);
 	BUILD_BUG_ON(sizeof(struct nvmf_property_get_command) != 64);
 	BUILD_BUG_ON(sizeof(struct nvmf_property_set_command) != 64);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_send_command) != 64);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_receive_command) != 64);
 	BUILD_BUG_ON(sizeof(struct nvmf_connect_data) != 1024);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_negotiate_data) != 8);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_challenge_data) != 16);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_reply_data) != 16);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_success1_data) != 16);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_success2_data) != 16);
 }
 
 MODULE_LICENSE("GPL v2");
diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h
index 46d6e194ac2b..a6e22116e139 100644
--- a/drivers/nvme/host/fabrics.h
+++ b/drivers/nvme/host/fabrics.h
@@ -68,6 +68,8 @@ enum {
 	NVMF_OPT_FAIL_FAST_TMO	= 1 << 20,
 	NVMF_OPT_HOST_IFACE	= 1 << 21,
 	NVMF_OPT_DISCOVERY	= 1 << 22,
+	NVMF_OPT_DHCHAP_SECRET	= 1 << 23,
+	NVMF_OPT_DHCHAP_CTRL_SECRET = 1 << 24,
 };
 
 /**
@@ -97,6 +99,9 @@ enum {
  * @max_reconnects: maximum number of allowed reconnect attempts before removing
  *              the controller, (-1) means reconnect forever, zero means remove
  *              immediately;
+ * @dhchap_secret: DH-HMAC-CHAP secret
+ * @dhchap_ctrl_secret: DH-HMAC-CHAP controller secret for bi-directional
+ *              authentication
  * @disable_sqflow: disable controller sq flow control
  * @hdr_digest: generate/verify header digest (TCP)
  * @data_digest: generate/verify data digest (TCP)
@@ -121,6 +126,8 @@ struct nvmf_ctrl_options {
 	unsigned int		kato;
 	struct nvmf_host	*host;
 	int			max_reconnects;
+	char			*dhchap_secret;
+	char			*dhchap_ctrl_secret;
 	bool			disable_sqflow;
 	bool			hdr_digest;
 	bool			data_digest;
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index e4612dd0b420..e9350bf7b2d1 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -328,6 +328,15 @@ struct nvme_ctrl {
 	struct work_struct ana_work;
 #endif
 
+#ifdef CONFIG_NVME_AUTH
+	struct work_struct dhchap_auth_work;
+	struct list_head dhchap_auth_list;
+	struct mutex dhchap_auth_mutex;
+	struct nvme_dhchap_key *host_key;
+	struct nvme_dhchap_key *ctrl_key;
+	u16 transaction;
+#endif
+
 	/* Power saving configuration */
 	u64 ps_max_latency_us;
 	bool apst_enabled;
@@ -991,6 +1000,27 @@ static inline bool nvme_ctrl_sgl_supported(struct nvme_ctrl *ctrl)
 	return ctrl->sgls & ((1 << 0) | (1 << 1));
 }
 
+#ifdef CONFIG_NVME_AUTH
+void nvme_auth_init_ctrl(struct nvme_ctrl *ctrl);
+void nvme_auth_stop(struct nvme_ctrl *ctrl);
+int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid);
+int nvme_auth_wait(struct nvme_ctrl *ctrl, int qid);
+void nvme_auth_reset(struct nvme_ctrl *ctrl);
+void nvme_auth_free(struct nvme_ctrl *ctrl);
+#else
+static inline void nvme_auth_init_ctrl(struct nvme_ctrl *ctrl) {};
+static inline void nvme_auth_stop(struct nvme_ctrl *ctrl) {};
+static inline int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid)
+{
+	return -EPROTONOSUPPORT;
+}
+static inline int nvme_auth_wait(struct nvme_ctrl *ctrl, int qid)
+{
+	return NVME_SC_AUTH_REQUIRED;
+}
+static inline void nvme_auth_free(struct nvme_ctrl *ctrl) {};
+#endif
+
 u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 			 u8 opcode);
 int nvme_execute_passthru_rq(struct request *rq);
diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index f2a5e1ea508a..84ce3347d158 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -1197,6 +1197,7 @@ static void nvme_rdma_error_recovery_work(struct work_struct *work)
 	struct nvme_rdma_ctrl *ctrl = container_of(work,
 			struct nvme_rdma_ctrl, err_work);
 
+	nvme_auth_stop(&ctrl->ctrl);
 	nvme_stop_keep_alive(&ctrl->ctrl);
 	flush_work(&ctrl->ctrl.async_event_work);
 	nvme_rdma_teardown_io_queues(ctrl, false);
diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index bb67538d241b..a7848e430a5c 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -2174,6 +2174,7 @@ static void nvme_tcp_error_recovery_work(struct work_struct *work)
 				struct nvme_tcp_ctrl, err_work);
 	struct nvme_ctrl *ctrl = &tcp_ctrl->ctrl;
 
+	nvme_auth_stop(ctrl);
 	nvme_stop_keep_alive(ctrl);
 	flush_work(&ctrl->async_event_work);
 	nvme_tcp_teardown_io_queues(ctrl, false);
diff --git a/drivers/nvme/host/trace.c b/drivers/nvme/host/trace.c
index 2a89c5aa0790..1c36fcedea20 100644
--- a/drivers/nvme/host/trace.c
+++ b/drivers/nvme/host/trace.c
@@ -287,6 +287,34 @@ static const char *nvme_trace_fabrics_property_get(struct trace_seq *p, u8 *spc)
 	return ret;
 }
 
+static const char *nvme_trace_fabrics_auth_send(struct trace_seq *p, u8 *spc)
+{
+	const char *ret = trace_seq_buffer_ptr(p);
+	u8 spsp0 = spc[1];
+	u8 spsp1 = spc[2];
+	u8 secp = spc[3];
+	u32 tl = get_unaligned_le32(spc + 4);
+
+	trace_seq_printf(p, "spsp0=%02x, spsp1=%02x, secp=%02x, tl=%u",
+			 spsp0, spsp1, secp, tl);
+	trace_seq_putc(p, 0);
+	return ret;
+}
+
+static const char *nvme_trace_fabrics_auth_receive(struct trace_seq *p, u8 *spc)
+{
+	const char *ret = trace_seq_buffer_ptr(p);
+	u8 spsp0 = spc[1];
+	u8 spsp1 = spc[2];
+	u8 secp = spc[3];
+	u32 al = get_unaligned_le32(spc + 4);
+
+	trace_seq_printf(p, "spsp0=%02x, spsp1=%02x, secp=%02x, al=%u",
+			 spsp0, spsp1, secp, al);
+	trace_seq_putc(p, 0);
+	return ret;
+}
+
 static const char *nvme_trace_fabrics_common(struct trace_seq *p, u8 *spc)
 {
 	const char *ret = trace_seq_buffer_ptr(p);
@@ -306,6 +334,10 @@ const char *nvme_trace_parse_fabrics_cmd(struct trace_seq *p,
 		return nvme_trace_fabrics_connect(p, spc);
 	case nvme_fabrics_type_property_get:
 		return nvme_trace_fabrics_property_get(p, spc);
+	case nvme_fabrics_type_auth_send:
+		return nvme_trace_fabrics_auth_send(p, spc);
+	case nvme_fabrics_type_auth_receive:
+		return nvme_trace_fabrics_auth_receive(p, spc);
 	default:
 		return nvme_trace_fabrics_common(p, spc);
 	}
diff --git a/include/linux/nvme-auth.h b/include/linux/nvme-auth.h
new file mode 100644
index 000000000000..354456826221
--- /dev/null
+++ b/include/linux/nvme-auth.h
@@ -0,0 +1,33 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2021 Hannes Reinecke, SUSE Software Solutions
+ */
+
+#ifndef _NVME_AUTH_H
+#define _NVME_AUTH_H
+
+#include <crypto/kpp.h>
+
+struct nvme_dhchap_key {
+	u8 *key;
+	size_t len;
+	u8 hash;
+};
+
+u32 nvme_auth_get_seqnum(void);
+const char *nvme_auth_dhgroup_name(u8 dhgroup_id);
+const char *nvme_auth_dhgroup_kpp(u8 dhgroup_id);
+u8 nvme_auth_dhgroup_id(const char *dhgroup_name);
+
+const char *nvme_auth_hmac_name(u8 hmac_id);
+const char *nvme_auth_digest_name(u8 hmac_id);
+size_t nvme_auth_hmac_hash_len(u8 hmac_id);
+u8 nvme_auth_hmac_id(const char *hmac_name);
+
+struct nvme_dhchap_key *nvme_auth_extract_key(unsigned char *secret,
+					      u8 key_hash);
+void nvme_auth_free_key(struct nvme_dhchap_key *key);
+u8 *nvme_auth_transform_key(struct nvme_dhchap_key *key, char *nqn);
+int nvme_auth_generate_key(u8 *secret, struct nvme_dhchap_key **ret_key);
+
+#endif /* _NVME_AUTH_H */
-- 
2.29.2

[PATCH 07/11] nvme-auth: Diffie-Hellman key exchange support

[Hannes Reinecke]

Implement Diffie-Hellman key exchange using FFDHE groups
for NVMe In-Band Authentication.

Signed-off-by: Hannes Reinecke <hare@suse.de>
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
---
 drivers/nvme/common/auth.c | 153 ++++++++++++++++++++++++++++
 drivers/nvme/host/Kconfig  |   2 +
 drivers/nvme/host/auth.c   | 199 +++++++++++++++++++++++++++++++++++--
 include/linux/nvme-auth.h  |   8 ++
 4 files changed, 356 insertions(+), 6 deletions(-)

diff --git a/drivers/nvme/common/auth.c b/drivers/nvme/common/auth.c
index 1cccaf6afdd2..945f6bb6eb1f 100644
--- a/drivers/nvme/common/auth.c
+++ b/drivers/nvme/common/auth.c
@@ -301,6 +301,159 @@ u8 *nvme_auth_transform_key(struct nvme_dhchap_key *key, char *nqn)
 }
 EXPORT_SYMBOL_GPL(nvme_auth_transform_key);
 
+static int nvme_auth_hash_skey(int hmac_id, u8 *skey, size_t skey_len, u8 *hkey)
+{
+	const char *digest_name;
+	struct crypto_shash *tfm;
+	int ret;
+
+	digest_name = nvme_auth_digest_name(hmac_id);
+	if (!digest_name) {
+		pr_debug("%s: failed to get digest for %d\n", __func__,
+			 hmac_id);
+		return -EINVAL;
+	}
+	tfm = crypto_alloc_shash(digest_name, 0, 0);
+	if (IS_ERR(tfm))
+		return -ENOMEM;
+
+	ret = crypto_shash_tfm_digest(tfm, skey, skey_len, hkey);
+	if (ret < 0)
+		pr_debug("%s: Failed to hash digest len %zu\n", __func__,
+			 skey_len);
+
+	crypto_free_shash(tfm);
+	return ret;
+}
+
+int nvme_auth_augmented_challenge(u8 hmac_id, u8 *skey, size_t skey_len,
+		u8 *challenge, u8 *aug, size_t hlen)
+{
+	struct crypto_shash *tfm;
+	struct shash_desc *desc;
+	u8 *hashed_key;
+	const char *hmac_name;
+	int ret;
+
+	hashed_key = kmalloc(hlen, GFP_KERNEL);
+	if (!hashed_key)
+		return -ENOMEM;
+
+	ret = nvme_auth_hash_skey(hmac_id, skey,
+				  skey_len, hashed_key);
+	if (ret < 0)
+		goto out_free_key;
+
+	hmac_name = nvme_auth_hmac_name(hmac_id);
+	if (!hmac_name) {
+		pr_warn("%s: invalid hash algoritm %d\n",
+			__func__, hmac_id);
+		ret = -EINVAL;
+		goto out_free_key;
+	}
+
+	tfm = crypto_alloc_shash(hmac_name, 0, 0);
+	if (IS_ERR(tfm)) {
+		ret = PTR_ERR(tfm);
+		goto out_free_key;
+	}
+
+	desc = kmalloc(sizeof(struct shash_desc) + crypto_shash_descsize(tfm),
+		       GFP_KERNEL);
+	if (!desc) {
+		ret = -ENOMEM;
+		goto out_free_hash;
+	}
+	desc->tfm = tfm;
+
+	ret = crypto_shash_setkey(tfm, hashed_key, hlen);
+	if (ret)
+		goto out_free_desc;
+
+	ret = crypto_shash_init(desc);
+	if (ret)
+		goto out_free_desc;
+
+	ret = crypto_shash_update(desc, challenge, hlen);
+	if (ret)
+		goto out_free_desc;
+
+	ret = crypto_shash_final(desc, aug);
+out_free_desc:
+	kfree_sensitive(desc);
+out_free_hash:
+	crypto_free_shash(tfm);
+out_free_key:
+	kfree_sensitive(hashed_key);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_augmented_challenge);
+
+int nvme_auth_gen_privkey(struct crypto_kpp *dh_tfm, u8 dh_gid)
+{
+	int ret;
+
+	ret = crypto_kpp_set_secret(dh_tfm, NULL, 0);
+	if (ret)
+		pr_debug("failed to set private key, error %d\n", ret);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_gen_privkey);
+
+int nvme_auth_gen_pubkey(struct crypto_kpp *dh_tfm,
+		u8 *host_key, size_t host_key_len)
+{
+	struct kpp_request *req;
+	struct crypto_wait wait;
+	struct scatterlist dst;
+	int ret;
+
+	req = kpp_request_alloc(dh_tfm, GFP_KERNEL);
+	if (!req)
+		return -ENOMEM;
+
+	crypto_init_wait(&wait);
+	kpp_request_set_input(req, NULL, 0);
+	sg_init_one(&dst, host_key, host_key_len);
+	kpp_request_set_output(req, &dst, host_key_len);
+	kpp_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG,
+				 crypto_req_done, &wait);
+
+	ret = crypto_wait_req(crypto_kpp_generate_public_key(req), &wait);
+	kpp_request_free(req);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_gen_pubkey);
+
+int nvme_auth_gen_shared_secret(struct crypto_kpp *dh_tfm,
+		u8 *ctrl_key, size_t ctrl_key_len,
+		u8 *sess_key, size_t sess_key_len)
+{
+	struct kpp_request *req;
+	struct crypto_wait wait;
+	struct scatterlist src, dst;
+	int ret;
+
+	req = kpp_request_alloc(dh_tfm, GFP_KERNEL);
+	if (!req)
+		return -ENOMEM;
+
+	crypto_init_wait(&wait);
+	sg_init_one(&src, ctrl_key, ctrl_key_len);
+	kpp_request_set_input(req, &src, ctrl_key_len);
+	sg_init_one(&dst, sess_key, sess_key_len);
+	kpp_request_set_output(req, &dst, sess_key_len);
+	kpp_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG,
+				 crypto_req_done, &wait);
+
+	ret = crypto_wait_req(crypto_kpp_compute_shared_secret(req), &wait);
+
+	kpp_request_free(req);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_gen_shared_secret);
+
 int nvme_auth_generate_key(u8 *secret, struct nvme_dhchap_key **ret_key)
 {
 	struct nvme_dhchap_key *key;
diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig
index 6c503f42f3c6..2f6a7f8c94e8 100644
--- a/drivers/nvme/host/Kconfig
+++ b/drivers/nvme/host/Kconfig
@@ -100,6 +100,8 @@ config NVME_AUTH
 	select CRYPTO_HMAC
 	select CRYPTO_SHA256
 	select CRYPTO_SHA512
+	select CRYPTO_DH
+	select CRYPTO_DH_RFC7919_GROUPS
 	help
 	  This provides support for NVMe over Fabrics In-Band Authentication.
 
diff --git a/drivers/nvme/host/auth.c b/drivers/nvme/host/auth.c
index c0353783fe27..df82f5182c40 100644
--- a/drivers/nvme/host/auth.c
+++ b/drivers/nvme/host/auth.c
@@ -18,6 +18,7 @@ struct nvme_dhchap_queue_context {
 	struct work_struct auth_work;
 	struct nvme_ctrl *ctrl;
 	struct crypto_shash *shash_tfm;
+	struct crypto_kpp *dh_tfm;
 	void *buf;
 	size_t buf_size;
 	int qid;
@@ -33,6 +34,12 @@ struct nvme_dhchap_queue_context {
 	u8 c2[64];
 	u8 response[64];
 	u8 *host_response;
+	u8 *ctrl_key;
+	int ctrl_key_len;
+	u8 *host_key;
+	int host_key_len;
+	u8 *sess_key;
+	int sess_key_len;
 };
 
 #define nvme_auth_flags_from_qid(qid) \
@@ -137,6 +144,7 @@ static int nvme_auth_process_dhchap_challenge(struct nvme_ctrl *ctrl,
 	struct nvmf_auth_dhchap_challenge_data *data = chap->buf;
 	u16 dhvlen = le16_to_cpu(data->dhvlen);
 	size_t size = sizeof(*data) + data->hl + dhvlen;
+	const char *gid_name = nvme_auth_dhgroup_name(data->dhgid);
 	const char *hmac_name, *kpp_name;
 
 	if (chap->buf_size < size) {
@@ -207,15 +215,54 @@ static int nvme_auth_process_dhchap_challenge(struct nvme_ctrl *ctrl,
 			 "qid %d: invalid DH group id %d\n",
 			 chap->qid, data->dhgid);
 		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
+		/* Leave previous dh_tfm intact */
 		return NVME_SC_AUTH_REQUIRED;
 	}
 
+	/* Clear host and controller key to avoid accidental reuse */
+	kfree_sensitive(chap->host_key);
+	chap->host_key = NULL;
+	chap->host_key_len = 0;
+	kfree_sensitive(chap->ctrl_key);
+	chap->ctrl_key = NULL;
+	chap->ctrl_key_len = 0;
+
+	if (chap->dhgroup_id == data->dhgid &&
+	    (data->dhgid == NVME_AUTH_DHGROUP_NULL || chap->dh_tfm)) {
+		dev_dbg(ctrl->device,
+			"qid %d: reuse existing DH group %s\n",
+			chap->qid, gid_name);
+		goto skip_kpp;
+	}
+
+	/* Reset dh_tfm if it can't be reused */
+	if (chap->dh_tfm) {
+		crypto_free_kpp(chap->dh_tfm);
+		chap->dh_tfm = NULL;
+	}
+
 	if (data->dhgid != NVME_AUTH_DHGROUP_NULL) {
-		dev_warn(ctrl->device,
-			 "qid %d: unsupported DH group %s\n",
-			 chap->qid, kpp_name);
-		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
-		return NVME_SC_AUTH_REQUIRED;
+		if (dhvlen == 0) {
+			dev_warn(ctrl->device,
+				 "qid %d: empty DH value\n",
+				 chap->qid);
+			chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
+			return NVME_SC_INVALID_FIELD;
+		}
+
+		chap->dh_tfm = crypto_alloc_kpp(kpp_name, 0, 0);
+		if (IS_ERR(chap->dh_tfm)) {
+			int ret = PTR_ERR(chap->dh_tfm);
+
+			dev_warn(ctrl->device,
+				 "qid %d: error %d initializing DH group %s\n",
+				 chap->qid, ret, gid_name);
+			chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
+			chap->dh_tfm = NULL;
+			return NVME_SC_AUTH_REQUIRED;
+		}
+		dev_dbg(ctrl->device, "qid %d: selected DH group %s\n",
+			chap->qid, gid_name);
 	} else if (dhvlen != 0) {
 		dev_warn(ctrl->device,
 			 "qid %d: invalid DH value for NULL DH\n",
@@ -225,8 +272,21 @@ static int nvme_auth_process_dhchap_challenge(struct nvme_ctrl *ctrl,
 	}
 	chap->dhgroup_id = data->dhgid;
 
+skip_kpp:
 	chap->s1 = le32_to_cpu(data->seqnum);
 	memcpy(chap->c1, data->cval, chap->hash_len);
+	if (dhvlen) {
+		chap->ctrl_key = kmalloc(dhvlen, GFP_KERNEL);
+		if (!chap->ctrl_key) {
+			chap->status = NVME_AUTH_DHCHAP_FAILURE_FAILED;
+			return NVME_SC_AUTH_REQUIRED;
+		}
+		chap->ctrl_key_len = dhvlen;
+		memcpy(chap->ctrl_key, data->cval + chap->hash_len,
+		       dhvlen);
+		dev_dbg(ctrl->device, "ctrl public key %*ph\n",
+			 (int)chap->ctrl_key_len, chap->ctrl_key);
+	}
 
 	return 0;
 }
@@ -239,6 +299,9 @@ static int nvme_auth_set_dhchap_reply_data(struct nvme_ctrl *ctrl,
 
 	size += 2 * chap->hash_len;
 
+	if (chap->host_key_len)
+		size += chap->host_key_len;
+
 	if (chap->buf_size < size) {
 		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
 		return -EINVAL;
@@ -249,7 +312,7 @@ static int nvme_auth_set_dhchap_reply_data(struct nvme_ctrl *ctrl,
 	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_REPLY;
 	data->t_id = cpu_to_le16(chap->transaction);
 	data->hl = chap->hash_len;
-	data->dhvlen = 0;
+	data->dhvlen = cpu_to_le16(chap->host_key_len);
 	memcpy(data->rval, chap->response, chap->hash_len);
 	if (ctrl->opts->dhchap_ctrl_secret) {
 		get_random_bytes(chap->c2, chap->hash_len);
@@ -264,6 +327,14 @@ static int nvme_auth_set_dhchap_reply_data(struct nvme_ctrl *ctrl,
 		chap->s2 = 0;
 	}
 	data->seqnum = cpu_to_le32(chap->s2);
+	if (chap->host_key_len) {
+		dev_dbg(ctrl->device, "%s: qid %d host public key %*ph\n",
+			__func__, chap->qid,
+			chap->host_key_len, chap->host_key);
+		memcpy(data->rval + 2 * chap->hash_len, chap->host_key,
+		       chap->host_key_len);
+	}
+
 	return size;
 }
 
@@ -381,6 +452,21 @@ static int nvme_auth_dhchap_setup_host_response(struct nvme_ctrl *ctrl,
 		goto out;
 	}
 
+	if (chap->dh_tfm) {
+		challenge = kmalloc(chap->hash_len, GFP_KERNEL);
+		if (!challenge) {
+			ret = -ENOMEM;
+			goto out;
+		}
+		ret = nvme_auth_augmented_challenge(chap->hash_id,
+						    chap->sess_key,
+						    chap->sess_key_len,
+						    chap->c1, challenge,
+						    chap->hash_len);
+		if (ret)
+			goto out;
+	}
+
 	shash->tfm = chap->shash_tfm;
 	ret = crypto_shash_init(shash);
 	if (ret)
@@ -443,6 +529,20 @@ static int nvme_auth_dhchap_setup_ctrl_response(struct nvme_ctrl *ctrl,
 		goto out;
 	}
 
+	if (chap->dh_tfm) {
+		challenge = kmalloc(chap->hash_len, GFP_KERNEL);
+		if (!challenge) {
+			ret = -ENOMEM;
+			goto out;
+		}
+		ret = nvme_auth_augmented_challenge(chap->hash_id,
+						    chap->sess_key,
+						    chap->sess_key_len,
+						    chap->c2, challenge,
+						    chap->hash_len);
+		if (ret)
+			goto out;
+	}
 	dev_dbg(ctrl->device, "%s: qid %d ctrl response seq %u transaction %d\n",
 		__func__, chap->qid, chap->s2, chap->transaction);
 	dev_dbg(ctrl->device, "%s: qid %d challenge %*ph\n",
@@ -491,8 +591,81 @@ static int nvme_auth_dhchap_setup_ctrl_response(struct nvme_ctrl *ctrl,
 	return ret;
 }
 
+static int nvme_auth_dhchap_exponential(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	int ret;
+
+	if (chap->host_key && chap->host_key_len) {
+		dev_dbg(ctrl->device,
+			"qid %d: reusing host key\n", chap->qid);
+		goto gen_sesskey;
+	}
+	ret = nvme_auth_gen_privkey(chap->dh_tfm, chap->dhgroup_id);
+	if (ret < 0) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return ret;
+	}
+
+	chap->host_key_len = crypto_kpp_maxsize(chap->dh_tfm);
+
+	chap->host_key = kzalloc(chap->host_key_len, GFP_KERNEL);
+	if (!chap->host_key) {
+		chap->host_key_len = 0;
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_FAILED;
+		return -ENOMEM;
+	}
+	ret = nvme_auth_gen_pubkey(chap->dh_tfm,
+				   chap->host_key, chap->host_key_len);
+	if (ret) {
+		dev_dbg(ctrl->device,
+			"failed to generate public key, error %d\n", ret);
+		kfree(chap->host_key);
+		chap->host_key = NULL;
+		chap->host_key_len = 0;
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return ret;
+	}
+
+gen_sesskey:
+	chap->sess_key_len = chap->host_key_len;
+	chap->sess_key = kmalloc(chap->sess_key_len, GFP_KERNEL);
+	if (!chap->sess_key) {
+		chap->sess_key_len = 0;
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_FAILED;
+		return -ENOMEM;
+	}
+
+	ret = nvme_auth_gen_shared_secret(chap->dh_tfm,
+					  chap->ctrl_key, chap->ctrl_key_len,
+					  chap->sess_key, chap->sess_key_len);
+	if (ret) {
+		dev_dbg(ctrl->device,
+			"failed to generate shared secret, error %d\n", ret);
+		kfree_sensitive(chap->sess_key);
+		chap->sess_key = NULL;
+		chap->sess_key_len = 0;
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return ret;
+	}
+	dev_dbg(ctrl->device, "shared secret %*ph\n",
+		(int)chap->sess_key_len, chap->sess_key);
+	return 0;
+}
+
 static void __nvme_auth_reset(struct nvme_dhchap_queue_context *chap)
 {
+	kfree_sensitive(chap->host_response);
+	chap->host_response = NULL;
+	kfree_sensitive(chap->host_key);
+	chap->host_key = NULL;
+	chap->host_key_len = 0;
+	kfree_sensitive(chap->ctrl_key);
+	chap->ctrl_key = NULL;
+	chap->ctrl_key_len = 0;
+	kfree_sensitive(chap->sess_key);
+	chap->sess_key = NULL;
+	chap->sess_key_len = 0;
 	chap->status = 0;
 	chap->error = 0;
 	chap->s1 = 0;
@@ -507,6 +680,11 @@ static void __nvme_auth_free(struct nvme_dhchap_queue_context *chap)
 	__nvme_auth_reset(chap);
 	if (chap->shash_tfm)
 		crypto_free_shash(chap->shash_tfm);
+	if (chap->dh_tfm)
+		crypto_free_kpp(chap->dh_tfm);
+	kfree_sensitive(chap->ctrl_key);
+	kfree_sensitive(chap->host_key);
+	kfree_sensitive(chap->sess_key);
 	kfree_sensitive(chap->host_response);
 	kfree(chap->buf);
 	kfree(chap);
@@ -564,6 +742,15 @@ static void __nvme_auth_work(struct work_struct *work)
 		goto fail2;
 	}
 
+	if (chap->ctrl_key_len) {
+		dev_dbg(ctrl->device,
+			"%s: qid %d DH exponential\n",
+			__func__, chap->qid);
+		ret = nvme_auth_dhchap_exponential(ctrl, chap);
+		if (ret)
+			goto fail2;
+	}
+
 	dev_dbg(ctrl->device, "%s: qid %d host response\n",
 		__func__, chap->qid);
 	ret = nvme_auth_dhchap_setup_host_response(ctrl, chap);
diff --git a/include/linux/nvme-auth.h b/include/linux/nvme-auth.h
index 354456826221..dcb8030062dd 100644
--- a/include/linux/nvme-auth.h
+++ b/include/linux/nvme-auth.h
@@ -29,5 +29,13 @@ struct nvme_dhchap_key *nvme_auth_extract_key(unsigned char *secret,
 void nvme_auth_free_key(struct nvme_dhchap_key *key);
 u8 *nvme_auth_transform_key(struct nvme_dhchap_key *key, char *nqn);
 int nvme_auth_generate_key(u8 *secret, struct nvme_dhchap_key **ret_key);
+int nvme_auth_augmented_challenge(u8 hmac_id, u8 *skey, size_t skey_len,
+				  u8 *challenge, u8 *aug, size_t hlen);
+int nvme_auth_gen_privkey(struct crypto_kpp *dh_tfm, u8 dh_gid);
+int nvme_auth_gen_pubkey(struct crypto_kpp *dh_tfm,
+			 u8 *host_key, size_t host_key_len);
+int nvme_auth_gen_shared_secret(struct crypto_kpp *dh_tfm,
+				u8 *ctrl_key, size_t ctrl_key_len,
+				u8 *sess_key, size_t sess_key_len);
 
 #endif /* _NVME_AUTH_H */
-- 
2.29.2

[PATCH 08/11] nvmet: parse fabrics commands on io queues

[Hannes Reinecke]

Some fabrics commands can be sent via io queues, so add a new
function nvmet_parse_fabrics_io_cmd() and rename the existing
nvmet_parse_fabrics_cmd() to nvmet_parse_fabrics_admin_cmd().

Signed-off-by: Hannes Reinecke <hare@suse.de>
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
---
 drivers/nvme/target/admin-cmd.c   |  2 +-
 drivers/nvme/target/core.c        |  4 ++++
 drivers/nvme/target/fabrics-cmd.c | 17 ++++++++++++++++-
 drivers/nvme/target/nvmet.h       |  3 ++-
 4 files changed, 23 insertions(+), 3 deletions(-)

diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c
index 397daaf51f1b..31df40ac828f 100644
--- a/drivers/nvme/target/admin-cmd.c
+++ b/drivers/nvme/target/admin-cmd.c
@@ -1017,7 +1017,7 @@ u16 nvmet_parse_admin_cmd(struct nvmet_req *req)
 	u16 ret;
 
 	if (nvme_is_fabrics(cmd))
-		return nvmet_parse_fabrics_cmd(req);
+		return nvmet_parse_fabrics_admin_cmd(req);
 	if (nvmet_is_disc_subsys(nvmet_req_subsys(req)))
 		return nvmet_parse_discovery_cmd(req);
 
diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index 90e75324dae0..792f15621173 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -865,8 +865,12 @@ static inline u16 nvmet_io_cmd_check_access(struct nvmet_req *req)
 
 static u16 nvmet_parse_io_cmd(struct nvmet_req *req)
 {
+	struct nvme_command *cmd = req->cmd;
 	u16 ret;
 
+	if (nvme_is_fabrics(cmd))
+		return nvmet_parse_fabrics_io_cmd(req);
+
 	ret = nvmet_check_ctrl_status(req);
 	if (unlikely(ret))
 		return ret;
diff --git a/drivers/nvme/target/fabrics-cmd.c b/drivers/nvme/target/fabrics-cmd.c
index 70fb587e9413..f23c28729908 100644
--- a/drivers/nvme/target/fabrics-cmd.c
+++ b/drivers/nvme/target/fabrics-cmd.c
@@ -82,7 +82,7 @@ static void nvmet_execute_prop_get(struct nvmet_req *req)
 	nvmet_req_complete(req, status);
 }
 
-u16 nvmet_parse_fabrics_cmd(struct nvmet_req *req)
+u16 nvmet_parse_fabrics_admin_cmd(struct nvmet_req *req)
 {
 	struct nvme_command *cmd = req->cmd;
 
@@ -103,6 +103,21 @@ u16 nvmet_parse_fabrics_cmd(struct nvmet_req *req)
 	return 0;
 }
 
+u16 nvmet_parse_fabrics_io_cmd(struct nvmet_req *req)
+{
+	struct nvme_command *cmd = req->cmd;
+
+	switch (cmd->fabrics.fctype) {
+	default:
+		pr_debug("received unknown capsule type 0x%x\n",
+			cmd->fabrics.fctype);
+		req->error_loc = offsetof(struct nvmf_common_command, fctype);
+		return NVME_SC_INVALID_OPCODE | NVME_SC_DNR;
+	}
+
+	return 0;
+}
+
 static u16 nvmet_install_queue(struct nvmet_ctrl *ctrl, struct nvmet_req *req)
 {
 	struct nvmf_connect_command *c = &req->cmd->connect;
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index 69818752a33a..c37f41eafc2f 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -419,7 +419,8 @@ u16 nvmet_file_parse_io_cmd(struct nvmet_req *req);
 u16 nvmet_bdev_zns_parse_io_cmd(struct nvmet_req *req);
 u16 nvmet_parse_admin_cmd(struct nvmet_req *req);
 u16 nvmet_parse_discovery_cmd(struct nvmet_req *req);
-u16 nvmet_parse_fabrics_cmd(struct nvmet_req *req);
+u16 nvmet_parse_fabrics_admin_cmd(struct nvmet_req *req);
+u16 nvmet_parse_fabrics_io_cmd(struct nvmet_req *req);
 
 bool nvmet_req_init(struct nvmet_req *req, struct nvmet_cq *cq,
 		struct nvmet_sq *sq, const struct nvmet_fabrics_ops *ops);
-- 
2.29.2

[PATCH 09/11] nvmet: Implement basic In-Band Authentication

[Hannes Reinecke]

Implement NVMe-oF In-Band authentication according to NVMe TPAR 8006.
This patch adds three additional configfs entries 'dhchap_key',
'dhchap_ctrl_key', and 'dhchap_hash' to the 'host' configfs directory.
The 'dhchap_key' and 'dhchap_ctrl_key' entries need to be in the ASCII
format as specified in NVMe Base Specification v2.0 section 8.13.5.8
'Secret representation'.
'dhchap_hash' defaults to 'hmac(sha256)', and can be written to to
switch to a different HMAC algorithm.

Signed-off-by: Hannes Reinecke <hare@suse.de>
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
---
 drivers/nvme/target/Kconfig            |  13 +
 drivers/nvme/target/Makefile           |   1 +
 drivers/nvme/target/admin-cmd.c        |   2 +
 drivers/nvme/target/auth.c             | 367 ++++++++++++++++++
 drivers/nvme/target/configfs.c         | 107 +++++-
 drivers/nvme/target/core.c             |  11 +
 drivers/nvme/target/fabrics-cmd-auth.c | 502 +++++++++++++++++++++++++
 drivers/nvme/target/fabrics-cmd.c      |  38 +-
 drivers/nvme/target/nvmet.h            |  62 +++
 9 files changed, 1100 insertions(+), 3 deletions(-)
 create mode 100644 drivers/nvme/target/auth.c
 create mode 100644 drivers/nvme/target/fabrics-cmd-auth.c

diff --git a/drivers/nvme/target/Kconfig b/drivers/nvme/target/Kconfig
index 973561c93888..df526b59b509 100644
--- a/drivers/nvme/target/Kconfig
+++ b/drivers/nvme/target/Kconfig
@@ -83,3 +83,16 @@ config NVME_TARGET_TCP
 	  devices over TCP.
 
 	  If unsure, say N.
+
+config NVME_TARGET_AUTH
+	bool "NVMe over Fabrics In-band Authentication support"
+	depends on NVME_TARGET
+	select NVME_COMMON
+	select CRYPTO
+	select CRYPTO_HMAC
+	select CRYPTO_SHA256
+	select CRYPTO_SHA512
+	help
+	  This enables support for NVMe over Fabrics In-band Authentication
+
+	  If unsure, say N.
diff --git a/drivers/nvme/target/Makefile b/drivers/nvme/target/Makefile
index 9837e580fa7e..c66820102493 100644
--- a/drivers/nvme/target/Makefile
+++ b/drivers/nvme/target/Makefile
@@ -13,6 +13,7 @@ nvmet-y		+= core.o configfs.o admin-cmd.o fabrics-cmd.o \
 			discovery.o io-cmd-file.o io-cmd-bdev.o
 nvmet-$(CONFIG_NVME_TARGET_PASSTHRU)	+= passthru.o
 nvmet-$(CONFIG_BLK_DEV_ZONED)		+= zns.o
+nvmet-$(CONFIG_NVME_TARGET_AUTH)	+= fabrics-cmd-auth.o auth.o
 nvme-loop-y	+= loop.o
 nvmet-rdma-y	+= rdma.o
 nvmet-fc-y	+= fc.o
diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c
index 31df40ac828f..fc8a957fad0a 100644
--- a/drivers/nvme/target/admin-cmd.c
+++ b/drivers/nvme/target/admin-cmd.c
@@ -1018,6 +1018,8 @@ u16 nvmet_parse_admin_cmd(struct nvmet_req *req)
 
 	if (nvme_is_fabrics(cmd))
 		return nvmet_parse_fabrics_admin_cmd(req);
+	if (unlikely(!nvmet_check_auth_status(req)))
+		return NVME_SC_AUTH_REQUIRED | NVME_SC_DNR;
 	if (nvmet_is_disc_subsys(nvmet_req_subsys(req)))
 		return nvmet_parse_discovery_cmd(req);
 
diff --git a/drivers/nvme/target/auth.c b/drivers/nvme/target/auth.c
new file mode 100644
index 000000000000..5cdd23c34185
--- /dev/null
+++ b/drivers/nvme/target/auth.c
@@ -0,0 +1,367 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * NVMe over Fabrics DH-HMAC-CHAP authentication.
+ * Copyright (c) 2020 Hannes Reinecke, SUSE Software Solutions.
+ * All rights reserved.
+ */
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+#include <linux/module.h>
+#include <linux/init.h>
+#include <linux/slab.h>
+#include <linux/err.h>
+#include <crypto/hash.h>
+#include <linux/crc32.h>
+#include <linux/base64.h>
+#include <linux/ctype.h>
+#include <linux/random.h>
+#include <linux/nvme-auth.h>
+#include <asm/unaligned.h>
+
+#include "nvmet.h"
+
+int nvmet_auth_set_key(struct nvmet_host *host, const char *secret,
+		       bool set_ctrl)
+{
+	unsigned char key_hash;
+	char *dhchap_secret;
+
+	if (sscanf(secret, "DHHC-1:%hhd:%*s", &key_hash) != 1)
+		return -EINVAL;
+	if (key_hash > 3) {
+		pr_warn("Invalid DH-HMAC-CHAP hash id %d\n",
+			 key_hash);
+		return -EINVAL;
+	}
+	if (key_hash > 0) {
+		/* Validate selected hash algorithm */
+		const char *hmac = nvme_auth_hmac_name(key_hash);
+
+		if (!crypto_has_shash(hmac, 0, 0)) {
+			pr_err("DH-HMAC-CHAP hash %s unsupported\n", hmac);
+			return -ENOTSUPP;
+		}
+	}
+	dhchap_secret = kstrdup(secret, GFP_KERNEL);
+	if (!dhchap_secret)
+		return -ENOMEM;
+	if (set_ctrl) {
+		host->dhchap_ctrl_secret = strim(dhchap_secret);
+		host->dhchap_ctrl_key_hash = key_hash;
+	} else {
+		host->dhchap_secret = strim(dhchap_secret);
+		host->dhchap_key_hash = key_hash;
+	}
+	return 0;
+}
+
+int nvmet_setup_auth(struct nvmet_ctrl *ctrl)
+{
+	int ret = 0;
+	struct nvmet_host_link *p;
+	struct nvmet_host *host = NULL;
+	const char *hash_name;
+
+	down_read(&nvmet_config_sem);
+	if (nvmet_is_disc_subsys(ctrl->subsys))
+		goto out_unlock;
+
+	if (ctrl->subsys->allow_any_host)
+		goto out_unlock;
+
+	list_for_each_entry(p, &ctrl->subsys->hosts, entry) {
+		pr_debug("check %s\n", nvmet_host_name(p->host));
+		if (strcmp(nvmet_host_name(p->host), ctrl->hostnqn))
+			continue;
+		host = p->host;
+		break;
+	}
+	if (!host) {
+		pr_debug("host %s not found\n", ctrl->hostnqn);
+		ret = -EPERM;
+		goto out_unlock;
+	}
+
+	if (!host->dhchap_secret) {
+		pr_debug("No authentication provided\n");
+		goto out_unlock;
+	}
+
+	if (host->dhchap_hash_id == ctrl->shash_id) {
+		pr_debug("Re-use existing hash ID %d\n",
+			 ctrl->shash_id);
+	} else {
+		hash_name = nvme_auth_hmac_name(host->dhchap_hash_id);
+		if (!hash_name) {
+			pr_warn("Hash ID %d invalid\n", host->dhchap_hash_id);
+			ret = -EINVAL;
+			goto out_unlock;
+		}
+		ctrl->shash_id = host->dhchap_hash_id;
+	}
+
+	/* Skip the 'DHHC-1:XX:' prefix */
+	nvme_auth_free_key(ctrl->host_key);
+	ctrl->host_key = nvme_auth_extract_key(host->dhchap_secret + 10,
+					       host->dhchap_key_hash);
+	if (IS_ERR(ctrl->host_key)) {
+		ret = PTR_ERR(ctrl->host_key);
+		ctrl->host_key = NULL;
+		goto out_free_hash;
+	}
+	pr_debug("%s: using hash %s key %*ph\n", __func__,
+		 ctrl->host_key->hash > 0 ?
+		 nvme_auth_hmac_name(ctrl->host_key->hash) : "none",
+		 (int)ctrl->host_key->len, ctrl->host_key->key);
+
+	nvme_auth_free_key(ctrl->ctrl_key);
+	if (!host->dhchap_ctrl_secret) {
+		ctrl->ctrl_key = NULL;
+		goto out_unlock;
+	}
+
+	ctrl->ctrl_key = nvme_auth_extract_key(host->dhchap_ctrl_secret + 10,
+					       host->dhchap_ctrl_key_hash);
+	if (IS_ERR(ctrl->ctrl_key)) {
+		ret = PTR_ERR(ctrl->ctrl_key);
+		ctrl->ctrl_key = NULL;
+	}
+	pr_debug("%s: using ctrl hash %s key %*ph\n", __func__,
+		 ctrl->ctrl_key->hash > 0 ?
+		 nvme_auth_hmac_name(ctrl->ctrl_key->hash) : "none",
+		 (int)ctrl->ctrl_key->len, ctrl->ctrl_key->key);
+
+out_free_hash:
+	if (ret) {
+		if (ctrl->host_key) {
+			nvme_auth_free_key(ctrl->host_key);
+			ctrl->host_key = NULL;
+		}
+		ctrl->shash_id = 0;
+	}
+out_unlock:
+	up_read(&nvmet_config_sem);
+
+	return ret;
+}
+
+void nvmet_auth_sq_free(struct nvmet_sq *sq)
+{
+	kfree(sq->dhchap_c1);
+	sq->dhchap_c1 = NULL;
+	kfree(sq->dhchap_c2);
+	sq->dhchap_c2 = NULL;
+	kfree(sq->dhchap_skey);
+	sq->dhchap_skey = NULL;
+}
+
+void nvmet_destroy_auth(struct nvmet_ctrl *ctrl)
+{
+	ctrl->shash_id = 0;
+
+	if (ctrl->host_key) {
+		nvme_auth_free_key(ctrl->host_key);
+		ctrl->host_key = NULL;
+	}
+	if (ctrl->ctrl_key) {
+		nvme_auth_free_key(ctrl->ctrl_key);
+		ctrl->ctrl_key = NULL;
+	}
+}
+
+bool nvmet_check_auth_status(struct nvmet_req *req)
+{
+	if (req->sq->ctrl->host_key &&
+	    !req->sq->authenticated)
+		return false;
+	return true;
+}
+
+int nvmet_auth_host_hash(struct nvmet_req *req, u8 *response,
+			 unsigned int shash_len)
+{
+	struct crypto_shash *shash_tfm;
+	struct shash_desc *shash;
+	struct nvmet_ctrl *ctrl = req->sq->ctrl;
+	const char *hash_name;
+	u8 *challenge = req->sq->dhchap_c1, *host_response;
+	u8 buf[4];
+	int ret;
+
+	hash_name = nvme_auth_hmac_name(ctrl->shash_id);
+	if (!hash_name) {
+		pr_warn("Hash ID %d invalid\n", ctrl->shash_id);
+		return -EINVAL;
+	}
+
+	shash_tfm = crypto_alloc_shash(hash_name, 0, 0);
+	if (IS_ERR(shash_tfm)) {
+		pr_err("failed to allocate shash %s\n", hash_name);
+		return PTR_ERR(shash_tfm);
+	}
+
+	if (shash_len != crypto_shash_digestsize(shash_tfm)) {
+		pr_debug("%s: hash len mismatch (len %d digest %d)\n",
+			 __func__, shash_len,
+			 crypto_shash_digestsize(shash_tfm));
+		ret = -EINVAL;
+		goto out_free_tfm;
+	}
+
+	host_response = nvme_auth_transform_key(ctrl->host_key, ctrl->hostnqn);
+	if (IS_ERR(host_response)) {
+		ret = PTR_ERR(host_response);
+		goto out_free_tfm;
+	}
+
+	ret = crypto_shash_setkey(shash_tfm, host_response,
+				  ctrl->host_key->len);
+	if (ret)
+		goto out_free_response;
+
+	pr_debug("ctrl %d qid %d host response seq %u transaction %d\n",
+		 ctrl->cntlid, req->sq->qid, req->sq->dhchap_s1,
+		 req->sq->dhchap_tid);
+
+	shash = kzalloc(sizeof(*shash) + crypto_shash_descsize(shash_tfm),
+			GFP_KERNEL);
+	if (!shash) {
+		ret = -ENOMEM;
+		goto out_free_response;
+	}
+	shash->tfm = shash_tfm;
+	ret = crypto_shash_init(shash);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, challenge, shash_len);
+	if (ret)
+		goto out;
+	put_unaligned_le32(req->sq->dhchap_s1, buf);
+	ret = crypto_shash_update(shash, buf, 4);
+	if (ret)
+		goto out;
+	put_unaligned_le16(req->sq->dhchap_tid, buf);
+	ret = crypto_shash_update(shash, buf, 2);
+	if (ret)
+		goto out;
+	memset(buf, 0, 4);
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, "HostHost", 8);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->hostnqn, strlen(ctrl->hostnqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->subsysnqn,
+				  strlen(ctrl->subsysnqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_final(shash, response);
+out:
+	if (challenge != req->sq->dhchap_c1)
+		kfree(challenge);
+	kfree(shash);
+out_free_response:
+	kfree_sensitive(host_response);
+out_free_tfm:
+	crypto_free_shash(shash_tfm);
+	return 0;
+}
+
+int nvmet_auth_ctrl_hash(struct nvmet_req *req, u8 *response,
+			 unsigned int shash_len)
+{
+	struct crypto_shash *shash_tfm;
+	struct shash_desc *shash;
+	struct nvmet_ctrl *ctrl = req->sq->ctrl;
+	const char *hash_name;
+	u8 *challenge = req->sq->dhchap_c2, *ctrl_response;
+	u8 buf[4];
+	int ret;
+
+	hash_name = nvme_auth_hmac_name(ctrl->shash_id);
+	if (!hash_name) {
+		pr_warn("Hash ID %d invalid\n", ctrl->shash_id);
+		return -EINVAL;
+	}
+
+	shash_tfm = crypto_alloc_shash(hash_name, 0, 0);
+	if (IS_ERR(shash_tfm)) {
+		pr_err("failed to allocate shash %s\n", hash_name);
+		return PTR_ERR(shash_tfm);
+	}
+
+	if (shash_len != crypto_shash_digestsize(shash_tfm)) {
+		pr_debug("%s: hash len mismatch (len %d digest %d)\n",
+			 __func__, shash_len,
+			 crypto_shash_digestsize(shash_tfm));
+		ret = -EINVAL;
+		goto out_free_tfm;
+	}
+
+	ctrl_response = nvme_auth_transform_key(ctrl->ctrl_key,
+						ctrl->subsysnqn);
+	if (IS_ERR(ctrl_response)) {
+		ret = PTR_ERR(ctrl_response);
+		goto out_free_tfm;
+	}
+
+	ret = crypto_shash_setkey(shash_tfm, ctrl_response,
+				  ctrl->ctrl_key->len);
+	if (ret)
+		goto out_free_response;
+
+	shash = kzalloc(sizeof(*shash) + crypto_shash_descsize(shash_tfm),
+			GFP_KERNEL);
+	if (!shash) {
+		ret = -ENOMEM;
+		goto out_free_response;
+	}
+	shash->tfm = shash_tfm;
+
+	ret = crypto_shash_init(shash);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, challenge, shash_len);
+	if (ret)
+		goto out;
+	put_unaligned_le32(req->sq->dhchap_s2, buf);
+	ret = crypto_shash_update(shash, buf, 4);
+	if (ret)
+		goto out;
+	put_unaligned_le16(req->sq->dhchap_tid, buf);
+	ret = crypto_shash_update(shash, buf, 2);
+	if (ret)
+		goto out;
+	memset(buf, 0, 4);
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, "Controller", 10);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->subsysnqn,
+			    strlen(ctrl->subsysnqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->hostnqn, strlen(ctrl->hostnqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_final(shash, response);
+out:
+	if (challenge != req->sq->dhchap_c2)
+		kfree(challenge);
+	kfree(shash);
+out_free_response:
+	kfree_sensitive(ctrl_response);
+out_free_tfm:
+	crypto_free_shash(shash_tfm);
+	return 0;
+}
diff --git a/drivers/nvme/target/configfs.c b/drivers/nvme/target/configfs.c
index e44b2988759e..9a7d91c64fcd 100644
--- a/drivers/nvme/target/configfs.c
+++ b/drivers/nvme/target/configfs.c
@@ -11,6 +11,11 @@
 #include <linux/ctype.h>
 #include <linux/pci.h>
 #include <linux/pci-p2pdma.h>
+#ifdef CONFIG_NVME_TARGET_AUTH
+#include <linux/nvme-auth.h>
+#endif
+#include <crypto/hash.h>
+#include <crypto/kpp.h>
 
 #include "nvmet.h"
 
@@ -1660,10 +1665,102 @@ static const struct config_item_type nvmet_ports_type = {
 static struct config_group nvmet_subsystems_group;
 static struct config_group nvmet_ports_group;
 
-static void nvmet_host_release(struct config_item *item)
+#ifdef CONFIG_NVME_TARGET_AUTH
+static ssize_t nvmet_host_dhchap_key_show(struct config_item *item,
+		char *page)
+{
+	u8 *dhchap_secret = to_host(item)->dhchap_secret;
+
+	if (!dhchap_secret)
+		return sprintf(page, "\n");
+	return sprintf(page, "%s\n", dhchap_secret);
+}
+
+static ssize_t nvmet_host_dhchap_key_store(struct config_item *item,
+		const char *page, size_t count)
+{
+	struct nvmet_host *host = to_host(item);
+	int ret;
+
+	ret = nvmet_auth_set_key(host, page, false);
+	/*
+	 * Re-authentication is a soft state, so keep the
+	 * current authentication valid until the host
+	 * requests re-authentication.
+	 */
+	return ret < 0 ? ret : count;
+}
+
+CONFIGFS_ATTR(nvmet_host_, dhchap_key);
+
+static ssize_t nvmet_host_dhchap_ctrl_key_show(struct config_item *item,
+		char *page)
+{
+	u8 *dhchap_secret = to_host(item)->dhchap_ctrl_secret;
+
+	if (!dhchap_secret)
+		return sprintf(page, "\n");
+	return sprintf(page, "%s\n", dhchap_secret);
+}
+
+static ssize_t nvmet_host_dhchap_ctrl_key_store(struct config_item *item,
+		const char *page, size_t count)
+{
+	struct nvmet_host *host = to_host(item);
+	int ret;
+
+	ret = nvmet_auth_set_key(host, page, true);
+	/*
+	 * Re-authentication is a soft state, so keep the
+	 * current authentication valid until the host
+	 * requests re-authentication.
+	 */
+	return ret < 0 ? ret : count;
+}
+
+CONFIGFS_ATTR(nvmet_host_, dhchap_ctrl_key);
+
+static ssize_t nvmet_host_dhchap_hash_show(struct config_item *item,
+		char *page)
 {
 	struct nvmet_host *host = to_host(item);
+	const char *hash_name = nvme_auth_hmac_name(host->dhchap_hash_id);
 
+	return sprintf(page, "%s\n", hash_name ? hash_name : "none");
+}
+
+static ssize_t nvmet_host_dhchap_hash_store(struct config_item *item,
+		const char *page, size_t count)
+{
+	struct nvmet_host *host = to_host(item);
+	u8 hmac_id;
+
+	hmac_id = nvme_auth_hmac_id(page);
+	if (hmac_id == NVME_AUTH_HASH_INVALID)
+		return -EINVAL;
+	if (!crypto_has_shash(nvme_auth_hmac_name(hmac_id), 0, 0))
+		return -ENOTSUPP;
+	host->dhchap_hash_id = hmac_id;
+	return count;
+}
+
+CONFIGFS_ATTR(nvmet_host_, dhchap_hash);
+
+static struct configfs_attribute *nvmet_host_attrs[] = {
+	&nvmet_host_attr_dhchap_key,
+	&nvmet_host_attr_dhchap_ctrl_key,
+	&nvmet_host_attr_dhchap_hash,
+	NULL,
+};
+#endif /* CONFIG_NVME_TARGET_AUTH */
+
+static void nvmet_host_release(struct config_item *item)
+{
+	struct nvmet_host *host = to_host(item);
+#ifdef CONFIG_NVME_TARGET_AUTH
+	if (host->dhchap_secret)
+		kfree(host->dhchap_secret);
+#endif
 	kfree(host);
 }
 
@@ -1673,6 +1770,9 @@ static struct configfs_item_operations nvmet_host_item_ops = {
 
 static const struct config_item_type nvmet_host_type = {
 	.ct_item_ops		= &nvmet_host_item_ops,
+#ifdef CONFIG_NVME_TARGET_AUTH
+	.ct_attrs		= nvmet_host_attrs,
+#endif
 	.ct_owner		= THIS_MODULE,
 };
 
@@ -1685,6 +1785,11 @@ static struct config_group *nvmet_hosts_make_group(struct config_group *group,
 	if (!host)
 		return ERR_PTR(-ENOMEM);
 
+#ifdef CONFIG_NVME_TARGET_AUTH
+	/* Default to SHA256 */
+	host->dhchap_hash_id = NVME_AUTH_HASH_SHA256;
+#endif
+
 	config_group_init_type_name(&host->group, name, &nvmet_host_type);
 
 	return &host->group;
diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index 792f15621173..eec0351e1022 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -795,6 +795,7 @@ void nvmet_sq_destroy(struct nvmet_sq *sq)
 	wait_for_completion(&sq->confirm_done);
 	wait_for_completion(&sq->free_done);
 	percpu_ref_exit(&sq->ref);
+	nvmet_auth_sq_free(sq);
 
 	if (ctrl) {
 		/*
@@ -871,6 +872,9 @@ static u16 nvmet_parse_io_cmd(struct nvmet_req *req)
 	if (nvme_is_fabrics(cmd))
 		return nvmet_parse_fabrics_io_cmd(req);
 
+	if (unlikely(!nvmet_check_auth_status(req)))
+		return NVME_SC_AUTH_REQUIRED | NVME_SC_DNR;
+
 	ret = nvmet_check_ctrl_status(req);
 	if (unlikely(ret))
 		return ret;
@@ -1275,6 +1279,11 @@ u16 nvmet_check_ctrl_status(struct nvmet_req *req)
 		       req->cmd->common.opcode, req->sq->qid);
 		return NVME_SC_CMD_SEQ_ERROR | NVME_SC_DNR;
 	}
+
+	if (unlikely(!nvmet_check_auth_status(req))) {
+		pr_warn("qid %d not authenticated\n", req->sq->qid);
+		return NVME_SC_AUTH_REQUIRED | NVME_SC_DNR;
+	}
 	return 0;
 }
 
@@ -1465,6 +1474,8 @@ static void nvmet_ctrl_free(struct kref *ref)
 	flush_work(&ctrl->async_event_work);
 	cancel_work_sync(&ctrl->fatal_err_work);
 
+	nvmet_destroy_auth(ctrl);
+
 	ida_free(&cntlid_ida, ctrl->cntlid);
 
 	nvmet_async_events_free(ctrl);
diff --git a/drivers/nvme/target/fabrics-cmd-auth.c b/drivers/nvme/target/fabrics-cmd-auth.c
new file mode 100644
index 000000000000..776073a10e04
--- /dev/null
+++ b/drivers/nvme/target/fabrics-cmd-auth.c
@@ -0,0 +1,502 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * NVMe over Fabrics DH-HMAC-CHAP authentication command handling.
+ * Copyright (c) 2020 Hannes Reinecke, SUSE Software Solutions.
+ * All rights reserved.
+ */
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+#include <linux/blkdev.h>
+#include <linux/random.h>
+#include <linux/nvme-auth.h>
+#include <crypto/hash.h>
+#include <crypto/kpp.h>
+#include "nvmet.h"
+
+void nvmet_init_auth(struct nvmet_ctrl *ctrl, struct nvmet_req *req)
+{
+	u32 result = le32_to_cpu(req->cqe->result.u32);
+
+	/* Initialize in-band authentication */
+	req->sq->authenticated = false;
+	req->sq->dhchap_step = NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE;
+	result |= (u32)NVME_CONNECT_AUTHREQ_ATR << 16;
+	req->cqe->result.u32 = cpu_to_le32(result);
+}
+
+static u16 nvmet_auth_negotiate(struct nvmet_req *req, void *d)
+{
+	struct nvmet_ctrl *ctrl = req->sq->ctrl;
+	struct nvmf_auth_dhchap_negotiate_data *data = d;
+	int i, hash_id = 0, fallback_hash_id = 0, dhgid;
+
+	pr_debug("%s: ctrl %d qid %d: data sc_d %d napd %d authid %d halen %d dhlen %d\n",
+		 __func__, ctrl->cntlid, req->sq->qid,
+		 data->sc_c, data->napd, data->auth_protocol[0].dhchap.authid,
+		 data->auth_protocol[0].dhchap.halen,
+		 data->auth_protocol[0].dhchap.dhlen);
+	req->sq->dhchap_tid = le16_to_cpu(data->t_id);
+	if (data->sc_c)
+		return NVME_AUTH_DHCHAP_FAILURE_CONCAT_MISMATCH;
+
+	if (data->napd != 1)
+		return NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
+
+	if (data->auth_protocol[0].dhchap.authid !=
+	    NVME_AUTH_DHCHAP_AUTH_ID)
+		return NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+
+	for (i = 0; i < data->auth_protocol[0].dhchap.halen; i++) {
+		u8 host_hmac_id = data->auth_protocol[0].dhchap.idlist[i];
+
+		if (!fallback_hash_id &&
+		    crypto_has_shash(nvme_auth_hmac_name(host_hmac_id), 0, 0))
+			fallback_hash_id = host_hmac_id;
+		if (ctrl->shash_id != host_hmac_id)
+			continue;
+		hash_id = ctrl->shash_id;
+		break;
+	}
+	if (hash_id == 0) {
+		if (fallback_hash_id == 0) {
+			pr_debug("%s: ctrl %d qid %d: no usable hash found\n",
+				 __func__, ctrl->cntlid, req->sq->qid);
+			return NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
+		}
+		pr_debug("%s: ctrl %d qid %d: no usable hash found, falling back to %s\n",
+			 __func__, ctrl->cntlid, req->sq->qid,
+			 nvme_auth_hmac_name(fallback_hash_id));
+		ctrl->shash_id = fallback_hash_id;
+	}
+
+	dhgid = -1;
+	for (i = 0; i < data->auth_protocol[0].dhchap.dhlen; i++) {
+		int tmp_dhgid = data->auth_protocol[0].dhchap.idlist[i + 30];
+
+		if (tmp_dhgid == NVME_AUTH_DHGROUP_NULL) {
+			dhgid = tmp_dhgid;
+			break;
+		}
+	}
+	if (dhgid < 0) {
+		pr_debug("%s: ctrl %d qid %d: no usable DH group found\n",
+				 __func__, ctrl->cntlid, req->sq->qid);
+		return NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
+	}
+	pr_debug("%s: ctrl %d qid %d: selected DH group %s (%d)\n",
+		 __func__, ctrl->cntlid, req->sq->qid,
+		 nvme_auth_dhgroup_name(dhgid), dhgid);
+	return 0;
+}
+
+static u16 nvmet_auth_reply(struct nvmet_req *req, void *d)
+{
+	struct nvmet_ctrl *ctrl = req->sq->ctrl;
+	struct nvmf_auth_dhchap_reply_data *data = d;
+	u16 dhvlen = le16_to_cpu(data->dhvlen);
+	u8 *response;
+
+	pr_debug("%s: ctrl %d qid %d: data hl %d cvalid %d dhvlen %u\n",
+		 __func__, ctrl->cntlid, req->sq->qid,
+		 data->hl, data->cvalid, dhvlen);
+
+	if (dhvlen) {
+		return NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+	}
+
+	response = kmalloc(data->hl, GFP_KERNEL);
+	if (!response)
+		return NVME_AUTH_DHCHAP_FAILURE_FAILED;
+
+	if (!ctrl->host_key) {
+		pr_warn("ctrl %d qid %d no host key\n",
+			ctrl->cntlid, req->sq->qid);
+		kfree(response);
+		return NVME_AUTH_DHCHAP_FAILURE_FAILED;
+	}
+	if (nvmet_auth_host_hash(req, response, data->hl) < 0) {
+		pr_debug("ctrl %d qid %d host hash failed\n",
+			 ctrl->cntlid, req->sq->qid);
+		kfree(response);
+		return NVME_AUTH_DHCHAP_FAILURE_FAILED;
+	}
+
+	if (memcmp(data->rval, response, data->hl)) {
+		pr_info("ctrl %d qid %d host response mismatch\n",
+			ctrl->cntlid, req->sq->qid);
+		kfree(response);
+		return NVME_AUTH_DHCHAP_FAILURE_FAILED;
+	}
+	kfree(response);
+	pr_debug("%s: ctrl %d qid %d host authenticated\n",
+		 __func__, ctrl->cntlid, req->sq->qid);
+	if (data->cvalid) {
+		req->sq->dhchap_c2 = kmalloc(data->hl, GFP_KERNEL);
+		if (!req->sq->dhchap_c2)
+			return NVME_AUTH_DHCHAP_FAILURE_FAILED;
+		memcpy(req->sq->dhchap_c2, data->rval + data->hl, data->hl);
+
+		pr_debug("%s: ctrl %d qid %d challenge %*ph\n",
+			 __func__, ctrl->cntlid, req->sq->qid, data->hl,
+			 req->sq->dhchap_c2);
+		req->sq->dhchap_s2 = le32_to_cpu(data->seqnum);
+	} else {
+		req->sq->authenticated = true;
+		req->sq->dhchap_c2 = NULL;
+	}
+
+	return 0;
+}
+
+static u16 nvmet_auth_failure2(struct nvmet_req *req, void *d)
+{
+	struct nvmf_auth_dhchap_failure_data *data = d;
+
+	return data->rescode_exp;
+}
+
+void nvmet_execute_auth_send(struct nvmet_req *req)
+{
+	struct nvmet_ctrl *ctrl = req->sq->ctrl;
+	struct nvmf_auth_dhchap_success2_data *data;
+	void *d;
+	u32 tl;
+	u16 status = 0;
+
+	if (req->cmd->auth_send.secp != NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER) {
+		status = NVME_SC_INVALID_FIELD | NVME_SC_DNR;
+		req->error_loc =
+			offsetof(struct nvmf_auth_send_command, secp);
+		goto done;
+	}
+	if (req->cmd->auth_send.spsp0 != 0x01) {
+		status = NVME_SC_INVALID_FIELD | NVME_SC_DNR;
+		req->error_loc =
+			offsetof(struct nvmf_auth_send_command, spsp0);
+		goto done;
+	}
+	if (req->cmd->auth_send.spsp1 != 0x01) {
+		status = NVME_SC_INVALID_FIELD | NVME_SC_DNR;
+		req->error_loc =
+			offsetof(struct nvmf_auth_send_command, spsp1);
+		goto done;
+	}
+	tl = le32_to_cpu(req->cmd->auth_send.tl);
+	if (!tl) {
+		status = NVME_SC_INVALID_FIELD | NVME_SC_DNR;
+		req->error_loc =
+			offsetof(struct nvmf_auth_send_command, tl);
+		goto done;
+	}
+	if (!nvmet_check_transfer_len(req, tl)) {
+		pr_debug("%s: transfer length mismatch (%u)\n", __func__, tl);
+		return;
+	}
+
+	d = kmalloc(tl, GFP_KERNEL);
+	if (!d) {
+		status = NVME_SC_INTERNAL;
+		goto done;
+	}
+
+	status = nvmet_copy_from_sgl(req, 0, d, tl);
+	if (status) {
+		kfree(d);
+		goto done;
+	}
+
+	data = d;
+	pr_debug("%s: ctrl %d qid %d type %d id %d step %x\n", __func__,
+		 ctrl->cntlid, req->sq->qid, data->auth_type, data->auth_id,
+		 req->sq->dhchap_step);
+	if (data->auth_type != NVME_AUTH_COMMON_MESSAGES &&
+	    data->auth_type != NVME_AUTH_DHCHAP_MESSAGES)
+		goto done_failure1;
+	if (data->auth_type == NVME_AUTH_COMMON_MESSAGES) {
+		if (data->auth_id == NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE) {
+			/* Restart negotiation */
+			pr_debug("%s: ctrl %d qid %d reset negotiation\n", __func__,
+				 ctrl->cntlid, req->sq->qid);
+			if (!req->sq->qid) {
+				status = nvmet_setup_auth(ctrl);
+				if (status < 0) {
+					pr_err("ctrl %d qid 0 failed to setup"
+					       "re-authentication",
+					       ctrl->cntlid);
+					goto done_failure1;
+				}
+			}
+			req->sq->dhchap_step = NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE;
+		} else if (data->auth_id != req->sq->dhchap_step)
+			goto done_failure1;
+		/* Validate negotiation parameters */
+		status = nvmet_auth_negotiate(req, d);
+		if (status == 0)
+			req->sq->dhchap_step =
+				NVME_AUTH_DHCHAP_MESSAGE_CHALLENGE;
+		else {
+			req->sq->dhchap_step =
+				NVME_AUTH_DHCHAP_MESSAGE_FAILURE1;
+			req->sq->dhchap_status = status;
+			status = 0;
+		}
+		goto done_kfree;
+	}
+	if (data->auth_id != req->sq->dhchap_step) {
+		pr_debug("%s: ctrl %d qid %d step mismatch (%d != %d)\n",
+			 __func__, ctrl->cntlid, req->sq->qid,
+			 data->auth_id, req->sq->dhchap_step);
+		goto done_failure1;
+	}
+	if (le16_to_cpu(data->t_id) != req->sq->dhchap_tid) {
+		pr_debug("%s: ctrl %d qid %d invalid transaction %d (expected %d)\n",
+			 __func__, ctrl->cntlid, req->sq->qid,
+			 le16_to_cpu(data->t_id),
+			 req->sq->dhchap_tid);
+		req->sq->dhchap_step =
+			NVME_AUTH_DHCHAP_MESSAGE_FAILURE1;
+		req->sq->dhchap_status =
+			NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		goto done_kfree;
+	}
+
+	switch (data->auth_id) {
+	case NVME_AUTH_DHCHAP_MESSAGE_REPLY:
+		status = nvmet_auth_reply(req, d);
+		if (status == 0)
+			req->sq->dhchap_step =
+				NVME_AUTH_DHCHAP_MESSAGE_SUCCESS1;
+		else {
+			req->sq->dhchap_step =
+				NVME_AUTH_DHCHAP_MESSAGE_FAILURE1;
+			req->sq->dhchap_status = status;
+			status = 0;
+		}
+		goto done_kfree;
+		break;
+	case NVME_AUTH_DHCHAP_MESSAGE_SUCCESS2:
+		req->sq->authenticated = true;
+		pr_debug("%s: ctrl %d qid %d ctrl authenticated\n",
+			 __func__, ctrl->cntlid, req->sq->qid);
+		goto done_kfree;
+		break;
+	case NVME_AUTH_DHCHAP_MESSAGE_FAILURE2:
+		status = nvmet_auth_failure2(req, d);
+		if (status) {
+			pr_warn("ctrl %d qid %d: authentication failed (%d)\n",
+				ctrl->cntlid, req->sq->qid, status);
+			req->sq->dhchap_status = status;
+			req->sq->authenticated = false;
+			status = 0;
+		}
+		goto done_kfree;
+		break;
+	default:
+		req->sq->dhchap_status =
+			NVME_AUTH_DHCHAP_FAILURE_INCORRECT_MESSAGE;
+		req->sq->dhchap_step =
+			NVME_AUTH_DHCHAP_MESSAGE_FAILURE2;
+		req->sq->authenticated = false;
+		goto done_kfree;
+		break;
+	}
+done_failure1:
+	req->sq->dhchap_status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_MESSAGE;
+	req->sq->dhchap_step = NVME_AUTH_DHCHAP_MESSAGE_FAILURE2;
+
+done_kfree:
+	kfree(d);
+done:
+	pr_debug("%s: ctrl %d qid %d dhchap status %x step %x\n", __func__,
+		 ctrl->cntlid, req->sq->qid,
+		 req->sq->dhchap_status, req->sq->dhchap_step);
+	if (status)
+		pr_debug("%s: ctrl %d qid %d nvme status %x error loc %d\n",
+			 __func__, ctrl->cntlid, req->sq->qid,
+			 status, req->error_loc);
+	req->cqe->result.u64 = 0;
+	nvmet_req_complete(req, status);
+	if (req->sq->dhchap_step != NVME_AUTH_DHCHAP_MESSAGE_SUCCESS2 &&
+	    req->sq->dhchap_step != NVME_AUTH_DHCHAP_MESSAGE_FAILURE2)
+		return;
+	/* Final states, clear up variables */
+	nvmet_auth_sq_free(req->sq);
+	if (req->sq->dhchap_step == NVME_AUTH_DHCHAP_MESSAGE_FAILURE2)
+		nvmet_ctrl_fatal_error(ctrl);
+}
+
+static int nvmet_auth_challenge(struct nvmet_req *req, void *d, int al)
+{
+	struct nvmf_auth_dhchap_challenge_data *data = d;
+	struct nvmet_ctrl *ctrl = req->sq->ctrl;
+	int ret = 0;
+	int hash_len = nvme_auth_hmac_hash_len(ctrl->shash_id);
+	int data_size = sizeof(*d) + hash_len;
+
+	if (al < data_size) {
+		pr_debug("%s: buffer too small (al %d need %d)\n", __func__,
+			 al, data_size);
+		return -EINVAL;
+	}
+	memset(data, 0, data_size);
+	req->sq->dhchap_s1 = nvme_auth_get_seqnum();
+	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_CHALLENGE;
+	data->t_id = cpu_to_le16(req->sq->dhchap_tid);
+	data->hashid = ctrl->shash_id;
+	data->hl = hash_len;
+	data->seqnum = cpu_to_le32(req->sq->dhchap_s1);
+	req->sq->dhchap_c1 = kmalloc(data->hl, GFP_KERNEL);
+	if (!req->sq->dhchap_c1)
+		return -ENOMEM;
+	get_random_bytes(req->sq->dhchap_c1, data->hl);
+	memcpy(data->cval, req->sq->dhchap_c1, data->hl);
+	pr_debug("%s: ctrl %d qid %d seq %u transaction %d hl %d dhvlen %u\n",
+		 __func__, ctrl->cntlid, req->sq->qid, req->sq->dhchap_s1,
+		 req->sq->dhchap_tid, data->hl, 0);
+	return ret;
+}
+
+static int nvmet_auth_success1(struct nvmet_req *req, void *d, int al)
+{
+	struct nvmf_auth_dhchap_success1_data *data = d;
+	struct nvmet_ctrl *ctrl = req->sq->ctrl;
+	int hash_len = nvme_auth_hmac_hash_len(ctrl->shash_id);
+
+	WARN_ON(al < sizeof(*data));
+	memset(data, 0, sizeof(*data));
+	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_SUCCESS1;
+	data->t_id = cpu_to_le16(req->sq->dhchap_tid);
+	data->hl = hash_len;
+	if (req->sq->dhchap_c2) {
+		if (!ctrl->ctrl_key) {
+			pr_warn("ctrl %d qid %d no ctrl key\n",
+				ctrl->cntlid, req->sq->qid);
+			return NVME_AUTH_DHCHAP_FAILURE_FAILED;
+		}
+		if (nvmet_auth_ctrl_hash(req, data->rval, data->hl))
+			return NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
+		data->rvalid = 1;
+		pr_debug("ctrl %d qid %d response %*ph\n",
+			 ctrl->cntlid, req->sq->qid, data->hl, data->rval);
+	}
+	return 0;
+}
+
+static void nvmet_auth_failure1(struct nvmet_req *req, void *d, int al)
+{
+	struct nvmf_auth_dhchap_failure_data *data = d;
+
+	WARN_ON(al < sizeof(*data));
+	data->auth_type = NVME_AUTH_COMMON_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_FAILURE1;
+	data->t_id = cpu_to_le16(req->sq->dhchap_tid);
+	data->rescode = NVME_AUTH_DHCHAP_FAILURE_REASON_FAILED;
+	data->rescode_exp = req->sq->dhchap_status;
+}
+
+void nvmet_execute_auth_receive(struct nvmet_req *req)
+{
+	struct nvmet_ctrl *ctrl = req->sq->ctrl;
+	void *d;
+	u32 al;
+	u16 status = 0;
+
+	if (req->cmd->auth_receive.secp != NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER) {
+		status = NVME_SC_INVALID_FIELD | NVME_SC_DNR;
+		req->error_loc =
+			offsetof(struct nvmf_auth_receive_command, secp);
+		goto done;
+	}
+	if (req->cmd->auth_receive.spsp0 != 0x01) {
+		status = NVME_SC_INVALID_FIELD | NVME_SC_DNR;
+		req->error_loc =
+			offsetof(struct nvmf_auth_receive_command, spsp0);
+		goto done;
+	}
+	if (req->cmd->auth_receive.spsp1 != 0x01) {
+		status = NVME_SC_INVALID_FIELD | NVME_SC_DNR;
+		req->error_loc =
+			offsetof(struct nvmf_auth_receive_command, spsp1);
+		goto done;
+	}
+	al = le32_to_cpu(req->cmd->auth_receive.al);
+	if (!al) {
+		status = NVME_SC_INVALID_FIELD | NVME_SC_DNR;
+		req->error_loc =
+			offsetof(struct nvmf_auth_receive_command, al);
+		goto done;
+	}
+	if (!nvmet_check_transfer_len(req, al)) {
+		pr_debug("%s: transfer length mismatch (%u)\n", __func__, al);
+		return;
+	}
+
+	d = kmalloc(al, GFP_KERNEL);
+	if (!d) {
+		status = NVME_SC_INTERNAL;
+		goto done;
+	}
+	pr_debug("%s: ctrl %d qid %d step %x\n", __func__,
+		 ctrl->cntlid, req->sq->qid, req->sq->dhchap_step);
+	switch (req->sq->dhchap_step) {
+	case NVME_AUTH_DHCHAP_MESSAGE_CHALLENGE:
+		status = nvmet_auth_challenge(req, d, al);
+		if (status < 0) {
+			pr_warn("ctrl %d qid %d: challenge error (%d)\n",
+				ctrl->cntlid, req->sq->qid, status);
+			status = NVME_SC_INTERNAL;
+			break;
+		}
+		if (status) {
+			req->sq->dhchap_status = status;
+			nvmet_auth_failure1(req, d, al);
+			pr_warn("ctrl %d qid %d: challenge status (%x)\n",
+				ctrl->cntlid, req->sq->qid,
+				req->sq->dhchap_status);
+			status = 0;
+			break;
+		}
+		req->sq->dhchap_step = NVME_AUTH_DHCHAP_MESSAGE_REPLY;
+		break;
+	case NVME_AUTH_DHCHAP_MESSAGE_SUCCESS1:
+		status = nvmet_auth_success1(req, d, al);
+		if (status) {
+			req->sq->dhchap_status = status;
+			req->sq->authenticated = false;
+			nvmet_auth_failure1(req, d, al);
+			pr_warn("ctrl %d qid %d: success1 status (%x)\n",
+				ctrl->cntlid, req->sq->qid,
+				req->sq->dhchap_status);
+			break;
+		}
+		req->sq->dhchap_step = NVME_AUTH_DHCHAP_MESSAGE_SUCCESS2;
+		break;
+	case NVME_AUTH_DHCHAP_MESSAGE_FAILURE1:
+		req->sq->authenticated = false;
+		nvmet_auth_failure1(req, d, al);
+		pr_warn("ctrl %d qid %d failure1 (%x)\n",
+			ctrl->cntlid, req->sq->qid, req->sq->dhchap_status);
+		break;
+	default:
+		pr_warn("ctrl %d qid %d unhandled step (%d)\n",
+			ctrl->cntlid, req->sq->qid, req->sq->dhchap_step);
+		req->sq->dhchap_step = NVME_AUTH_DHCHAP_MESSAGE_FAILURE1;
+		req->sq->dhchap_status = NVME_AUTH_DHCHAP_FAILURE_FAILED;
+		nvmet_auth_failure1(req, d, al);
+		status = 0;
+		break;
+	}
+
+	status = nvmet_copy_to_sgl(req, 0, d, al);
+	kfree(d);
+done:
+	req->cqe->result.u64 = 0;
+	nvmet_req_complete(req, status);
+	if (req->sq->dhchap_step == NVME_AUTH_DHCHAP_MESSAGE_SUCCESS2)
+		nvmet_auth_sq_free(req->sq);
+	else if (req->sq->dhchap_step == NVME_AUTH_DHCHAP_MESSAGE_FAILURE1) {
+		nvmet_auth_sq_free(req->sq);
+		nvmet_ctrl_fatal_error(ctrl);
+	}
+}
diff --git a/drivers/nvme/target/fabrics-cmd.c b/drivers/nvme/target/fabrics-cmd.c
index f23c28729908..f91a56180d3d 100644
--- a/drivers/nvme/target/fabrics-cmd.c
+++ b/drivers/nvme/target/fabrics-cmd.c
@@ -93,6 +93,14 @@ u16 nvmet_parse_fabrics_admin_cmd(struct nvmet_req *req)
 	case nvme_fabrics_type_property_get:
 		req->execute = nvmet_execute_prop_get;
 		break;
+#ifdef CONFIG_NVME_TARGET_AUTH
+	case nvme_fabrics_type_auth_send:
+		req->execute = nvmet_execute_auth_send;
+		break;
+	case nvme_fabrics_type_auth_receive:
+		req->execute = nvmet_execute_auth_receive;
+		break;
+#endif
 	default:
 		pr_debug("received unknown capsule type 0x%x\n",
 			cmd->fabrics.fctype);
@@ -108,6 +116,14 @@ u16 nvmet_parse_fabrics_io_cmd(struct nvmet_req *req)
 	struct nvme_command *cmd = req->cmd;
 
 	switch (cmd->fabrics.fctype) {
+#ifdef CONFIG_NVME_TARGET_AUTH
+	case nvme_fabrics_type_auth_send:
+		req->execute = nvmet_execute_auth_send;
+		break;
+	case nvme_fabrics_type_auth_receive:
+		req->execute = nvmet_execute_auth_receive;
+		break;
+#endif
 	default:
 		pr_debug("received unknown capsule type 0x%x\n",
 			cmd->fabrics.fctype);
@@ -188,6 +204,7 @@ static void nvmet_execute_admin_connect(struct nvmet_req *req)
 	struct nvmf_connect_data *d;
 	struct nvmet_ctrl *ctrl = NULL;
 	u16 status = 0;
+	int ret;
 
 	if (!nvmet_check_transfer_len(req, sizeof(struct nvmf_connect_data)))
 		return;
@@ -230,18 +247,32 @@ static void nvmet_execute_admin_connect(struct nvmet_req *req)
 
 	uuid_copy(&ctrl->hostid, &d->hostid);
 
+	ret = nvmet_setup_auth(ctrl);
+	if (ret < 0) {
+		pr_err("Failed to setup authentication, error %d\n", ret);
+		nvmet_ctrl_put(ctrl);
+		if (ret == -EPERM)
+			status = (NVME_SC_CONNECT_INVALID_HOST | NVME_SC_DNR);
+		else
+			status = NVME_SC_INTERNAL;
+		goto out;
+	}
+
 	status = nvmet_install_queue(ctrl, req);
 	if (status) {
 		nvmet_ctrl_put(ctrl);
 		goto out;
 	}
 
-	pr_info("creating %s controller %d for subsystem %s for NQN %s%s.\n",
+	pr_info("creating %s controller %d for subsystem %s for NQN %s%s%s.\n",
 		nvmet_is_disc_subsys(ctrl->subsys) ? "discovery" : "nvm",
 		ctrl->cntlid, ctrl->subsys->subsysnqn, ctrl->hostnqn,
-		ctrl->pi_support ? " T10-PI is enabled" : "");
+		ctrl->pi_support ? " T10-PI is enabled" : "",
+		nvmet_has_auth(ctrl) ? " with DH-HMAC-CHAP" : "");
 	req->cqe->result.u16 = cpu_to_le16(ctrl->cntlid);
 
+	if (nvmet_has_auth(ctrl))
+		nvmet_init_auth(ctrl, req);
 out:
 	kfree(d);
 complete:
@@ -301,6 +332,9 @@ static void nvmet_execute_io_connect(struct nvmet_req *req)
 	req->cqe->result.u16 = cpu_to_le16(ctrl->cntlid);
 
 	pr_debug("adding queue %d to ctrl %d.\n", qid, ctrl->cntlid);
+	req->cqe->result.u16 = cpu_to_le16(ctrl->cntlid);
+	if (nvmet_has_auth(ctrl))
+		nvmet_init_auth(ctrl, req);
 
 out:
 	kfree(d);
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index c37f41eafc2f..765db7541a87 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -108,6 +108,18 @@ struct nvmet_sq {
 	u16			size;
 	u32			sqhd;
 	bool			sqhd_disabled;
+#ifdef CONFIG_NVME_TARGET_AUTH
+	bool			authenticated;
+	u16			dhchap_tid;
+	u16			dhchap_status;
+	int			dhchap_step;
+	u8			*dhchap_c1;
+	u8			*dhchap_c2;
+	u32			dhchap_s1;
+	u32			dhchap_s2;
+	u8			*dhchap_skey;
+	int			dhchap_skey_len;
+#endif
 	struct completion	free_done;
 	struct completion	confirm_done;
 };
@@ -209,6 +221,11 @@ struct nvmet_ctrl {
 	u64			err_counter;
 	struct nvme_error_slot	slots[NVMET_ERROR_LOG_SLOTS];
 	bool			pi_support;
+#ifdef CONFIG_NVME_TARGET_AUTH
+	struct nvme_dhchap_key	*host_key;
+	struct nvme_dhchap_key	*ctrl_key;
+	u8			shash_id;
+#endif
 };
 
 struct nvmet_subsys {
@@ -270,6 +287,12 @@ static inline struct nvmet_subsys *namespaces_to_subsys(
 
 struct nvmet_host {
 	struct config_group	group;
+	u8			*dhchap_secret;
+	u8			*dhchap_ctrl_secret;
+	u8			dhchap_key_hash;
+	u8			dhchap_ctrl_key_hash;
+	u8			dhchap_hash_id;
+	u8			dhchap_dhgroup_id;
 };
 
 static inline struct nvmet_host *to_host(struct config_item *item)
@@ -668,4 +691,43 @@ static inline void nvmet_req_bio_put(struct nvmet_req *req, struct bio *bio)
 		bio_put(bio);
 }
 
+#ifdef CONFIG_NVME_TARGET_AUTH
+void nvmet_execute_auth_send(struct nvmet_req *req);
+void nvmet_execute_auth_receive(struct nvmet_req *req);
+int nvmet_auth_set_key(struct nvmet_host *host, const char *secret,
+		       bool set_ctrl);
+int nvmet_auth_set_host_hash(struct nvmet_host *host, const char *hash);
+int nvmet_setup_auth(struct nvmet_ctrl *ctrl);
+void nvmet_init_auth(struct nvmet_ctrl *ctrl, struct nvmet_req *req);
+void nvmet_destroy_auth(struct nvmet_ctrl *ctrl);
+void nvmet_auth_sq_free(struct nvmet_sq *sq);
+bool nvmet_check_auth_status(struct nvmet_req *req);
+int nvmet_auth_host_hash(struct nvmet_req *req, u8 *response,
+			 unsigned int hash_len);
+int nvmet_auth_ctrl_hash(struct nvmet_req *req, u8 *response,
+			 unsigned int hash_len);
+static inline bool nvmet_has_auth(struct nvmet_ctrl *ctrl)
+{
+	return ctrl->host_key != NULL;
+}
+#else
+static inline int nvmet_setup_auth(struct nvmet_ctrl *ctrl)
+{
+	return 0;
+}
+static inline void nvmet_init_auth(struct nvmet_ctrl *ctrl,
+				   struct nvmet_req *req) {};
+static inline void nvmet_destroy_auth(struct nvmet_ctrl *ctrl) {};
+static inline void nvmet_auth_sq_free(struct nvmet_sq *sq) {};
+static inline bool nvmet_check_auth_status(struct nvmet_req *req)
+{
+	return true;
+}
+static inline bool nvmet_has_auth(struct nvmet_ctrl *ctrl)
+{
+	return false;
+}
+static inline const char *nvmet_dhchap_dhgroup_name(u8 dhgid) { return NULL; }
+#endif
+
 #endif /* _NVMET_H */
-- 
2.29.2

[PATCH 10/11] nvmet-auth: Diffie-Hellman key exchange support

[Hannes Reinecke]

Implement Diffie-Hellman key exchange using FFDHE groups for NVMe
In-Band Authentication.
This patch adds a new host configfs attribute 'dhchap_dhgroup' to
select the FFDHE group to use.

Signed-off-by: Hannes Reinecke <hare@suse.de>
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
---
 drivers/nvme/target/Kconfig            |   2 +
 drivers/nvme/target/auth.c             | 157 +++++++++++++++++++++++++
 drivers/nvme/target/configfs.c         |  31 +++++
 drivers/nvme/target/fabrics-cmd-auth.c |  41 +++++--
 drivers/nvme/target/nvmet.h            |   9 ++
 5 files changed, 232 insertions(+), 8 deletions(-)

diff --git a/drivers/nvme/target/Kconfig b/drivers/nvme/target/Kconfig
index df526b59b509..f0c91f7686a3 100644
--- a/drivers/nvme/target/Kconfig
+++ b/drivers/nvme/target/Kconfig
@@ -92,6 +92,8 @@ config NVME_TARGET_AUTH
 	select CRYPTO_HMAC
 	select CRYPTO_SHA256
 	select CRYPTO_SHA512
+	select CRYPTO_DH
+	select CRYPTO_DH_GROUPS_RFC7919
 	help
 	  This enables support for NVMe over Fabrics In-band Authentication
 
diff --git a/drivers/nvme/target/auth.c b/drivers/nvme/target/auth.c
index 5cdd23c34185..d5624bdf834b 100644
--- a/drivers/nvme/target/auth.c
+++ b/drivers/nvme/target/auth.c
@@ -54,6 +54,74 @@ int nvmet_auth_set_key(struct nvmet_host *host, const char *secret,
 	return 0;
 }
 
+int nvmet_setup_dhgroup(struct nvmet_ctrl *ctrl, u8 dhgroup_id)
+{
+	const char *dhgroup_kpp;
+	int ret = 0;
+
+	pr_debug("%s: ctrl %d selecting dhgroup %d\n",
+		 __func__, ctrl->cntlid, dhgroup_id);
+
+	if (ctrl->dh_tfm) {
+		if (ctrl->dh_gid == dhgroup_id) {
+			pr_debug("%s: ctrl %d reuse existing DH group %d\n",
+				 __func__, ctrl->cntlid, dhgroup_id);
+			return 0;
+		}
+		crypto_free_kpp(ctrl->dh_tfm);
+		ctrl->dh_tfm = NULL;
+		ctrl->dh_gid = 0;
+	}
+
+	if (dhgroup_id == NVME_AUTH_DHGROUP_NULL)
+		return 0;
+
+	dhgroup_kpp = nvme_auth_dhgroup_kpp(dhgroup_id);
+	if (!dhgroup_kpp) {
+		pr_debug("%s: ctrl %d invalid DH group %d\n",
+			 __func__, ctrl->cntlid, dhgroup_id);
+		return -EINVAL;
+	}
+	ctrl->dh_tfm = crypto_alloc_kpp(dhgroup_kpp, 0, 0);
+	if (IS_ERR(ctrl->dh_tfm)) {
+		pr_debug("%s: ctrl %d failed to setup DH group %d, err %ld\n",
+			 __func__, ctrl->cntlid, dhgroup_id,
+			 PTR_ERR(ctrl->dh_tfm));
+		ret = PTR_ERR(ctrl->dh_tfm);
+		ctrl->dh_tfm = NULL;
+		ctrl->dh_gid = 0;
+	} else {
+		ctrl->dh_gid = dhgroup_id;
+		pr_debug("%s: ctrl %d setup DH group %d\n",
+			 __func__, ctrl->cntlid, ctrl->dh_gid);
+		ret = nvme_auth_gen_privkey(ctrl->dh_tfm, ctrl->dh_gid);
+		if (ret < 0) {
+			pr_debug("%s: ctrl %d failed to generate private key, err %d\n",
+				 __func__, ctrl->cntlid, ret);
+			kfree_sensitive(ctrl->dh_key);
+			return ret;
+		}
+		ctrl->dh_keysize = crypto_kpp_maxsize(ctrl->dh_tfm);
+		kfree_sensitive(ctrl->dh_key);
+		ctrl->dh_key = kzalloc(ctrl->dh_keysize, GFP_KERNEL);
+		if (!ctrl->dh_key) {
+			pr_warn("ctrl %d failed to allocate public key\n",
+				ctrl->cntlid);
+			return -ENOMEM;
+		}
+		ret = nvme_auth_gen_pubkey(ctrl->dh_tfm, ctrl->dh_key,
+					   ctrl->dh_keysize);
+		if (ret < 0) {
+			pr_warn("ctrl %d failed to generate public key\n",
+				ctrl->cntlid);
+			kfree(ctrl->dh_key);
+			ctrl->dh_key = NULL;
+		}
+	}
+
+	return ret;
+}
+
 int nvmet_setup_auth(struct nvmet_ctrl *ctrl)
 {
 	int ret = 0;
@@ -81,6 +149,10 @@ int nvmet_setup_auth(struct nvmet_ctrl *ctrl)
 		goto out_unlock;
 	}
 
+	ret = nvmet_setup_dhgroup(ctrl, host->dhchap_dhgroup_id);
+	if (ret < 0)
+		pr_warn("Failed to setup DH group");
+
 	if (!host->dhchap_secret) {
 		pr_debug("No authentication provided\n");
 		goto out_unlock;
@@ -158,6 +230,14 @@ void nvmet_destroy_auth(struct nvmet_ctrl *ctrl)
 {
 	ctrl->shash_id = 0;
 
+	if (ctrl->dh_tfm) {
+		crypto_free_kpp(ctrl->dh_tfm);
+		ctrl->dh_tfm = NULL;
+		ctrl->dh_gid = 0;
+	}
+	kfree_sensitive(ctrl->dh_key);
+	ctrl->dh_key = NULL;
+
 	if (ctrl->host_key) {
 		nvme_auth_free_key(ctrl->host_key);
 		ctrl->host_key = NULL;
@@ -218,6 +298,21 @@ int nvmet_auth_host_hash(struct nvmet_req *req, u8 *response,
 	if (ret)
 		goto out_free_response;
 
+	if (ctrl->dh_gid != NVME_AUTH_DHGROUP_NULL) {
+		challenge = kmalloc(shash_len, GFP_KERNEL);
+		if (!challenge) {
+			ret = -ENOMEM;
+			goto out_free_response;
+		}
+		ret = nvme_auth_augmented_challenge(ctrl->shash_id,
+						    req->sq->dhchap_skey,
+						    req->sq->dhchap_skey_len,
+						    req->sq->dhchap_c1,
+						    challenge, shash_len);
+		if (ret)
+			goto out_free_response;
+	}
+
 	pr_debug("ctrl %d qid %d host response seq %u transaction %d\n",
 		 ctrl->cntlid, req->sq->qid, req->sq->dhchap_s1,
 		 req->sq->dhchap_tid);
@@ -315,6 +410,21 @@ int nvmet_auth_ctrl_hash(struct nvmet_req *req, u8 *response,
 	if (ret)
 		goto out_free_response;
 
+	if (ctrl->dh_gid != NVME_AUTH_DHGROUP_NULL) {
+		challenge = kmalloc(shash_len, GFP_KERNEL);
+		if (!challenge) {
+			ret = -ENOMEM;
+			goto out_free_response;
+		}
+		ret = nvme_auth_augmented_challenge(ctrl->shash_id,
+						    req->sq->dhchap_skey,
+						    req->sq->dhchap_skey_len,
+						    req->sq->dhchap_c2,
+						    challenge, shash_len);
+		if (ret)
+			goto out_free_response;
+	}
+
 	shash = kzalloc(sizeof(*shash) + crypto_shash_descsize(shash_tfm),
 			GFP_KERNEL);
 	if (!shash) {
@@ -365,3 +475,50 @@ int nvmet_auth_ctrl_hash(struct nvmet_req *req, u8 *response,
 	crypto_free_shash(shash_tfm);
 	return 0;
 }
+
+int nvmet_auth_ctrl_exponential(struct nvmet_req *req,
+				u8 *buf, int buf_size)
+{
+	struct nvmet_ctrl *ctrl = req->sq->ctrl;
+	int ret = 0;
+
+	if (!ctrl->dh_key) {
+		pr_warn("ctrl %d no DH public key!\n", ctrl->cntlid);
+		return -ENOKEY;
+	}
+	if (buf_size != ctrl->dh_keysize) {
+		pr_warn("ctrl %d DH public key size mismatch, need %lu is %d\n",
+			ctrl->cntlid, ctrl->dh_keysize, buf_size);
+		ret = -EINVAL;
+	} else {
+		memcpy(buf, ctrl->dh_key, buf_size);
+		pr_debug("%s: ctrl %d public key %*ph\n", __func__,
+			 ctrl->cntlid, (int)buf_size, buf);
+	}
+
+	return ret;
+}
+
+int nvmet_auth_ctrl_sesskey(struct nvmet_req *req,
+			    u8 *pkey, int pkey_size)
+{
+	struct nvmet_ctrl *ctrl = req->sq->ctrl;
+	int ret;
+
+	req->sq->dhchap_skey_len = ctrl->dh_keysize;
+	req->sq->dhchap_skey = kzalloc(req->sq->dhchap_skey_len, GFP_KERNEL);
+	if (!req->sq->dhchap_skey)
+		return -ENOMEM;
+	ret = nvme_auth_gen_shared_secret(ctrl->dh_tfm,
+					  pkey, pkey_size,
+					  req->sq->dhchap_skey,
+					  req->sq->dhchap_skey_len);
+	if (ret)
+		pr_debug("failed to compute shared secred, err %d\n", ret);
+	else
+		pr_debug("%s: shared secret %*ph\n", __func__,
+			 (int)req->sq->dhchap_skey_len,
+			 req->sq->dhchap_skey);
+
+	return ret;
+}
diff --git a/drivers/nvme/target/configfs.c b/drivers/nvme/target/configfs.c
index 9a7d91c64fcd..1b11f6a83bb6 100644
--- a/drivers/nvme/target/configfs.c
+++ b/drivers/nvme/target/configfs.c
@@ -1746,10 +1746,41 @@ static ssize_t nvmet_host_dhchap_hash_store(struct config_item *item,
 
 CONFIGFS_ATTR(nvmet_host_, dhchap_hash);
 
+static ssize_t nvmet_host_dhchap_dhgroup_show(struct config_item *item,
+		char *page)
+{
+	struct nvmet_host *host = to_host(item);
+	const char *dhgroup = nvme_auth_dhgroup_name(host->dhchap_dhgroup_id);
+
+	return sprintf(page, "%s\n", dhgroup ? dhgroup : "none");
+}
+
+static ssize_t nvmet_host_dhchap_dhgroup_store(struct config_item *item,
+		const char *page, size_t count)
+{
+	struct nvmet_host *host = to_host(item);
+	int dhgroup_id;
+
+	dhgroup_id = nvme_auth_dhgroup_id(page);
+	if (dhgroup_id == NVME_AUTH_DHGROUP_INVALID)
+		return -EINVAL;
+	if (dhgroup_id != NVME_AUTH_DHGROUP_NULL) {
+		const char *kpp = nvme_auth_dhgroup_kpp(dhgroup_id);
+
+		if (!crypto_has_kpp(kpp, 0, 0))
+			return -EINVAL;
+	}
+	host->dhchap_dhgroup_id = dhgroup_id;
+	return count;
+}
+
+CONFIGFS_ATTR(nvmet_host_, dhchap_dhgroup);
+
 static struct configfs_attribute *nvmet_host_attrs[] = {
 	&nvmet_host_attr_dhchap_key,
 	&nvmet_host_attr_dhchap_ctrl_key,
 	&nvmet_host_attr_dhchap_hash,
+	&nvmet_host_attr_dhchap_dhgroup,
 	NULL,
 };
 #endif /* CONFIG_NVME_TARGET_AUTH */
diff --git a/drivers/nvme/target/fabrics-cmd-auth.c b/drivers/nvme/target/fabrics-cmd-auth.c
index 776073a10e04..5b1be7e607e2 100644
--- a/drivers/nvme/target/fabrics-cmd-auth.c
+++ b/drivers/nvme/target/fabrics-cmd-auth.c
@@ -27,7 +27,7 @@ static u16 nvmet_auth_negotiate(struct nvmet_req *req, void *d)
 {
 	struct nvmet_ctrl *ctrl = req->sq->ctrl;
 	struct nvmf_auth_dhchap_negotiate_data *data = d;
-	int i, hash_id = 0, fallback_hash_id = 0, dhgid;
+	int i, hash_id = 0, fallback_hash_id = 0, dhgid, fallback_dhgid;
 
 	pr_debug("%s: ctrl %d qid %d: data sc_d %d napd %d authid %d halen %d dhlen %d\n",
 		 __func__, ctrl->cntlid, req->sq->qid,
@@ -69,22 +69,35 @@ static u16 nvmet_auth_negotiate(struct nvmet_req *req, void *d)
 	}
 
 	dhgid = -1;
+	fallback_dhgid = -1;
 	for (i = 0; i < data->auth_protocol[0].dhchap.dhlen; i++) {
 		int tmp_dhgid = data->auth_protocol[0].dhchap.idlist[i + 30];
 
-		if (tmp_dhgid == NVME_AUTH_DHGROUP_NULL) {
+		if (tmp_dhgid != ctrl->dh_gid) {
 			dhgid = tmp_dhgid;
 			break;
 		}
+		if (fallback_dhgid < 0) {
+			const char *kpp = nvme_auth_dhgroup_kpp(tmp_dhgid);
+
+			if (crypto_has_kpp(kpp, 0, 0))
+				fallback_dhgid = tmp_dhgid;
+		}
 	}
 	if (dhgid < 0) {
-		pr_debug("%s: ctrl %d qid %d: no usable DH group found\n",
+		if (fallback_dhgid < 0) {
+			pr_debug("%s: ctrl %d qid %d: no usable DH group found\n",
 				 __func__, ctrl->cntlid, req->sq->qid);
-		return NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
+			return NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
+		}
+		pr_debug("%s: ctrl %d qid %d: configured DH group %s not found\n",
+			 __func__, ctrl->cntlid, req->sq->qid,
+			 nvme_auth_dhgroup_name(fallback_dhgid));
+		ctrl->dh_gid = fallback_dhgid;
 	}
 	pr_debug("%s: ctrl %d qid %d: selected DH group %s (%d)\n",
 		 __func__, ctrl->cntlid, req->sq->qid,
-		 nvme_auth_dhgroup_name(dhgid), dhgid);
+		 nvme_auth_dhgroup_name(ctrl->dh_gid), ctrl->dh_gid);
 	return 0;
 }
 
@@ -100,7 +113,11 @@ static u16 nvmet_auth_reply(struct nvmet_req *req, void *d)
 		 data->hl, data->cvalid, dhvlen);
 
 	if (dhvlen) {
-		return NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		if (!ctrl->dh_tfm)
+			return NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		if (nvmet_auth_ctrl_sesskey(req, data->rval + 2 * data->hl,
+					    dhvlen) < 0)
+			return NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
 	}
 
 	response = kmalloc(data->hl, GFP_KERNEL);
@@ -332,6 +349,8 @@ static int nvmet_auth_challenge(struct nvmet_req *req, void *d, int al)
 	int hash_len = nvme_auth_hmac_hash_len(ctrl->shash_id);
 	int data_size = sizeof(*d) + hash_len;
 
+	if (ctrl->dh_tfm)
+		data_size += ctrl->dh_keysize;
 	if (al < data_size) {
 		pr_debug("%s: buffer too small (al %d need %d)\n", __func__,
 			 al, data_size);
@@ -350,9 +369,15 @@ static int nvmet_auth_challenge(struct nvmet_req *req, void *d, int al)
 		return -ENOMEM;
 	get_random_bytes(req->sq->dhchap_c1, data->hl);
 	memcpy(data->cval, req->sq->dhchap_c1, data->hl);
-	pr_debug("%s: ctrl %d qid %d seq %u transaction %d hl %d dhvlen %u\n",
+	if (ctrl->dh_tfm) {
+		data->dhgid = ctrl->dh_gid;
+		data->dhvlen = cpu_to_le16(ctrl->dh_keysize);
+		ret = nvmet_auth_ctrl_exponential(req, data->cval + data->hl,
+						  ctrl->dh_keysize);
+	}
+	pr_debug("%s: ctrl %d qid %d seq %d transaction %d hl %d dhvlen %zu\n",
 		 __func__, ctrl->cntlid, req->sq->qid, req->sq->dhchap_s1,
-		 req->sq->dhchap_tid, data->hl, 0);
+		 req->sq->dhchap_tid, data->hl, ctrl->dh_keysize);
 	return ret;
 }
 
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index 765db7541a87..8b239aec3ca2 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -225,6 +225,10 @@ struct nvmet_ctrl {
 	struct nvme_dhchap_key	*host_key;
 	struct nvme_dhchap_key	*ctrl_key;
 	u8			shash_id;
+	struct crypto_kpp	*dh_tfm;
+	u8			dh_gid;
+	u8			*dh_key;
+	size_t			dh_keysize;
 #endif
 };
 
@@ -701,6 +705,7 @@ int nvmet_setup_auth(struct nvmet_ctrl *ctrl);
 void nvmet_init_auth(struct nvmet_ctrl *ctrl, struct nvmet_req *req);
 void nvmet_destroy_auth(struct nvmet_ctrl *ctrl);
 void nvmet_auth_sq_free(struct nvmet_sq *sq);
+int nvmet_setup_dhgroup(struct nvmet_ctrl *ctrl, u8 dhgroup_id);
 bool nvmet_check_auth_status(struct nvmet_req *req);
 int nvmet_auth_host_hash(struct nvmet_req *req, u8 *response,
 			 unsigned int hash_len);
@@ -710,6 +715,10 @@ static inline bool nvmet_has_auth(struct nvmet_ctrl *ctrl)
 {
 	return ctrl->host_key != NULL;
 }
+int nvmet_auth_ctrl_exponential(struct nvmet_req *req,
+				u8 *buf, int buf_size);
+int nvmet_auth_ctrl_sesskey(struct nvmet_req *req,
+			    u8 *buf, int buf_size);
 #else
 static inline int nvmet_setup_auth(struct nvmet_ctrl *ctrl)
 {
-- 
2.29.2

[PATCH 11/11] nvmet-auth: expire authentication sessions

[Hannes Reinecke]

Each authentication step is required to be completed within the
KATO interval (or two minutes if not set). So add a workqueue function
to reset the transaction ID and the expected next protocol step;
this will automatically the next authentication command referring
to the terminated authentication.

Signed-off-by: Hannes Reinecke <hare@suse.de>
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
---
 drivers/nvme/target/auth.c             |  1 +
 drivers/nvme/target/fabrics-cmd-auth.c | 20 +++++++++++++++++++-
 drivers/nvme/target/nvmet.h            |  1 +
 3 files changed, 21 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/target/auth.c b/drivers/nvme/target/auth.c
index d5624bdf834b..bf92435c783c 100644
--- a/drivers/nvme/target/auth.c
+++ b/drivers/nvme/target/auth.c
@@ -218,6 +218,7 @@ int nvmet_setup_auth(struct nvmet_ctrl *ctrl)
 
 void nvmet_auth_sq_free(struct nvmet_sq *sq)
 {
+	cancel_delayed_work(&sq->auth_expired_work);
 	kfree(sq->dhchap_c1);
 	sq->dhchap_c1 = NULL;
 	kfree(sq->dhchap_c2);
diff --git a/drivers/nvme/target/fabrics-cmd-auth.c b/drivers/nvme/target/fabrics-cmd-auth.c
index 5b1be7e607e2..cc56e8c821ce 100644
--- a/drivers/nvme/target/fabrics-cmd-auth.c
+++ b/drivers/nvme/target/fabrics-cmd-auth.c
@@ -12,11 +12,24 @@
 #include <crypto/kpp.h>
 #include "nvmet.h"
 
+static void nvmet_auth_expired_work(struct work_struct *work)
+{
+	struct nvmet_sq *sq = container_of(to_delayed_work(work),
+			struct nvmet_sq, auth_expired_work);
+
+	pr_debug("%s: ctrl %d qid %d transaction %u expired, resetting\n",
+		 __func__, sq->ctrl->cntlid, sq->qid, sq->dhchap_tid);
+	sq->dhchap_step = NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE;
+	sq->dhchap_tid = -1;
+}
+
 void nvmet_init_auth(struct nvmet_ctrl *ctrl, struct nvmet_req *req)
 {
 	u32 result = le32_to_cpu(req->cqe->result.u32);
 
 	/* Initialize in-band authentication */
+	INIT_DELAYED_WORK(&req->sq->auth_expired_work,
+			  nvmet_auth_expired_work);
 	req->sq->authenticated = false;
 	req->sq->dhchap_step = NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE;
 	result |= (u32)NVME_CONNECT_AUTHREQ_ATR << 16;
@@ -333,8 +346,13 @@ void nvmet_execute_auth_send(struct nvmet_req *req)
 	req->cqe->result.u64 = 0;
 	nvmet_req_complete(req, status);
 	if (req->sq->dhchap_step != NVME_AUTH_DHCHAP_MESSAGE_SUCCESS2 &&
-	    req->sq->dhchap_step != NVME_AUTH_DHCHAP_MESSAGE_FAILURE2)
+	    req->sq->dhchap_step != NVME_AUTH_DHCHAP_MESSAGE_FAILURE2) {
+		unsigned long auth_expire_secs = ctrl->kato ? ctrl->kato : 120;
+
+		mod_delayed_work(system_wq, &req->sq->auth_expired_work,
+				 auth_expire_secs * HZ);
 		return;
+	}
 	/* Final states, clear up variables */
 	nvmet_auth_sq_free(req->sq);
 	if (req->sq->dhchap_step == NVME_AUTH_DHCHAP_MESSAGE_FAILURE2)
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index 8b239aec3ca2..829fb1d78ee1 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -109,6 +109,7 @@ struct nvmet_sq {
 	u32			sqhd;
 	bool			sqhd_disabled;
 #ifdef CONFIG_NVME_TARGET_AUTH
+	struct delayed_work	auth_expired_work;
 	bool			authenticated;
 	u16			dhchap_tid;
 	u16			dhchap_status;
-- 
2.29.2

Re: [PATCHv16 00/11] nvme: In-band authentication support

[Sagi Grimberg]

> Hi all,
> 
> recent updates to the NVMe spec have added definitions for in-band
> authentication, and seeing that it provides some real benefit
> especially for NVMe-TCP here's an attempt to implement it.
> 
> Thanks to Nicolai Stange the crypto DH framework has been upgraded
> to provide us with a FFDHE implementation; I've updated the patchset
> to use the ephemeral key generation provided there.
> 
> Note that this is just for in-band authentication. Secure
> concatenation (ie starting TLS with the negotiated parameters)
> requires a TLS handshake, which the in-kernel TLS implementation
> does not provide. This is being worked on with a different patchset
> which is still WIP.
> 
> The nvme-cli support has already been merged; please use the latest
> nvme-cli git repository to build the most recent version.
> 
> A copy of this patchset can be found at
> git://git.kernel.org/pub/scm/linux/kernel/git/hare/scsi-devel
> branch auth.v15
> 
> The patchset is being cut against nvme-5.20.
> 
> As usual, comments and reviews are welcome.

Hannes, did you see my panic report on a malformed dhchap_ctrl_key?

Also, why does the dhchap_ctrl_key not passed when connecting
via discovery?

I have in the target:
--
# grep -r '' /sys/kernel/config/nvmet/hosts/
/sys/kernel/config/nvmet/hosts/nqn.2014-08.org.nvmexpress:uuid:302ae323-4acd-465d-ace4-3d4102e9d11f/dhchap_dhgroup:null
/sys/kernel/config/nvmet/hosts/nqn.2014-08.org.nvmexpress:uuid:302ae323-4acd-465d-ace4-3d4102e9d11f/dhchap_hash:hmac(sha256)
/sys/kernel/config/nvmet/hosts/nqn.2014-08.org.nvmexpress:uuid:302ae323-4acd-465d-ace4-3d4102e9d11f/dhchap_ctrl_key:DHHC-1:00:Jc/My1o0qtLCWRp+sHhAVN6mFaS7YQOMYhk9zSmlatobqB8C:
/sys/kernel/config/nvmet/hosts/nqn.2014-08.org.nvmexpress:uuid:302ae323-4acd-465d-ace4-3d4102e9d11f/dhchap_key:DHHC-1:00:QpxVGpctx5J+4SeW2MClUI8rfZO3WdP1llImvsPsx7e3TK+I:
--

Then on the host I have:
--
# cat /etc/nvme/config.json
[
   {
     "hostnqn": 
"nqn.2014-08.org.nvmexpress:uuid:302ae323-4acd-465d-ace4-3d4102e9d11f",
     "hostid": "14f15c4e-f6cb-434b-90cd-7c1f84f0c194",
     "dhchap_key": 
"DHHC-1:00:QpxVGpctx5J+4SeW2MClUI8rfZO3WdP1llImvsPsx7e3TK+I:",
     "subsystems": [
       {
         "nqn": "testnqn1",
         "ports": [
           {
             "transport": "tcp",
             "traddr": "192.168.123.1",
             "trsvcid": "8009",
             "dhchap_key": 
"DHHC-1:00:Jc/My1o0qtLCWRp+sHhAVN6mFaS7YQOMYhk9zSmlatobqB8C:"
           }
         ]
       }
     ]
   }
]
--

And when I do connect-all (i.e. connect via the discovery log page:
--
# grep -r '' /sys/class/nvme/nvme1/dhchap*
/sys/class/nvme/nvme1/dhchap_ctrl_secret:none
/sys/class/nvme/nvme1/dhchap_secret:DHHC-1:00:QpxVGpctx5J+4SeW2MClUI8rfZO3WdP1llImvsPsx7e3TK+I:
--

This means that I can corrupt the dhchap_ctrl_key entry in the config
and no one will care (because it is not authenticating the ctrl if
dhchap_ctrl_key is not passed)

I think this is something wrong with nvme-cli/libnvme though...

Re: [PATCHv16 00/11] nvme: In-band authentication support

[Hannes Reinecke]

On 6/21/22 23:22, Sagi Grimberg wrote:
> 
>> Hi all,
>>
>> recent updates to the NVMe spec have added definitions for in-band
>> authentication, and seeing that it provides some real benefit
>> especially for NVMe-TCP here's an attempt to implement it.
>>
>> Thanks to Nicolai Stange the crypto DH framework has been upgraded
>> to provide us with a FFDHE implementation; I've updated the patchset
>> to use the ephemeral key generation provided there.
>>
>> Note that this is just for in-band authentication. Secure
>> concatenation (ie starting TLS with the negotiated parameters)
>> requires a TLS handshake, which the in-kernel TLS implementation
>> does not provide. This is being worked on with a different patchset
>> which is still WIP.
>>
>> The nvme-cli support has already been merged; please use the latest
>> nvme-cli git repository to build the most recent version.
>>
>> A copy of this patchset can be found at
>> git://git.kernel.org/pub/scm/linux/kernel/git/hare/scsi-devel
>> branch auth.v15
>>
>> The patchset is being cut against nvme-5.20.
>>
>> As usual, comments and reviews are welcome.
> 
> Hannes, did you see my panic report on a malformed dhchap_ctrl_key?
> 
> Also, why does the dhchap_ctrl_key not passed when connecting
> via discovery?
> 
> I have in the target:
> -- 
> # grep -r '' /sys/kernel/config/nvmet/hosts/
> /sys/kernel/config/nvmet/hosts/nqn.2014-08.org.nvmexpress:uuid:302ae323-4acd-465d-ace4-3d4102e9d11f/dhchap_dhgroup:null 
> 
> /sys/kernel/config/nvmet/hosts/nqn.2014-08.org.nvmexpress:uuid:302ae323-4acd-465d-ace4-3d4102e9d11f/dhchap_hash:hmac(sha256) 
> 
> /sys/kernel/config/nvmet/hosts/nqn.2014-08.org.nvmexpress:uuid:302ae323-4acd-465d-ace4-3d4102e9d11f/dhchap_ctrl_key:DHHC-1:00:Jc/My1o0qtLCWRp+sHhAVN6mFaS7YQOMYhk9zSmlatobqB8C: 
> 
> /sys/kernel/config/nvmet/hosts/nqn.2014-08.org.nvmexpress:uuid:302ae323-4acd-465d-ace4-3d4102e9d11f/dhchap_key:DHHC-1:00:QpxVGpctx5J+4SeW2MClUI8rfZO3WdP1llImvsPsx7e3TK+I: 
> 
> -- 
> 
> Then on the host I have:
> -- 
> # cat /etc/nvme/config.json
> [
>    {
>      "hostnqn": 
> "nqn.2014-08.org.nvmexpress:uuid:302ae323-4acd-465d-ace4-3d4102e9d11f",
>      "hostid": "14f15c4e-f6cb-434b-90cd-7c1f84f0c194",
>      "dhchap_key": 
> "DHHC-1:00:QpxVGpctx5J+4SeW2MClUI8rfZO3WdP1llImvsPsx7e3TK+I:",
>      "subsystems": [
>        {
>          "nqn": "testnqn1",
>          "ports": [
>            {
>              "transport": "tcp",
>              "traddr": "192.168.123.1",
>              "trsvcid": "8009",
>              "dhchap_key": 
> "DHHC-1:00:Jc/My1o0qtLCWRp+sHhAVN6mFaS7YQOMYhk9zSmlatobqB8C:"
>            }
>          ]
>        }
>      ]
>    }
> ]
> -- 
> 
> And when I do connect-all (i.e. connect via the discovery log page:
> -- 
> # grep -r '' /sys/class/nvme/nvme1/dhchap*
> /sys/class/nvme/nvme1/dhchap_ctrl_secret:none
> /sys/class/nvme/nvme1/dhchap_secret:DHHC-1:00:QpxVGpctx5J+4SeW2MClUI8rfZO3WdP1llImvsPsx7e3TK+I: 
> 
> -- 
> 
> This means that I can corrupt the dhchap_ctrl_key entry in the config
> and no one will care (because it is not authenticating the ctrl if
> dhchap_ctrl_key is not passed)
> 
Unfortunate design on both ends.
It's the host who requests controller authentication.
So if the host doesn't request it nothing will happen.
But that also means that controller authentication is optional, and so I 
didn't feel comfortable to refuse connections for an optional feature.

But we can make that a real error, and refuse the 'connect' call if the 
controller key is invalid. Might be a better choice after all.

Something like this would help:

diff --git a/drivers/nvme/host/auth.c b/drivers/nvme/host/auth.c
index 53184ac76240..a03f41fa146e 100644
--- a/drivers/nvme/host/auth.c
+++ b/drivers/nvme/host/auth.c
@@ -842,6 +842,10 @@ int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int 
qid)
                 dev_warn(ctrl->device, "qid %d: no key\n", qid);
                 return -ENOKEY;
         }
+       if (ctrl->opts->dhchap_ctrl_secret && !ctrl->ctrl_key) {
+               dev_warn(ctrl->device, "qid %d: invalid ctrl key\n", qid);
+               return -ENOKEY;
+       }

         mutex_lock(&ctrl->dhchap_auth_mutex);
         /* Check if the context is already queued */


Is this what you had in mind?

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman

Re: [PATCHv16 00/11] nvme: In-band authentication support

[Sagi Grimberg]

>>> Hi all,
>>>
>>> recent updates to the NVMe spec have added definitions for in-band
>>> authentication, and seeing that it provides some real benefit
>>> especially for NVMe-TCP here's an attempt to implement it.
>>>
>>> Thanks to Nicolai Stange the crypto DH framework has been upgraded
>>> to provide us with a FFDHE implementation; I've updated the patchset
>>> to use the ephemeral key generation provided there.
>>>
>>> Note that this is just for in-band authentication. Secure
>>> concatenation (ie starting TLS with the negotiated parameters)
>>> requires a TLS handshake, which the in-kernel TLS implementation
>>> does not provide. This is being worked on with a different patchset
>>> which is still WIP.
>>>
>>> The nvme-cli support has already been merged; please use the latest
>>> nvme-cli git repository to build the most recent version.
>>>
>>> A copy of this patchset can be found at
>>> git://git.kernel.org/pub/scm/linux/kernel/git/hare/scsi-devel
>>> branch auth.v15
>>>
>>> The patchset is being cut against nvme-5.20.
>>>
>>> As usual, comments and reviews are welcome.
>>
>> Hannes, did you see my panic report on a malformed dhchap_ctrl_key?
>>
>> Also, why does the dhchap_ctrl_key not passed when connecting
>> via discovery?
>>
>> I have in the target:
>> -- 
>> # grep -r '' /sys/kernel/config/nvmet/hosts/
>> /sys/kernel/config/nvmet/hosts/nqn.2014-08.org.nvmexpress:uuid:302ae323-4acd-465d-ace4-3d4102e9d11f/dhchap_dhgroup:null 
>>
>> /sys/kernel/config/nvmet/hosts/nqn.2014-08.org.nvmexpress:uuid:302ae323-4acd-465d-ace4-3d4102e9d11f/dhchap_hash:hmac(sha256) 
>>
>> /sys/kernel/config/nvmet/hosts/nqn.2014-08.org.nvmexpress:uuid:302ae323-4acd-465d-ace4-3d4102e9d11f/dhchap_ctrl_key:DHHC-1:00:Jc/My1o0qtLCWRp+sHhAVN6mFaS7YQOMYhk9zSmlatobqB8C: 
>>
>> /sys/kernel/config/nvmet/hosts/nqn.2014-08.org.nvmexpress:uuid:302ae323-4acd-465d-ace4-3d4102e9d11f/dhchap_key:DHHC-1:00:QpxVGpctx5J+4SeW2MClUI8rfZO3WdP1llImvsPsx7e3TK+I: 
>>
>> -- 
>>
>> Then on the host I have:
>> -- 
>> # cat /etc/nvme/config.json
>> [
>>    {
>>      "hostnqn": 
>> "nqn.2014-08.org.nvmexpress:uuid:302ae323-4acd-465d-ace4-3d4102e9d11f",
>>      "hostid": "14f15c4e-f6cb-434b-90cd-7c1f84f0c194",
>>      "dhchap_key": 
>> "DHHC-1:00:QpxVGpctx5J+4SeW2MClUI8rfZO3WdP1llImvsPsx7e3TK+I:",
>>      "subsystems": [
>>        {
>>          "nqn": "testnqn1",
>>          "ports": [
>>            {
>>              "transport": "tcp",
>>              "traddr": "192.168.123.1",
>>              "trsvcid": "8009",
>>              "dhchap_key": 
>> "DHHC-1:00:Jc/My1o0qtLCWRp+sHhAVN6mFaS7YQOMYhk9zSmlatobqB8C:"
>>            }
>>          ]
>>        }
>>      ]
>>    }
>> ]
>> -- 
>>
>> And when I do connect-all (i.e. connect via the discovery log page:
>> -- 
>> # grep -r '' /sys/class/nvme/nvme1/dhchap*
>> /sys/class/nvme/nvme1/dhchap_ctrl_secret:none
>> /sys/class/nvme/nvme1/dhchap_secret:DHHC-1:00:QpxVGpctx5J+4SeW2MClUI8rfZO3WdP1llImvsPsx7e3TK+I: 
>>
>> -- 
>>
>> This means that I can corrupt the dhchap_ctrl_key entry in the config
>> and no one will care (because it is not authenticating the ctrl if
>> dhchap_ctrl_key is not passed)
>>
> Unfortunate design on both ends.
> It's the host who requests controller authentication.

Yes.

> So if the host doesn't request it nothing will happen.

Correct.

> But that also means that controller authentication is optional, and so I 
> didn't feel comfortable to refuse connections for an optional feature.

If the ctrl authentication is unsuccessful then of course we need to
fail it. If the host requested the ctrl to authenticate, it needs to be
successful.

But I don't think that it explains what I'm seeing.

> 
> But we can make that a real error, and refuse the 'connect' call if the 
> controller key is invalid. Might be a better choice after all.

In my mind it is mandatory, its not really an interpretation thing...
But I think that if I pass a wrong dhchap_ctrl_key to connect I get
a failure.

> 
> Something like this would help:
> 
> diff --git a/drivers/nvme/host/auth.c b/drivers/nvme/host/auth.c
> index 53184ac76240..a03f41fa146e 100644
> --- a/drivers/nvme/host/auth.c
> +++ b/drivers/nvme/host/auth.c
> @@ -842,6 +842,10 @@ int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int 
> qid)
>                  dev_warn(ctrl->device, "qid %d: no key\n", qid);
>                  return -ENOKEY;
>          }
> +       if (ctrl->opts->dhchap_ctrl_secret && !ctrl->ctrl_key) {
> +               dev_warn(ctrl->device, "qid %d: invalid ctrl key\n", qid);
> +               return -ENOKEY;
> +       }
> 
>          mutex_lock(&ctrl->dhchap_auth_mutex);
>          /* Check if the context is already queued */
> 
> 
> Is this what you had in mind?

So you are saying that nvme-cli does pass the ctrl key in the above
case? And even so when I read /sys/class/nvme/nvme1/dhchap_ctrl_secret
I see "none"? I'll clarify, in my test the ctrl dhchap_key is _valid_
and still it behaves like the ctrl dhchap key is ignored...

Re: [PATCHv16 00/11] nvme: In-band authentication support

[Sagi Grimberg]

On 6/22/22 11:33, Sagi Grimberg wrote:
> 
>>>> Hi all,
>>>>
>>>> recent updates to the NVMe spec have added definitions for in-band
>>>> authentication, and seeing that it provides some real benefit
>>>> especially for NVMe-TCP here's an attempt to implement it.
>>>>
>>>> Thanks to Nicolai Stange the crypto DH framework has been upgraded
>>>> to provide us with a FFDHE implementation; I've updated the patchset
>>>> to use the ephemeral key generation provided there.
>>>>
>>>> Note that this is just for in-band authentication. Secure
>>>> concatenation (ie starting TLS with the negotiated parameters)
>>>> requires a TLS handshake, which the in-kernel TLS implementation
>>>> does not provide. This is being worked on with a different patchset
>>>> which is still WIP.
>>>>
>>>> The nvme-cli support has already been merged; please use the latest
>>>> nvme-cli git repository to build the most recent version.
>>>>
>>>> A copy of this patchset can be found at
>>>> git://git.kernel.org/pub/scm/linux/kernel/git/hare/scsi-devel
>>>> branch auth.v15
>>>>
>>>> The patchset is being cut against nvme-5.20.
>>>>
>>>> As usual, comments and reviews are welcome.
>>>
>>> Hannes, did you see my panic report on a malformed dhchap_ctrl_key?
>>>
>>> Also, why does the dhchap_ctrl_key not passed when connecting
>>> via discovery?
>>>
>>> I have in the target:
>>> -- 
>>> # grep -r '' /sys/kernel/config/nvmet/hosts/
>>> /sys/kernel/config/nvmet/hosts/nqn.2014-08.org.nvmexpress:uuid:302ae323-4acd-465d-ace4-3d4102e9d11f/dhchap_dhgroup:null 
>>>
>>> /sys/kernel/config/nvmet/hosts/nqn.2014-08.org.nvmexpress:uuid:302ae323-4acd-465d-ace4-3d4102e9d11f/dhchap_hash:hmac(sha256) 
>>>
>>> /sys/kernel/config/nvmet/hosts/nqn.2014-08.org.nvmexpress:uuid:302ae323-4acd-465d-ace4-3d4102e9d11f/dhchap_ctrl_key:DHHC-1:00:Jc/My1o0qtLCWRp+sHhAVN6mFaS7YQOMYhk9zSmlatobqB8C: 
>>>
>>> /sys/kernel/config/nvmet/hosts/nqn.2014-08.org.nvmexpress:uuid:302ae323-4acd-465d-ace4-3d4102e9d11f/dhchap_key:DHHC-1:00:QpxVGpctx5J+4SeW2MClUI8rfZO3WdP1llImvsPsx7e3TK+I: 
>>>
>>> -- 
>>>
>>> Then on the host I have:
>>> -- 
>>> # cat /etc/nvme/config.json
>>> [
>>>    {
>>>      "hostnqn": 
>>> "nqn.2014-08.org.nvmexpress:uuid:302ae323-4acd-465d-ace4-3d4102e9d11f",
>>>      "hostid": "14f15c4e-f6cb-434b-90cd-7c1f84f0c194",
>>>      "dhchap_key": 
>>> "DHHC-1:00:QpxVGpctx5J+4SeW2MClUI8rfZO3WdP1llImvsPsx7e3TK+I:",
>>>      "subsystems": [
>>>        {
>>>          "nqn": "testnqn1",
>>>          "ports": [
>>>            {
>>>              "transport": "tcp",
>>>              "traddr": "192.168.123.1",
>>>              "trsvcid": "8009",
>>>              "dhchap_key": 
>>> "DHHC-1:00:Jc/My1o0qtLCWRp+sHhAVN6mFaS7YQOMYhk9zSmlatobqB8C:"
>>>            }
>>>          ]
>>>        }
>>>      ]
>>>    }
>>> ]
>>> -- 
>>>
>>> And when I do connect-all (i.e. connect via the discovery log page:
>>> -- 
>>> # grep -r '' /sys/class/nvme/nvme1/dhchap*
>>> /sys/class/nvme/nvme1/dhchap_ctrl_secret:none
>>> /sys/class/nvme/nvme1/dhchap_secret:DHHC-1:00:QpxVGpctx5J+4SeW2MClUI8rfZO3WdP1llImvsPsx7e3TK+I: 
>>>
>>> -- 
>>>
>>> This means that I can corrupt the dhchap_ctrl_key entry in the config
>>> and no one will care (because it is not authenticating the ctrl if
>>> dhchap_ctrl_key is not passed)
>>>
>> Unfortunate design on both ends.
>> It's the host who requests controller authentication.
> 
> Yes.
> 
>> So if the host doesn't request it nothing will happen.
> 
> Correct.
> 
>> But that also means that controller authentication is optional, and so 
>> I didn't feel comfortable to refuse connections for an optional feature.
> 
> If the ctrl authentication is unsuccessful then of course we need to
> fail it. If the host requested the ctrl to authenticate, it needs to be
> successful.
> 
> But I don't think that it explains what I'm seeing.
> 
>>
>> But we can make that a real error, and refuse the 'connect' call if 
>> the controller key is invalid. Might be a better choice after all.
> 
> In my mind it is mandatory, its not really an interpretation thing...
> But I think that if I pass a wrong dhchap_ctrl_key to connect I get
> a failure.

Yes, when I send a valid ctrl dhchap_key, but just wrong, I do get
a failure:
--
# nvme connect -a 192.168.123.1 -t tcp  -s 8009 -n testnqn1 -S 
"DHHC-1:00:QpxVGpctx5J+4SeW2MClUI8rfZO3WdP1llImvsPsx7e3TK+I:" -C 
DHHC-1:00:Ic6ygXy+/+IZQ3jato0k1GbcQWj/pnjo6kmjhbFofQnRHSyg:
no controller found: failed to write to nvme-fabrics device
# dmesg
[63655.518509] nvme nvme0: qid 0: authenticated with hash hmac(sha256) 
dhgroup null
[63655.518514] nvme nvme0: qid 0: controller authentication failed
[63655.519068] nvme nvme0: qid 0: authentication failed
[63655.519103] nvme nvme0: failed to connect queue: 0 ret=401
--

Please have a look into the case I originally mentioned.

Re: [PATCHv16 00/11] nvme: In-band authentication support

[Hannes Reinecke]

On 6/21/22 23:22, Sagi Grimberg wrote:
> 
>> Hi all,
>>
>> recent updates to the NVMe spec have added definitions for in-band
>> authentication, and seeing that it provides some real benefit
>> especially for NVMe-TCP here's an attempt to implement it.
>>
>> Thanks to Nicolai Stange the crypto DH framework has been upgraded
>> to provide us with a FFDHE implementation; I've updated the patchset
>> to use the ephemeral key generation provided there.
>>
>> Note that this is just for in-band authentication. Secure
>> concatenation (ie starting TLS with the negotiated parameters)
>> requires a TLS handshake, which the in-kernel TLS implementation
>> does not provide. This is being worked on with a different patchset
>> which is still WIP.
>>
>> The nvme-cli support has already been merged; please use the latest
>> nvme-cli git repository to build the most recent version.
>>
>> A copy of this patchset can be found at
>> git://git.kernel.org/pub/scm/linux/kernel/git/hare/scsi-devel
>> branch auth.v15
>>
>> The patchset is being cut against nvme-5.20.
>>
>> As usual, comments and reviews are welcome.
> 
> Hannes, did you see my panic report on a malformed dhchap_ctrl_key?
> 
> Also, why does the dhchap_ctrl_key not passed when connecting
> via discovery?
> 
> I have in the target:
> -- 
> # grep -r '' /sys/kernel/config/nvmet/hosts/
> /sys/kernel/config/nvmet/hosts/nqn.2014-08.org.nvmexpress:uuid:302ae323-4acd-465d-ace4-3d4102e9d11f/dhchap_dhgroup:null 
> 
> /sys/kernel/config/nvmet/hosts/nqn.2014-08.org.nvmexpress:uuid:302ae323-4acd-465d-ace4-3d4102e9d11f/dhchap_hash:hmac(sha256) 
> 
> /sys/kernel/config/nvmet/hosts/nqn.2014-08.org.nvmexpress:uuid:302ae323-4acd-465d-ace4-3d4102e9d11f/dhchap_ctrl_key:DHHC-1:00:Jc/My1o0qtLCWRp+sHhAVN6mFaS7YQOMYhk9zSmlatobqB8C: 
> 
> /sys/kernel/config/nvmet/hosts/nqn.2014-08.org.nvmexpress:uuid:302ae323-4acd-465d-ace4-3d4102e9d11f/dhchap_key:DHHC-1:00:QpxVGpctx5J+4SeW2MClUI8rfZO3WdP1llImvsPsx7e3TK+I: 
> 
> -- 
> 
> Then on the host I have:
> -- 
> # cat /etc/nvme/config.json
> [
>    {
>      "hostnqn": 
> "nqn.2014-08.org.nvmexpress:uuid:302ae323-4acd-465d-ace4-3d4102e9d11f",
>      "hostid": "14f15c4e-f6cb-434b-90cd-7c1f84f0c194",
>      "dhchap_key": 
> "DHHC-1:00:QpxVGpctx5J+4SeW2MClUI8rfZO3WdP1llImvsPsx7e3TK+I:",
>      "subsystems": [
>        {
>          "nqn": "testnqn1",
>          "ports": [
>            {
>              "transport": "tcp",
>              "traddr": "192.168.123.1",
>              "trsvcid": "8009",
>              "dhchap_key": 
> "DHHC-1:00:Jc/My1o0qtLCWRp+sHhAVN6mFaS7YQOMYhk9zSmlatobqB8C:"
>            }
>          ]
>        }
>      ]
>    }
> ]
> -- 
> 
> And when I do connect-all (i.e. connect via the discovery log page:
> -- 
> # grep -r '' /sys/class/nvme/nvme1/dhchap*
> /sys/class/nvme/nvme1/dhchap_ctrl_secret:none
> /sys/class/nvme/nvme1/dhchap_secret:DHHC-1:00:QpxVGpctx5J+4SeW2MClUI8rfZO3WdP1llImvsPsx7e3TK+I: 
> 
> -- 
> 
> This means that I can corrupt the dhchap_ctrl_key entry in the config
> and no one will care (because it is not authenticating the ctrl if
> dhchap_ctrl_key is not passed)
> 
> I think this is something wrong with nvme-cli/libnvme though...

Using the latest patches (and latest nvme-cli) it works if I specify:

nvme connect-all -t tcp -a 127.0.0.1 -s 4420 \
   --hostnqn 
nqn.2014-08.org.nvmexpress:uuid:302ae323-4acd-465d-ace4-3d4102e9d11f \
   --hostid 14f15c4e-f6cb-434b-90cd-7c1f84f0c194 --config config.json

(You have to specify --config to instruct nvme-cli to use the 
configuration file)

We have had some fixes for the config handling recently, so maybe it's 
enough to update nvme-cli.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		           Kernel Storage Architect
hare@suse.de			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), GF: Felix Imendörffer

Re: [PATCHv16 00/11] nvme: In-band authentication support

[Sagi Grimberg]

On 6/22/22 13:03, Hannes Reinecke wrote:
> On 6/21/22 23:22, Sagi Grimberg wrote:
>>
>>> Hi all,
>>>
>>> recent updates to the NVMe spec have added definitions for in-band
>>> authentication, and seeing that it provides some real benefit
>>> especially for NVMe-TCP here's an attempt to implement it.
>>>
>>> Thanks to Nicolai Stange the crypto DH framework has been upgraded
>>> to provide us with a FFDHE implementation; I've updated the patchset
>>> to use the ephemeral key generation provided there.
>>>
>>> Note that this is just for in-band authentication. Secure
>>> concatenation (ie starting TLS with the negotiated parameters)
>>> requires a TLS handshake, which the in-kernel TLS implementation
>>> does not provide. This is being worked on with a different patchset
>>> which is still WIP.
>>>
>>> The nvme-cli support has already been merged; please use the latest
>>> nvme-cli git repository to build the most recent version.
>>>
>>> A copy of this patchset can be found at
>>> git://git.kernel.org/pub/scm/linux/kernel/git/hare/scsi-devel
>>> branch auth.v15
>>>
>>> The patchset is being cut against nvme-5.20.
>>>
>>> As usual, comments and reviews are welcome.
>>
>> Hannes, did you see my panic report on a malformed dhchap_ctrl_key?
>>
>> Also, why does the dhchap_ctrl_key not passed when connecting
>> via discovery?
>>
>> I have in the target:
>> -- 
>> # grep -r '' /sys/kernel/config/nvmet/hosts/
>> /sys/kernel/config/nvmet/hosts/nqn.2014-08.org.nvmexpress:uuid:302ae323-4acd-465d-ace4-3d4102e9d11f/dhchap_dhgroup:null 
>>
>> /sys/kernel/config/nvmet/hosts/nqn.2014-08.org.nvmexpress:uuid:302ae323-4acd-465d-ace4-3d4102e9d11f/dhchap_hash:hmac(sha256) 
>>
>> /sys/kernel/config/nvmet/hosts/nqn.2014-08.org.nvmexpress:uuid:302ae323-4acd-465d-ace4-3d4102e9d11f/dhchap_ctrl_key:DHHC-1:00:Jc/My1o0qtLCWRp+sHhAVN6mFaS7YQOMYhk9zSmlatobqB8C: 
>>
>> /sys/kernel/config/nvmet/hosts/nqn.2014-08.org.nvmexpress:uuid:302ae323-4acd-465d-ace4-3d4102e9d11f/dhchap_key:DHHC-1:00:QpxVGpctx5J+4SeW2MClUI8rfZO3WdP1llImvsPsx7e3TK+I: 
>>
>> -- 
>>
>> Then on the host I have:
>> -- 
>> # cat /etc/nvme/config.json
>> [
>>    {
>>      "hostnqn": 
>> "nqn.2014-08.org.nvmexpress:uuid:302ae323-4acd-465d-ace4-3d4102e9d11f",
>>      "hostid": "14f15c4e-f6cb-434b-90cd-7c1f84f0c194",
>>      "dhchap_key": 
>> "DHHC-1:00:QpxVGpctx5J+4SeW2MClUI8rfZO3WdP1llImvsPsx7e3TK+I:",
>>      "subsystems": [
>>        {
>>          "nqn": "testnqn1",
>>          "ports": [
>>            {
>>              "transport": "tcp",
>>              "traddr": "192.168.123.1",
>>              "trsvcid": "8009",
>>              "dhchap_key": 
>> "DHHC-1:00:Jc/My1o0qtLCWRp+sHhAVN6mFaS7YQOMYhk9zSmlatobqB8C:"
>>            }
>>          ]
>>        }
>>      ]
>>    }
>> ]
>> -- 
>>
>> And when I do connect-all (i.e. connect via the discovery log page:
>> -- 
>> # grep -r '' /sys/class/nvme/nvme1/dhchap*
>> /sys/class/nvme/nvme1/dhchap_ctrl_secret:none
>> /sys/class/nvme/nvme1/dhchap_secret:DHHC-1:00:QpxVGpctx5J+4SeW2MClUI8rfZO3WdP1llImvsPsx7e3TK+I: 
>>
>> -- 
>>
>> This means that I can corrupt the dhchap_ctrl_key entry in the config
>> and no one will care (because it is not authenticating the ctrl if
>> dhchap_ctrl_key is not passed)
>>
>> I think this is something wrong with nvme-cli/libnvme though...
> 
> Using the latest patches (and latest nvme-cli) it works if I specify:
> 
> nvme connect-all -t tcp -a 127.0.0.1 -s 4420 \
>    --hostnqn 
> nqn.2014-08.org.nvmexpress:uuid:302ae323-4acd-465d-ace4-3d4102e9d11f \
>    --hostid 14f15c4e-f6cb-434b-90cd-7c1f84f0c194 --config config.json

No it doesn't, see my patch in the other thread.

> (You have to specify --config to instruct nvme-cli to use the 
> configuration file)

If the config is in the default path you don't need to pass it.

> We have had some fixes for the config handling recently, so maybe it's 
> enough to update nvme-cli.

It isn't

Re: [PATCH 06/11] nvme: Implement In-Band authentication

[Sagi Grimberg]

> @@ -552,6 +586,8 @@ static const match_table_t opt_tokens = {
>   	{ NVMF_OPT_TOS,			"tos=%d"		},
>   	{ NVMF_OPT_FAIL_FAST_TMO,	"fast_io_fail_tmo=%d"	},
>   	{ NVMF_OPT_DISCOVERY,		"discovery"		},
> +	{ NVMF_OPT_DHCHAP_SECRET,	"dhchap_secret=%s"	},
> +	{ NVMF_OPT_DHCHAP_CTRL_SECRET,	"dhchap_ctrl_secret=%s"	},
>   	{ NVMF_OPT_ERR,			NULL			}
>   };
>   
> @@ -833,6 +869,34 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
>   		case NVMF_OPT_DISCOVERY:
>   			opts->discovery_nqn = true;
>   			break;
> +		case NVMF_OPT_DHCHAP_SECRET:
> +			p = match_strdup(args);
> +			if (!p) {
> +				ret = -ENOMEM;
> +				goto out;
> +			}
> +			if (strlen(p) < 11 || strncmp(p, "DHHC-1:", 7)) {
> +				pr_err("Invalid DH-CHAP secret %s\n", p);
> +				ret = -EINVAL;
> +				goto out;
> +			}
> +			kfree(opts->dhchap_secret);
> +			opts->dhchap_secret = p;
> +			break;
> +		case NVMF_OPT_DHCHAP_CTRL_SECRET:
> +			p = match_strdup(args);
> +			if (!p) {
> +				ret = -ENOMEM;
> +				goto out;
> +			}
> +			if (strlen(p) < 11 || strncmp(p, "DHHC-1:", 7)) {
> +				pr_err("Invalid DH-CHAP secret %s\n", p);
> +				ret = -EINVAL;
> +				goto out;
> +			}
> +			kfree(opts->dhchap_ctrl_secret);
> +			opts->dhchap_ctrl_secret = p;
> +			break;
>   		default:
>   			pr_warn("unknown parameter or missing value '%s' in ctrl creation request\n",
>   				p);
> @@ -951,6 +1015,7 @@ void nvmf_free_options(struct nvmf_ctrl_options *opts)
>   	kfree(opts->subsysnqn);
>   	kfree(opts->host_traddr);
>   	kfree(opts->host_iface);
> +	kfree(opts->dhchap_secret);

I think you need to free ctrl_dhchap_secret as well.

I see kmemleak complaits:
--
unreferenced object 0xffff9797b529f140 (size 64):
   comm "nvme", pid 7070, jiffies 4362827766 (age 27851.164s)
   hex dump (first 32 bytes):
     44 48 48 43 2d 31 3a 30 30 3a 6a 63 2f 4d 79 31  DHHC-1:00:jc/My1
     6f 30 71 74 4c 43 57 52 70 2b 73 48 68 41 56 4e  o0qtLCWRp+sHhAVN
   backtrace:
     [<00000000a8aa18e4>] kmemdup_nul+0x22/0x50
     [<000000009798b50f>] nvmf_parse_options+0x208/0x790 [nvme_fabrics]
     [<0000000049fd6c0a>] nvmf_create_ctrl+0x3e/0x230 [nvme_fabrics]
     [<000000000a5ab7b3>] nvmf_dev_write+0x7d/0xe0 [nvme_fabrics]
     [<000000006c0feca3>] vfs_write+0xb5/0x290
     [<00000000868859e4>] ksys_write+0x5f/0xe0
     [<00000000d670ec58>] do_syscall_64+0x3b/0x90
     [<00000000a2feb9a4>] entry_SYSCALL_64_after_hwframe+0x46/0xb0
--

[PATCH 06/11] nvme: Implement In-Band authentication

[Hannes Reinecke]

Implement NVMe-oF In-Band authentication according to NVMe TPAR 8006.
This patch adds two new fabric options 'dhchap_secret' to specify the
pre-shared key (in ASCII respresentation according to NVMe 2.0 section
8.13.5.8 'Secret representation') and 'dhchap_ctrl_secret' to specify
the pre-shared controller key for bi-directional authentication of both
the host and the controller.
Re-authentication can be triggered by writing the PSK into the new
controller sysfs attribute 'dhchap_secret' or 'dhchap_ctrl_secret'.

Signed-off-by: Hannes Reinecke <hare@suse.de>
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
---
 drivers/nvme/Kconfig         |   1 +
 drivers/nvme/Makefile        |   1 +
 drivers/nvme/common/Kconfig  |   4 +
 drivers/nvme/common/Makefile |   7 +
 drivers/nvme/common/auth.c   | 329 ++++++++++++++
 drivers/nvme/host/Kconfig    |  13 +
 drivers/nvme/host/Makefile   |   1 +
 drivers/nvme/host/auth.c     | 828 +++++++++++++++++++++++++++++++++++
 drivers/nvme/host/core.c     | 143 +++++-
 drivers/nvme/host/fabrics.c  |  80 +++-
 drivers/nvme/host/fabrics.h  |   7 +
 drivers/nvme/host/nvme.h     |  30 ++
 drivers/nvme/host/rdma.c     |   1 +
 drivers/nvme/host/tcp.c      |   1 +
 drivers/nvme/host/trace.c    |  32 ++
 include/linux/nvme-auth.h    |  33 ++
 16 files changed, 1504 insertions(+), 7 deletions(-)
 create mode 100644 drivers/nvme/common/Kconfig
 create mode 100644 drivers/nvme/common/Makefile
 create mode 100644 drivers/nvme/common/auth.c
 create mode 100644 drivers/nvme/host/auth.c
 create mode 100644 include/linux/nvme-auth.h

diff --git a/drivers/nvme/Kconfig b/drivers/nvme/Kconfig
index 87ae409a32b9..656e46d938da 100644
--- a/drivers/nvme/Kconfig
+++ b/drivers/nvme/Kconfig
@@ -1,6 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0-only
 menu "NVME Support"
 
+source "drivers/nvme/common/Kconfig"
 source "drivers/nvme/host/Kconfig"
 source "drivers/nvme/target/Kconfig"
 
diff --git a/drivers/nvme/Makefile b/drivers/nvme/Makefile
index fb42c44609a8..eedca8c72098 100644
--- a/drivers/nvme/Makefile
+++ b/drivers/nvme/Makefile
@@ -1,4 +1,5 @@
 # SPDX-License-Identifier: GPL-2.0-only
 
+obj-$(CONFIG_NVME_COMMON)		+= common/
 obj-y		+= host/
 obj-y		+= target/
diff --git a/drivers/nvme/common/Kconfig b/drivers/nvme/common/Kconfig
new file mode 100644
index 000000000000..4514f44362dd
--- /dev/null
+++ b/drivers/nvme/common/Kconfig
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: GPL-2.0-only
+
+config NVME_COMMON
+       tristate
diff --git a/drivers/nvme/common/Makefile b/drivers/nvme/common/Makefile
new file mode 100644
index 000000000000..720c625b8a52
--- /dev/null
+++ b/drivers/nvme/common/Makefile
@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: GPL-2.0
+
+ccflags-y			+= -I$(src)
+
+obj-$(CONFIG_NVME_COMMON)	+= nvme-common.o
+
+nvme-common-y			+= auth.o
diff --git a/drivers/nvme/common/auth.c b/drivers/nvme/common/auth.c
new file mode 100644
index 000000000000..01adb29947d4
--- /dev/null
+++ b/drivers/nvme/common/auth.c
@@ -0,0 +1,329 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2020 Hannes Reinecke, SUSE Linux
+ */
+
+#include <linux/module.h>
+#include <linux/crc32.h>
+#include <linux/base64.h>
+#include <linux/prandom.h>
+#include <linux/scatterlist.h>
+#include <asm/unaligned.h>
+#include <crypto/hash.h>
+#include <crypto/dh.h>
+#include <linux/nvme.h>
+#include <linux/nvme-auth.h>
+
+static u32 nvme_dhchap_seqnum;
+static DEFINE_MUTEX(nvme_dhchap_mutex);
+
+u32 nvme_auth_get_seqnum(void)
+{
+	u32 seqnum;
+
+	mutex_lock(&nvme_dhchap_mutex);
+	if (!nvme_dhchap_seqnum)
+		nvme_dhchap_seqnum = prandom_u32();
+	else {
+		nvme_dhchap_seqnum++;
+		if (!nvme_dhchap_seqnum)
+			nvme_dhchap_seqnum++;
+	}
+	seqnum = nvme_dhchap_seqnum;
+	mutex_unlock(&nvme_dhchap_mutex);
+	return seqnum;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_get_seqnum);
+
+static struct nvme_auth_dhgroup_map {
+	const char name[16];
+	const char kpp[16];
+} dhgroup_map[] = {
+	[NVME_AUTH_DHGROUP_NULL] = {
+		.name = "null", .kpp = "null" },
+	[NVME_AUTH_DHGROUP_2048] = {
+		.name = "ffdhe2048", .kpp = "ffdhe2048(dh)" },
+	[NVME_AUTH_DHGROUP_3072] = {
+		.name = "ffdhe3072", .kpp = "ffdhe3072(dh)" },
+	[NVME_AUTH_DHGROUP_4096] = {
+		.name = "ffdhe4096", .kpp = "ffdhe4096(dh)" },
+	[NVME_AUTH_DHGROUP_6144] = {
+		.name = "ffdhe6144", .kpp = "ffdhe6144(dh)" },
+	[NVME_AUTH_DHGROUP_8192] = {
+		.name = "ffdhe8192", .kpp = "ffdhe8192(dh)" },
+};
+
+const char *nvme_auth_dhgroup_name(u8 dhgroup_id)
+{
+	if ((dhgroup_id > ARRAY_SIZE(dhgroup_map)) ||
+	    !dhgroup_map[dhgroup_id].name ||
+	    !strlen(dhgroup_map[dhgroup_id].name))
+		return NULL;
+	return dhgroup_map[dhgroup_id].name;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_dhgroup_name);
+
+const char *nvme_auth_dhgroup_kpp(u8 dhgroup_id)
+{
+	if ((dhgroup_id > ARRAY_SIZE(dhgroup_map)) ||
+	    !dhgroup_map[dhgroup_id].kpp ||
+	    !strlen(dhgroup_map[dhgroup_id].kpp))
+		return NULL;
+	return dhgroup_map[dhgroup_id].kpp;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_dhgroup_kpp);
+
+u8 nvme_auth_dhgroup_id(const char *dhgroup_name)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(dhgroup_map); i++) {
+		if (!dhgroup_map[i].name ||
+		    !strlen(dhgroup_map[i].name))
+			continue;
+		if (!strncmp(dhgroup_map[i].name, dhgroup_name,
+			     strlen(dhgroup_map[i].name)))
+			return i;
+	}
+	return NVME_AUTH_DHGROUP_INVALID;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_dhgroup_id);
+
+static struct nvme_dhchap_hash_map {
+	int len;
+	const char hmac[15];
+	const char digest[8];
+} hash_map[] = {
+	[NVME_AUTH_HASH_SHA256] = {
+		.len = 32,
+		.hmac = "hmac(sha256)",
+		.digest = "sha256",
+	},
+	[NVME_AUTH_HASH_SHA384] = {
+		.len = 48,
+		.hmac = "hmac(sha384)",
+		.digest = "sha384",
+	},
+	[NVME_AUTH_HASH_SHA512] = {
+		.len = 64,
+		.hmac = "hmac(sha512)",
+		.digest = "sha512",
+	},
+};
+
+const char *nvme_auth_hmac_name(u8 hmac_id)
+{
+	if ((hmac_id > ARRAY_SIZE(hash_map)) ||
+	    !hash_map[hmac_id].hmac ||
+	    !strlen(hash_map[hmac_id].hmac))
+		return NULL;
+	return hash_map[hmac_id].hmac;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_hmac_name);
+
+const char *nvme_auth_digest_name(u8 hmac_id)
+{
+	if ((hmac_id > ARRAY_SIZE(hash_map)) ||
+	    !hash_map[hmac_id].digest ||
+	    !strlen(hash_map[hmac_id].digest))
+		return NULL;
+	return hash_map[hmac_id].digest;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_digest_name);
+
+u8 nvme_auth_hmac_id(const char *hmac_name)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
+		if (!hash_map[i].hmac || !strlen(hash_map[i].hmac))
+			continue;
+		if (!strncmp(hash_map[i].hmac, hmac_name,
+			     strlen(hash_map[i].hmac)))
+			return i;
+	}
+	return NVME_AUTH_HASH_INVALID;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_hmac_id);
+
+size_t nvme_auth_hmac_hash_len(u8 hmac_id)
+{
+	if ((hmac_id > ARRAY_SIZE(hash_map)) ||
+	    !hash_map[hmac_id].hmac ||
+	    !strlen(hash_map[hmac_id].hmac))
+		return 0;
+	return hash_map[hmac_id].len;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_hmac_hash_len);
+
+struct nvme_dhchap_key *nvme_auth_extract_key(unsigned char *secret,
+					      u8 key_hash)
+{
+	struct nvme_dhchap_key *key;
+	unsigned char *p;
+	u32 crc;
+	int ret, key_len;
+	size_t allocated_len = strlen(secret);
+
+	/* Secret might be affixed with a ':' */
+	p = strrchr(secret, ':');
+	if (p)
+		allocated_len = p - secret;
+	key = kzalloc(sizeof(*key), GFP_KERNEL);
+	if (!key)
+		return ERR_PTR(-ENOMEM);
+	key->key = kzalloc(allocated_len, GFP_KERNEL);
+	if (!key->key) {
+		ret = -ENOMEM;
+		goto out_free_key;
+	}
+
+	key_len = base64_decode(secret, allocated_len, key->key);
+	if (key_len < 0) {
+		pr_debug("base64 key decoding error %d\n",
+			 key_len);
+		ret = key_len;
+		goto out_free_secret;
+	}
+
+	if (key_len != 36 && key_len != 52 &&
+	    key_len != 68) {
+		pr_err("Invalid key len %d\n", key_len);
+		ret = -EINVAL;
+		goto out_free_secret;
+	}
+
+	if (key_hash > 0 &&
+	    (key_len - 4) != nvme_auth_hmac_hash_len(key_hash)) {
+		pr_err("Mismatched key len %d for %s\n", key_len,
+		       nvme_auth_hmac_name(key_hash));
+		ret = -EINVAL;
+		goto out_free_secret;
+	}
+
+	/* The last four bytes is the CRC in little-endian format */
+	key_len -= 4;
+	/*
+	 * The linux implementation doesn't do pre- and post-increments,
+	 * so we have to do it manually.
+	 */
+	crc = ~crc32(~0, key->key, key_len);
+
+	if (get_unaligned_le32(key->key + key_len) != crc) {
+		pr_err("key crc mismatch (key %08x, crc %08x)\n",
+		       get_unaligned_le32(key->key + key_len), crc);
+		ret = -EKEYREJECTED;
+		goto out_free_secret;
+	}
+	key->len = key_len;
+	key->hash = key_hash;
+	return key;
+out_free_secret:
+	kfree_sensitive(key->key);
+out_free_key:
+	kfree(key);
+	return ERR_PTR(ret);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_extract_key);
+
+void nvme_auth_free_key(struct nvme_dhchap_key *key)
+{
+	if (!key)
+		return;
+	kfree_sensitive(key->key);
+	kfree(key);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_free_key);
+
+u8 *nvme_auth_transform_key(struct nvme_dhchap_key *key, char *nqn)
+{
+	const char *hmac_name;
+	struct crypto_shash *key_tfm;
+	struct shash_desc *shash;
+	u8 *transformed_key;
+	int ret;
+
+	if (!key || !key->key) {
+		pr_warn("No key specified\n");
+		return ERR_PTR(-ENOKEY);
+	}
+	if (key->hash == 0) {
+		transformed_key = kmemdup(key->key, key->len, GFP_KERNEL);
+		return transformed_key ? transformed_key : ERR_PTR(-ENOMEM);
+	}
+	hmac_name = nvme_auth_hmac_name(key->hash);
+	if (!hmac_name) {
+		pr_warn("Invalid key hash id %d\n", key->hash);
+		return ERR_PTR(-EINVAL);
+	}
+
+	key_tfm = crypto_alloc_shash(hmac_name, 0, 0);
+	if (IS_ERR(key_tfm))
+		return (u8 *)key_tfm;
+
+	shash = kmalloc(sizeof(struct shash_desc) +
+			crypto_shash_descsize(key_tfm),
+			GFP_KERNEL);
+	if (!shash) {
+		ret = -ENOMEM;
+		goto out_free_key;
+	}
+
+	transformed_key = kzalloc(crypto_shash_digestsize(key_tfm), GFP_KERNEL);
+	if (!transformed_key) {
+		ret = -ENOMEM;
+		goto out_free_shash;
+	}
+
+	shash->tfm = key_tfm;
+	ret = crypto_shash_setkey(key_tfm, key->key, key->len);
+	if (ret < 0)
+		goto out_free_shash;
+	ret = crypto_shash_init(shash);
+	if (ret < 0)
+		goto out_free_shash;
+	ret = crypto_shash_update(shash, nqn, strlen(nqn));
+	if (ret < 0)
+		goto out_free_shash;
+	ret = crypto_shash_update(shash, "NVMe-over-Fabrics", 17);
+	if (ret < 0)
+		goto out_free_shash;
+	ret = crypto_shash_final(shash, transformed_key);
+out_free_shash:
+	kfree(shash);
+out_free_key:
+	crypto_free_shash(key_tfm);
+	if (ret < 0) {
+		kfree_sensitive(transformed_key);
+		return ERR_PTR(ret);
+	}
+	return transformed_key;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_transform_key);
+
+int nvme_auth_generate_key(u8 *secret, struct nvme_dhchap_key **ret_key)
+{
+	struct nvme_dhchap_key *key;
+	u8 key_hash;
+
+	if (!secret) {
+		*ret_key = NULL;
+		return 0;
+	}
+
+	if (sscanf(secret, "DHHC-1:%hhd:%*s:", &key_hash) != 1)
+		return -EINVAL;
+
+	/* Pass in the secret without the 'DHHC-1:XX:' prefix */
+	key = nvme_auth_extract_key(secret + 10, key_hash);
+	if (IS_ERR(key)) {
+		*ret_key = NULL;
+		return PTR_ERR(key);
+	}
+
+	*ret_key = key;
+	return 0;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_generate_key);
+
+MODULE_LICENSE("GPL v2");
diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig
index 877d2ec4ea9f..6c503f42f3c6 100644
--- a/drivers/nvme/host/Kconfig
+++ b/drivers/nvme/host/Kconfig
@@ -92,6 +92,19 @@ config NVME_TCP
 
 	  If unsure, say N.
 
+config NVME_AUTH
+	bool "NVM Express over Fabrics In-Band Authentication"
+	depends on NVME_CORE
+	select NVME_COMMON
+	select CRYPTO
+	select CRYPTO_HMAC
+	select CRYPTO_SHA256
+	select CRYPTO_SHA512
+	help
+	  This provides support for NVMe over Fabrics In-Band Authentication.
+
+	  If unsure, say N.
+
 config NVME_APPLE
 	tristate "Apple ANS2 NVM Express host driver"
 	depends on OF && BLOCK
diff --git a/drivers/nvme/host/Makefile b/drivers/nvme/host/Makefile
index a36ae1612059..a3e88f32f560 100644
--- a/drivers/nvme/host/Makefile
+++ b/drivers/nvme/host/Makefile
@@ -16,6 +16,7 @@ nvme-core-$(CONFIG_NVME_MULTIPATH)	+= multipath.o
 nvme-core-$(CONFIG_BLK_DEV_ZONED)	+= zns.o
 nvme-core-$(CONFIG_FAULT_INJECTION_DEBUG_FS)	+= fault_inject.o
 nvme-core-$(CONFIG_NVME_HWMON)		+= hwmon.o
+nvme-core-$(CONFIG_NVME_AUTH)		+= auth.o
 
 nvme-y					+= pci.o
 
diff --git a/drivers/nvme/host/auth.c b/drivers/nvme/host/auth.c
new file mode 100644
index 000000000000..9766bfffecac
--- /dev/null
+++ b/drivers/nvme/host/auth.c
@@ -0,0 +1,828 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2020 Hannes Reinecke, SUSE Linux
+ */
+
+#include <linux/crc32.h>
+#include <linux/base64.h>
+#include <linux/prandom.h>
+#include <asm/unaligned.h>
+#include <crypto/hash.h>
+#include <crypto/dh.h>
+#include "nvme.h"
+#include "fabrics.h"
+#include <linux/nvme-auth.h>
+
+struct nvme_dhchap_queue_context {
+	struct list_head entry;
+	struct work_struct auth_work;
+	struct nvme_ctrl *ctrl;
+	struct crypto_shash *shash_tfm;
+	void *buf;
+	size_t buf_size;
+	int qid;
+	int error;
+	u32 s1;
+	u32 s2;
+	u16 transaction;
+	u8 status;
+	u8 hash_id;
+	size_t hash_len;
+	u8 dhgroup_id;
+	u8 c1[64];
+	u8 c2[64];
+	u8 response[64];
+	u8 *host_response;
+};
+
+#define nvme_auth_flags_from_qid(qid) \
+	(qid == 0) ? 0 : BLK_MQ_REQ_NOWAIT | BLK_MQ_REQ_RESERVED
+#define nvme_auth_queue_from_qid(ctrl, qid) \
+	(qid == 0) ? (ctrl)->fabrics_q : (ctrl)->connect_q
+
+static int nvme_auth_submit(struct nvme_ctrl *ctrl, int qid,
+			    void *data, size_t data_len, bool auth_send)
+{
+	struct nvme_command cmd = {};
+	blk_mq_req_flags_t flags = nvme_auth_flags_from_qid(qid);
+	struct request_queue *q = nvme_auth_queue_from_qid(ctrl, qid);
+	int ret;
+
+	cmd.auth_common.opcode = nvme_fabrics_command;
+	cmd.auth_common.secp = NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER;
+	cmd.auth_common.spsp0 = 0x01;
+	cmd.auth_common.spsp1 = 0x01;
+	if (auth_send) {
+		cmd.auth_send.fctype = nvme_fabrics_type_auth_send;
+		cmd.auth_send.tl = cpu_to_le32(data_len);
+	} else {
+		cmd.auth_receive.fctype = nvme_fabrics_type_auth_receive;
+		cmd.auth_receive.al = cpu_to_le32(data_len);
+	}
+
+	ret = __nvme_submit_sync_cmd(q, &cmd, NULL, data, data_len,
+				     qid == 0 ? NVME_QID_ANY : qid,
+				     0, flags);
+	if (ret > 0)
+		dev_warn(ctrl->device,
+			"qid %d auth_send failed with status %d\n", qid, ret);
+	else if (ret < 0)
+		dev_err(ctrl->device,
+			"qid %d auth_send failed with error %d\n", qid, ret);
+	return ret;
+}
+
+static int nvme_auth_receive_validate(struct nvme_ctrl *ctrl, int qid,
+		struct nvmf_auth_dhchap_failure_data *data,
+		u16 transaction, u8 expected_msg)
+{
+	dev_dbg(ctrl->device, "%s: qid %d auth_type %d auth_id %x\n",
+		__func__, qid, data->auth_type, data->auth_id);
+
+	if (data->auth_type == NVME_AUTH_COMMON_MESSAGES &&
+	    data->auth_id == NVME_AUTH_DHCHAP_MESSAGE_FAILURE1) {
+		return data->rescode_exp;
+	}
+	if (data->auth_type != NVME_AUTH_DHCHAP_MESSAGES ||
+	    data->auth_id != expected_msg) {
+		dev_warn(ctrl->device,
+			 "qid %d invalid message %02x/%02x\n",
+			 qid, data->auth_type, data->auth_id);
+		return NVME_AUTH_DHCHAP_FAILURE_INCORRECT_MESSAGE;
+	}
+	if (le16_to_cpu(data->t_id) != transaction) {
+		dev_warn(ctrl->device,
+			 "qid %d invalid transaction ID %d\n",
+			 qid, le16_to_cpu(data->t_id));
+		return NVME_AUTH_DHCHAP_FAILURE_INCORRECT_MESSAGE;
+	}
+	return 0;
+}
+
+static int nvme_auth_set_dhchap_negotiate_data(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_negotiate_data *data = chap->buf;
+	size_t size = sizeof(*data) + sizeof(union nvmf_auth_protocol);
+
+	if (chap->buf_size < size) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return -EINVAL;
+	}
+	memset((u8 *)chap->buf, 0, size);
+	data->auth_type = NVME_AUTH_COMMON_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE;
+	data->t_id = cpu_to_le16(chap->transaction);
+	data->sc_c = 0; /* No secure channel concatenation */
+	data->napd = 1;
+	data->auth_protocol[0].dhchap.authid = NVME_AUTH_DHCHAP_AUTH_ID;
+	data->auth_protocol[0].dhchap.halen = 3;
+	data->auth_protocol[0].dhchap.dhlen = 6;
+	data->auth_protocol[0].dhchap.idlist[0] = NVME_AUTH_HASH_SHA256;
+	data->auth_protocol[0].dhchap.idlist[1] = NVME_AUTH_HASH_SHA384;
+	data->auth_protocol[0].dhchap.idlist[2] = NVME_AUTH_HASH_SHA512;
+	data->auth_protocol[0].dhchap.idlist[30] = NVME_AUTH_DHGROUP_NULL;
+	data->auth_protocol[0].dhchap.idlist[31] = NVME_AUTH_DHGROUP_2048;
+	data->auth_protocol[0].dhchap.idlist[32] = NVME_AUTH_DHGROUP_3072;
+	data->auth_protocol[0].dhchap.idlist[33] = NVME_AUTH_DHGROUP_4096;
+	data->auth_protocol[0].dhchap.idlist[34] = NVME_AUTH_DHGROUP_6144;
+	data->auth_protocol[0].dhchap.idlist[35] = NVME_AUTH_DHGROUP_8192;
+
+	return size;
+}
+
+static int nvme_auth_process_dhchap_challenge(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_challenge_data *data = chap->buf;
+	u16 dhvlen = le16_to_cpu(data->dhvlen);
+	size_t size = sizeof(*data) + data->hl + dhvlen;
+	const char *hmac_name, *kpp_name;
+
+	if (chap->buf_size < size) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return NVME_SC_INVALID_FIELD;
+	}
+
+	hmac_name = nvme_auth_hmac_name(data->hashid);
+	if (!hmac_name) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid HASH ID %d\n",
+			 chap->qid, data->hashid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
+		return NVME_SC_INVALID_FIELD;
+	}
+
+	if (chap->hash_id == data->hashid && chap->shash_tfm &&
+	    !strcmp(crypto_shash_alg_name(chap->shash_tfm), hmac_name) &&
+	    crypto_shash_digestsize(chap->shash_tfm) == data->hl) {
+		dev_dbg(ctrl->device,
+			"qid %d: reuse existing hash %s\n",
+			chap->qid, hmac_name);
+		goto select_kpp;
+	}
+
+	/* Reset if hash cannot be reused */
+	if (chap->shash_tfm) {
+		crypto_free_shash(chap->shash_tfm);
+		chap->hash_id = 0;
+		chap->hash_len = 0;
+	}
+	chap->shash_tfm = crypto_alloc_shash(hmac_name, 0,
+					     CRYPTO_ALG_ALLOCATES_MEMORY);
+	if (IS_ERR(chap->shash_tfm)) {
+		dev_warn(ctrl->device,
+			 "qid %d: failed to allocate hash %s, error %ld\n",
+			 chap->qid, hmac_name, PTR_ERR(chap->shash_tfm));
+		chap->shash_tfm = NULL;
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_FAILED;
+		return NVME_SC_AUTH_REQUIRED;
+	}
+
+	if (crypto_shash_digestsize(chap->shash_tfm) != data->hl) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid hash length %d\n",
+			 chap->qid, data->hl);
+		crypto_free_shash(chap->shash_tfm);
+		chap->shash_tfm = NULL;
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
+		return NVME_SC_AUTH_REQUIRED;
+	}
+
+	/* Reset host response if the hash had been changed */
+	if (chap->hash_id != data->hashid) {
+		kfree(chap->host_response);
+		chap->host_response = NULL;
+	}
+
+	chap->hash_id = data->hashid;
+	chap->hash_len = data->hl;
+	dev_dbg(ctrl->device, "qid %d: selected hash %s\n",
+		chap->qid, hmac_name);
+
+select_kpp:
+	kpp_name = nvme_auth_dhgroup_kpp(data->dhgid);
+	if (!kpp_name) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid DH group id %d\n",
+			 chap->qid, data->dhgid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
+		return NVME_SC_AUTH_REQUIRED;
+	}
+
+	if (data->dhgid != NVME_AUTH_DHGROUP_NULL) {
+		dev_warn(ctrl->device,
+			 "qid %d: unsupported DH group %s\n",
+			 chap->qid, kpp_name);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
+		return NVME_SC_AUTH_REQUIRED;
+	} else if (dhvlen != 0) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid DH value for NULL DH\n",
+			 chap->qid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return NVME_SC_INVALID_FIELD;
+	}
+	chap->dhgroup_id = data->dhgid;
+
+	chap->s1 = le32_to_cpu(data->seqnum);
+	memcpy(chap->c1, data->cval, chap->hash_len);
+
+	return 0;
+}
+
+static int nvme_auth_set_dhchap_reply_data(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_reply_data *data = chap->buf;
+	size_t size = sizeof(*data);
+
+	size += 2 * chap->hash_len;
+
+	if (chap->buf_size < size) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return -EINVAL;
+	}
+
+	memset(chap->buf, 0, size);
+	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_REPLY;
+	data->t_id = cpu_to_le16(chap->transaction);
+	data->hl = chap->hash_len;
+	data->dhvlen = 0;
+	memcpy(data->rval, chap->response, chap->hash_len);
+	if (ctrl->ctrl_key) {
+		get_random_bytes(chap->c2, chap->hash_len);
+		data->cvalid = 1;
+		chap->s2 = nvme_auth_get_seqnum();
+		memcpy(data->rval + chap->hash_len, chap->c2,
+		       chap->hash_len);
+		dev_dbg(ctrl->device, "%s: qid %d ctrl challenge %*ph\n",
+			__func__, chap->qid, (int)chap->hash_len, chap->c2);
+	} else {
+		memset(chap->c2, 0, chap->hash_len);
+		chap->s2 = 0;
+	}
+	data->seqnum = cpu_to_le32(chap->s2);
+	return size;
+}
+
+static int nvme_auth_process_dhchap_success1(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_success1_data *data = chap->buf;
+	size_t size = sizeof(*data);
+
+	if (ctrl->ctrl_key)
+		size += chap->hash_len;
+
+	if (chap->buf_size < size) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return NVME_SC_INVALID_FIELD;
+	}
+
+	if (data->hl != chap->hash_len) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid hash length %u\n",
+			 chap->qid, data->hl);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
+		return NVME_SC_INVALID_FIELD;
+	}
+
+	/* Just print out information for the admin queue */
+	if (chap->qid == 0)
+		dev_info(ctrl->device,
+			 "qid 0: authenticated with hash %s dhgroup %s\n",
+			 nvme_auth_hmac_name(chap->hash_id),
+			 nvme_auth_dhgroup_name(chap->dhgroup_id));
+
+	if (!data->rvalid)
+		return 0;
+
+	/* Validate controller response */
+	if (memcmp(chap->response, data->rval, data->hl)) {
+		dev_dbg(ctrl->device, "%s: qid %d ctrl response %*ph\n",
+			__func__, chap->qid, (int)chap->hash_len, data->rval);
+		dev_dbg(ctrl->device, "%s: qid %d host response %*ph\n",
+			__func__, chap->qid, (int)chap->hash_len,
+			chap->response);
+		dev_warn(ctrl->device,
+			 "qid %d: controller authentication failed\n",
+			 chap->qid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_FAILED;
+		return NVME_SC_AUTH_REQUIRED;
+	}
+
+	/* Just print out information for the admin queue */
+	if (chap->qid == 0)
+		dev_info(ctrl->device,
+			 "qid 0: controller authenticated\n");
+	return 0;
+}
+
+static int nvme_auth_set_dhchap_success2_data(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_success2_data *data = chap->buf;
+	size_t size = sizeof(*data);
+
+	memset(chap->buf, 0, size);
+	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_SUCCESS2;
+	data->t_id = cpu_to_le16(chap->transaction);
+
+	return size;
+}
+
+static int nvme_auth_set_dhchap_failure2_data(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_failure_data *data = chap->buf;
+	size_t size = sizeof(*data);
+
+	memset(chap->buf, 0, size);
+	data->auth_type = NVME_AUTH_COMMON_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_FAILURE2;
+	data->t_id = cpu_to_le16(chap->transaction);
+	data->rescode = NVME_AUTH_DHCHAP_FAILURE_REASON_FAILED;
+	data->rescode_exp = chap->status;
+
+	return size;
+}
+
+static int nvme_auth_dhchap_setup_host_response(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
+	u8 buf[4], *challenge = chap->c1;
+	int ret;
+
+	dev_dbg(ctrl->device, "%s: qid %d host response seq %u transaction %d\n",
+		__func__, chap->qid, chap->s1, chap->transaction);
+
+	if (!chap->host_response) {
+		chap->host_response = nvme_auth_transform_key(ctrl->host_key,
+						ctrl->opts->host->nqn);
+		if (IS_ERR(chap->host_response)) {
+			ret = PTR_ERR(chap->host_response);
+			chap->host_response = NULL;
+			return ret;
+		}
+	} else {
+		dev_dbg(ctrl->device, "%s: qid %d re-using host response\n",
+			__func__, chap->qid);
+	}
+
+	ret = crypto_shash_setkey(chap->shash_tfm,
+			chap->host_response, ctrl->host_key->len);
+	if (ret) {
+		dev_warn(ctrl->device, "qid %d: failed to set key, error %d\n",
+			 chap->qid, ret);
+		goto out;
+	}
+
+	shash->tfm = chap->shash_tfm;
+	ret = crypto_shash_init(shash);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, challenge, chap->hash_len);
+	if (ret)
+		goto out;
+	put_unaligned_le32(chap->s1, buf);
+	ret = crypto_shash_update(shash, buf, 4);
+	if (ret)
+		goto out;
+	put_unaligned_le16(chap->transaction, buf);
+	ret = crypto_shash_update(shash, buf, 2);
+	if (ret)
+		goto out;
+	memset(buf, 0, sizeof(buf));
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, "HostHost", 8);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
+				  strlen(ctrl->opts->host->nqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
+			    strlen(ctrl->opts->subsysnqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_final(shash, chap->response);
+out:
+	if (challenge != chap->c1)
+		kfree(challenge);
+	return ret;
+}
+
+static int nvme_auth_dhchap_setup_ctrl_response(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
+	u8 *ctrl_response;
+	u8 buf[4], *challenge = chap->c2;
+	int ret;
+
+	ctrl_response = nvme_auth_transform_key(ctrl->ctrl_key,
+				ctrl->opts->subsysnqn);
+	if (IS_ERR(ctrl_response)) {
+		ret = PTR_ERR(ctrl_response);
+		return ret;
+	}
+	ret = crypto_shash_setkey(chap->shash_tfm,
+			ctrl_response, ctrl->ctrl_key->len);
+	if (ret) {
+		dev_warn(ctrl->device, "qid %d: failed to set key, error %d\n",
+			 chap->qid, ret);
+		goto out;
+	}
+
+	dev_dbg(ctrl->device, "%s: qid %d ctrl response seq %u transaction %d\n",
+		__func__, chap->qid, chap->s2, chap->transaction);
+	dev_dbg(ctrl->device, "%s: qid %d challenge %*ph\n",
+		__func__, chap->qid, (int)chap->hash_len, challenge);
+	dev_dbg(ctrl->device, "%s: qid %d subsysnqn %s\n",
+		__func__, chap->qid, ctrl->opts->subsysnqn);
+	dev_dbg(ctrl->device, "%s: qid %d hostnqn %s\n",
+		__func__, chap->qid, ctrl->opts->host->nqn);
+	shash->tfm = chap->shash_tfm;
+	ret = crypto_shash_init(shash);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, challenge, chap->hash_len);
+	if (ret)
+		goto out;
+	put_unaligned_le32(chap->s2, buf);
+	ret = crypto_shash_update(shash, buf, 4);
+	if (ret)
+		goto out;
+	put_unaligned_le16(chap->transaction, buf);
+	ret = crypto_shash_update(shash, buf, 2);
+	if (ret)
+		goto out;
+	memset(buf, 0, 4);
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, "Controller", 10);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
+				  strlen(ctrl->opts->subsysnqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
+				  strlen(ctrl->opts->host->nqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_final(shash, chap->response);
+out:
+	if (challenge != chap->c2)
+		kfree(challenge);
+	kfree(ctrl_response);
+	return ret;
+}
+
+static void __nvme_auth_reset(struct nvme_dhchap_queue_context *chap)
+{
+	chap->status = 0;
+	chap->error = 0;
+	chap->s1 = 0;
+	chap->s2 = 0;
+	chap->transaction = 0;
+	memset(chap->c1, 0, sizeof(chap->c1));
+	memset(chap->c2, 0, sizeof(chap->c2));
+}
+
+static void __nvme_auth_free(struct nvme_dhchap_queue_context *chap)
+{
+	__nvme_auth_reset(chap);
+	if (chap->shash_tfm)
+		crypto_free_shash(chap->shash_tfm);
+	kfree_sensitive(chap->host_response);
+	kfree(chap->buf);
+	kfree(chap);
+}
+
+static void __nvme_auth_work(struct work_struct *work)
+{
+	struct nvme_dhchap_queue_context *chap =
+		container_of(work, struct nvme_dhchap_queue_context, auth_work);
+	struct nvme_ctrl *ctrl = chap->ctrl;
+	size_t tl;
+	int ret = 0;
+
+	chap->transaction = ctrl->transaction++;
+
+	/* DH-HMAC-CHAP Step 1: send negotiate */
+	dev_dbg(ctrl->device, "%s: qid %d send negotiate\n",
+		__func__, chap->qid);
+	ret = nvme_auth_set_dhchap_negotiate_data(ctrl, chap);
+	if (ret < 0) {
+		chap->error = ret;
+		return;
+	}
+	tl = ret;
+	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, tl, true);
+	if (ret) {
+		chap->error = ret;
+		return;
+	}
+
+	/* DH-HMAC-CHAP Step 2: receive challenge */
+	dev_dbg(ctrl->device, "%s: qid %d receive challenge\n",
+		__func__, chap->qid);
+
+	memset(chap->buf, 0, chap->buf_size);
+	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, chap->buf_size, false);
+	if (ret) {
+		dev_warn(ctrl->device,
+			 "qid %d failed to receive challenge, %s %d\n",
+			 chap->qid, ret < 0 ? "error" : "nvme status", ret);
+		chap->error = ret;
+		return;
+	}
+	ret = nvme_auth_receive_validate(ctrl, chap->qid, chap->buf, chap->transaction,
+					 NVME_AUTH_DHCHAP_MESSAGE_CHALLENGE);
+	if (ret) {
+		chap->status = ret;
+		chap->error = NVME_SC_AUTH_REQUIRED;
+		return;
+	}
+
+	ret = nvme_auth_process_dhchap_challenge(ctrl, chap);
+	if (ret) {
+		/* Invalid challenge parameters */
+		chap->error = ret;
+		goto fail2;
+	}
+
+	dev_dbg(ctrl->device, "%s: qid %d host response\n",
+		__func__, chap->qid);
+	ret = nvme_auth_dhchap_setup_host_response(ctrl, chap);
+	if (ret) {
+		chap->error = ret;
+		goto fail2;
+	}
+
+	/* DH-HMAC-CHAP Step 3: send reply */
+	dev_dbg(ctrl->device, "%s: qid %d send reply\n",
+		__func__, chap->qid);
+	ret = nvme_auth_set_dhchap_reply_data(ctrl, chap);
+	if (ret < 0) {
+		chap->error = ret;
+		goto fail2;
+	}
+
+	tl = ret;
+	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, tl, true);
+	if (ret) {
+		chap->error = ret;
+		goto fail2;
+	}
+
+	/* DH-HMAC-CHAP Step 4: receive success1 */
+	dev_dbg(ctrl->device, "%s: qid %d receive success1\n",
+		__func__, chap->qid);
+
+	memset(chap->buf, 0, chap->buf_size);
+	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, chap->buf_size, false);
+	if (ret) {
+		dev_warn(ctrl->device,
+			 "qid %d failed to receive success1, %s %d\n",
+			 chap->qid, ret < 0 ? "error" : "nvme status", ret);
+		chap->error = ret;
+		return;
+	}
+	ret = nvme_auth_receive_validate(ctrl, chap->qid,
+					 chap->buf, chap->transaction,
+					 NVME_AUTH_DHCHAP_MESSAGE_SUCCESS1);
+	if (ret) {
+		chap->status = ret;
+		chap->error = NVME_SC_AUTH_REQUIRED;
+		return;
+	}
+
+	if (ctrl->ctrl_key) {
+		dev_dbg(ctrl->device,
+			"%s: qid %d controller response\n",
+			__func__, chap->qid);
+		ret = nvme_auth_dhchap_setup_ctrl_response(ctrl, chap);
+		if (ret) {
+			chap->error = ret;
+			goto fail2;
+		}
+	}
+
+	ret = nvme_auth_process_dhchap_success1(ctrl, chap);
+	if (ret) {
+		/* Controller authentication failed */
+		chap->error = NVME_SC_AUTH_REQUIRED;
+		goto fail2;
+	}
+
+	if (ctrl->ctrl_key) {
+		/* DH-HMAC-CHAP Step 5: send success2 */
+		dev_dbg(ctrl->device, "%s: qid %d send success2\n",
+			__func__, chap->qid);
+		tl = nvme_auth_set_dhchap_success2_data(ctrl, chap);
+		ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, tl, true);
+		if (ret)
+			chap->error = ret;
+	}
+	if (!ret) {
+		chap->error = 0;
+		return;
+	}
+
+fail2:
+	dev_dbg(ctrl->device, "%s: qid %d send failure2, status %x\n",
+		__func__, chap->qid, chap->status);
+	tl = nvme_auth_set_dhchap_failure2_data(ctrl, chap);
+	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, tl, true);
+	/*
+	 * only update error if send failure2 failed and no other
+	 * error had been set during authentication.
+	 */
+	if (ret && !chap->error)
+		chap->error = ret;
+}
+
+int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid)
+{
+	struct nvme_dhchap_queue_context *chap;
+
+	if (!ctrl->host_key) {
+		dev_warn(ctrl->device, "qid %d: no key\n", qid);
+		return -ENOKEY;
+	}
+
+	if (ctrl->opts->dhchap_ctrl_secret && !ctrl->ctrl_key) {
+		dev_warn(ctrl->device, "qid %d: invalid ctrl key\n", qid);
+		return -ENOKEY;
+	}
+
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	/* Check if the context is already queued */
+	list_for_each_entry(chap, &ctrl->dhchap_auth_list, entry) {
+		WARN_ON(!chap->buf);
+		if (chap->qid == qid) {
+			dev_dbg(ctrl->device, "qid %d: re-using context\n", qid);
+			mutex_unlock(&ctrl->dhchap_auth_mutex);
+			flush_work(&chap->auth_work);
+			__nvme_auth_reset(chap);
+			queue_work(nvme_wq, &chap->auth_work);
+			return 0;
+		}
+	}
+	chap = kzalloc(sizeof(*chap), GFP_KERNEL);
+	if (!chap) {
+		mutex_unlock(&ctrl->dhchap_auth_mutex);
+		return -ENOMEM;
+	}
+	chap->qid = (qid == NVME_QID_ANY) ? 0 : qid;
+	chap->ctrl = ctrl;
+
+	/*
+	 * Allocate a large enough buffer for the entire negotiation:
+	 * 4k should be enough to ffdhe8192.
+	 */
+	chap->buf_size = 4096;
+	chap->buf = kzalloc(chap->buf_size, GFP_KERNEL);
+	if (!chap->buf) {
+		mutex_unlock(&ctrl->dhchap_auth_mutex);
+		kfree(chap);
+		return -ENOMEM;
+	}
+
+	INIT_WORK(&chap->auth_work, __nvme_auth_work);
+	list_add(&chap->entry, &ctrl->dhchap_auth_list);
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+	queue_work(nvme_wq, &chap->auth_work);
+	return 0;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_negotiate);
+
+int nvme_auth_wait(struct nvme_ctrl *ctrl, int qid)
+{
+	struct nvme_dhchap_queue_context *chap;
+	int ret;
+
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	list_for_each_entry(chap, &ctrl->dhchap_auth_list, entry) {
+		if (chap->qid != qid)
+			continue;
+		mutex_unlock(&ctrl->dhchap_auth_mutex);
+		flush_work(&chap->auth_work);
+		ret = chap->error;
+		return ret;
+	}
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+	return -ENXIO;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_wait);
+
+void nvme_auth_reset(struct nvme_ctrl *ctrl)
+{
+	struct nvme_dhchap_queue_context *chap;
+
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	list_for_each_entry(chap, &ctrl->dhchap_auth_list, entry) {
+		mutex_unlock(&ctrl->dhchap_auth_mutex);
+		flush_work(&chap->auth_work);
+		__nvme_auth_reset(chap);
+	}
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_reset);
+
+static void nvme_dhchap_auth_work(struct work_struct *work)
+{
+	struct nvme_ctrl *ctrl =
+		container_of(work, struct nvme_ctrl, dhchap_auth_work);
+	int ret, q;
+
+	/* Authenticate admin queue first */
+	ret = nvme_auth_negotiate(ctrl, 0);
+	if (ret) {
+		dev_warn(ctrl->device,
+			 "qid 0: error %d setting up authentication\n", ret);
+		return;
+	}
+	ret = nvme_auth_wait(ctrl, 0);
+	if (ret) {
+		dev_warn(ctrl->device,
+			 "qid 0: authentication failed\n");
+		return;
+	}
+
+	for (q = 1; q < ctrl->queue_count; q++) {
+		ret = nvme_auth_negotiate(ctrl, q);
+		if (ret) {
+			dev_warn(ctrl->device,
+				 "qid %d: error %d setting up authentication\n",
+				 q, ret);
+			break;
+		}
+	}
+
+	/*
+	 * Failure is a soft-state; credentials remain valid until
+	 * the controller terminates the connection.
+	 */
+}
+
+void nvme_auth_init_ctrl(struct nvme_ctrl *ctrl)
+{
+	INIT_LIST_HEAD(&ctrl->dhchap_auth_list);
+	INIT_WORK(&ctrl->dhchap_auth_work, nvme_dhchap_auth_work);
+	mutex_init(&ctrl->dhchap_auth_mutex);
+	if (!ctrl->opts)
+		return;
+	nvme_auth_generate_key(ctrl->opts->dhchap_secret, &ctrl->host_key);
+	nvme_auth_generate_key(ctrl->opts->dhchap_ctrl_secret, &ctrl->ctrl_key);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_init_ctrl);
+
+void nvme_auth_stop(struct nvme_ctrl *ctrl)
+{
+	struct nvme_dhchap_queue_context *chap = NULL, *tmp;
+
+	cancel_work_sync(&ctrl->dhchap_auth_work);
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	list_for_each_entry_safe(chap, tmp, &ctrl->dhchap_auth_list, entry)
+		cancel_work_sync(&chap->auth_work);
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_stop);
+
+void nvme_auth_free(struct nvme_ctrl *ctrl)
+{
+	struct nvme_dhchap_queue_context *chap = NULL, *tmp;
+
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	list_for_each_entry_safe(chap, tmp, &ctrl->dhchap_auth_list, entry) {
+		list_del_init(&chap->entry);
+		flush_work(&chap->auth_work);
+		__nvme_auth_free(chap);
+	}
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+	if (ctrl->host_key) {
+		nvme_auth_free_key(ctrl->host_key);
+		ctrl->host_key = NULL;
+	}
+	if (ctrl->ctrl_key) {
+		nvme_auth_free_key(ctrl->ctrl_key);
+		ctrl->ctrl_key = NULL;
+	}
+}
+EXPORT_SYMBOL_GPL(nvme_auth_free);
diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 57480075550c..9071a9798996 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -24,6 +24,7 @@
 
 #include "nvme.h"
 #include "fabrics.h"
+#include <linux/nvme-auth.h>
 
 #define CREATE_TRACE_POINTS
 #include "trace.h"
@@ -330,6 +331,7 @@ enum nvme_disposition {
 	COMPLETE,
 	RETRY,
 	FAILOVER,
+	AUTHENTICATE,
 };
 
 static inline enum nvme_disposition nvme_decide_disposition(struct request *req)
@@ -337,6 +339,9 @@ static inline enum nvme_disposition nvme_decide_disposition(struct request *req)
 	if (likely(nvme_req(req)->status == 0))
 		return COMPLETE;
 
+	if ((nvme_req(req)->status & 0x7ff) == NVME_SC_AUTH_REQUIRED)
+		return AUTHENTICATE;
+
 	if (blk_noretry_request(req) ||
 	    (nvme_req(req)->status & NVME_SC_DNR) ||
 	    nvme_req(req)->retries >= nvme_max_retries)
@@ -375,11 +380,13 @@ static inline void nvme_end_req(struct request *req)
 
 void nvme_complete_rq(struct request *req)
 {
+	struct nvme_ctrl *ctrl = nvme_req(req)->ctrl;
+
 	trace_nvme_complete_rq(req);
 	nvme_cleanup_cmd(req);
 
-	if (nvme_req(req)->ctrl->kas)
-		nvme_req(req)->ctrl->comp_seen = true;
+	if (ctrl->kas)
+		ctrl->comp_seen = true;
 
 	switch (nvme_decide_disposition(req)) {
 	case COMPLETE:
@@ -391,6 +398,14 @@ void nvme_complete_rq(struct request *req)
 	case FAILOVER:
 		nvme_failover_req(req);
 		return;
+	case AUTHENTICATE:
+#ifdef CONFIG_NVME_AUTH
+		queue_work(nvme_wq, &ctrl->dhchap_auth_work);
+		nvme_retry_req(req);
+#else
+		nvme_end_req(req);
+#endif
+		return;
 	}
 }
 EXPORT_SYMBOL_GPL(nvme_complete_rq);
@@ -702,7 +717,9 @@ bool __nvme_check_ready(struct nvme_ctrl *ctrl, struct request *rq,
 		switch (ctrl->state) {
 		case NVME_CTRL_CONNECTING:
 			if (blk_rq_is_passthrough(rq) && nvme_is_fabrics(req->cmd) &&
-			    req->cmd->fabrics.fctype == nvme_fabrics_type_connect)
+			    (req->cmd->fabrics.fctype == nvme_fabrics_type_connect ||
+			     req->cmd->fabrics.fctype == nvme_fabrics_type_auth_send ||
+			     req->cmd->fabrics.fctype == nvme_fabrics_type_auth_receive))
 				return true;
 			break;
 		default:
@@ -3595,6 +3612,108 @@ static ssize_t dctype_show(struct device *dev,
 }
 static DEVICE_ATTR_RO(dctype);
 
+#ifdef CONFIG_NVME_AUTH
+static ssize_t nvme_ctrl_dhchap_secret_show(struct device *dev,
+		struct device_attribute *attr, char *buf)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	struct nvmf_ctrl_options *opts = ctrl->opts;
+
+	if (!opts->dhchap_secret)
+		return sysfs_emit(buf, "none\n");
+	return sysfs_emit(buf, "%s\n", opts->dhchap_secret);
+}
+
+static ssize_t nvme_ctrl_dhchap_secret_store(struct device *dev,
+		struct device_attribute *attr, const char *buf, size_t count)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	struct nvmf_ctrl_options *opts = ctrl->opts;
+	char *dhchap_secret;
+
+	if (!ctrl->opts->dhchap_secret)
+		return -EINVAL;
+	if (count < 7)
+		return -EINVAL;
+	if (memcmp(buf, "DHHC-1:", 7))
+		return -EINVAL;
+
+	dhchap_secret = kzalloc(count + 1, GFP_KERNEL);
+	if (!dhchap_secret)
+		return -ENOMEM;
+	memcpy(dhchap_secret, buf, count);
+	nvme_auth_stop(ctrl);
+	if (strcmp(dhchap_secret, opts->dhchap_secret)) {
+		int ret;
+
+		ret = nvme_auth_generate_key(dhchap_secret, &ctrl->host_key);
+		if (ret)
+			return ret;
+		kfree(opts->dhchap_secret);
+		opts->dhchap_secret = dhchap_secret;
+		/* Key has changed; re-authentication with new key */
+		nvme_auth_reset(ctrl);
+	}
+	/* Start re-authentication */
+	dev_info(ctrl->device, "re-authenticating controller\n");
+	queue_work(nvme_wq, &ctrl->dhchap_auth_work);
+
+	return count;
+}
+static DEVICE_ATTR(dhchap_secret, S_IRUGO | S_IWUSR,
+	nvme_ctrl_dhchap_secret_show, nvme_ctrl_dhchap_secret_store);
+
+static ssize_t nvme_ctrl_dhchap_ctrl_secret_show(struct device *dev,
+		struct device_attribute *attr, char *buf)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	struct nvmf_ctrl_options *opts = ctrl->opts;
+
+	if (!opts->dhchap_ctrl_secret)
+		return sysfs_emit(buf, "none\n");
+	return sysfs_emit(buf, "%s\n", opts->dhchap_ctrl_secret);
+}
+
+static ssize_t nvme_ctrl_dhchap_ctrl_secret_store(struct device *dev,
+		struct device_attribute *attr, const char *buf, size_t count)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	struct nvmf_ctrl_options *opts = ctrl->opts;
+	char *dhchap_secret;
+
+	if (!ctrl->opts->dhchap_ctrl_secret)
+		return -EINVAL;
+	if (count < 7)
+		return -EINVAL;
+	if (memcmp(buf, "DHHC-1:", 7))
+		return -EINVAL;
+
+	dhchap_secret = kzalloc(count + 1, GFP_KERNEL);
+	if (!dhchap_secret)
+		return -ENOMEM;
+	memcpy(dhchap_secret, buf, count);
+	nvme_auth_stop(ctrl);
+	if (strcmp(dhchap_secret, opts->dhchap_ctrl_secret)) {
+		int ret;
+
+		ret = nvme_auth_generate_key(dhchap_secret, &ctrl->ctrl_key);
+		if (ret)
+			return ret;
+		kfree(opts->dhchap_ctrl_secret);
+		opts->dhchap_ctrl_secret = dhchap_secret;
+		/* Key has changed; re-authentication with new key */
+		nvme_auth_reset(ctrl);
+	}
+	/* Start re-authentication */
+	dev_info(ctrl->device, "re-authenticating controller\n");
+	queue_work(nvme_wq, &ctrl->dhchap_auth_work);
+
+	return count;
+}
+static DEVICE_ATTR(dhchap_ctrl_secret, S_IRUGO | S_IWUSR,
+	nvme_ctrl_dhchap_ctrl_secret_show, nvme_ctrl_dhchap_ctrl_secret_store);
+#endif
+
 static struct attribute *nvme_dev_attrs[] = {
 	&dev_attr_reset_controller.attr,
 	&dev_attr_rescan_controller.attr,
@@ -3618,6 +3737,10 @@ static struct attribute *nvme_dev_attrs[] = {
 	&dev_attr_kato.attr,
 	&dev_attr_cntrltype.attr,
 	&dev_attr_dctype.attr,
+#ifdef CONFIG_NVME_AUTH
+	&dev_attr_dhchap_secret.attr,
+	&dev_attr_dhchap_ctrl_secret.attr,
+#endif
 	NULL
 };
 
@@ -3641,6 +3764,12 @@ static umode_t nvme_dev_attrs_are_visible(struct kobject *kobj,
 		return 0;
 	if (a == &dev_attr_fast_io_fail_tmo.attr && !ctrl->opts)
 		return 0;
+#ifdef CONFIG_NVME_AUTH
+	if (a == &dev_attr_dhchap_secret.attr && !ctrl->opts)
+		return 0;
+	if (a == &dev_attr_dhchap_ctrl_secret.attr && !ctrl->opts)
+		return 0;
+#endif
 
 	return a->mode;
 }
@@ -4535,8 +4664,10 @@ static void nvme_handle_aen_notice(struct nvme_ctrl *ctrl, u32 result)
 		 * recovery actions from interfering with the controller's
 		 * firmware activation.
 		 */
-		if (nvme_change_ctrl_state(ctrl, NVME_CTRL_RESETTING))
+		if (nvme_change_ctrl_state(ctrl, NVME_CTRL_RESETTING)) {
+			nvme_auth_stop(ctrl);
 			queue_work(nvme_wq, &ctrl->fw_act_work);
+		}
 		break;
 #ifdef CONFIG_NVME_MULTIPATH
 	case NVME_AER_NOTICE_ANA:
@@ -4600,6 +4731,7 @@ EXPORT_SYMBOL_GPL(nvme_complete_async_event);
 void nvme_stop_ctrl(struct nvme_ctrl *ctrl)
 {
 	nvme_mpath_stop(ctrl);
+	nvme_auth_stop(ctrl);
 	nvme_stop_keep_alive(ctrl);
 	nvme_stop_failfast_work(ctrl);
 	flush_work(&ctrl->async_event_work);
@@ -4657,6 +4789,8 @@ static void nvme_free_ctrl(struct device *dev)
 
 	nvme_free_cels(ctrl);
 	nvme_mpath_uninit(ctrl);
+	nvme_auth_stop(ctrl);
+	nvme_auth_free(ctrl);
 	__free_page(ctrl->discard_page);
 
 	if (subsys) {
@@ -4747,6 +4881,7 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev,
 
 	nvme_fault_inject_init(&ctrl->fault_inject, dev_name(ctrl->device));
 	nvme_mpath_init_ctrl(ctrl);
+	nvme_auth_init_ctrl(ctrl);
 
 	return 0;
 out_free_name:
diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index e4b1520862d8..5207a2348257 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -369,6 +369,7 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)
 	union nvme_result res;
 	struct nvmf_connect_data *data;
 	int ret;
+	u32 result;
 
 	cmd.connect.opcode = nvme_fabrics_command;
 	cmd.connect.fctype = nvme_fabrics_type_connect;
@@ -401,8 +402,25 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)
 		goto out_free_data;
 	}
 
-	ctrl->cntlid = le16_to_cpu(res.u16);
-
+	result = le32_to_cpu(res.u32);
+	ctrl->cntlid = result & 0xFFFF;
+	if ((result >> 16) & 0x3) {
+		/* Authentication required */
+		ret = nvme_auth_negotiate(ctrl, 0);
+		if (ret) {
+			dev_warn(ctrl->device,
+				 "qid 0: authentication setup failed\n");
+			ret = NVME_SC_AUTH_REQUIRED;
+			goto out_free_data;
+		}
+		ret = nvme_auth_wait(ctrl, 0);
+		if (ret)
+			dev_warn(ctrl->device,
+				 "qid 0: authentication failed\n");
+		else
+			dev_info(ctrl->device,
+				 "qid 0: authenticated\n");
+	}
 out_free_data:
 	kfree(data);
 	return ret;
@@ -435,6 +453,7 @@ int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, u16 qid)
 	struct nvmf_connect_data *data;
 	union nvme_result res;
 	int ret;
+	u32 result;
 
 	cmd.connect.opcode = nvme_fabrics_command;
 	cmd.connect.fctype = nvme_fabrics_type_connect;
@@ -460,6 +479,21 @@ int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, u16 qid)
 		nvmf_log_connect_error(ctrl, ret, le32_to_cpu(res.u32),
 				       &cmd, data);
 	}
+	result = le32_to_cpu(res.u32);
+	if ((result >> 16) & 2) {
+		/* Authentication required */
+		ret = nvme_auth_negotiate(ctrl, qid);
+		if (ret) {
+			dev_warn(ctrl->device,
+				 "qid %d: authentication setup failed\n", qid);
+			ret = NVME_SC_AUTH_REQUIRED;
+		} else {
+			ret = nvme_auth_wait(ctrl, qid);
+			if (ret)
+				dev_warn(ctrl->device,
+					 "qid %u: authentication failed\n", qid);
+		}
+	}
 	kfree(data);
 	return ret;
 }
@@ -552,6 +586,8 @@ static const match_table_t opt_tokens = {
 	{ NVMF_OPT_TOS,			"tos=%d"		},
 	{ NVMF_OPT_FAIL_FAST_TMO,	"fast_io_fail_tmo=%d"	},
 	{ NVMF_OPT_DISCOVERY,		"discovery"		},
+	{ NVMF_OPT_DHCHAP_SECRET,	"dhchap_secret=%s"	},
+	{ NVMF_OPT_DHCHAP_CTRL_SECRET,	"dhchap_ctrl_secret=%s"	},
 	{ NVMF_OPT_ERR,			NULL			}
 };
 
@@ -833,6 +869,34 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
 		case NVMF_OPT_DISCOVERY:
 			opts->discovery_nqn = true;
 			break;
+		case NVMF_OPT_DHCHAP_SECRET:
+			p = match_strdup(args);
+			if (!p) {
+				ret = -ENOMEM;
+				goto out;
+			}
+			if (strlen(p) < 11 || strncmp(p, "DHHC-1:", 7)) {
+				pr_err("Invalid DH-CHAP secret %s\n", p);
+				ret = -EINVAL;
+				goto out;
+			}
+			kfree(opts->dhchap_secret);
+			opts->dhchap_secret = p;
+			break;
+		case NVMF_OPT_DHCHAP_CTRL_SECRET:
+			p = match_strdup(args);
+			if (!p) {
+				ret = -ENOMEM;
+				goto out;
+			}
+			if (strlen(p) < 11 || strncmp(p, "DHHC-1:", 7)) {
+				pr_err("Invalid DH-CHAP secret %s\n", p);
+				ret = -EINVAL;
+				goto out;
+			}
+			kfree(opts->dhchap_ctrl_secret);
+			opts->dhchap_ctrl_secret = p;
+			break;
 		default:
 			pr_warn("unknown parameter or missing value '%s' in ctrl creation request\n",
 				p);
@@ -951,6 +1015,8 @@ void nvmf_free_options(struct nvmf_ctrl_options *opts)
 	kfree(opts->subsysnqn);
 	kfree(opts->host_traddr);
 	kfree(opts->host_iface);
+	kfree(opts->dhchap_secret);
+	kfree(opts->dhchap_ctrl_secret);
 	kfree(opts);
 }
 EXPORT_SYMBOL_GPL(nvmf_free_options);
@@ -960,7 +1026,8 @@ EXPORT_SYMBOL_GPL(nvmf_free_options);
 				 NVMF_OPT_KATO | NVMF_OPT_HOSTNQN | \
 				 NVMF_OPT_HOST_ID | NVMF_OPT_DUP_CONNECT |\
 				 NVMF_OPT_DISABLE_SQFLOW | NVMF_OPT_DISCOVERY |\
-				 NVMF_OPT_FAIL_FAST_TMO)
+				 NVMF_OPT_FAIL_FAST_TMO | NVMF_OPT_DHCHAP_SECRET |\
+				 NVMF_OPT_DHCHAP_CTRL_SECRET)
 
 static struct nvme_ctrl *
 nvmf_create_ctrl(struct device *dev, const char *buf)
@@ -1196,7 +1263,14 @@ static void __exit nvmf_exit(void)
 	BUILD_BUG_ON(sizeof(struct nvmf_connect_command) != 64);
 	BUILD_BUG_ON(sizeof(struct nvmf_property_get_command) != 64);
 	BUILD_BUG_ON(sizeof(struct nvmf_property_set_command) != 64);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_send_command) != 64);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_receive_command) != 64);
 	BUILD_BUG_ON(sizeof(struct nvmf_connect_data) != 1024);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_negotiate_data) != 8);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_challenge_data) != 16);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_reply_data) != 16);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_success1_data) != 16);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_success2_data) != 16);
 }
 
 MODULE_LICENSE("GPL v2");
diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h
index 46d6e194ac2b..a6e22116e139 100644
--- a/drivers/nvme/host/fabrics.h
+++ b/drivers/nvme/host/fabrics.h
@@ -68,6 +68,8 @@ enum {
 	NVMF_OPT_FAIL_FAST_TMO	= 1 << 20,
 	NVMF_OPT_HOST_IFACE	= 1 << 21,
 	NVMF_OPT_DISCOVERY	= 1 << 22,
+	NVMF_OPT_DHCHAP_SECRET	= 1 << 23,
+	NVMF_OPT_DHCHAP_CTRL_SECRET = 1 << 24,
 };
 
 /**
@@ -97,6 +99,9 @@ enum {
  * @max_reconnects: maximum number of allowed reconnect attempts before removing
  *              the controller, (-1) means reconnect forever, zero means remove
  *              immediately;
+ * @dhchap_secret: DH-HMAC-CHAP secret
+ * @dhchap_ctrl_secret: DH-HMAC-CHAP controller secret for bi-directional
+ *              authentication
  * @disable_sqflow: disable controller sq flow control
  * @hdr_digest: generate/verify header digest (TCP)
  * @data_digest: generate/verify data digest (TCP)
@@ -121,6 +126,8 @@ struct nvmf_ctrl_options {
 	unsigned int		kato;
 	struct nvmf_host	*host;
 	int			max_reconnects;
+	char			*dhchap_secret;
+	char			*dhchap_ctrl_secret;
 	bool			disable_sqflow;
 	bool			hdr_digest;
 	bool			data_digest;
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index e4612dd0b420..e9350bf7b2d1 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -328,6 +328,15 @@ struct nvme_ctrl {
 	struct work_struct ana_work;
 #endif
 
+#ifdef CONFIG_NVME_AUTH
+	struct work_struct dhchap_auth_work;
+	struct list_head dhchap_auth_list;
+	struct mutex dhchap_auth_mutex;
+	struct nvme_dhchap_key *host_key;
+	struct nvme_dhchap_key *ctrl_key;
+	u16 transaction;
+#endif
+
 	/* Power saving configuration */
 	u64 ps_max_latency_us;
 	bool apst_enabled;
@@ -991,6 +1000,27 @@ static inline bool nvme_ctrl_sgl_supported(struct nvme_ctrl *ctrl)
 	return ctrl->sgls & ((1 << 0) | (1 << 1));
 }
 
+#ifdef CONFIG_NVME_AUTH
+void nvme_auth_init_ctrl(struct nvme_ctrl *ctrl);
+void nvme_auth_stop(struct nvme_ctrl *ctrl);
+int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid);
+int nvme_auth_wait(struct nvme_ctrl *ctrl, int qid);
+void nvme_auth_reset(struct nvme_ctrl *ctrl);
+void nvme_auth_free(struct nvme_ctrl *ctrl);
+#else
+static inline void nvme_auth_init_ctrl(struct nvme_ctrl *ctrl) {};
+static inline void nvme_auth_stop(struct nvme_ctrl *ctrl) {};
+static inline int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid)
+{
+	return -EPROTONOSUPPORT;
+}
+static inline int nvme_auth_wait(struct nvme_ctrl *ctrl, int qid)
+{
+	return NVME_SC_AUTH_REQUIRED;
+}
+static inline void nvme_auth_free(struct nvme_ctrl *ctrl) {};
+#endif
+
 u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 			 u8 opcode);
 int nvme_execute_passthru_rq(struct request *rq);
diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index f2a5e1ea508a..84ce3347d158 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -1197,6 +1197,7 @@ static void nvme_rdma_error_recovery_work(struct work_struct *work)
 	struct nvme_rdma_ctrl *ctrl = container_of(work,
 			struct nvme_rdma_ctrl, err_work);
 
+	nvme_auth_stop(&ctrl->ctrl);
 	nvme_stop_keep_alive(&ctrl->ctrl);
 	flush_work(&ctrl->ctrl.async_event_work);
 	nvme_rdma_teardown_io_queues(ctrl, false);
diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index bb67538d241b..a7848e430a5c 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -2174,6 +2174,7 @@ static void nvme_tcp_error_recovery_work(struct work_struct *work)
 				struct nvme_tcp_ctrl, err_work);
 	struct nvme_ctrl *ctrl = &tcp_ctrl->ctrl;
 
+	nvme_auth_stop(ctrl);
 	nvme_stop_keep_alive(ctrl);
 	flush_work(&ctrl->async_event_work);
 	nvme_tcp_teardown_io_queues(ctrl, false);
diff --git a/drivers/nvme/host/trace.c b/drivers/nvme/host/trace.c
index 2a89c5aa0790..1c36fcedea20 100644
--- a/drivers/nvme/host/trace.c
+++ b/drivers/nvme/host/trace.c
@@ -287,6 +287,34 @@ static const char *nvme_trace_fabrics_property_get(struct trace_seq *p, u8 *spc)
 	return ret;
 }
 
+static const char *nvme_trace_fabrics_auth_send(struct trace_seq *p, u8 *spc)
+{
+	const char *ret = trace_seq_buffer_ptr(p);
+	u8 spsp0 = spc[1];
+	u8 spsp1 = spc[2];
+	u8 secp = spc[3];
+	u32 tl = get_unaligned_le32(spc + 4);
+
+	trace_seq_printf(p, "spsp0=%02x, spsp1=%02x, secp=%02x, tl=%u",
+			 spsp0, spsp1, secp, tl);
+	trace_seq_putc(p, 0);
+	return ret;
+}
+
+static const char *nvme_trace_fabrics_auth_receive(struct trace_seq *p, u8 *spc)
+{
+	const char *ret = trace_seq_buffer_ptr(p);
+	u8 spsp0 = spc[1];
+	u8 spsp1 = spc[2];
+	u8 secp = spc[3];
+	u32 al = get_unaligned_le32(spc + 4);
+
+	trace_seq_printf(p, "spsp0=%02x, spsp1=%02x, secp=%02x, al=%u",
+			 spsp0, spsp1, secp, al);
+	trace_seq_putc(p, 0);
+	return ret;
+}
+
 static const char *nvme_trace_fabrics_common(struct trace_seq *p, u8 *spc)
 {
 	const char *ret = trace_seq_buffer_ptr(p);
@@ -306,6 +334,10 @@ const char *nvme_trace_parse_fabrics_cmd(struct trace_seq *p,
 		return nvme_trace_fabrics_connect(p, spc);
 	case nvme_fabrics_type_property_get:
 		return nvme_trace_fabrics_property_get(p, spc);
+	case nvme_fabrics_type_auth_send:
+		return nvme_trace_fabrics_auth_send(p, spc);
+	case nvme_fabrics_type_auth_receive:
+		return nvme_trace_fabrics_auth_receive(p, spc);
 	default:
 		return nvme_trace_fabrics_common(p, spc);
 	}
diff --git a/include/linux/nvme-auth.h b/include/linux/nvme-auth.h
new file mode 100644
index 000000000000..354456826221
--- /dev/null
+++ b/include/linux/nvme-auth.h
@@ -0,0 +1,33 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2021 Hannes Reinecke, SUSE Software Solutions
+ */
+
+#ifndef _NVME_AUTH_H
+#define _NVME_AUTH_H
+
+#include <crypto/kpp.h>
+
+struct nvme_dhchap_key {
+	u8 *key;
+	size_t len;
+	u8 hash;
+};
+
+u32 nvme_auth_get_seqnum(void);
+const char *nvme_auth_dhgroup_name(u8 dhgroup_id);
+const char *nvme_auth_dhgroup_kpp(u8 dhgroup_id);
+u8 nvme_auth_dhgroup_id(const char *dhgroup_name);
+
+const char *nvme_auth_hmac_name(u8 hmac_id);
+const char *nvme_auth_digest_name(u8 hmac_id);
+size_t nvme_auth_hmac_hash_len(u8 hmac_id);
+u8 nvme_auth_hmac_id(const char *hmac_name);
+
+struct nvme_dhchap_key *nvme_auth_extract_key(unsigned char *secret,
+					      u8 key_hash);
+void nvme_auth_free_key(struct nvme_dhchap_key *key);
+u8 *nvme_auth_transform_key(struct nvme_dhchap_key *key, char *nqn);
+int nvme_auth_generate_key(u8 *secret, struct nvme_dhchap_key **ret_key);
+
+#endif /* _NVME_AUTH_H */
-- 
2.29.2

[PATCH 06/11] nvme: Implement In-Band authentication

[Hannes Reinecke]

Implement NVMe-oF In-Band authentication according to NVMe TPAR 8006.
This patch adds two new fabric options 'dhchap_secret' to specify the
pre-shared key (in ASCII respresentation according to NVMe 2.0 section
8.13.5.8 'Secret representation') and 'dhchap_ctrl_secret' to specify
the pre-shared controller key for bi-directional authentication of both
the host and the controller.
Re-authentication can be triggered by writing the PSK into the new
controller sysfs attribute 'dhchap_secret' or 'dhchap_ctrl_secret'.

Signed-off-by: Hannes Reinecke <hare@suse.de>
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
---
 drivers/nvme/Kconfig         |   1 +
 drivers/nvme/Makefile        |   1 +
 drivers/nvme/common/Kconfig  |   4 +
 drivers/nvme/common/Makefile |   7 +
 drivers/nvme/common/auth.c   | 329 ++++++++++++++
 drivers/nvme/host/Kconfig    |  13 +
 drivers/nvme/host/Makefile   |   1 +
 drivers/nvme/host/auth.c     | 819 +++++++++++++++++++++++++++++++++++
 drivers/nvme/host/core.c     | 143 +++++-
 drivers/nvme/host/fabrics.c  |  80 +++-
 drivers/nvme/host/fabrics.h  |   7 +
 drivers/nvme/host/nvme.h     |  30 ++
 drivers/nvme/host/rdma.c     |   1 +
 drivers/nvme/host/tcp.c      |   1 +
 drivers/nvme/host/trace.c    |  32 ++
 include/linux/nvme-auth.h    |  33 ++
 16 files changed, 1495 insertions(+), 7 deletions(-)
 create mode 100644 drivers/nvme/common/Kconfig
 create mode 100644 drivers/nvme/common/Makefile
 create mode 100644 drivers/nvme/common/auth.c
 create mode 100644 drivers/nvme/host/auth.c
 create mode 100644 include/linux/nvme-auth.h

diff --git a/drivers/nvme/Kconfig b/drivers/nvme/Kconfig
index 87ae409a32b9..656e46d938da 100644
--- a/drivers/nvme/Kconfig
+++ b/drivers/nvme/Kconfig
@@ -1,6 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0-only
 menu "NVME Support"
 
+source "drivers/nvme/common/Kconfig"
 source "drivers/nvme/host/Kconfig"
 source "drivers/nvme/target/Kconfig"
 
diff --git a/drivers/nvme/Makefile b/drivers/nvme/Makefile
index fb42c44609a8..eedca8c72098 100644
--- a/drivers/nvme/Makefile
+++ b/drivers/nvme/Makefile
@@ -1,4 +1,5 @@
 # SPDX-License-Identifier: GPL-2.0-only
 
+obj-$(CONFIG_NVME_COMMON)		+= common/
 obj-y		+= host/
 obj-y		+= target/
diff --git a/drivers/nvme/common/Kconfig b/drivers/nvme/common/Kconfig
new file mode 100644
index 000000000000..4514f44362dd
--- /dev/null
+++ b/drivers/nvme/common/Kconfig
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: GPL-2.0-only
+
+config NVME_COMMON
+       tristate
diff --git a/drivers/nvme/common/Makefile b/drivers/nvme/common/Makefile
new file mode 100644
index 000000000000..720c625b8a52
--- /dev/null
+++ b/drivers/nvme/common/Makefile
@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: GPL-2.0
+
+ccflags-y			+= -I$(src)
+
+obj-$(CONFIG_NVME_COMMON)	+= nvme-common.o
+
+nvme-common-y			+= auth.o
diff --git a/drivers/nvme/common/auth.c b/drivers/nvme/common/auth.c
new file mode 100644
index 000000000000..01adb29947d4
--- /dev/null
+++ b/drivers/nvme/common/auth.c
@@ -0,0 +1,329 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2020 Hannes Reinecke, SUSE Linux
+ */
+
+#include <linux/module.h>
+#include <linux/crc32.h>
+#include <linux/base64.h>
+#include <linux/prandom.h>
+#include <linux/scatterlist.h>
+#include <asm/unaligned.h>
+#include <crypto/hash.h>
+#include <crypto/dh.h>
+#include <linux/nvme.h>
+#include <linux/nvme-auth.h>
+
+static u32 nvme_dhchap_seqnum;
+static DEFINE_MUTEX(nvme_dhchap_mutex);
+
+u32 nvme_auth_get_seqnum(void)
+{
+	u32 seqnum;
+
+	mutex_lock(&nvme_dhchap_mutex);
+	if (!nvme_dhchap_seqnum)
+		nvme_dhchap_seqnum = prandom_u32();
+	else {
+		nvme_dhchap_seqnum++;
+		if (!nvme_dhchap_seqnum)
+			nvme_dhchap_seqnum++;
+	}
+	seqnum = nvme_dhchap_seqnum;
+	mutex_unlock(&nvme_dhchap_mutex);
+	return seqnum;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_get_seqnum);
+
+static struct nvme_auth_dhgroup_map {
+	const char name[16];
+	const char kpp[16];
+} dhgroup_map[] = {
+	[NVME_AUTH_DHGROUP_NULL] = {
+		.name = "null", .kpp = "null" },
+	[NVME_AUTH_DHGROUP_2048] = {
+		.name = "ffdhe2048", .kpp = "ffdhe2048(dh)" },
+	[NVME_AUTH_DHGROUP_3072] = {
+		.name = "ffdhe3072", .kpp = "ffdhe3072(dh)" },
+	[NVME_AUTH_DHGROUP_4096] = {
+		.name = "ffdhe4096", .kpp = "ffdhe4096(dh)" },
+	[NVME_AUTH_DHGROUP_6144] = {
+		.name = "ffdhe6144", .kpp = "ffdhe6144(dh)" },
+	[NVME_AUTH_DHGROUP_8192] = {
+		.name = "ffdhe8192", .kpp = "ffdhe8192(dh)" },
+};
+
+const char *nvme_auth_dhgroup_name(u8 dhgroup_id)
+{
+	if ((dhgroup_id > ARRAY_SIZE(dhgroup_map)) ||
+	    !dhgroup_map[dhgroup_id].name ||
+	    !strlen(dhgroup_map[dhgroup_id].name))
+		return NULL;
+	return dhgroup_map[dhgroup_id].name;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_dhgroup_name);
+
+const char *nvme_auth_dhgroup_kpp(u8 dhgroup_id)
+{
+	if ((dhgroup_id > ARRAY_SIZE(dhgroup_map)) ||
+	    !dhgroup_map[dhgroup_id].kpp ||
+	    !strlen(dhgroup_map[dhgroup_id].kpp))
+		return NULL;
+	return dhgroup_map[dhgroup_id].kpp;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_dhgroup_kpp);
+
+u8 nvme_auth_dhgroup_id(const char *dhgroup_name)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(dhgroup_map); i++) {
+		if (!dhgroup_map[i].name ||
+		    !strlen(dhgroup_map[i].name))
+			continue;
+		if (!strncmp(dhgroup_map[i].name, dhgroup_name,
+			     strlen(dhgroup_map[i].name)))
+			return i;
+	}
+	return NVME_AUTH_DHGROUP_INVALID;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_dhgroup_id);
+
+static struct nvme_dhchap_hash_map {
+	int len;
+	const char hmac[15];
+	const char digest[8];
+} hash_map[] = {
+	[NVME_AUTH_HASH_SHA256] = {
+		.len = 32,
+		.hmac = "hmac(sha256)",
+		.digest = "sha256",
+	},
+	[NVME_AUTH_HASH_SHA384] = {
+		.len = 48,
+		.hmac = "hmac(sha384)",
+		.digest = "sha384",
+	},
+	[NVME_AUTH_HASH_SHA512] = {
+		.len = 64,
+		.hmac = "hmac(sha512)",
+		.digest = "sha512",
+	},
+};
+
+const char *nvme_auth_hmac_name(u8 hmac_id)
+{
+	if ((hmac_id > ARRAY_SIZE(hash_map)) ||
+	    !hash_map[hmac_id].hmac ||
+	    !strlen(hash_map[hmac_id].hmac))
+		return NULL;
+	return hash_map[hmac_id].hmac;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_hmac_name);
+
+const char *nvme_auth_digest_name(u8 hmac_id)
+{
+	if ((hmac_id > ARRAY_SIZE(hash_map)) ||
+	    !hash_map[hmac_id].digest ||
+	    !strlen(hash_map[hmac_id].digest))
+		return NULL;
+	return hash_map[hmac_id].digest;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_digest_name);
+
+u8 nvme_auth_hmac_id(const char *hmac_name)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
+		if (!hash_map[i].hmac || !strlen(hash_map[i].hmac))
+			continue;
+		if (!strncmp(hash_map[i].hmac, hmac_name,
+			     strlen(hash_map[i].hmac)))
+			return i;
+	}
+	return NVME_AUTH_HASH_INVALID;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_hmac_id);
+
+size_t nvme_auth_hmac_hash_len(u8 hmac_id)
+{
+	if ((hmac_id > ARRAY_SIZE(hash_map)) ||
+	    !hash_map[hmac_id].hmac ||
+	    !strlen(hash_map[hmac_id].hmac))
+		return 0;
+	return hash_map[hmac_id].len;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_hmac_hash_len);
+
+struct nvme_dhchap_key *nvme_auth_extract_key(unsigned char *secret,
+					      u8 key_hash)
+{
+	struct nvme_dhchap_key *key;
+	unsigned char *p;
+	u32 crc;
+	int ret, key_len;
+	size_t allocated_len = strlen(secret);
+
+	/* Secret might be affixed with a ':' */
+	p = strrchr(secret, ':');
+	if (p)
+		allocated_len = p - secret;
+	key = kzalloc(sizeof(*key), GFP_KERNEL);
+	if (!key)
+		return ERR_PTR(-ENOMEM);
+	key->key = kzalloc(allocated_len, GFP_KERNEL);
+	if (!key->key) {
+		ret = -ENOMEM;
+		goto out_free_key;
+	}
+
+	key_len = base64_decode(secret, allocated_len, key->key);
+	if (key_len < 0) {
+		pr_debug("base64 key decoding error %d\n",
+			 key_len);
+		ret = key_len;
+		goto out_free_secret;
+	}
+
+	if (key_len != 36 && key_len != 52 &&
+	    key_len != 68) {
+		pr_err("Invalid key len %d\n", key_len);
+		ret = -EINVAL;
+		goto out_free_secret;
+	}
+
+	if (key_hash > 0 &&
+	    (key_len - 4) != nvme_auth_hmac_hash_len(key_hash)) {
+		pr_err("Mismatched key len %d for %s\n", key_len,
+		       nvme_auth_hmac_name(key_hash));
+		ret = -EINVAL;
+		goto out_free_secret;
+	}
+
+	/* The last four bytes is the CRC in little-endian format */
+	key_len -= 4;
+	/*
+	 * The linux implementation doesn't do pre- and post-increments,
+	 * so we have to do it manually.
+	 */
+	crc = ~crc32(~0, key->key, key_len);
+
+	if (get_unaligned_le32(key->key + key_len) != crc) {
+		pr_err("key crc mismatch (key %08x, crc %08x)\n",
+		       get_unaligned_le32(key->key + key_len), crc);
+		ret = -EKEYREJECTED;
+		goto out_free_secret;
+	}
+	key->len = key_len;
+	key->hash = key_hash;
+	return key;
+out_free_secret:
+	kfree_sensitive(key->key);
+out_free_key:
+	kfree(key);
+	return ERR_PTR(ret);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_extract_key);
+
+void nvme_auth_free_key(struct nvme_dhchap_key *key)
+{
+	if (!key)
+		return;
+	kfree_sensitive(key->key);
+	kfree(key);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_free_key);
+
+u8 *nvme_auth_transform_key(struct nvme_dhchap_key *key, char *nqn)
+{
+	const char *hmac_name;
+	struct crypto_shash *key_tfm;
+	struct shash_desc *shash;
+	u8 *transformed_key;
+	int ret;
+
+	if (!key || !key->key) {
+		pr_warn("No key specified\n");
+		return ERR_PTR(-ENOKEY);
+	}
+	if (key->hash == 0) {
+		transformed_key = kmemdup(key->key, key->len, GFP_KERNEL);
+		return transformed_key ? transformed_key : ERR_PTR(-ENOMEM);
+	}
+	hmac_name = nvme_auth_hmac_name(key->hash);
+	if (!hmac_name) {
+		pr_warn("Invalid key hash id %d\n", key->hash);
+		return ERR_PTR(-EINVAL);
+	}
+
+	key_tfm = crypto_alloc_shash(hmac_name, 0, 0);
+	if (IS_ERR(key_tfm))
+		return (u8 *)key_tfm;
+
+	shash = kmalloc(sizeof(struct shash_desc) +
+			crypto_shash_descsize(key_tfm),
+			GFP_KERNEL);
+	if (!shash) {
+		ret = -ENOMEM;
+		goto out_free_key;
+	}
+
+	transformed_key = kzalloc(crypto_shash_digestsize(key_tfm), GFP_KERNEL);
+	if (!transformed_key) {
+		ret = -ENOMEM;
+		goto out_free_shash;
+	}
+
+	shash->tfm = key_tfm;
+	ret = crypto_shash_setkey(key_tfm, key->key, key->len);
+	if (ret < 0)
+		goto out_free_shash;
+	ret = crypto_shash_init(shash);
+	if (ret < 0)
+		goto out_free_shash;
+	ret = crypto_shash_update(shash, nqn, strlen(nqn));
+	if (ret < 0)
+		goto out_free_shash;
+	ret = crypto_shash_update(shash, "NVMe-over-Fabrics", 17);
+	if (ret < 0)
+		goto out_free_shash;
+	ret = crypto_shash_final(shash, transformed_key);
+out_free_shash:
+	kfree(shash);
+out_free_key:
+	crypto_free_shash(key_tfm);
+	if (ret < 0) {
+		kfree_sensitive(transformed_key);
+		return ERR_PTR(ret);
+	}
+	return transformed_key;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_transform_key);
+
+int nvme_auth_generate_key(u8 *secret, struct nvme_dhchap_key **ret_key)
+{
+	struct nvme_dhchap_key *key;
+	u8 key_hash;
+
+	if (!secret) {
+		*ret_key = NULL;
+		return 0;
+	}
+
+	if (sscanf(secret, "DHHC-1:%hhd:%*s:", &key_hash) != 1)
+		return -EINVAL;
+
+	/* Pass in the secret without the 'DHHC-1:XX:' prefix */
+	key = nvme_auth_extract_key(secret + 10, key_hash);
+	if (IS_ERR(key)) {
+		*ret_key = NULL;
+		return PTR_ERR(key);
+	}
+
+	*ret_key = key;
+	return 0;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_generate_key);
+
+MODULE_LICENSE("GPL v2");
diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig
index 877d2ec4ea9f..6c503f42f3c6 100644
--- a/drivers/nvme/host/Kconfig
+++ b/drivers/nvme/host/Kconfig
@@ -92,6 +92,19 @@ config NVME_TCP
 
 	  If unsure, say N.
 
+config NVME_AUTH
+	bool "NVM Express over Fabrics In-Band Authentication"
+	depends on NVME_CORE
+	select NVME_COMMON
+	select CRYPTO
+	select CRYPTO_HMAC
+	select CRYPTO_SHA256
+	select CRYPTO_SHA512
+	help
+	  This provides support for NVMe over Fabrics In-Band Authentication.
+
+	  If unsure, say N.
+
 config NVME_APPLE
 	tristate "Apple ANS2 NVM Express host driver"
 	depends on OF && BLOCK
diff --git a/drivers/nvme/host/Makefile b/drivers/nvme/host/Makefile
index a36ae1612059..a3e88f32f560 100644
--- a/drivers/nvme/host/Makefile
+++ b/drivers/nvme/host/Makefile
@@ -16,6 +16,7 @@ nvme-core-$(CONFIG_NVME_MULTIPATH)	+= multipath.o
 nvme-core-$(CONFIG_BLK_DEV_ZONED)	+= zns.o
 nvme-core-$(CONFIG_FAULT_INJECTION_DEBUG_FS)	+= fault_inject.o
 nvme-core-$(CONFIG_NVME_HWMON)		+= hwmon.o
+nvme-core-$(CONFIG_NVME_AUTH)		+= auth.o
 
 nvme-y					+= pci.o
 
diff --git a/drivers/nvme/host/auth.c b/drivers/nvme/host/auth.c
new file mode 100644
index 000000000000..dc739c340cc3
--- /dev/null
+++ b/drivers/nvme/host/auth.c
@@ -0,0 +1,819 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2020 Hannes Reinecke, SUSE Linux
+ */
+
+#include <linux/crc32.h>
+#include <linux/base64.h>
+#include <linux/prandom.h>
+#include <asm/unaligned.h>
+#include <crypto/hash.h>
+#include <crypto/dh.h>
+#include "nvme.h"
+#include "fabrics.h"
+#include <linux/nvme-auth.h>
+
+struct nvme_dhchap_queue_context {
+	struct list_head entry;
+	struct work_struct auth_work;
+	struct nvme_ctrl *ctrl;
+	struct crypto_shash *shash_tfm;
+	void *buf;
+	size_t buf_size;
+	int qid;
+	int error;
+	u32 s1;
+	u32 s2;
+	u16 transaction;
+	u8 status;
+	u8 hash_id;
+	size_t hash_len;
+	u8 dhgroup_id;
+	u8 c1[64];
+	u8 c2[64];
+	u8 response[64];
+	u8 *host_response;
+};
+
+#define nvme_auth_flags_from_qid(qid) \
+	(qid == 0) ? 0 : BLK_MQ_REQ_NOWAIT | BLK_MQ_REQ_RESERVED
+#define nvme_auth_queue_from_qid(ctrl, qid) \
+	(qid == 0) ? (ctrl)->fabrics_q : (ctrl)->connect_q
+
+static int nvme_auth_submit(struct nvme_ctrl *ctrl, int qid,
+			    void *data, size_t data_len, bool auth_send)
+{
+	struct nvme_command cmd = {};
+	blk_mq_req_flags_t flags = nvme_auth_flags_from_qid(qid);
+	struct request_queue *q = nvme_auth_queue_from_qid(ctrl, qid);
+	int ret;
+
+	cmd.auth_common.opcode = nvme_fabrics_command;
+	cmd.auth_common.secp = NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER;
+	cmd.auth_common.spsp0 = 0x01;
+	cmd.auth_common.spsp1 = 0x01;
+	if (auth_send) {
+		cmd.auth_send.fctype = nvme_fabrics_type_auth_send;
+		cmd.auth_send.tl = cpu_to_le32(data_len);
+	} else {
+		cmd.auth_receive.fctype = nvme_fabrics_type_auth_receive;
+		cmd.auth_receive.al = cpu_to_le32(data_len);
+	}
+
+	ret = __nvme_submit_sync_cmd(q, &cmd, NULL, data, data_len,
+				     qid == 0 ? NVME_QID_ANY : qid,
+				     0, flags);
+	if (ret > 0)
+		dev_warn(ctrl->device,
+			"qid %d auth_send failed with status %d\n", qid, ret);
+	else if (ret < 0)
+		dev_err(ctrl->device,
+			"qid %d auth_send failed with error %d\n", qid, ret);
+	return ret;
+}
+
+static int nvme_auth_receive_validate(struct nvme_ctrl *ctrl, int qid,
+		struct nvmf_auth_dhchap_failure_data *data,
+		u16 transaction, u8 expected_msg)
+{
+	dev_dbg(ctrl->device, "%s: qid %d auth_type %d auth_id %x\n",
+		__func__, qid, data->auth_type, data->auth_id);
+
+	if (data->auth_type == NVME_AUTH_COMMON_MESSAGES &&
+	    data->auth_id == NVME_AUTH_DHCHAP_MESSAGE_FAILURE1) {
+		return data->rescode_exp;
+	}
+	if (data->auth_type != NVME_AUTH_DHCHAP_MESSAGES ||
+	    data->auth_id != expected_msg) {
+		dev_warn(ctrl->device,
+			 "qid %d invalid message %02x/%02x\n",
+			 qid, data->auth_type, data->auth_id);
+		return NVME_AUTH_DHCHAP_FAILURE_INCORRECT_MESSAGE;
+	}
+	if (le16_to_cpu(data->t_id) != transaction) {
+		dev_warn(ctrl->device,
+			 "qid %d invalid transaction ID %d\n",
+			 qid, le16_to_cpu(data->t_id));
+		return NVME_AUTH_DHCHAP_FAILURE_INCORRECT_MESSAGE;
+	}
+	return 0;
+}
+
+static int nvme_auth_set_dhchap_negotiate_data(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_negotiate_data *data = chap->buf;
+	size_t size = sizeof(*data) + sizeof(union nvmf_auth_protocol);
+
+	if (chap->buf_size < size) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return -EINVAL;
+	}
+	memset((u8 *)chap->buf, 0, size);
+	data->auth_type = NVME_AUTH_COMMON_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE;
+	data->t_id = cpu_to_le16(chap->transaction);
+	data->sc_c = 0; /* No secure channel concatenation */
+	data->napd = 1;
+	data->auth_protocol[0].dhchap.authid = NVME_AUTH_DHCHAP_AUTH_ID;
+	data->auth_protocol[0].dhchap.halen = 3;
+	data->auth_protocol[0].dhchap.dhlen = 6;
+	data->auth_protocol[0].dhchap.idlist[0] = NVME_AUTH_HASH_SHA256;
+	data->auth_protocol[0].dhchap.idlist[1] = NVME_AUTH_HASH_SHA384;
+	data->auth_protocol[0].dhchap.idlist[2] = NVME_AUTH_HASH_SHA512;
+	data->auth_protocol[0].dhchap.idlist[30] = NVME_AUTH_DHGROUP_NULL;
+	data->auth_protocol[0].dhchap.idlist[31] = NVME_AUTH_DHGROUP_2048;
+	data->auth_protocol[0].dhchap.idlist[32] = NVME_AUTH_DHGROUP_3072;
+	data->auth_protocol[0].dhchap.idlist[33] = NVME_AUTH_DHGROUP_4096;
+	data->auth_protocol[0].dhchap.idlist[34] = NVME_AUTH_DHGROUP_6144;
+	data->auth_protocol[0].dhchap.idlist[35] = NVME_AUTH_DHGROUP_8192;
+
+	return size;
+}
+
+static int nvme_auth_process_dhchap_challenge(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_challenge_data *data = chap->buf;
+	u16 dhvlen = le16_to_cpu(data->dhvlen);
+	size_t size = sizeof(*data) + data->hl + dhvlen;
+	const char *hmac_name, *kpp_name;
+
+	if (chap->buf_size < size) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return NVME_SC_INVALID_FIELD;
+	}
+
+	hmac_name = nvme_auth_hmac_name(data->hashid);
+	if (!hmac_name) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid HASH ID %d\n",
+			 chap->qid, data->hashid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
+		return NVME_SC_INVALID_FIELD;
+	}
+
+	if (chap->hash_id == data->hashid && chap->shash_tfm &&
+	    !strcmp(crypto_shash_alg_name(chap->shash_tfm), hmac_name) &&
+	    crypto_shash_digestsize(chap->shash_tfm) == data->hl) {
+		dev_dbg(ctrl->device,
+			"qid %d: reuse existing hash %s\n",
+			chap->qid, hmac_name);
+		goto select_kpp;
+	}
+
+	/* Reset if hash cannot be reused */
+	if (chap->shash_tfm) {
+		crypto_free_shash(chap->shash_tfm);
+		chap->hash_id = 0;
+		chap->hash_len = 0;
+	}
+	chap->shash_tfm = crypto_alloc_shash(hmac_name, 0,
+					     CRYPTO_ALG_ALLOCATES_MEMORY);
+	if (IS_ERR(chap->shash_tfm)) {
+		dev_warn(ctrl->device,
+			 "qid %d: failed to allocate hash %s, error %ld\n",
+			 chap->qid, hmac_name, PTR_ERR(chap->shash_tfm));
+		chap->shash_tfm = NULL;
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_FAILED;
+		return NVME_SC_AUTH_REQUIRED;
+	}
+
+	if (crypto_shash_digestsize(chap->shash_tfm) != data->hl) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid hash length %d\n",
+			 chap->qid, data->hl);
+		crypto_free_shash(chap->shash_tfm);
+		chap->shash_tfm = NULL;
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
+		return NVME_SC_AUTH_REQUIRED;
+	}
+
+	/* Reset host response if the hash had been changed */
+	if (chap->hash_id != data->hashid) {
+		kfree(chap->host_response);
+		chap->host_response = NULL;
+	}
+
+	chap->hash_id = data->hashid;
+	chap->hash_len = data->hl;
+	dev_dbg(ctrl->device, "qid %d: selected hash %s\n",
+		chap->qid, hmac_name);
+
+select_kpp:
+	kpp_name = nvme_auth_dhgroup_kpp(data->dhgid);
+	if (!kpp_name) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid DH group id %d\n",
+			 chap->qid, data->dhgid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
+		return NVME_SC_AUTH_REQUIRED;
+	}
+
+	if (data->dhgid != NVME_AUTH_DHGROUP_NULL) {
+		dev_warn(ctrl->device,
+			 "qid %d: unsupported DH group %s\n",
+			 chap->qid, kpp_name);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
+		return NVME_SC_AUTH_REQUIRED;
+	} else if (dhvlen != 0) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid DH value for NULL DH\n",
+			 chap->qid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return NVME_SC_INVALID_FIELD;
+	}
+	chap->dhgroup_id = data->dhgid;
+
+	chap->s1 = le32_to_cpu(data->seqnum);
+	memcpy(chap->c1, data->cval, chap->hash_len);
+
+	return 0;
+}
+
+static int nvme_auth_set_dhchap_reply_data(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_reply_data *data = chap->buf;
+	size_t size = sizeof(*data);
+
+	size += 2 * chap->hash_len;
+
+	if (chap->buf_size < size) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return -EINVAL;
+	}
+
+	memset(chap->buf, 0, size);
+	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_REPLY;
+	data->t_id = cpu_to_le16(chap->transaction);
+	data->hl = chap->hash_len;
+	data->dhvlen = 0;
+	memcpy(data->rval, chap->response, chap->hash_len);
+	if (ctrl->ctrl_key) {
+		get_random_bytes(chap->c2, chap->hash_len);
+		data->cvalid = 1;
+		chap->s2 = nvme_auth_get_seqnum();
+		memcpy(data->rval + chap->hash_len, chap->c2,
+		       chap->hash_len);
+		dev_dbg(ctrl->device, "%s: qid %d ctrl challenge %*ph\n",
+			__func__, chap->qid, (int)chap->hash_len, chap->c2);
+	} else {
+		memset(chap->c2, 0, chap->hash_len);
+		chap->s2 = 0;
+	}
+	data->seqnum = cpu_to_le32(chap->s2);
+	return size;
+}
+
+static int nvme_auth_process_dhchap_success1(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_success1_data *data = chap->buf;
+	size_t size = sizeof(*data);
+
+	if (ctrl->ctrl_key)
+		size += chap->hash_len;
+
+	if (chap->buf_size < size) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return NVME_SC_INVALID_FIELD;
+	}
+
+	if (data->hl != chap->hash_len) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid hash length %u\n",
+			 chap->qid, data->hl);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
+		return NVME_SC_INVALID_FIELD;
+	}
+
+	/* Just print out information for the admin queue */
+	if (chap->qid == 0)
+		dev_info(ctrl->device,
+			 "qid 0: authenticated with hash %s dhgroup %s\n",
+			 nvme_auth_hmac_name(chap->hash_id),
+			 nvme_auth_dhgroup_name(chap->dhgroup_id));
+
+	if (!data->rvalid)
+		return 0;
+
+	/* Validate controller response */
+	if (memcmp(chap->response, data->rval, data->hl)) {
+		dev_dbg(ctrl->device, "%s: qid %d ctrl response %*ph\n",
+			__func__, chap->qid, (int)chap->hash_len, data->rval);
+		dev_dbg(ctrl->device, "%s: qid %d host response %*ph\n",
+			__func__, chap->qid, (int)chap->hash_len,
+			chap->response);
+		dev_warn(ctrl->device,
+			 "qid %d: controller authentication failed\n",
+			 chap->qid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_FAILED;
+		return NVME_SC_AUTH_REQUIRED;
+	}
+
+	/* Just print out information for the admin queue */
+	if (chap->qid == 0)
+		dev_info(ctrl->device,
+			 "qid 0: controller authenticated\n");
+	return 0;
+}
+
+static int nvme_auth_set_dhchap_success2_data(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_success2_data *data = chap->buf;
+	size_t size = sizeof(*data);
+
+	memset(chap->buf, 0, size);
+	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_SUCCESS2;
+	data->t_id = cpu_to_le16(chap->transaction);
+
+	return size;
+}
+
+static int nvme_auth_set_dhchap_failure2_data(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_failure_data *data = chap->buf;
+	size_t size = sizeof(*data);
+
+	memset(chap->buf, 0, size);
+	data->auth_type = NVME_AUTH_COMMON_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_FAILURE2;
+	data->t_id = cpu_to_le16(chap->transaction);
+	data->rescode = NVME_AUTH_DHCHAP_FAILURE_REASON_FAILED;
+	data->rescode_exp = chap->status;
+
+	return size;
+}
+
+static int nvme_auth_dhchap_setup_host_response(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
+	u8 buf[4], *challenge = chap->c1;
+	int ret;
+
+	dev_dbg(ctrl->device, "%s: qid %d host response seq %u transaction %d\n",
+		__func__, chap->qid, chap->s1, chap->transaction);
+
+	if (!chap->host_response) {
+		chap->host_response = nvme_auth_transform_key(ctrl->host_key,
+						ctrl->opts->host->nqn);
+		if (IS_ERR(chap->host_response)) {
+			ret = PTR_ERR(chap->host_response);
+			chap->host_response = NULL;
+			return ret;
+		}
+	} else {
+		dev_dbg(ctrl->device, "%s: qid %d re-using host response\n",
+			__func__, chap->qid);
+	}
+
+	ret = crypto_shash_setkey(chap->shash_tfm,
+			chap->host_response, ctrl->host_key->len);
+	if (ret) {
+		dev_warn(ctrl->device, "qid %d: failed to set key, error %d\n",
+			 chap->qid, ret);
+		goto out;
+	}
+
+	shash->tfm = chap->shash_tfm;
+	ret = crypto_shash_init(shash);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, challenge, chap->hash_len);
+	if (ret)
+		goto out;
+	put_unaligned_le32(chap->s1, buf);
+	ret = crypto_shash_update(shash, buf, 4);
+	if (ret)
+		goto out;
+	put_unaligned_le16(chap->transaction, buf);
+	ret = crypto_shash_update(shash, buf, 2);
+	if (ret)
+		goto out;
+	memset(buf, 0, sizeof(buf));
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, "HostHost", 8);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
+				  strlen(ctrl->opts->host->nqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
+			    strlen(ctrl->opts->subsysnqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_final(shash, chap->response);
+out:
+	if (challenge != chap->c1)
+		kfree(challenge);
+	return ret;
+}
+
+static int nvme_auth_dhchap_setup_ctrl_response(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
+	u8 *ctrl_response;
+	u8 buf[4], *challenge = chap->c2;
+	int ret;
+
+	ctrl_response = nvme_auth_transform_key(ctrl->ctrl_key,
+				ctrl->opts->subsysnqn);
+	if (IS_ERR(ctrl_response)) {
+		ret = PTR_ERR(ctrl_response);
+		return ret;
+	}
+	ret = crypto_shash_setkey(chap->shash_tfm,
+			ctrl_response, ctrl->ctrl_key->len);
+	if (ret) {
+		dev_warn(ctrl->device, "qid %d: failed to set key, error %d\n",
+			 chap->qid, ret);
+		goto out;
+	}
+
+	dev_dbg(ctrl->device, "%s: qid %d ctrl response seq %u transaction %d\n",
+		__func__, chap->qid, chap->s2, chap->transaction);
+	dev_dbg(ctrl->device, "%s: qid %d challenge %*ph\n",
+		__func__, chap->qid, (int)chap->hash_len, challenge);
+	dev_dbg(ctrl->device, "%s: qid %d subsysnqn %s\n",
+		__func__, chap->qid, ctrl->opts->subsysnqn);
+	dev_dbg(ctrl->device, "%s: qid %d hostnqn %s\n",
+		__func__, chap->qid, ctrl->opts->host->nqn);
+	shash->tfm = chap->shash_tfm;
+	ret = crypto_shash_init(shash);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, challenge, chap->hash_len);
+	if (ret)
+		goto out;
+	put_unaligned_le32(chap->s2, buf);
+	ret = crypto_shash_update(shash, buf, 4);
+	if (ret)
+		goto out;
+	put_unaligned_le16(chap->transaction, buf);
+	ret = crypto_shash_update(shash, buf, 2);
+	if (ret)
+		goto out;
+	memset(buf, 0, 4);
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, "Controller", 10);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
+				  strlen(ctrl->opts->subsysnqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
+				  strlen(ctrl->opts->host->nqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_final(shash, chap->response);
+out:
+	if (challenge != chap->c2)
+		kfree(challenge);
+	kfree(ctrl_response);
+	return ret;
+}
+
+static void __nvme_auth_reset(struct nvme_dhchap_queue_context *chap)
+{
+	chap->status = 0;
+	chap->error = 0;
+	chap->s1 = 0;
+	chap->s2 = 0;
+	chap->transaction = 0;
+	memset(chap->c1, 0, sizeof(chap->c1));
+	memset(chap->c2, 0, sizeof(chap->c2));
+}
+
+static void __nvme_auth_free(struct nvme_dhchap_queue_context *chap)
+{
+	__nvme_auth_reset(chap);
+	if (chap->shash_tfm)
+		crypto_free_shash(chap->shash_tfm);
+	kfree_sensitive(chap->host_response);
+	kfree(chap->buf);
+	kfree(chap);
+}
+
+static void __nvme_auth_work(struct work_struct *work)
+{
+	struct nvme_dhchap_queue_context *chap =
+		container_of(work, struct nvme_dhchap_queue_context, auth_work);
+	struct nvme_ctrl *ctrl = chap->ctrl;
+	size_t tl;
+	int ret = 0;
+
+	chap->transaction = ctrl->transaction++;
+
+	/* DH-HMAC-CHAP Step 1: send negotiate */
+	dev_dbg(ctrl->device, "%s: qid %d send negotiate\n",
+		__func__, chap->qid);
+	ret = nvme_auth_set_dhchap_negotiate_data(ctrl, chap);
+	if (ret < 0) {
+		chap->error = ret;
+		return;
+	}
+	tl = ret;
+	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, tl, true);
+	if (ret) {
+		chap->error = ret;
+		return;
+	}
+
+	/* DH-HMAC-CHAP Step 2: receive challenge */
+	dev_dbg(ctrl->device, "%s: qid %d receive challenge\n",
+		__func__, chap->qid);
+
+	memset(chap->buf, 0, chap->buf_size);
+	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, chap->buf_size, false);
+	if (ret) {
+		dev_warn(ctrl->device,
+			 "qid %d failed to receive challenge, %s %d\n",
+			 chap->qid, ret < 0 ? "error" : "nvme status", ret);
+		chap->error = ret;
+		return;
+	}
+	ret = nvme_auth_receive_validate(ctrl, chap->qid, chap->buf, chap->transaction,
+					 NVME_AUTH_DHCHAP_MESSAGE_CHALLENGE);
+	if (ret) {
+		chap->status = ret;
+		chap->error = NVME_SC_AUTH_REQUIRED;
+		return;
+	}
+
+	ret = nvme_auth_process_dhchap_challenge(ctrl, chap);
+	if (ret) {
+		/* Invalid challenge parameters */
+		goto fail2;
+	}
+
+	dev_dbg(ctrl->device, "%s: qid %d host response\n",
+		__func__, chap->qid);
+	ret = nvme_auth_dhchap_setup_host_response(ctrl, chap);
+	if (ret)
+		goto fail2;
+
+	/* DH-HMAC-CHAP Step 3: send reply */
+	dev_dbg(ctrl->device, "%s: qid %d send reply\n",
+		__func__, chap->qid);
+	ret = nvme_auth_set_dhchap_reply_data(ctrl, chap);
+	if (ret < 0)
+		goto fail2;
+
+	tl = ret;
+	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, tl, true);
+	if (ret)
+		goto fail2;
+
+	/* DH-HMAC-CHAP Step 4: receive success1 */
+	dev_dbg(ctrl->device, "%s: qid %d receive success1\n",
+		__func__, chap->qid);
+
+	memset(chap->buf, 0, chap->buf_size);
+	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, chap->buf_size, false);
+	if (ret) {
+		dev_warn(ctrl->device,
+			 "qid %d failed to receive success1, %s %d\n",
+			 chap->qid, ret < 0 ? "error" : "nvme status", ret);
+		chap->error = ret;
+		return;
+	}
+	ret = nvme_auth_receive_validate(ctrl, chap->qid,
+					 chap->buf, chap->transaction,
+					 NVME_AUTH_DHCHAP_MESSAGE_SUCCESS1);
+	if (ret) {
+		chap->status = ret;
+		chap->error = NVME_SC_AUTH_REQUIRED;
+		return;
+	}
+
+	if (ctrl->ctrl_key) {
+		dev_dbg(ctrl->device,
+			"%s: qid %d controller response\n",
+			__func__, chap->qid);
+		ret = nvme_auth_dhchap_setup_ctrl_response(ctrl, chap);
+		if (ret) {
+			chap->error = ret;
+			goto fail2;
+		}
+	}
+
+	ret = nvme_auth_process_dhchap_success1(ctrl, chap);
+	if (ret) {
+		/* Controller authentication failed */
+		chap->error = NVME_SC_AUTH_REQUIRED;
+		goto fail2;
+	}
+
+	if (ctrl->ctrl_key) {
+		/* DH-HMAC-CHAP Step 5: send success2 */
+		dev_dbg(ctrl->device, "%s: qid %d send success2\n",
+			__func__, chap->qid);
+		tl = nvme_auth_set_dhchap_success2_data(ctrl, chap);
+		ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, tl, true);
+	}
+	if (!ret) {
+		chap->error = 0;
+		return;
+	}
+
+fail2:
+	dev_dbg(ctrl->device, "%s: qid %d send failure2, status %x\n",
+		__func__, chap->qid, chap->status);
+	tl = nvme_auth_set_dhchap_failure2_data(ctrl, chap);
+	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, tl, true);
+	/*
+	 * only update error if send failure2 failed and no other
+	 * error had been set during authentication.
+	 */
+	if (ret && !chap->error)
+		chap->error = ret;
+}
+
+int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid)
+{
+	struct nvme_dhchap_queue_context *chap;
+
+	if (!ctrl->host_key) {
+		dev_warn(ctrl->device, "qid %d: no key\n", qid);
+		return -ENOKEY;
+	}
+
+	if (ctrl->opts->dhchap_ctrl_secret && !ctrl->ctrl_key) {
+		dev_warn(ctrl->device, "qid %d: invalid ctrl key\n", qid);
+		return -ENOKEY;
+	}
+
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	/* Check if the context is already queued */
+	list_for_each_entry(chap, &ctrl->dhchap_auth_list, entry) {
+		WARN_ON(!chap->buf);
+		if (chap->qid == qid) {
+			dev_dbg(ctrl->device, "qid %d: re-using context\n", qid);
+			mutex_unlock(&ctrl->dhchap_auth_mutex);
+			flush_work(&chap->auth_work);
+			__nvme_auth_reset(chap);
+			queue_work(nvme_wq, &chap->auth_work);
+			return 0;
+		}
+	}
+	chap = kzalloc(sizeof(*chap), GFP_KERNEL);
+	if (!chap) {
+		mutex_unlock(&ctrl->dhchap_auth_mutex);
+		return -ENOMEM;
+	}
+	chap->qid = (qid == NVME_QID_ANY) ? 0 : qid;
+	chap->ctrl = ctrl;
+
+	/*
+	 * Allocate a large enough buffer for the entire negotiation:
+	 * 4k should be enough to ffdhe8192.
+	 */
+	chap->buf_size = 4096;
+	chap->buf = kzalloc(chap->buf_size, GFP_KERNEL);
+	if (!chap->buf) {
+		mutex_unlock(&ctrl->dhchap_auth_mutex);
+		kfree(chap);
+		return -ENOMEM;
+	}
+
+	INIT_WORK(&chap->auth_work, __nvme_auth_work);
+	list_add(&chap->entry, &ctrl->dhchap_auth_list);
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+	queue_work(nvme_wq, &chap->auth_work);
+	return 0;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_negotiate);
+
+int nvme_auth_wait(struct nvme_ctrl *ctrl, int qid)
+{
+	struct nvme_dhchap_queue_context *chap;
+	int ret;
+
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	list_for_each_entry(chap, &ctrl->dhchap_auth_list, entry) {
+		if (chap->qid != qid)
+			continue;
+		mutex_unlock(&ctrl->dhchap_auth_mutex);
+		flush_work(&chap->auth_work);
+		ret = chap->error;
+		return ret;
+	}
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+	return -ENXIO;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_wait);
+
+void nvme_auth_reset(struct nvme_ctrl *ctrl)
+{
+	struct nvme_dhchap_queue_context *chap;
+
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	list_for_each_entry(chap, &ctrl->dhchap_auth_list, entry) {
+		mutex_unlock(&ctrl->dhchap_auth_mutex);
+		flush_work(&chap->auth_work);
+		__nvme_auth_reset(chap);
+	}
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_reset);
+
+static void nvme_dhchap_auth_work(struct work_struct *work)
+{
+	struct nvme_ctrl *ctrl =
+		container_of(work, struct nvme_ctrl, dhchap_auth_work);
+	int ret, q;
+
+	/* Authenticate admin queue first */
+	ret = nvme_auth_negotiate(ctrl, 0);
+	if (ret) {
+		dev_warn(ctrl->device,
+			 "qid 0: error %d setting up authentication\n", ret);
+		return;
+	}
+	ret = nvme_auth_wait(ctrl, 0);
+	if (ret) {
+		dev_warn(ctrl->device,
+			 "qid 0: authentication failed\n");
+		return;
+	}
+
+	for (q = 1; q < ctrl->queue_count; q++) {
+		ret = nvme_auth_negotiate(ctrl, q);
+		if (ret) {
+			dev_warn(ctrl->device,
+				 "qid %d: error %d setting up authentication\n",
+				 q, ret);
+			break;
+		}
+	}
+
+	/*
+	 * Failure is a soft-state; credentials remain valid until
+	 * the controller terminates the connection.
+	 */
+}
+
+void nvme_auth_init_ctrl(struct nvme_ctrl *ctrl)
+{
+	INIT_LIST_HEAD(&ctrl->dhchap_auth_list);
+	INIT_WORK(&ctrl->dhchap_auth_work, nvme_dhchap_auth_work);
+	mutex_init(&ctrl->dhchap_auth_mutex);
+	if (!ctrl->opts)
+		return;
+	nvme_auth_generate_key(ctrl->opts->dhchap_secret, &ctrl->host_key);
+	nvme_auth_generate_key(ctrl->opts->dhchap_ctrl_secret, &ctrl->ctrl_key);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_init_ctrl);
+
+void nvme_auth_stop(struct nvme_ctrl *ctrl)
+{
+	struct nvme_dhchap_queue_context *chap = NULL, *tmp;
+
+	cancel_work_sync(&ctrl->dhchap_auth_work);
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	list_for_each_entry_safe(chap, tmp, &ctrl->dhchap_auth_list, entry)
+		cancel_work_sync(&chap->auth_work);
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_stop);
+
+void nvme_auth_free(struct nvme_ctrl *ctrl)
+{
+	struct nvme_dhchap_queue_context *chap = NULL, *tmp;
+
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	list_for_each_entry_safe(chap, tmp, &ctrl->dhchap_auth_list, entry) {
+		list_del_init(&chap->entry);
+		flush_work(&chap->auth_work);
+		__nvme_auth_free(chap);
+	}
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+	if (ctrl->host_key) {
+		nvme_auth_free_key(ctrl->host_key);
+		ctrl->host_key = NULL;
+	}
+	if (ctrl->ctrl_key) {
+		nvme_auth_free_key(ctrl->ctrl_key);
+		ctrl->ctrl_key = NULL;
+	}
+}
+EXPORT_SYMBOL_GPL(nvme_auth_free);
diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 57480075550c..9071a9798996 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -24,6 +24,7 @@
 
 #include "nvme.h"
 #include "fabrics.h"
+#include <linux/nvme-auth.h>
 
 #define CREATE_TRACE_POINTS
 #include "trace.h"
@@ -330,6 +331,7 @@ enum nvme_disposition {
 	COMPLETE,
 	RETRY,
 	FAILOVER,
+	AUTHENTICATE,
 };
 
 static inline enum nvme_disposition nvme_decide_disposition(struct request *req)
@@ -337,6 +339,9 @@ static inline enum nvme_disposition nvme_decide_disposition(struct request *req)
 	if (likely(nvme_req(req)->status == 0))
 		return COMPLETE;
 
+	if ((nvme_req(req)->status & 0x7ff) == NVME_SC_AUTH_REQUIRED)
+		return AUTHENTICATE;
+
 	if (blk_noretry_request(req) ||
 	    (nvme_req(req)->status & NVME_SC_DNR) ||
 	    nvme_req(req)->retries >= nvme_max_retries)
@@ -375,11 +380,13 @@ static inline void nvme_end_req(struct request *req)
 
 void nvme_complete_rq(struct request *req)
 {
+	struct nvme_ctrl *ctrl = nvme_req(req)->ctrl;
+
 	trace_nvme_complete_rq(req);
 	nvme_cleanup_cmd(req);
 
-	if (nvme_req(req)->ctrl->kas)
-		nvme_req(req)->ctrl->comp_seen = true;
+	if (ctrl->kas)
+		ctrl->comp_seen = true;
 
 	switch (nvme_decide_disposition(req)) {
 	case COMPLETE:
@@ -391,6 +398,14 @@ void nvme_complete_rq(struct request *req)
 	case FAILOVER:
 		nvme_failover_req(req);
 		return;
+	case AUTHENTICATE:
+#ifdef CONFIG_NVME_AUTH
+		queue_work(nvme_wq, &ctrl->dhchap_auth_work);
+		nvme_retry_req(req);
+#else
+		nvme_end_req(req);
+#endif
+		return;
 	}
 }
 EXPORT_SYMBOL_GPL(nvme_complete_rq);
@@ -702,7 +717,9 @@ bool __nvme_check_ready(struct nvme_ctrl *ctrl, struct request *rq,
 		switch (ctrl->state) {
 		case NVME_CTRL_CONNECTING:
 			if (blk_rq_is_passthrough(rq) && nvme_is_fabrics(req->cmd) &&
-			    req->cmd->fabrics.fctype == nvme_fabrics_type_connect)
+			    (req->cmd->fabrics.fctype == nvme_fabrics_type_connect ||
+			     req->cmd->fabrics.fctype == nvme_fabrics_type_auth_send ||
+			     req->cmd->fabrics.fctype == nvme_fabrics_type_auth_receive))
 				return true;
 			break;
 		default:
@@ -3595,6 +3612,108 @@ static ssize_t dctype_show(struct device *dev,
 }
 static DEVICE_ATTR_RO(dctype);
 
+#ifdef CONFIG_NVME_AUTH
+static ssize_t nvme_ctrl_dhchap_secret_show(struct device *dev,
+		struct device_attribute *attr, char *buf)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	struct nvmf_ctrl_options *opts = ctrl->opts;
+
+	if (!opts->dhchap_secret)
+		return sysfs_emit(buf, "none\n");
+	return sysfs_emit(buf, "%s\n", opts->dhchap_secret);
+}
+
+static ssize_t nvme_ctrl_dhchap_secret_store(struct device *dev,
+		struct device_attribute *attr, const char *buf, size_t count)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	struct nvmf_ctrl_options *opts = ctrl->opts;
+	char *dhchap_secret;
+
+	if (!ctrl->opts->dhchap_secret)
+		return -EINVAL;
+	if (count < 7)
+		return -EINVAL;
+	if (memcmp(buf, "DHHC-1:", 7))
+		return -EINVAL;
+
+	dhchap_secret = kzalloc(count + 1, GFP_KERNEL);
+	if (!dhchap_secret)
+		return -ENOMEM;
+	memcpy(dhchap_secret, buf, count);
+	nvme_auth_stop(ctrl);
+	if (strcmp(dhchap_secret, opts->dhchap_secret)) {
+		int ret;
+
+		ret = nvme_auth_generate_key(dhchap_secret, &ctrl->host_key);
+		if (ret)
+			return ret;
+		kfree(opts->dhchap_secret);
+		opts->dhchap_secret = dhchap_secret;
+		/* Key has changed; re-authentication with new key */
+		nvme_auth_reset(ctrl);
+	}
+	/* Start re-authentication */
+	dev_info(ctrl->device, "re-authenticating controller\n");
+	queue_work(nvme_wq, &ctrl->dhchap_auth_work);
+
+	return count;
+}
+static DEVICE_ATTR(dhchap_secret, S_IRUGO | S_IWUSR,
+	nvme_ctrl_dhchap_secret_show, nvme_ctrl_dhchap_secret_store);
+
+static ssize_t nvme_ctrl_dhchap_ctrl_secret_show(struct device *dev,
+		struct device_attribute *attr, char *buf)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	struct nvmf_ctrl_options *opts = ctrl->opts;
+
+	if (!opts->dhchap_ctrl_secret)
+		return sysfs_emit(buf, "none\n");
+	return sysfs_emit(buf, "%s\n", opts->dhchap_ctrl_secret);
+}
+
+static ssize_t nvme_ctrl_dhchap_ctrl_secret_store(struct device *dev,
+		struct device_attribute *attr, const char *buf, size_t count)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	struct nvmf_ctrl_options *opts = ctrl->opts;
+	char *dhchap_secret;
+
+	if (!ctrl->opts->dhchap_ctrl_secret)
+		return -EINVAL;
+	if (count < 7)
+		return -EINVAL;
+	if (memcmp(buf, "DHHC-1:", 7))
+		return -EINVAL;
+
+	dhchap_secret = kzalloc(count + 1, GFP_KERNEL);
+	if (!dhchap_secret)
+		return -ENOMEM;
+	memcpy(dhchap_secret, buf, count);
+	nvme_auth_stop(ctrl);
+	if (strcmp(dhchap_secret, opts->dhchap_ctrl_secret)) {
+		int ret;
+
+		ret = nvme_auth_generate_key(dhchap_secret, &ctrl->ctrl_key);
+		if (ret)
+			return ret;
+		kfree(opts->dhchap_ctrl_secret);
+		opts->dhchap_ctrl_secret = dhchap_secret;
+		/* Key has changed; re-authentication with new key */
+		nvme_auth_reset(ctrl);
+	}
+	/* Start re-authentication */
+	dev_info(ctrl->device, "re-authenticating controller\n");
+	queue_work(nvme_wq, &ctrl->dhchap_auth_work);
+
+	return count;
+}
+static DEVICE_ATTR(dhchap_ctrl_secret, S_IRUGO | S_IWUSR,
+	nvme_ctrl_dhchap_ctrl_secret_show, nvme_ctrl_dhchap_ctrl_secret_store);
+#endif
+
 static struct attribute *nvme_dev_attrs[] = {
 	&dev_attr_reset_controller.attr,
 	&dev_attr_rescan_controller.attr,
@@ -3618,6 +3737,10 @@ static struct attribute *nvme_dev_attrs[] = {
 	&dev_attr_kato.attr,
 	&dev_attr_cntrltype.attr,
 	&dev_attr_dctype.attr,
+#ifdef CONFIG_NVME_AUTH
+	&dev_attr_dhchap_secret.attr,
+	&dev_attr_dhchap_ctrl_secret.attr,
+#endif
 	NULL
 };
 
@@ -3641,6 +3764,12 @@ static umode_t nvme_dev_attrs_are_visible(struct kobject *kobj,
 		return 0;
 	if (a == &dev_attr_fast_io_fail_tmo.attr && !ctrl->opts)
 		return 0;
+#ifdef CONFIG_NVME_AUTH
+	if (a == &dev_attr_dhchap_secret.attr && !ctrl->opts)
+		return 0;
+	if (a == &dev_attr_dhchap_ctrl_secret.attr && !ctrl->opts)
+		return 0;
+#endif
 
 	return a->mode;
 }
@@ -4535,8 +4664,10 @@ static void nvme_handle_aen_notice(struct nvme_ctrl *ctrl, u32 result)
 		 * recovery actions from interfering with the controller's
 		 * firmware activation.
 		 */
-		if (nvme_change_ctrl_state(ctrl, NVME_CTRL_RESETTING))
+		if (nvme_change_ctrl_state(ctrl, NVME_CTRL_RESETTING)) {
+			nvme_auth_stop(ctrl);
 			queue_work(nvme_wq, &ctrl->fw_act_work);
+		}
 		break;
 #ifdef CONFIG_NVME_MULTIPATH
 	case NVME_AER_NOTICE_ANA:
@@ -4600,6 +4731,7 @@ EXPORT_SYMBOL_GPL(nvme_complete_async_event);
 void nvme_stop_ctrl(struct nvme_ctrl *ctrl)
 {
 	nvme_mpath_stop(ctrl);
+	nvme_auth_stop(ctrl);
 	nvme_stop_keep_alive(ctrl);
 	nvme_stop_failfast_work(ctrl);
 	flush_work(&ctrl->async_event_work);
@@ -4657,6 +4789,8 @@ static void nvme_free_ctrl(struct device *dev)
 
 	nvme_free_cels(ctrl);
 	nvme_mpath_uninit(ctrl);
+	nvme_auth_stop(ctrl);
+	nvme_auth_free(ctrl);
 	__free_page(ctrl->discard_page);
 
 	if (subsys) {
@@ -4747,6 +4881,7 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev,
 
 	nvme_fault_inject_init(&ctrl->fault_inject, dev_name(ctrl->device));
 	nvme_mpath_init_ctrl(ctrl);
+	nvme_auth_init_ctrl(ctrl);
 
 	return 0;
 out_free_name:
diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index e4b1520862d8..5207a2348257 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -369,6 +369,7 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)
 	union nvme_result res;
 	struct nvmf_connect_data *data;
 	int ret;
+	u32 result;
 
 	cmd.connect.opcode = nvme_fabrics_command;
 	cmd.connect.fctype = nvme_fabrics_type_connect;
@@ -401,8 +402,25 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)
 		goto out_free_data;
 	}
 
-	ctrl->cntlid = le16_to_cpu(res.u16);
-
+	result = le32_to_cpu(res.u32);
+	ctrl->cntlid = result & 0xFFFF;
+	if ((result >> 16) & 0x3) {
+		/* Authentication required */
+		ret = nvme_auth_negotiate(ctrl, 0);
+		if (ret) {
+			dev_warn(ctrl->device,
+				 "qid 0: authentication setup failed\n");
+			ret = NVME_SC_AUTH_REQUIRED;
+			goto out_free_data;
+		}
+		ret = nvme_auth_wait(ctrl, 0);
+		if (ret)
+			dev_warn(ctrl->device,
+				 "qid 0: authentication failed\n");
+		else
+			dev_info(ctrl->device,
+				 "qid 0: authenticated\n");
+	}
 out_free_data:
 	kfree(data);
 	return ret;
@@ -435,6 +453,7 @@ int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, u16 qid)
 	struct nvmf_connect_data *data;
 	union nvme_result res;
 	int ret;
+	u32 result;
 
 	cmd.connect.opcode = nvme_fabrics_command;
 	cmd.connect.fctype = nvme_fabrics_type_connect;
@@ -460,6 +479,21 @@ int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, u16 qid)
 		nvmf_log_connect_error(ctrl, ret, le32_to_cpu(res.u32),
 				       &cmd, data);
 	}
+	result = le32_to_cpu(res.u32);
+	if ((result >> 16) & 2) {
+		/* Authentication required */
+		ret = nvme_auth_negotiate(ctrl, qid);
+		if (ret) {
+			dev_warn(ctrl->device,
+				 "qid %d: authentication setup failed\n", qid);
+			ret = NVME_SC_AUTH_REQUIRED;
+		} else {
+			ret = nvme_auth_wait(ctrl, qid);
+			if (ret)
+				dev_warn(ctrl->device,
+					 "qid %u: authentication failed\n", qid);
+		}
+	}
 	kfree(data);
 	return ret;
 }
@@ -552,6 +586,8 @@ static const match_table_t opt_tokens = {
 	{ NVMF_OPT_TOS,			"tos=%d"		},
 	{ NVMF_OPT_FAIL_FAST_TMO,	"fast_io_fail_tmo=%d"	},
 	{ NVMF_OPT_DISCOVERY,		"discovery"		},
+	{ NVMF_OPT_DHCHAP_SECRET,	"dhchap_secret=%s"	},
+	{ NVMF_OPT_DHCHAP_CTRL_SECRET,	"dhchap_ctrl_secret=%s"	},
 	{ NVMF_OPT_ERR,			NULL			}
 };
 
@@ -833,6 +869,34 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
 		case NVMF_OPT_DISCOVERY:
 			opts->discovery_nqn = true;
 			break;
+		case NVMF_OPT_DHCHAP_SECRET:
+			p = match_strdup(args);
+			if (!p) {
+				ret = -ENOMEM;
+				goto out;
+			}
+			if (strlen(p) < 11 || strncmp(p, "DHHC-1:", 7)) {
+				pr_err("Invalid DH-CHAP secret %s\n", p);
+				ret = -EINVAL;
+				goto out;
+			}
+			kfree(opts->dhchap_secret);
+			opts->dhchap_secret = p;
+			break;
+		case NVMF_OPT_DHCHAP_CTRL_SECRET:
+			p = match_strdup(args);
+			if (!p) {
+				ret = -ENOMEM;
+				goto out;
+			}
+			if (strlen(p) < 11 || strncmp(p, "DHHC-1:", 7)) {
+				pr_err("Invalid DH-CHAP secret %s\n", p);
+				ret = -EINVAL;
+				goto out;
+			}
+			kfree(opts->dhchap_ctrl_secret);
+			opts->dhchap_ctrl_secret = p;
+			break;
 		default:
 			pr_warn("unknown parameter or missing value '%s' in ctrl creation request\n",
 				p);
@@ -951,6 +1015,8 @@ void nvmf_free_options(struct nvmf_ctrl_options *opts)
 	kfree(opts->subsysnqn);
 	kfree(opts->host_traddr);
 	kfree(opts->host_iface);
+	kfree(opts->dhchap_secret);
+	kfree(opts->dhchap_ctrl_secret);
 	kfree(opts);
 }
 EXPORT_SYMBOL_GPL(nvmf_free_options);
@@ -960,7 +1026,8 @@ EXPORT_SYMBOL_GPL(nvmf_free_options);
 				 NVMF_OPT_KATO | NVMF_OPT_HOSTNQN | \
 				 NVMF_OPT_HOST_ID | NVMF_OPT_DUP_CONNECT |\
 				 NVMF_OPT_DISABLE_SQFLOW | NVMF_OPT_DISCOVERY |\
-				 NVMF_OPT_FAIL_FAST_TMO)
+				 NVMF_OPT_FAIL_FAST_TMO | NVMF_OPT_DHCHAP_SECRET |\
+				 NVMF_OPT_DHCHAP_CTRL_SECRET)
 
 static struct nvme_ctrl *
 nvmf_create_ctrl(struct device *dev, const char *buf)
@@ -1196,7 +1263,14 @@ static void __exit nvmf_exit(void)
 	BUILD_BUG_ON(sizeof(struct nvmf_connect_command) != 64);
 	BUILD_BUG_ON(sizeof(struct nvmf_property_get_command) != 64);
 	BUILD_BUG_ON(sizeof(struct nvmf_property_set_command) != 64);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_send_command) != 64);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_receive_command) != 64);
 	BUILD_BUG_ON(sizeof(struct nvmf_connect_data) != 1024);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_negotiate_data) != 8);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_challenge_data) != 16);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_reply_data) != 16);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_success1_data) != 16);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_success2_data) != 16);
 }
 
 MODULE_LICENSE("GPL v2");
diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h
index 46d6e194ac2b..a6e22116e139 100644
--- a/drivers/nvme/host/fabrics.h
+++ b/drivers/nvme/host/fabrics.h
@@ -68,6 +68,8 @@ enum {
 	NVMF_OPT_FAIL_FAST_TMO	= 1 << 20,
 	NVMF_OPT_HOST_IFACE	= 1 << 21,
 	NVMF_OPT_DISCOVERY	= 1 << 22,
+	NVMF_OPT_DHCHAP_SECRET	= 1 << 23,
+	NVMF_OPT_DHCHAP_CTRL_SECRET = 1 << 24,
 };
 
 /**
@@ -97,6 +99,9 @@ enum {
  * @max_reconnects: maximum number of allowed reconnect attempts before removing
  *              the controller, (-1) means reconnect forever, zero means remove
  *              immediately;
+ * @dhchap_secret: DH-HMAC-CHAP secret
+ * @dhchap_ctrl_secret: DH-HMAC-CHAP controller secret for bi-directional
+ *              authentication
  * @disable_sqflow: disable controller sq flow control
  * @hdr_digest: generate/verify header digest (TCP)
  * @data_digest: generate/verify data digest (TCP)
@@ -121,6 +126,8 @@ struct nvmf_ctrl_options {
 	unsigned int		kato;
 	struct nvmf_host	*host;
 	int			max_reconnects;
+	char			*dhchap_secret;
+	char			*dhchap_ctrl_secret;
 	bool			disable_sqflow;
 	bool			hdr_digest;
 	bool			data_digest;
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index e4612dd0b420..e9350bf7b2d1 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -328,6 +328,15 @@ struct nvme_ctrl {
 	struct work_struct ana_work;
 #endif
 
+#ifdef CONFIG_NVME_AUTH
+	struct work_struct dhchap_auth_work;
+	struct list_head dhchap_auth_list;
+	struct mutex dhchap_auth_mutex;
+	struct nvme_dhchap_key *host_key;
+	struct nvme_dhchap_key *ctrl_key;
+	u16 transaction;
+#endif
+
 	/* Power saving configuration */
 	u64 ps_max_latency_us;
 	bool apst_enabled;
@@ -991,6 +1000,27 @@ static inline bool nvme_ctrl_sgl_supported(struct nvme_ctrl *ctrl)
 	return ctrl->sgls & ((1 << 0) | (1 << 1));
 }
 
+#ifdef CONFIG_NVME_AUTH
+void nvme_auth_init_ctrl(struct nvme_ctrl *ctrl);
+void nvme_auth_stop(struct nvme_ctrl *ctrl);
+int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid);
+int nvme_auth_wait(struct nvme_ctrl *ctrl, int qid);
+void nvme_auth_reset(struct nvme_ctrl *ctrl);
+void nvme_auth_free(struct nvme_ctrl *ctrl);
+#else
+static inline void nvme_auth_init_ctrl(struct nvme_ctrl *ctrl) {};
+static inline void nvme_auth_stop(struct nvme_ctrl *ctrl) {};
+static inline int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid)
+{
+	return -EPROTONOSUPPORT;
+}
+static inline int nvme_auth_wait(struct nvme_ctrl *ctrl, int qid)
+{
+	return NVME_SC_AUTH_REQUIRED;
+}
+static inline void nvme_auth_free(struct nvme_ctrl *ctrl) {};
+#endif
+
 u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 			 u8 opcode);
 int nvme_execute_passthru_rq(struct request *rq);
diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index f2a5e1ea508a..84ce3347d158 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -1197,6 +1197,7 @@ static void nvme_rdma_error_recovery_work(struct work_struct *work)
 	struct nvme_rdma_ctrl *ctrl = container_of(work,
 			struct nvme_rdma_ctrl, err_work);
 
+	nvme_auth_stop(&ctrl->ctrl);
 	nvme_stop_keep_alive(&ctrl->ctrl);
 	flush_work(&ctrl->ctrl.async_event_work);
 	nvme_rdma_teardown_io_queues(ctrl, false);
diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index bb67538d241b..a7848e430a5c 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -2174,6 +2174,7 @@ static void nvme_tcp_error_recovery_work(struct work_struct *work)
 				struct nvme_tcp_ctrl, err_work);
 	struct nvme_ctrl *ctrl = &tcp_ctrl->ctrl;
 
+	nvme_auth_stop(ctrl);
 	nvme_stop_keep_alive(ctrl);
 	flush_work(&ctrl->async_event_work);
 	nvme_tcp_teardown_io_queues(ctrl, false);
diff --git a/drivers/nvme/host/trace.c b/drivers/nvme/host/trace.c
index 2a89c5aa0790..1c36fcedea20 100644
--- a/drivers/nvme/host/trace.c
+++ b/drivers/nvme/host/trace.c
@@ -287,6 +287,34 @@ static const char *nvme_trace_fabrics_property_get(struct trace_seq *p, u8 *spc)
 	return ret;
 }
 
+static const char *nvme_trace_fabrics_auth_send(struct trace_seq *p, u8 *spc)
+{
+	const char *ret = trace_seq_buffer_ptr(p);
+	u8 spsp0 = spc[1];
+	u8 spsp1 = spc[2];
+	u8 secp = spc[3];
+	u32 tl = get_unaligned_le32(spc + 4);
+
+	trace_seq_printf(p, "spsp0=%02x, spsp1=%02x, secp=%02x, tl=%u",
+			 spsp0, spsp1, secp, tl);
+	trace_seq_putc(p, 0);
+	return ret;
+}
+
+static const char *nvme_trace_fabrics_auth_receive(struct trace_seq *p, u8 *spc)
+{
+	const char *ret = trace_seq_buffer_ptr(p);
+	u8 spsp0 = spc[1];
+	u8 spsp1 = spc[2];
+	u8 secp = spc[3];
+	u32 al = get_unaligned_le32(spc + 4);
+
+	trace_seq_printf(p, "spsp0=%02x, spsp1=%02x, secp=%02x, al=%u",
+			 spsp0, spsp1, secp, al);
+	trace_seq_putc(p, 0);
+	return ret;
+}
+
 static const char *nvme_trace_fabrics_common(struct trace_seq *p, u8 *spc)
 {
 	const char *ret = trace_seq_buffer_ptr(p);
@@ -306,6 +334,10 @@ const char *nvme_trace_parse_fabrics_cmd(struct trace_seq *p,
 		return nvme_trace_fabrics_connect(p, spc);
 	case nvme_fabrics_type_property_get:
 		return nvme_trace_fabrics_property_get(p, spc);
+	case nvme_fabrics_type_auth_send:
+		return nvme_trace_fabrics_auth_send(p, spc);
+	case nvme_fabrics_type_auth_receive:
+		return nvme_trace_fabrics_auth_receive(p, spc);
 	default:
 		return nvme_trace_fabrics_common(p, spc);
 	}
diff --git a/include/linux/nvme-auth.h b/include/linux/nvme-auth.h
new file mode 100644
index 000000000000..354456826221
--- /dev/null
+++ b/include/linux/nvme-auth.h
@@ -0,0 +1,33 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2021 Hannes Reinecke, SUSE Software Solutions
+ */
+
+#ifndef _NVME_AUTH_H
+#define _NVME_AUTH_H
+
+#include <crypto/kpp.h>
+
+struct nvme_dhchap_key {
+	u8 *key;
+	size_t len;
+	u8 hash;
+};
+
+u32 nvme_auth_get_seqnum(void);
+const char *nvme_auth_dhgroup_name(u8 dhgroup_id);
+const char *nvme_auth_dhgroup_kpp(u8 dhgroup_id);
+u8 nvme_auth_dhgroup_id(const char *dhgroup_name);
+
+const char *nvme_auth_hmac_name(u8 hmac_id);
+const char *nvme_auth_digest_name(u8 hmac_id);
+size_t nvme_auth_hmac_hash_len(u8 hmac_id);
+u8 nvme_auth_hmac_id(const char *hmac_name);
+
+struct nvme_dhchap_key *nvme_auth_extract_key(unsigned char *secret,
+					      u8 key_hash);
+void nvme_auth_free_key(struct nvme_dhchap_key *key);
+u8 *nvme_auth_transform_key(struct nvme_dhchap_key *key, char *nqn);
+int nvme_auth_generate_key(u8 *secret, struct nvme_dhchap_key **ret_key);
+
+#endif /* _NVME_AUTH_H */
-- 
2.29.2

Re: [PATCH 06/11] nvme: Implement In-Band authentication

[Sagi Grimberg]

On 6/22/22 13:26, Hannes Reinecke wrote:
> On 6/22/22 11:58, Sagi Grimberg wrote:
>>
>>
>> On 6/22/22 12:20, Hannes Reinecke wrote:
>>> On 6/22/22 11:09, Sagi Grimberg wrote:
>>>>
>>>>>>>>> Looks like if I pass a malformed ctrl key to nvme connect I am 
>>>>>>>>> able to
>>>>>>>>> crash the system:
>>>>>>>>
>>>>>>>> This was what I used in this:
>>>>>>>> $ nvme connect -a 192.168.123.1 -t tcp  -s 8009 -n testnqn1 -S 
>>>>>>>> "DHHC-1:00:QpxVGpctx5J+4SeW2MClUI8rfZO3WdP1llImvsPsx7e3TK+I:" -C 
>>>>>>>> "DHHC-1:00:Jc/My1o0qtLCWRp+sHhAVafdfaS7YQOMYhk9zSmlatobqB8C:"
>>>>>>>>
>>>>>>>> The dhchap_ctrl_key is the offending string...
>>>>>>>>
>>>>>>> Right. Should be fixed with the attached patch.
>>>>>>> Can you check?
>>>>>>
>>>>>> Yes, that works.
>>>>>
>>>>> So, to summarize: With this one and the tentative patch I've sent 
>>>>> earlier:
>>>>>
>>>>> diff --git a/drivers/nvme/host/auth.c b/drivers/nvme/host/auth.c
>>>>> index 53184ac76240..a03f41fa146e 100644
>>>>> --- a/drivers/nvme/host/auth.c
>>>>> +++ b/drivers/nvme/host/auth.c
>>>>> @@ -842,6 +842,10 @@ int nvme_auth_negotiate(struct nvme_ctrl 
>>>>> *ctrl, int qid)
>>>>>                  dev_warn(ctrl->device, "qid %d: no key\n", qid);
>>>>>                  return -ENOKEY;
>>>>>          }
>>>>> +       if (ctrl->opts->dhchap_ctrl_secret && !ctrl->ctrl_key) {
>>>>> +               dev_warn(ctrl->device, "qid %d: invalid ctrl 
>>>>> key\n", qid);
>>>>> +               return -ENOKEY;
>>>>> +       }
>>>>>
>>>>>          mutex_lock(&ctrl->dhchap_auth_mutex);
>>>>>          /* Check if the context is already queued */
>>>>>
>>>>>
>>>>>
>>>>> your issues are resolved?
>>>>> Just to clarify before I sent another round of patches ...
>>>>
>>>> No. I don't understand why this patch is needed. As I noted
>>>> earlier, when I pass a wrong ctrl key I fail to connect do
>>>> to authentication error. Also when I pass a malformed ctrl key.
>>>>
>>>> So what does this patch fix? Can you give me an example that should
>>>> have failed before and only with this patch it fails?
>>>>
>>> Reasoning is as follows:
>>> When an invalid controller key is passed via 
>>> 'ctrl->opts->dhchap_ctrl_secret' we'll end up with an empty 
>>> 'ctrl->ctrl_key' in nvme_auth_init_ctrl() (which is what caused the 
>>> original oops).
>>> But the negotiation will still start, and, seeing that 
>>> 'ctrl->ctrl_key' is empty, will _not_ attempt controller authentication.
>>> But as a controller key is passed from userland we should have 
>>> attempted it, and hence an error is in order.
>>
>> Yes, I see it now.
>> And the patch seems to work.
>>
>>>> And for the other issue, please read it again, I think it is related
>>>> to nvme-cli and not this patchset.
>>>
>>> Hmm. Okay, I'll check, and possibly have to create another blktest 
>>> for this ...
>>
>> Looks like a libnvme fix makes the behavior go away:
>> -- 
>> diff --git a/src/nvme/fabrics.c b/src/nvme/fabrics.c
>> index 53199a29e264..2dd863fb8b4c 100644
>> --- a/src/nvme/fabrics.c
>> +++ b/src/nvme/fabrics.c
>> @@ -596,8 +596,11 @@ int nvmf_add_ctrl(nvme_host_t h, nvme_ctrl_t c,
>>                                          nvme_ctrl_get_host_iface(c),
>>                                          nvme_ctrl_get_trsvcid(c),
>>                                          NULL);
>> -               if (fc)
>> +               if (fc) {
>>                          cfg = merge_config(c, nvme_ctrl_get_config(fc));
>> +                       if (fc->dhchap_key)
>> +                               nvme_ctrl_set_dhchap_key(c, 
>> fc->dhchap_key);
>> +               }
>>          }
>>
>>          nvme_ctrl_set_discovered(c, true);
>> -- 
>>
>> Looks like when we scan to see an existing controller in the config,
>> we find it, but merge_config does not touch the ctrl dhchap_key.
>>
>> Maybe it would be better to just add the dhchap_key to the ctrl cfg
>> and have merge_config merge it as well? Or we can live with dhchap_key
>> merged explicitly?
> 
> Hmm. That runs slightly afoul of the merge logic.
> 'merge_config' will overwrite any value _not_ being the default.

There is no ctrl dhchap_key default though...

> Consequently, if a non-default value is already set it won't be 
> overwritten.

Makes sense.

> Which means that if the 'cfg' structure already contained a controller 
> secret (which could've been used, say, for discovery) we won't overwrite 
> it in merge_config().

We probably shouldn't override it this way either if it was explicitly
set by the user no?

> Hence I fear we have to go with your suggestion.

Whatever you choose. Don't really care...

> 
> Will be preparing a pull request.
> 

Thanks

Re: [PATCH 06/11] nvme: Implement In-Band authentication

[Hannes Reinecke]

On 6/22/22 11:58, Sagi Grimberg wrote:
> 
> 
> On 6/22/22 12:20, Hannes Reinecke wrote:
>> On 6/22/22 11:09, Sagi Grimberg wrote:
>>>
>>>>>>>> Looks like if I pass a malformed ctrl key to nvme connect I am 
>>>>>>>> able to
>>>>>>>> crash the system:
>>>>>>>
>>>>>>> This was what I used in this:
>>>>>>> $ nvme connect -a 192.168.123.1 -t tcp  -s 8009 -n testnqn1 -S 
>>>>>>> "DHHC-1:00:QpxVGpctx5J+4SeW2MClUI8rfZO3WdP1llImvsPsx7e3TK+I:" -C 
>>>>>>> "DHHC-1:00:Jc/My1o0qtLCWRp+sHhAVafdfaS7YQOMYhk9zSmlatobqB8C:"
>>>>>>>
>>>>>>> The dhchap_ctrl_key is the offending string...
>>>>>>>
>>>>>> Right. Should be fixed with the attached patch.
>>>>>> Can you check?
>>>>>
>>>>> Yes, that works.
>>>>
>>>> So, to summarize: With this one and the tentative patch I've sent 
>>>> earlier:
>>>>
>>>> diff --git a/drivers/nvme/host/auth.c b/drivers/nvme/host/auth.c
>>>> index 53184ac76240..a03f41fa146e 100644
>>>> --- a/drivers/nvme/host/auth.c
>>>> +++ b/drivers/nvme/host/auth.c
>>>> @@ -842,6 +842,10 @@ int nvme_auth_negotiate(struct nvme_ctrl *ctrl, 
>>>> int qid)
>>>>                  dev_warn(ctrl->device, "qid %d: no key\n", qid);
>>>>                  return -ENOKEY;
>>>>          }
>>>> +       if (ctrl->opts->dhchap_ctrl_secret && !ctrl->ctrl_key) {
>>>> +               dev_warn(ctrl->device, "qid %d: invalid ctrl key\n", 
>>>> qid);
>>>> +               return -ENOKEY;
>>>> +       }
>>>>
>>>>          mutex_lock(&ctrl->dhchap_auth_mutex);
>>>>          /* Check if the context is already queued */
>>>>
>>>>
>>>>
>>>> your issues are resolved?
>>>> Just to clarify before I sent another round of patches ...
>>>
>>> No. I don't understand why this patch is needed. As I noted
>>> earlier, when I pass a wrong ctrl key I fail to connect do
>>> to authentication error. Also when I pass a malformed ctrl key.
>>>
>>> So what does this patch fix? Can you give me an example that should
>>> have failed before and only with this patch it fails?
>>>
>> Reasoning is as follows:
>> When an invalid controller key is passed via 
>> 'ctrl->opts->dhchap_ctrl_secret' we'll end up with an empty 
>> 'ctrl->ctrl_key' in nvme_auth_init_ctrl() (which is what caused the 
>> original oops).
>> But the negotiation will still start, and, seeing that 
>> 'ctrl->ctrl_key' is empty, will _not_ attempt controller authentication.
>> But as a controller key is passed from userland we should have 
>> attempted it, and hence an error is in order.
> 
> Yes, I see it now.
> And the patch seems to work.
> 
>>> And for the other issue, please read it again, I think it is related
>>> to nvme-cli and not this patchset.
>>
>> Hmm. Okay, I'll check, and possibly have to create another blktest for 
>> this ...
> 
> Looks like a libnvme fix makes the behavior go away:
> -- 
> diff --git a/src/nvme/fabrics.c b/src/nvme/fabrics.c
> index 53199a29e264..2dd863fb8b4c 100644
> --- a/src/nvme/fabrics.c
> +++ b/src/nvme/fabrics.c
> @@ -596,8 +596,11 @@ int nvmf_add_ctrl(nvme_host_t h, nvme_ctrl_t c,
>                                          nvme_ctrl_get_host_iface(c),
>                                          nvme_ctrl_get_trsvcid(c),
>                                          NULL);
> -               if (fc)
> +               if (fc) {
>                          cfg = merge_config(c, nvme_ctrl_get_config(fc));
> +                       if (fc->dhchap_key)
> +                               nvme_ctrl_set_dhchap_key(c, 
> fc->dhchap_key);
> +               }
>          }
> 
>          nvme_ctrl_set_discovered(c, true);
> -- 
> 
> Looks like when we scan to see an existing controller in the config,
> we find it, but merge_config does not touch the ctrl dhchap_key.
> 
> Maybe it would be better to just add the dhchap_key to the ctrl cfg
> and have merge_config merge it as well? Or we can live with dhchap_key
> merged explicitly?

Hmm. That runs slightly afoul of the merge logic.
'merge_config' will overwrite any value _not_ being the default.
Consequently, if a non-default value is already set it won't be overwritten.
Which means that if the 'cfg' structure already contained a controller 
secret (which could've been used, say, for discovery) we won't overwrite 
it in merge_config().
Hence I fear we have to go with your suggestion.

Will be preparing a pull request.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		           Kernel Storage Architect
hare@suse.de			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), GF: Felix Imendörffer

Re: [PATCH 06/11] nvme: Implement In-Band authentication

[Sagi Grimberg]

On 6/22/22 12:20, Hannes Reinecke wrote:
> On 6/22/22 11:09, Sagi Grimberg wrote:
>>
>>>>>>> Looks like if I pass a malformed ctrl key to nvme connect I am 
>>>>>>> able to
>>>>>>> crash the system:
>>>>>>
>>>>>> This was what I used in this:
>>>>>> $ nvme connect -a 192.168.123.1 -t tcp  -s 8009 -n testnqn1 -S 
>>>>>> "DHHC-1:00:QpxVGpctx5J+4SeW2MClUI8rfZO3WdP1llImvsPsx7e3TK+I:" -C 
>>>>>> "DHHC-1:00:Jc/My1o0qtLCWRp+sHhAVafdfaS7YQOMYhk9zSmlatobqB8C:"
>>>>>>
>>>>>> The dhchap_ctrl_key is the offending string...
>>>>>>
>>>>> Right. Should be fixed with the attached patch.
>>>>> Can you check?
>>>>
>>>> Yes, that works.
>>>
>>> So, to summarize: With this one and the tentative patch I've sent 
>>> earlier:
>>>
>>> diff --git a/drivers/nvme/host/auth.c b/drivers/nvme/host/auth.c
>>> index 53184ac76240..a03f41fa146e 100644
>>> --- a/drivers/nvme/host/auth.c
>>> +++ b/drivers/nvme/host/auth.c
>>> @@ -842,6 +842,10 @@ int nvme_auth_negotiate(struct nvme_ctrl *ctrl, 
>>> int qid)
>>>                  dev_warn(ctrl->device, "qid %d: no key\n", qid);
>>>                  return -ENOKEY;
>>>          }
>>> +       if (ctrl->opts->dhchap_ctrl_secret && !ctrl->ctrl_key) {
>>> +               dev_warn(ctrl->device, "qid %d: invalid ctrl key\n", 
>>> qid);
>>> +               return -ENOKEY;
>>> +       }
>>>
>>>          mutex_lock(&ctrl->dhchap_auth_mutex);
>>>          /* Check if the context is already queued */
>>>
>>>
>>>
>>> your issues are resolved?
>>> Just to clarify before I sent another round of patches ...
>>
>> No. I don't understand why this patch is needed. As I noted
>> earlier, when I pass a wrong ctrl key I fail to connect do
>> to authentication error. Also when I pass a malformed ctrl key.
>>
>> So what does this patch fix? Can you give me an example that should
>> have failed before and only with this patch it fails?
>>
> Reasoning is as follows:
> When an invalid controller key is passed via 
> 'ctrl->opts->dhchap_ctrl_secret' we'll end up with an empty 
> 'ctrl->ctrl_key' in nvme_auth_init_ctrl() (which is what caused the 
> original oops).
> But the negotiation will still start, and, seeing that 'ctrl->ctrl_key' 
> is empty, will _not_ attempt controller authentication.
> But as a controller key is passed from userland we should have attempted 
> it, and hence an error is in order.

Yes, I see it now.
And the patch seems to work.

>> And for the other issue, please read it again, I think it is related
>> to nvme-cli and not this patchset.
> 
> Hmm. Okay, I'll check, and possibly have to create another blktest for 
> this ...

Looks like a libnvme fix makes the behavior go away:
--
diff --git a/src/nvme/fabrics.c b/src/nvme/fabrics.c
index 53199a29e264..2dd863fb8b4c 100644
--- a/src/nvme/fabrics.c
+++ b/src/nvme/fabrics.c
@@ -596,8 +596,11 @@ int nvmf_add_ctrl(nvme_host_t h, nvme_ctrl_t c,
                                         nvme_ctrl_get_host_iface(c),
                                         nvme_ctrl_get_trsvcid(c),
                                         NULL);
-               if (fc)
+               if (fc) {
                         cfg = merge_config(c, nvme_ctrl_get_config(fc));
+                       if (fc->dhchap_key)
+                               nvme_ctrl_set_dhchap_key(c, fc->dhchap_key);
+               }
         }

         nvme_ctrl_set_discovered(c, true);
--

Looks like when we scan to see an existing controller in the config,
we find it, but merge_config does not touch the ctrl dhchap_key.

Maybe it would be better to just add the dhchap_key to the ctrl cfg
and have merge_config merge it as well? Or we can live with dhchap_key
merged explicitly?

Re: [PATCH 06/11] nvme: Implement In-Band authentication

[Hannes Reinecke]

On 6/22/22 11:09, Sagi Grimberg wrote:
> 
>>>>>> Looks like if I pass a malformed ctrl key to nvme connect I am 
>>>>>> able to
>>>>>> crash the system:
>>>>>
>>>>> This was what I used in this:
>>>>> $ nvme connect -a 192.168.123.1 -t tcp  -s 8009 -n testnqn1 -S 
>>>>> "DHHC-1:00:QpxVGpctx5J+4SeW2MClUI8rfZO3WdP1llImvsPsx7e3TK+I:" -C 
>>>>> "DHHC-1:00:Jc/My1o0qtLCWRp+sHhAVafdfaS7YQOMYhk9zSmlatobqB8C:"
>>>>>
>>>>> The dhchap_ctrl_key is the offending string...
>>>>>
>>>> Right. Should be fixed with the attached patch.
>>>> Can you check?
>>>
>>> Yes, that works.
>>
>> So, to summarize: With this one and the tentative patch I've sent 
>> earlier:
>>
>> diff --git a/drivers/nvme/host/auth.c b/drivers/nvme/host/auth.c
>> index 53184ac76240..a03f41fa146e 100644
>> --- a/drivers/nvme/host/auth.c
>> +++ b/drivers/nvme/host/auth.c
>> @@ -842,6 +842,10 @@ int nvme_auth_negotiate(struct nvme_ctrl *ctrl, 
>> int qid)
>>                  dev_warn(ctrl->device, "qid %d: no key\n", qid);
>>                  return -ENOKEY;
>>          }
>> +       if (ctrl->opts->dhchap_ctrl_secret && !ctrl->ctrl_key) {
>> +               dev_warn(ctrl->device, "qid %d: invalid ctrl key\n", 
>> qid);
>> +               return -ENOKEY;
>> +       }
>>
>>          mutex_lock(&ctrl->dhchap_auth_mutex);
>>          /* Check if the context is already queued */
>>
>>
>>
>> your issues are resolved?
>> Just to clarify before I sent another round of patches ...
> 
> No. I don't understand why this patch is needed. As I noted
> earlier, when I pass a wrong ctrl key I fail to connect do
> to authentication error. Also when I pass a malformed ctrl key.
> 
> So what does this patch fix? Can you give me an example that should
> have failed before and only with this patch it fails?
> 
Reasoning is as follows:
When an invalid controller key is passed via 
'ctrl->opts->dhchap_ctrl_secret' we'll end up with an empty 
'ctrl->ctrl_key' in nvme_auth_init_ctrl() (which is what caused the 
original oops).
But the negotiation will still start, and, seeing that 'ctrl->ctrl_key' 
is empty, will _not_ attempt controller authentication.
But as a controller key is passed from userland we should have attempted 
it, and hence an error is in order.

> And for the other issue, please read it again, I think it is related
> to nvme-cli and not this patchset.

Hmm. Okay, I'll check, and possibly have to create another blktest for 
this ...


Cheers,

Hannes
-- 
Dr. Hannes Reinecke		           Kernel Storage Architect
hare@suse.de			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), GF: Felix Imendörffer

Re: [PATCH 06/11] nvme: Implement In-Band authentication

[Sagi Grimberg]

>>>>> Looks like if I pass a malformed ctrl key to nvme connect I am able to
>>>>> crash the system:
>>>>
>>>> This was what I used in this:
>>>> $ nvme connect -a 192.168.123.1 -t tcp  -s 8009 -n testnqn1 -S 
>>>> "DHHC-1:00:QpxVGpctx5J+4SeW2MClUI8rfZO3WdP1llImvsPsx7e3TK+I:" -C 
>>>> "DHHC-1:00:Jc/My1o0qtLCWRp+sHhAVafdfaS7YQOMYhk9zSmlatobqB8C:"
>>>>
>>>> The dhchap_ctrl_key is the offending string...
>>>>
>>> Right. Should be fixed with the attached patch.
>>> Can you check?
>>
>> Yes, that works.
> 
> So, to summarize: With this one and the tentative patch I've sent earlier:
> 
> diff --git a/drivers/nvme/host/auth.c b/drivers/nvme/host/auth.c
> index 53184ac76240..a03f41fa146e 100644
> --- a/drivers/nvme/host/auth.c
> +++ b/drivers/nvme/host/auth.c
> @@ -842,6 +842,10 @@ int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int 
> qid)
>                  dev_warn(ctrl->device, "qid %d: no key\n", qid);
>                  return -ENOKEY;
>          }
> +       if (ctrl->opts->dhchap_ctrl_secret && !ctrl->ctrl_key) {
> +               dev_warn(ctrl->device, "qid %d: invalid ctrl key\n", qid);
> +               return -ENOKEY;
> +       }
> 
>          mutex_lock(&ctrl->dhchap_auth_mutex);
>          /* Check if the context is already queued */
> 
> 
> 
> your issues are resolved?
> Just to clarify before I sent another round of patches ...

No. I don't understand why this patch is needed. As I noted
earlier, when I pass a wrong ctrl key I fail to connect do
to authentication error. Also when I pass a malformed ctrl key.

So what does this patch fix? Can you give me an example that should
have failed before and only with this patch it fails?

And for the other issue, please read it again, I think it is related
to nvme-cli and not this patchset.

Re: [PATCH 06/11] nvme: Implement In-Band authentication

[Hannes Reinecke]

On 6/22/22 10:43, Sagi Grimberg wrote:
> 
>>>> Looks like if I pass a malformed ctrl key to nvme connect I am able to
>>>> crash the system:
>>>
>>> This was what I used in this:
>>> $ nvme connect -a 192.168.123.1 -t tcp  -s 8009 -n testnqn1 -S 
>>> "DHHC-1:00:QpxVGpctx5J+4SeW2MClUI8rfZO3WdP1llImvsPsx7e3TK+I:" -C 
>>> "DHHC-1:00:Jc/My1o0qtLCWRp+sHhAVafdfaS7YQOMYhk9zSmlatobqB8C:"
>>>
>>> The dhchap_ctrl_key is the offending string...
>>>
>> Right. Should be fixed with the attached patch.
>> Can you check?
> 
> Yes, that works.

So, to summarize: With this one and the tentative patch I've sent earlier:

diff --git a/drivers/nvme/host/auth.c b/drivers/nvme/host/auth.c
index 53184ac76240..a03f41fa146e 100644
--- a/drivers/nvme/host/auth.c
+++ b/drivers/nvme/host/auth.c
@@ -842,6 +842,10 @@ int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int 
qid)
                 dev_warn(ctrl->device, "qid %d: no key\n", qid);
                 return -ENOKEY;
         }
+       if (ctrl->opts->dhchap_ctrl_secret && !ctrl->ctrl_key) {
+               dev_warn(ctrl->device, "qid %d: invalid ctrl key\n", qid);
+               return -ENOKEY;
+       }

         mutex_lock(&ctrl->dhchap_auth_mutex);
         /* Check if the context is already queued */



your issues are resolved?
Just to clarify before I sent another round of patches ...

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		           Kernel Storage Architect
hare@suse.de			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), GF: Felix Imendörffer

Re: [PATCH 06/11] nvme: Implement In-Band authentication

[Sagi Grimberg]

>>> Looks like if I pass a malformed ctrl key to nvme connect I am able to
>>> crash the system:
>>
>> This was what I used in this:
>> $ nvme connect -a 192.168.123.1 -t tcp  -s 8009 -n testnqn1 -S 
>> "DHHC-1:00:QpxVGpctx5J+4SeW2MClUI8rfZO3WdP1llImvsPsx7e3TK+I:" -C 
>> "DHHC-1:00:Jc/My1o0qtLCWRp+sHhAVafdfaS7YQOMYhk9zSmlatobqB8C:"
>>
>> The dhchap_ctrl_key is the offending string...
>>
> Right. Should be fixed with the attached patch.
> Can you check?

Yes, that works.

Re: [PATCH 06/11] nvme: Implement In-Band authentication

[Hannes Reinecke]

[-- Attachment #1: Type: text/plain, Size: 800 bytes --]

On 6/21/22 16:59, Sagi Grimberg wrote:
> 
>> Looks like if I pass a malformed ctrl key to nvme connect I am able to
>> crash the system:
> 
> This was what I used in this:
> $ nvme connect -a 192.168.123.1 -t tcp  -s 8009 -n testnqn1 -S 
> "DHHC-1:00:QpxVGpctx5J+4SeW2MClUI8rfZO3WdP1llImvsPsx7e3TK+I:" -C 
> "DHHC-1:00:Jc/My1o0qtLCWRp+sHhAVafdfaS7YQOMYhk9zSmlatobqB8C:"
> 
> The dhchap_ctrl_key is the offending string...
> 
Right. Should be fixed with the attached patch.
Can you check?

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman

[-- Attachment #2: 0001-nvme-auth-do-not-use-ctrl-opts-dhchap_ctrl_secret.patch --]
[-- Type: text/x-patch, Size: 3356 bytes --]

From 1f4d34016e7cad41f6947143f07f802b05415e26 Mon Sep 17 00:00:00 2001
From: Hannes Reinecke <hare@suse.de>
Date: Wed, 22 Jun 2022 07:57:06 +0200
Subject: [PATCH] nvme-auth: do not use ctrl->opts->dhchap_ctrl_secret

The user might have passed in an invalid controller secret, causing
ctrl->opts->dhchap_ctrl_secret to be present, but ctrl->ctrl_key to
be empty. So always use ctrl->ctrl_key when checking if a valid
controller secret is present.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/nvme/common/auth.c | 13 +++++++------
 drivers/nvme/host/auth.c   |  8 ++++----
 2 files changed, 11 insertions(+), 10 deletions(-)

diff --git a/drivers/nvme/common/auth.c b/drivers/nvme/common/auth.c
index 945f6bb6eb1f..0c86ebce59d2 100644
--- a/drivers/nvme/common/auth.c
+++ b/drivers/nvme/common/auth.c
@@ -237,21 +237,21 @@ EXPORT_SYMBOL_GPL(nvme_auth_free_key);
 
 u8 *nvme_auth_transform_key(struct nvme_dhchap_key *key, char *nqn)
 {
-	const char *hmac_name = nvme_auth_hmac_name(key->hash);
+	const char *hmac_name;
 	struct crypto_shash *key_tfm;
 	struct shash_desc *shash;
 	u8 *transformed_key;
 	int ret;
 
-	if (key->hash == 0) {
-		transformed_key = kmemdup(key->key, key->len, GFP_KERNEL);
-		return transformed_key ? transformed_key : ERR_PTR(-ENOMEM);
-	}
-
 	if (!key || !key->key) {
 		pr_warn("No key specified\n");
 		return ERR_PTR(-ENOKEY);
 	}
+	if (key->hash == 0) {
+		transformed_key = kmemdup(key->key, key->len, GFP_KERNEL);
+		return transformed_key ? transformed_key : ERR_PTR(-ENOMEM);
+	}
+	hmac_name = nvme_auth_hmac_name(key->hash);
 	if (!hmac_name) {
 		pr_warn("Invalid key hash id %d\n", key->hash);
 		return ERR_PTR(-EINVAL);
@@ -470,6 +470,7 @@ int nvme_auth_generate_key(u8 *secret, struct nvme_dhchap_key **ret_key)
 	/* Pass in the secret without the 'DHHC-1:XX:' prefix */
 	key = nvme_auth_extract_key(secret + 10, key_hash);
 	if (IS_ERR(key)) {
+		*ret_key = NULL;
 		return PTR_ERR(key);
 	}
 
diff --git a/drivers/nvme/host/auth.c b/drivers/nvme/host/auth.c
index 9b84e1e54ca7..53184ac76240 100644
--- a/drivers/nvme/host/auth.c
+++ b/drivers/nvme/host/auth.c
@@ -314,7 +314,7 @@ static int nvme_auth_set_dhchap_reply_data(struct nvme_ctrl *ctrl,
 	data->hl = chap->hash_len;
 	data->dhvlen = cpu_to_le16(chap->host_key_len);
 	memcpy(data->rval, chap->response, chap->hash_len);
-	if (ctrl->opts->dhchap_ctrl_secret) {
+	if (ctrl->ctrl_key) {
 		get_random_bytes(chap->c2, chap->hash_len);
 		data->cvalid = 1;
 		chap->s2 = nvme_auth_get_seqnum();
@@ -344,7 +344,7 @@ static int nvme_auth_process_dhchap_success1(struct nvme_ctrl *ctrl,
 	struct nvmf_auth_dhchap_success1_data *data = chap->buf;
 	size_t size = sizeof(*data);
 
-	if (ctrl->opts->dhchap_ctrl_secret)
+	if (ctrl->ctrl_key)
 		size += chap->hash_len;
 
 	if (chap->buf_size < size) {
@@ -791,7 +791,7 @@ static void __nvme_auth_work(struct work_struct *work)
 		return;
 	}
 
-	if (ctrl->opts->dhchap_ctrl_secret) {
+	if (ctrl->ctrl_key) {
 		dev_dbg(ctrl->device,
 			"%s: qid %d controller response\n",
 			__func__, chap->qid);
@@ -809,7 +809,7 @@ static void __nvme_auth_work(struct work_struct *work)
 		goto fail2;
 	}
 
-	if (ctrl->opts->dhchap_ctrl_secret) {
+	if (ctrl->ctrl_key) {
 		/* DH-HMAC-CHAP Step 5: send success2 */
 		dev_dbg(ctrl->device, "%s: qid %d send success2\n",
 			__func__, chap->qid);
-- 
2.26.2

Re: [PATCH 06/11] nvme: Implement In-Band authentication

[Hannes Reinecke]

On 6/21/22 16:50, Sagi Grimberg wrote:
> 
> 
> On 6/21/22 17:26, Hannes Reinecke wrote:
>> On 6/21/22 16:24, Sagi Grimberg wrote:
>>> This one doesn't compile against nvme-5.20.
>>>
>>>> +    ret = __nvme_submit_sync_cmd(q, &cmd, NULL, data, data_len, 0,
>>>> +                     qid == 0 ? NVME_QID_ANY : qid,
>>>> +                     0, flags);
>>>
>>> here ^^^^^^^
>>
>> Sheesh. I've compiled it against nvme-5.19, where it works perfectly.
>> Alright, once more unto the breach ...
> 
> Looks like if I pass a malformed ctrl key to nvme connect I am able to
> crash the system:
> -- 
> [   84.793307] Workqueue: nvme-wq __nvme_auth_work [nvme_core]
> [   84.794790] RIP: 0010:nvme_auth_transform_key+0x19/0x1f0 [nvme_common]
> [   84.796468] Code: bc f4 ff ff ff eb bf e8 f5 2f e1 ee 0f 1f 44 00 00 
> 0f 1f 44 00 00 41 57 41 56 41 55 49 89 f5 41 54 55 53 48 89 fb 48 83 ec 
> 08 <0f> b6 7f 10 e8 ce f9 ff ff 48 89 c7 0f b6 43 10 84 c0 0f 84 4c 01
> [   84.800112] RSP: 0018:ffffae5f8047bc78 EFLAGS: 00010296
> [   84.801048] RAX: ffff973b854ca0c0 RBX: 0000000000000000 RCX: 
> 0000000000000000
> [   84.802289] RDX: ffff973b8355b000 RSI: ffff973b84f568a0 RDI: 
> 0000000000000000
> [   84.803447] RBP: ffff973b842dd6b9 R08: 0000000000000003 R09: 
> ffff973bbec308a8
> [   84.804638] R10: 0000000000000147 R11: 0000000000000000 R12: 
> ffff973b842dd600
> [   84.805767] R13: ffff973b84f568a0 R14: 0000000000000000 R15: 
> ffff973b9e94250d
> [   84.806929] FS:  0000000000000000(0000) GS:ffff973bbec00000(0000) 
> knlGS:0000000000000000
> [   84.808220] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [   84.809178] CR2: 0000000000000010 CR3: 00000000061ac003 CR4: 
> 0000000000370ef0
> [   84.810337] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
> 0000000000000000
> [   84.811432] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 
> 0000000000000400
> [   84.812527] Call Trace:
> [   84.812974]  <TASK>
> [   84.813372]  nvme_auth_dhchap_setup_ctrl_response+0x4b/0x3b0 [nvme_core]
> [   84.814355]  ? preempt_count_add+0x68/0xa0
> [   84.815080]  ? _raw_spin_unlock_irq+0x16/0x28
> [   84.815840]  ? __wait_for_common+0x19f/0x1d0
> [   84.816587]  ? firmware_map_remove+0x87/0x87
> [   84.817333]  ? blk_mq_hctx_has_pending+0x38/0x70
> [   84.818123]  ? blk_mq_run_hw_queue+0x7d/0xe0
> [   84.818784]  ? __blk_mq_free_request+0x9b/0xa0
> [   84.819482]  ? blk_queue_exit+0xe/0x40
> [   84.820124]  ? __nvme_submit_sync_cmd+0xe8/0x160 [nvme_core]
> [   84.821096]  ? nvme_auth_submit+0x8f/0xd0 [nvme_core]
> [   84.821970]  __nvme_auth_work+0x1fb/0x480 [nvme_core]
> [   84.822869]  process_one_work+0x1e5/0x3b0
> [   84.823608]  worker_thread+0x1c4/0x3a0
> [   84.824343]  ? rescuer_thread+0x390/0x390
> [   84.825044]  kthread+0xe8/0x110
> -- 

Malformed exactly how?
We should've captured any malformed key during nvme_auth_extract_key().
Can you share an example of your malformed key?

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman

Re: [PATCH 06/11] nvme: Implement In-Band authentication

[Sagi Grimberg]

> Looks like if I pass a malformed ctrl key to nvme connect I am able to
> crash the system:

This was what I used in this:
$ nvme connect -a 192.168.123.1 -t tcp  -s 8009 -n testnqn1 -S 
"DHHC-1:00:QpxVGpctx5J+4SeW2MClUI8rfZO3WdP1llImvsPsx7e3TK+I:" -C 
"DHHC-1:00:Jc/My1o0qtLCWRp+sHhAVafdfaS7YQOMYhk9zSmlatobqB8C:"

The dhchap_ctrl_key is the offending string...

> -- 
> [   84.793307] Workqueue: nvme-wq __nvme_auth_work [nvme_core]
> [   84.794790] RIP: 0010:nvme_auth_transform_key+0x19/0x1f0 [nvme_common]
> [   84.796468] Code: bc f4 ff ff ff eb bf e8 f5 2f e1 ee 0f 1f 44 00 00 
> 0f 1f 44 00 00 41 57 41 56 41 55 49 89 f5 41 54 55 53 48 89 fb 48 83 ec 
> 08 <0f> b6 7f 10 e8 ce f9 ff ff 48 89 c7 0f b6 43 10 84 c0 0f 84 4c 01
> [   84.800112] RSP: 0018:ffffae5f8047bc78 EFLAGS: 00010296
> [   84.801048] RAX: ffff973b854ca0c0 RBX: 0000000000000000 RCX: 
> 0000000000000000
> [   84.802289] RDX: ffff973b8355b000 RSI: ffff973b84f568a0 RDI: 
> 0000000000000000
> [   84.803447] RBP: ffff973b842dd6b9 R08: 0000000000000003 R09: 
> ffff973bbec308a8
> [   84.804638] R10: 0000000000000147 R11: 0000000000000000 R12: 
> ffff973b842dd600
> [   84.805767] R13: ffff973b84f568a0 R14: 0000000000000000 R15: 
> ffff973b9e94250d
> [   84.806929] FS:  0000000000000000(0000) GS:ffff973bbec00000(0000) 
> knlGS:0000000000000000
> [   84.808220] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [   84.809178] CR2: 0000000000000010 CR3: 00000000061ac003 CR4: 
> 0000000000370ef0
> [   84.810337] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
> 0000000000000000
> [   84.811432] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 
> 0000000000000400
> [   84.812527] Call Trace:
> [   84.812974]  <TASK>
> [   84.813372]  nvme_auth_dhchap_setup_ctrl_response+0x4b/0x3b0 [nvme_core]
> [   84.814355]  ? preempt_count_add+0x68/0xa0
> [   84.815080]  ? _raw_spin_unlock_irq+0x16/0x28
> [   84.815840]  ? __wait_for_common+0x19f/0x1d0
> [   84.816587]  ? firmware_map_remove+0x87/0x87
> [   84.817333]  ? blk_mq_hctx_has_pending+0x38/0x70
> [   84.818123]  ? blk_mq_run_hw_queue+0x7d/0xe0
> [   84.818784]  ? __blk_mq_free_request+0x9b/0xa0
> [   84.819482]  ? blk_queue_exit+0xe/0x40
> [   84.820124]  ? __nvme_submit_sync_cmd+0xe8/0x160 [nvme_core]
> [   84.821096]  ? nvme_auth_submit+0x8f/0xd0 [nvme_core]
> [   84.821970]  __nvme_auth_work+0x1fb/0x480 [nvme_core]
> [   84.822869]  process_one_work+0x1e5/0x3b0
> [   84.823608]  worker_thread+0x1c4/0x3a0
> [   84.824343]  ? rescuer_thread+0x390/0x390
> [   84.825044]  kthread+0xe8/0x110
> -- 

Re: [PATCH 06/11] nvme: Implement In-Band authentication

[Sagi Grimberg]

On 6/21/22 17:26, Hannes Reinecke wrote:
> On 6/21/22 16:24, Sagi Grimberg wrote:
>> This one doesn't compile against nvme-5.20.
>>
>>> +    ret = __nvme_submit_sync_cmd(q, &cmd, NULL, data, data_len, 0,
>>> +                     qid == 0 ? NVME_QID_ANY : qid,
>>> +                     0, flags);
>>
>> here ^^^^^^^
> 
> Sheesh. I've compiled it against nvme-5.19, where it works perfectly.
> Alright, once more unto the breach ...

Looks like if I pass a malformed ctrl key to nvme connect I am able to
crash the system:
--
[   84.793307] Workqueue: nvme-wq __nvme_auth_work [nvme_core]
[   84.794790] RIP: 0010:nvme_auth_transform_key+0x19/0x1f0 [nvme_common]
[   84.796468] Code: bc f4 ff ff ff eb bf e8 f5 2f e1 ee 0f 1f 44 00 00 
0f 1f 44 00 00 41 57 41 56 41 55 49 89 f5 41 54 55 53 48 89 fb 48 83 ec 
08 <0f> b6 7f 10 e8 ce f9 ff ff 48 89 c7 0f b6 43 10 84 c0 0f 84 4c 01
[   84.800112] RSP: 0018:ffffae5f8047bc78 EFLAGS: 00010296
[   84.801048] RAX: ffff973b854ca0c0 RBX: 0000000000000000 RCX: 
0000000000000000
[   84.802289] RDX: ffff973b8355b000 RSI: ffff973b84f568a0 RDI: 
0000000000000000
[   84.803447] RBP: ffff973b842dd6b9 R08: 0000000000000003 R09: 
ffff973bbec308a8
[   84.804638] R10: 0000000000000147 R11: 0000000000000000 R12: 
ffff973b842dd600
[   84.805767] R13: ffff973b84f568a0 R14: 0000000000000000 R15: 
ffff973b9e94250d
[   84.806929] FS:  0000000000000000(0000) GS:ffff973bbec00000(0000) 
knlGS:0000000000000000
[   84.808220] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   84.809178] CR2: 0000000000000010 CR3: 00000000061ac003 CR4: 
0000000000370ef0
[   84.810337] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[   84.811432] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 
0000000000000400
[   84.812527] Call Trace:
[   84.812974]  <TASK>
[   84.813372]  nvme_auth_dhchap_setup_ctrl_response+0x4b/0x3b0 [nvme_core]
[   84.814355]  ? preempt_count_add+0x68/0xa0
[   84.815080]  ? _raw_spin_unlock_irq+0x16/0x28
[   84.815840]  ? __wait_for_common+0x19f/0x1d0
[   84.816587]  ? firmware_map_remove+0x87/0x87
[   84.817333]  ? blk_mq_hctx_has_pending+0x38/0x70
[   84.818123]  ? blk_mq_run_hw_queue+0x7d/0xe0
[   84.818784]  ? __blk_mq_free_request+0x9b/0xa0
[   84.819482]  ? blk_queue_exit+0xe/0x40
[   84.820124]  ? __nvme_submit_sync_cmd+0xe8/0x160 [nvme_core]
[   84.821096]  ? nvme_auth_submit+0x8f/0xd0 [nvme_core]
[   84.821970]  __nvme_auth_work+0x1fb/0x480 [nvme_core]
[   84.822869]  process_one_work+0x1e5/0x3b0
[   84.823608]  worker_thread+0x1c4/0x3a0
[   84.824343]  ? rescuer_thread+0x390/0x390
[   84.825044]  kthread+0xe8/0x110
--

Re: [PATCH 06/11] nvme: Implement In-Band authentication

[Hannes Reinecke]

On 6/21/22 16:24, Sagi Grimberg wrote:
> This one doesn't compile against nvme-5.20.
> 
>> +    ret = __nvme_submit_sync_cmd(q, &cmd, NULL, data, data_len, 0,
>> +                     qid == 0 ? NVME_QID_ANY : qid,
>> +                     0, flags);
> 
> here ^^^^^^^

Sheesh. I've compiled it against nvme-5.19, where it works perfectly.
Alright, once more unto the breach ...

Cheers,

Hannes

Re: [PATCH 06/11] nvme: Implement In-Band authentication

[Sagi Grimberg]

This one doesn't compile against nvme-5.20.

> +	ret = __nvme_submit_sync_cmd(q, &cmd, NULL, data, data_len, 0,
> +				     qid == 0 ? NVME_QID_ANY : qid,
> +				     0, flags);

here ^^^^^^^

[PATCH 06/11] nvme: Implement In-Band authentication

[Hannes Reinecke]

Implement NVMe-oF In-Band authentication according to NVMe TPAR 8006.
This patch adds two new fabric options 'dhchap_secret' to specify the
pre-shared key (in ASCII respresentation according to NVMe 2.0 section
8.13.5.8 'Secret representation') and 'dhchap_ctrl_secret' to specify
the pre-shared controller key for bi-directional authentication of both
the host and the controller.
Re-authentication can be triggered by writing the PSK into the new
controller sysfs attribute 'dhchap_secret' or 'dhchap_ctrl_secret'.

Signed-off-by: Hannes Reinecke <hare@suse.de>
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
---
 drivers/nvme/Kconfig         |   1 +
 drivers/nvme/Makefile        |   1 +
 drivers/nvme/common/Kconfig  |   4 +
 drivers/nvme/common/Makefile |   7 +
 drivers/nvme/common/auth.c   | 328 ++++++++++++++
 drivers/nvme/host/Kconfig    |  13 +
 drivers/nvme/host/Makefile   |   1 +
 drivers/nvme/host/auth.c     | 813 +++++++++++++++++++++++++++++++++++
 drivers/nvme/host/core.c     | 143 +++++-
 drivers/nvme/host/fabrics.c  |  79 +++-
 drivers/nvme/host/fabrics.h  |   7 +
 drivers/nvme/host/nvme.h     |  30 ++
 drivers/nvme/host/rdma.c     |   1 +
 drivers/nvme/host/tcp.c      |   1 +
 drivers/nvme/host/trace.c    |  32 ++
 include/linux/nvme-auth.h    |  33 ++
 16 files changed, 1487 insertions(+), 7 deletions(-)
 create mode 100644 drivers/nvme/common/Kconfig
 create mode 100644 drivers/nvme/common/Makefile
 create mode 100644 drivers/nvme/common/auth.c
 create mode 100644 drivers/nvme/host/auth.c
 create mode 100644 include/linux/nvme-auth.h

diff --git a/drivers/nvme/Kconfig b/drivers/nvme/Kconfig
index 87ae409a32b9..656e46d938da 100644
--- a/drivers/nvme/Kconfig
+++ b/drivers/nvme/Kconfig
@@ -1,6 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0-only
 menu "NVME Support"
 
+source "drivers/nvme/common/Kconfig"
 source "drivers/nvme/host/Kconfig"
 source "drivers/nvme/target/Kconfig"
 
diff --git a/drivers/nvme/Makefile b/drivers/nvme/Makefile
index fb42c44609a8..eedca8c72098 100644
--- a/drivers/nvme/Makefile
+++ b/drivers/nvme/Makefile
@@ -1,4 +1,5 @@
 # SPDX-License-Identifier: GPL-2.0-only
 
+obj-$(CONFIG_NVME_COMMON)		+= common/
 obj-y		+= host/
 obj-y		+= target/
diff --git a/drivers/nvme/common/Kconfig b/drivers/nvme/common/Kconfig
new file mode 100644
index 000000000000..4514f44362dd
--- /dev/null
+++ b/drivers/nvme/common/Kconfig
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: GPL-2.0-only
+
+config NVME_COMMON
+       tristate
diff --git a/drivers/nvme/common/Makefile b/drivers/nvme/common/Makefile
new file mode 100644
index 000000000000..720c625b8a52
--- /dev/null
+++ b/drivers/nvme/common/Makefile
@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: GPL-2.0
+
+ccflags-y			+= -I$(src)
+
+obj-$(CONFIG_NVME_COMMON)	+= nvme-common.o
+
+nvme-common-y			+= auth.o
diff --git a/drivers/nvme/common/auth.c b/drivers/nvme/common/auth.c
new file mode 100644
index 000000000000..1cccaf6afdd2
--- /dev/null
+++ b/drivers/nvme/common/auth.c
@@ -0,0 +1,328 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2020 Hannes Reinecke, SUSE Linux
+ */
+
+#include <linux/module.h>
+#include <linux/crc32.h>
+#include <linux/base64.h>
+#include <linux/prandom.h>
+#include <linux/scatterlist.h>
+#include <asm/unaligned.h>
+#include <crypto/hash.h>
+#include <crypto/dh.h>
+#include <linux/nvme.h>
+#include <linux/nvme-auth.h>
+
+static u32 nvme_dhchap_seqnum;
+static DEFINE_MUTEX(nvme_dhchap_mutex);
+
+u32 nvme_auth_get_seqnum(void)
+{
+	u32 seqnum;
+
+	mutex_lock(&nvme_dhchap_mutex);
+	if (!nvme_dhchap_seqnum)
+		nvme_dhchap_seqnum = prandom_u32();
+	else {
+		nvme_dhchap_seqnum++;
+		if (!nvme_dhchap_seqnum)
+			nvme_dhchap_seqnum++;
+	}
+	seqnum = nvme_dhchap_seqnum;
+	mutex_unlock(&nvme_dhchap_mutex);
+	return seqnum;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_get_seqnum);
+
+static struct nvme_auth_dhgroup_map {
+	const char name[16];
+	const char kpp[16];
+} dhgroup_map[] = {
+	[NVME_AUTH_DHGROUP_NULL] = {
+		.name = "null", .kpp = "null" },
+	[NVME_AUTH_DHGROUP_2048] = {
+		.name = "ffdhe2048", .kpp = "ffdhe2048(dh)" },
+	[NVME_AUTH_DHGROUP_3072] = {
+		.name = "ffdhe3072", .kpp = "ffdhe3072(dh)" },
+	[NVME_AUTH_DHGROUP_4096] = {
+		.name = "ffdhe4096", .kpp = "ffdhe4096(dh)" },
+	[NVME_AUTH_DHGROUP_6144] = {
+		.name = "ffdhe6144", .kpp = "ffdhe6144(dh)" },
+	[NVME_AUTH_DHGROUP_8192] = {
+		.name = "ffdhe8192", .kpp = "ffdhe8192(dh)" },
+};
+
+const char *nvme_auth_dhgroup_name(u8 dhgroup_id)
+{
+	if ((dhgroup_id > ARRAY_SIZE(dhgroup_map)) ||
+	    !dhgroup_map[dhgroup_id].name ||
+	    !strlen(dhgroup_map[dhgroup_id].name))
+		return NULL;
+	return dhgroup_map[dhgroup_id].name;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_dhgroup_name);
+
+const char *nvme_auth_dhgroup_kpp(u8 dhgroup_id)
+{
+	if ((dhgroup_id > ARRAY_SIZE(dhgroup_map)) ||
+	    !dhgroup_map[dhgroup_id].kpp ||
+	    !strlen(dhgroup_map[dhgroup_id].kpp))
+		return NULL;
+	return dhgroup_map[dhgroup_id].kpp;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_dhgroup_kpp);
+
+u8 nvme_auth_dhgroup_id(const char *dhgroup_name)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(dhgroup_map); i++) {
+		if (!dhgroup_map[i].name ||
+		    !strlen(dhgroup_map[i].name))
+			continue;
+		if (!strncmp(dhgroup_map[i].name, dhgroup_name,
+			     strlen(dhgroup_map[i].name)))
+			return i;
+	}
+	return NVME_AUTH_DHGROUP_INVALID;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_dhgroup_id);
+
+static struct nvme_dhchap_hash_map {
+	int len;
+	const char hmac[15];
+	const char digest[8];
+} hash_map[] = {
+	[NVME_AUTH_HASH_SHA256] = {
+		.len = 32,
+		.hmac = "hmac(sha256)",
+		.digest = "sha256",
+	},
+	[NVME_AUTH_HASH_SHA384] = {
+		.len = 48,
+		.hmac = "hmac(sha384)",
+		.digest = "sha384",
+	},
+	[NVME_AUTH_HASH_SHA512] = {
+		.len = 64,
+		.hmac = "hmac(sha512)",
+		.digest = "sha512",
+	},
+};
+
+const char *nvme_auth_hmac_name(u8 hmac_id)
+{
+	if ((hmac_id > ARRAY_SIZE(hash_map)) ||
+	    !hash_map[hmac_id].hmac ||
+	    !strlen(hash_map[hmac_id].hmac))
+		return NULL;
+	return hash_map[hmac_id].hmac;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_hmac_name);
+
+const char *nvme_auth_digest_name(u8 hmac_id)
+{
+	if ((hmac_id > ARRAY_SIZE(hash_map)) ||
+	    !hash_map[hmac_id].digest ||
+	    !strlen(hash_map[hmac_id].digest))
+		return NULL;
+	return hash_map[hmac_id].digest;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_digest_name);
+
+u8 nvme_auth_hmac_id(const char *hmac_name)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
+		if (!hash_map[i].hmac || !strlen(hash_map[i].hmac))
+			continue;
+		if (!strncmp(hash_map[i].hmac, hmac_name,
+			     strlen(hash_map[i].hmac)))
+			return i;
+	}
+	return NVME_AUTH_HASH_INVALID;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_hmac_id);
+
+size_t nvme_auth_hmac_hash_len(u8 hmac_id)
+{
+	if ((hmac_id > ARRAY_SIZE(hash_map)) ||
+	    !hash_map[hmac_id].hmac ||
+	    !strlen(hash_map[hmac_id].hmac))
+		return 0;
+	return hash_map[hmac_id].len;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_hmac_hash_len);
+
+struct nvme_dhchap_key *nvme_auth_extract_key(unsigned char *secret,
+					      u8 key_hash)
+{
+	struct nvme_dhchap_key *key;
+	unsigned char *p;
+	u32 crc;
+	int ret, key_len;
+	size_t allocated_len = strlen(secret);
+
+	/* Secret might be affixed with a ':' */
+	p = strrchr(secret, ':');
+	if (p)
+		allocated_len = p - secret;
+	key = kzalloc(sizeof(*key), GFP_KERNEL);
+	if (!key)
+		return ERR_PTR(-ENOMEM);
+	key->key = kzalloc(allocated_len, GFP_KERNEL);
+	if (!key->key) {
+		ret = -ENOMEM;
+		goto out_free_key;
+	}
+
+	key_len = base64_decode(secret, allocated_len, key->key);
+	if (key_len < 0) {
+		pr_debug("base64 key decoding error %d\n",
+			 key_len);
+		ret = key_len;
+		goto out_free_secret;
+	}
+
+	if (key_len != 36 && key_len != 52 &&
+	    key_len != 68) {
+		pr_err("Invalid key len %d\n", key_len);
+		ret = -EINVAL;
+		goto out_free_secret;
+	}
+
+	if (key_hash > 0 &&
+	    (key_len - 4) != nvme_auth_hmac_hash_len(key_hash)) {
+		pr_err("Mismatched key len %d for %s\n", key_len,
+		       nvme_auth_hmac_name(key_hash));
+		ret = -EINVAL;
+		goto out_free_secret;
+	}
+
+	/* The last four bytes is the CRC in little-endian format */
+	key_len -= 4;
+	/*
+	 * The linux implementation doesn't do pre- and post-increments,
+	 * so we have to do it manually.
+	 */
+	crc = ~crc32(~0, key->key, key_len);
+
+	if (get_unaligned_le32(key->key + key_len) != crc) {
+		pr_err("key crc mismatch (key %08x, crc %08x)\n",
+		       get_unaligned_le32(key->key + key_len), crc);
+		ret = -EKEYREJECTED;
+		goto out_free_secret;
+	}
+	key->len = key_len;
+	key->hash = key_hash;
+	return key;
+out_free_secret:
+	kfree_sensitive(key->key);
+out_free_key:
+	kfree(key);
+	return ERR_PTR(ret);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_extract_key);
+
+void nvme_auth_free_key(struct nvme_dhchap_key *key)
+{
+	if (!key)
+		return;
+	kfree_sensitive(key->key);
+	kfree(key);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_free_key);
+
+u8 *nvme_auth_transform_key(struct nvme_dhchap_key *key, char *nqn)
+{
+	const char *hmac_name = nvme_auth_hmac_name(key->hash);
+	struct crypto_shash *key_tfm;
+	struct shash_desc *shash;
+	u8 *transformed_key;
+	int ret;
+
+	if (key->hash == 0) {
+		transformed_key = kmemdup(key->key, key->len, GFP_KERNEL);
+		return transformed_key ? transformed_key : ERR_PTR(-ENOMEM);
+	}
+
+	if (!key || !key->key) {
+		pr_warn("No key specified\n");
+		return ERR_PTR(-ENOKEY);
+	}
+	if (!hmac_name) {
+		pr_warn("Invalid key hash id %d\n", key->hash);
+		return ERR_PTR(-EINVAL);
+	}
+
+	key_tfm = crypto_alloc_shash(hmac_name, 0, 0);
+	if (IS_ERR(key_tfm))
+		return (u8 *)key_tfm;
+
+	shash = kmalloc(sizeof(struct shash_desc) +
+			crypto_shash_descsize(key_tfm),
+			GFP_KERNEL);
+	if (!shash) {
+		ret = -ENOMEM;
+		goto out_free_key;
+	}
+
+	transformed_key = kzalloc(crypto_shash_digestsize(key_tfm), GFP_KERNEL);
+	if (!transformed_key) {
+		ret = -ENOMEM;
+		goto out_free_shash;
+	}
+
+	shash->tfm = key_tfm;
+	ret = crypto_shash_setkey(key_tfm, key->key, key->len);
+	if (ret < 0)
+		goto out_free_shash;
+	ret = crypto_shash_init(shash);
+	if (ret < 0)
+		goto out_free_shash;
+	ret = crypto_shash_update(shash, nqn, strlen(nqn));
+	if (ret < 0)
+		goto out_free_shash;
+	ret = crypto_shash_update(shash, "NVMe-over-Fabrics", 17);
+	if (ret < 0)
+		goto out_free_shash;
+	ret = crypto_shash_final(shash, transformed_key);
+out_free_shash:
+	kfree(shash);
+out_free_key:
+	crypto_free_shash(key_tfm);
+	if (ret < 0) {
+		kfree_sensitive(transformed_key);
+		return ERR_PTR(ret);
+	}
+	return transformed_key;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_transform_key);
+
+int nvme_auth_generate_key(u8 *secret, struct nvme_dhchap_key **ret_key)
+{
+	struct nvme_dhchap_key *key;
+	u8 key_hash;
+
+	if (!secret) {
+		*ret_key = NULL;
+		return 0;
+	}
+
+	if (sscanf(secret, "DHHC-1:%hhd:%*s:", &key_hash) != 1)
+		return -EINVAL;
+
+	/* Pass in the secret without the 'DHHC-1:XX:' prefix */
+	key = nvme_auth_extract_key(secret + 10, key_hash);
+	if (IS_ERR(key)) {
+		return PTR_ERR(key);
+	}
+
+	*ret_key = key;
+	return 0;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_generate_key);
+
+MODULE_LICENSE("GPL v2");
diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig
index 877d2ec4ea9f..6c503f42f3c6 100644
--- a/drivers/nvme/host/Kconfig
+++ b/drivers/nvme/host/Kconfig
@@ -92,6 +92,19 @@ config NVME_TCP
 
 	  If unsure, say N.
 
+config NVME_AUTH
+	bool "NVM Express over Fabrics In-Band Authentication"
+	depends on NVME_CORE
+	select NVME_COMMON
+	select CRYPTO
+	select CRYPTO_HMAC
+	select CRYPTO_SHA256
+	select CRYPTO_SHA512
+	help
+	  This provides support for NVMe over Fabrics In-Band Authentication.
+
+	  If unsure, say N.
+
 config NVME_APPLE
 	tristate "Apple ANS2 NVM Express host driver"
 	depends on OF && BLOCK
diff --git a/drivers/nvme/host/Makefile b/drivers/nvme/host/Makefile
index a36ae1612059..a3e88f32f560 100644
--- a/drivers/nvme/host/Makefile
+++ b/drivers/nvme/host/Makefile
@@ -16,6 +16,7 @@ nvme-core-$(CONFIG_NVME_MULTIPATH)	+= multipath.o
 nvme-core-$(CONFIG_BLK_DEV_ZONED)	+= zns.o
 nvme-core-$(CONFIG_FAULT_INJECTION_DEBUG_FS)	+= fault_inject.o
 nvme-core-$(CONFIG_NVME_HWMON)		+= hwmon.o
+nvme-core-$(CONFIG_NVME_AUTH)		+= auth.o
 
 nvme-y					+= pci.o
 
diff --git a/drivers/nvme/host/auth.c b/drivers/nvme/host/auth.c
new file mode 100644
index 000000000000..c9f861b928da
--- /dev/null
+++ b/drivers/nvme/host/auth.c
@@ -0,0 +1,813 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2020 Hannes Reinecke, SUSE Linux
+ */
+
+#include <linux/crc32.h>
+#include <linux/base64.h>
+#include <linux/prandom.h>
+#include <asm/unaligned.h>
+#include <crypto/hash.h>
+#include <crypto/dh.h>
+#include "nvme.h"
+#include "fabrics.h"
+#include <linux/nvme-auth.h>
+
+struct nvme_dhchap_queue_context {
+	struct list_head entry;
+	struct work_struct auth_work;
+	struct nvme_ctrl *ctrl;
+	struct crypto_shash *shash_tfm;
+	void *buf;
+	size_t buf_size;
+	int qid;
+	int error;
+	u32 s1;
+	u32 s2;
+	u16 transaction;
+	u8 status;
+	u8 hash_id;
+	size_t hash_len;
+	u8 dhgroup_id;
+	u8 c1[64];
+	u8 c2[64];
+	u8 response[64];
+	u8 *host_response;
+};
+
+#define nvme_auth_flags_from_qid(qid) \
+	(qid == 0) ? 0 : BLK_MQ_REQ_NOWAIT | BLK_MQ_REQ_RESERVED
+#define nvme_auth_queue_from_qid(ctrl, qid) \
+	(qid == 0) ? (ctrl)->fabrics_q : (ctrl)->connect_q
+
+static int nvme_auth_submit(struct nvme_ctrl *ctrl, int qid,
+			    void *data, size_t data_len, bool auth_send)
+{
+	struct nvme_command cmd = {};
+	blk_mq_req_flags_t flags = nvme_auth_flags_from_qid(qid);
+	struct request_queue *q = nvme_auth_queue_from_qid(ctrl, qid);
+	int ret;
+
+	cmd.auth_common.opcode = nvme_fabrics_command;
+	cmd.auth_common.secp = NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER;
+	cmd.auth_common.spsp0 = 0x01;
+	cmd.auth_common.spsp1 = 0x01;
+	if (auth_send) {
+		cmd.auth_send.fctype = nvme_fabrics_type_auth_send;
+		cmd.auth_send.tl = cpu_to_le32(data_len);
+	} else {
+		cmd.auth_receive.fctype = nvme_fabrics_type_auth_receive;
+		cmd.auth_receive.al = cpu_to_le32(data_len);
+	}
+
+	ret = __nvme_submit_sync_cmd(q, &cmd, NULL, data, data_len, 0,
+				     qid == 0 ? NVME_QID_ANY : qid,
+				     0, flags);
+	if (ret > 0)
+		dev_warn(ctrl->device,
+			"qid %d auth_send failed with status %d\n", qid, ret);
+	else if (ret < 0)
+		dev_err(ctrl->device,
+			"qid %d auth_send failed with error %d\n", qid, ret);
+	return ret;
+}
+
+static int nvme_auth_receive_validate(struct nvme_ctrl *ctrl, int qid,
+		struct nvmf_auth_dhchap_failure_data *data,
+		u16 transaction, u8 expected_msg)
+{
+	dev_dbg(ctrl->device, "%s: qid %d auth_type %d auth_id %x\n",
+		__func__, qid, data->auth_type, data->auth_id);
+
+	if (data->auth_type == NVME_AUTH_COMMON_MESSAGES &&
+	    data->auth_id == NVME_AUTH_DHCHAP_MESSAGE_FAILURE1) {
+		return data->rescode_exp;
+	}
+	if (data->auth_type != NVME_AUTH_DHCHAP_MESSAGES ||
+	    data->auth_id != expected_msg) {
+		dev_warn(ctrl->device,
+			 "qid %d invalid message %02x/%02x\n",
+			 qid, data->auth_type, data->auth_id);
+		return NVME_AUTH_DHCHAP_FAILURE_INCORRECT_MESSAGE;
+	}
+	if (le16_to_cpu(data->t_id) != transaction) {
+		dev_warn(ctrl->device,
+			 "qid %d invalid transaction ID %d\n",
+			 qid, le16_to_cpu(data->t_id));
+		return NVME_AUTH_DHCHAP_FAILURE_INCORRECT_MESSAGE;
+	}
+	return 0;
+}
+
+static int nvme_auth_set_dhchap_negotiate_data(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_negotiate_data *data = chap->buf;
+	size_t size = sizeof(*data) + sizeof(union nvmf_auth_protocol);
+
+	if (chap->buf_size < size) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return -EINVAL;
+	}
+	memset((u8 *)chap->buf, 0, size);
+	data->auth_type = NVME_AUTH_COMMON_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE;
+	data->t_id = cpu_to_le16(chap->transaction);
+	data->sc_c = 0; /* No secure channel concatenation */
+	data->napd = 1;
+	data->auth_protocol[0].dhchap.authid = NVME_AUTH_DHCHAP_AUTH_ID;
+	data->auth_protocol[0].dhchap.halen = 3;
+	data->auth_protocol[0].dhchap.dhlen = 6;
+	data->auth_protocol[0].dhchap.idlist[0] = NVME_AUTH_HASH_SHA256;
+	data->auth_protocol[0].dhchap.idlist[1] = NVME_AUTH_HASH_SHA384;
+	data->auth_protocol[0].dhchap.idlist[2] = NVME_AUTH_HASH_SHA512;
+	data->auth_protocol[0].dhchap.idlist[30] = NVME_AUTH_DHGROUP_NULL;
+	data->auth_protocol[0].dhchap.idlist[31] = NVME_AUTH_DHGROUP_2048;
+	data->auth_protocol[0].dhchap.idlist[32] = NVME_AUTH_DHGROUP_3072;
+	data->auth_protocol[0].dhchap.idlist[33] = NVME_AUTH_DHGROUP_4096;
+	data->auth_protocol[0].dhchap.idlist[34] = NVME_AUTH_DHGROUP_6144;
+	data->auth_protocol[0].dhchap.idlist[35] = NVME_AUTH_DHGROUP_8192;
+
+	return size;
+}
+
+static int nvme_auth_process_dhchap_challenge(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_challenge_data *data = chap->buf;
+	u16 dhvlen = le16_to_cpu(data->dhvlen);
+	size_t size = sizeof(*data) + data->hl + dhvlen;
+	const char *hmac_name, *kpp_name;
+
+	if (chap->buf_size < size) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return NVME_SC_INVALID_FIELD;
+	}
+
+	hmac_name = nvme_auth_hmac_name(data->hashid);
+	if (!hmac_name) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid HASH ID %d\n",
+			 chap->qid, data->hashid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
+		return NVME_SC_INVALID_FIELD;
+	}
+
+	if (chap->hash_id == data->hashid && chap->shash_tfm &&
+	    !strcmp(crypto_shash_alg_name(chap->shash_tfm), hmac_name) &&
+	    crypto_shash_digestsize(chap->shash_tfm) == data->hl) {
+		dev_dbg(ctrl->device,
+			"qid %d: reuse existing hash %s\n",
+			chap->qid, hmac_name);
+		goto select_kpp;
+	}
+
+	/* Reset if hash cannot be reused */
+	if (chap->shash_tfm) {
+		crypto_free_shash(chap->shash_tfm);
+		chap->hash_id = 0;
+		chap->hash_len = 0;
+	}
+	chap->shash_tfm = crypto_alloc_shash(hmac_name, 0,
+					     CRYPTO_ALG_ALLOCATES_MEMORY);
+	if (IS_ERR(chap->shash_tfm)) {
+		dev_warn(ctrl->device,
+			 "qid %d: failed to allocate hash %s, error %ld\n",
+			 chap->qid, hmac_name, PTR_ERR(chap->shash_tfm));
+		chap->shash_tfm = NULL;
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_FAILED;
+		return NVME_SC_AUTH_REQUIRED;
+	}
+
+	if (crypto_shash_digestsize(chap->shash_tfm) != data->hl) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid hash length %d\n",
+			 chap->qid, data->hl);
+		crypto_free_shash(chap->shash_tfm);
+		chap->shash_tfm = NULL;
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
+		return NVME_SC_AUTH_REQUIRED;
+	}
+
+	/* Reset host response if the hash had been changed */
+	if (chap->hash_id != data->hashid) {
+		kfree(chap->host_response);
+		chap->host_response = NULL;
+	}
+
+	chap->hash_id = data->hashid;
+	chap->hash_len = data->hl;
+	dev_dbg(ctrl->device, "qid %d: selected hash %s\n",
+		chap->qid, hmac_name);
+
+select_kpp:
+	kpp_name = nvme_auth_dhgroup_kpp(data->dhgid);
+	if (!kpp_name) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid DH group id %d\n",
+			 chap->qid, data->dhgid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
+		return NVME_SC_AUTH_REQUIRED;
+	}
+
+	if (data->dhgid != NVME_AUTH_DHGROUP_NULL) {
+		dev_warn(ctrl->device,
+			 "qid %d: unsupported DH group %s\n",
+			 chap->qid, kpp_name);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
+		return NVME_SC_AUTH_REQUIRED;
+	} else if (dhvlen != 0) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid DH value for NULL DH\n",
+			 chap->qid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return NVME_SC_INVALID_FIELD;
+	}
+	chap->dhgroup_id = data->dhgid;
+
+	chap->s1 = le32_to_cpu(data->seqnum);
+	memcpy(chap->c1, data->cval, chap->hash_len);
+
+	return 0;
+}
+
+static int nvme_auth_set_dhchap_reply_data(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_reply_data *data = chap->buf;
+	size_t size = sizeof(*data);
+
+	size += 2 * chap->hash_len;
+
+	if (chap->buf_size < size) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return -EINVAL;
+	}
+
+	memset(chap->buf, 0, size);
+	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_REPLY;
+	data->t_id = cpu_to_le16(chap->transaction);
+	data->hl = chap->hash_len;
+	data->dhvlen = 0;
+	memcpy(data->rval, chap->response, chap->hash_len);
+	if (ctrl->opts->dhchap_ctrl_secret) {
+		get_random_bytes(chap->c2, chap->hash_len);
+		data->cvalid = 1;
+		chap->s2 = nvme_auth_get_seqnum();
+		memcpy(data->rval + chap->hash_len, chap->c2,
+		       chap->hash_len);
+		dev_dbg(ctrl->device, "%s: qid %d ctrl challenge %*ph\n",
+			__func__, chap->qid, (int)chap->hash_len, chap->c2);
+	} else {
+		memset(chap->c2, 0, chap->hash_len);
+		chap->s2 = 0;
+	}
+	data->seqnum = cpu_to_le32(chap->s2);
+	return size;
+}
+
+static int nvme_auth_process_dhchap_success1(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_success1_data *data = chap->buf;
+	size_t size = sizeof(*data);
+
+	if (ctrl->opts->dhchap_ctrl_secret)
+		size += chap->hash_len;
+
+	if (chap->buf_size < size) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return NVME_SC_INVALID_FIELD;
+	}
+
+	if (data->hl != chap->hash_len) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid hash length %u\n",
+			 chap->qid, data->hl);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
+		return NVME_SC_INVALID_FIELD;
+	}
+
+	/* Just print out information for the admin queue */
+	if (chap->qid == 0)
+		dev_info(ctrl->device,
+			 "qid 0: authenticated with hash %s dhgroup %s\n",
+			 nvme_auth_hmac_name(chap->hash_id),
+			 nvme_auth_dhgroup_name(chap->dhgroup_id));
+
+	if (!data->rvalid)
+		return 0;
+
+	/* Validate controller response */
+	if (memcmp(chap->response, data->rval, data->hl)) {
+		dev_dbg(ctrl->device, "%s: qid %d ctrl response %*ph\n",
+			__func__, chap->qid, (int)chap->hash_len, data->rval);
+		dev_dbg(ctrl->device, "%s: qid %d host response %*ph\n",
+			__func__, chap->qid, (int)chap->hash_len,
+			chap->response);
+		dev_warn(ctrl->device,
+			 "qid %d: controller authentication failed\n",
+			 chap->qid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_FAILED;
+		return NVME_SC_AUTH_REQUIRED;
+	}
+
+	/* Just print out information for the admin queue */
+	if (chap->qid == 0)
+		dev_info(ctrl->device,
+			 "qid 0: controller authenticated\n");
+	return 0;
+}
+
+static int nvme_auth_set_dhchap_success2_data(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_success2_data *data = chap->buf;
+	size_t size = sizeof(*data);
+
+	memset(chap->buf, 0, size);
+	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_SUCCESS2;
+	data->t_id = cpu_to_le16(chap->transaction);
+
+	return size;
+}
+
+static int nvme_auth_set_dhchap_failure2_data(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_failure_data *data = chap->buf;
+	size_t size = sizeof(*data);
+
+	memset(chap->buf, 0, size);
+	data->auth_type = NVME_AUTH_COMMON_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_FAILURE2;
+	data->t_id = cpu_to_le16(chap->transaction);
+	data->rescode = NVME_AUTH_DHCHAP_FAILURE_REASON_FAILED;
+	data->rescode_exp = chap->status;
+
+	return size;
+}
+
+static int nvme_auth_dhchap_setup_host_response(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
+	u8 buf[4], *challenge = chap->c1;
+	int ret;
+
+	dev_dbg(ctrl->device, "%s: qid %d host response seq %u transaction %d\n",
+		__func__, chap->qid, chap->s1, chap->transaction);
+
+	if (!chap->host_response) {
+		chap->host_response = nvme_auth_transform_key(ctrl->host_key,
+						ctrl->opts->host->nqn);
+		if (IS_ERR(chap->host_response)) {
+			ret = PTR_ERR(chap->host_response);
+			chap->host_response = NULL;
+			return ret;
+		}
+	} else {
+		dev_dbg(ctrl->device, "%s: qid %d re-using host response\n",
+			__func__, chap->qid);
+	}
+
+	ret = crypto_shash_setkey(chap->shash_tfm,
+			chap->host_response, ctrl->host_key->len);
+	if (ret) {
+		dev_warn(ctrl->device, "qid %d: failed to set key, error %d\n",
+			 chap->qid, ret);
+		goto out;
+	}
+
+	shash->tfm = chap->shash_tfm;
+	ret = crypto_shash_init(shash);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, challenge, chap->hash_len);
+	if (ret)
+		goto out;
+	put_unaligned_le32(chap->s1, buf);
+	ret = crypto_shash_update(shash, buf, 4);
+	if (ret)
+		goto out;
+	put_unaligned_le16(chap->transaction, buf);
+	ret = crypto_shash_update(shash, buf, 2);
+	if (ret)
+		goto out;
+	memset(buf, 0, sizeof(buf));
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, "HostHost", 8);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
+				  strlen(ctrl->opts->host->nqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
+			    strlen(ctrl->opts->subsysnqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_final(shash, chap->response);
+out:
+	if (challenge != chap->c1)
+		kfree(challenge);
+	return ret;
+}
+
+static int nvme_auth_dhchap_setup_ctrl_response(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
+	u8 *ctrl_response;
+	u8 buf[4], *challenge = chap->c2;
+	int ret;
+
+	ctrl_response = nvme_auth_transform_key(ctrl->ctrl_key,
+				ctrl->opts->subsysnqn);
+	if (IS_ERR(ctrl_response)) {
+		ret = PTR_ERR(ctrl_response);
+		return ret;
+	}
+	ret = crypto_shash_setkey(chap->shash_tfm,
+			ctrl_response, ctrl->ctrl_key->len);
+	if (ret) {
+		dev_warn(ctrl->device, "qid %d: failed to set key, error %d\n",
+			 chap->qid, ret);
+		goto out;
+	}
+
+	dev_dbg(ctrl->device, "%s: qid %d ctrl response seq %u transaction %d\n",
+		__func__, chap->qid, chap->s2, chap->transaction);
+	dev_dbg(ctrl->device, "%s: qid %d challenge %*ph\n",
+		__func__, chap->qid, (int)chap->hash_len, challenge);
+	dev_dbg(ctrl->device, "%s: qid %d subsysnqn %s\n",
+		__func__, chap->qid, ctrl->opts->subsysnqn);
+	dev_dbg(ctrl->device, "%s: qid %d hostnqn %s\n",
+		__func__, chap->qid, ctrl->opts->host->nqn);
+	shash->tfm = chap->shash_tfm;
+	ret = crypto_shash_init(shash);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, challenge, chap->hash_len);
+	if (ret)
+		goto out;
+	put_unaligned_le32(chap->s2, buf);
+	ret = crypto_shash_update(shash, buf, 4);
+	if (ret)
+		goto out;
+	put_unaligned_le16(chap->transaction, buf);
+	ret = crypto_shash_update(shash, buf, 2);
+	if (ret)
+		goto out;
+	memset(buf, 0, 4);
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, "Controller", 10);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
+				  strlen(ctrl->opts->subsysnqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
+				  strlen(ctrl->opts->host->nqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_final(shash, chap->response);
+out:
+	if (challenge != chap->c2)
+		kfree(challenge);
+	return ret;
+}
+
+static void __nvme_auth_reset(struct nvme_dhchap_queue_context *chap)
+{
+	chap->status = 0;
+	chap->error = 0;
+	chap->s1 = 0;
+	chap->s2 = 0;
+	chap->transaction = 0;
+	memset(chap->c1, 0, sizeof(chap->c1));
+	memset(chap->c2, 0, sizeof(chap->c2));
+}
+
+static void __nvme_auth_free(struct nvme_dhchap_queue_context *chap)
+{
+	__nvme_auth_reset(chap);
+	if (chap->shash_tfm)
+		crypto_free_shash(chap->shash_tfm);
+	kfree_sensitive(chap->host_response);
+	kfree(chap->buf);
+	kfree(chap);
+}
+
+static void __nvme_auth_work(struct work_struct *work)
+{
+	struct nvme_dhchap_queue_context *chap =
+		container_of(work, struct nvme_dhchap_queue_context, auth_work);
+	struct nvme_ctrl *ctrl = chap->ctrl;
+	size_t tl;
+	int ret = 0;
+
+	chap->transaction = ctrl->transaction++;
+
+	/* DH-HMAC-CHAP Step 1: send negotiate */
+	dev_dbg(ctrl->device, "%s: qid %d send negotiate\n",
+		__func__, chap->qid);
+	ret = nvme_auth_set_dhchap_negotiate_data(ctrl, chap);
+	if (ret < 0) {
+		chap->error = ret;
+		return;
+	}
+	tl = ret;
+	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, tl, true);
+	if (ret) {
+		chap->error = ret;
+		return;
+	}
+
+	/* DH-HMAC-CHAP Step 2: receive challenge */
+	dev_dbg(ctrl->device, "%s: qid %d receive challenge\n",
+		__func__, chap->qid);
+
+	memset(chap->buf, 0, chap->buf_size);
+	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, chap->buf_size, false);
+	if (ret) {
+		dev_warn(ctrl->device,
+			 "qid %d failed to receive challenge, %s %d\n",
+			 chap->qid, ret < 0 ? "error" : "nvme status", ret);
+		chap->error = ret;
+		return;
+	}
+	ret = nvme_auth_receive_validate(ctrl, chap->qid, chap->buf, chap->transaction,
+					 NVME_AUTH_DHCHAP_MESSAGE_CHALLENGE);
+	if (ret) {
+		chap->status = ret;
+		chap->error = NVME_SC_AUTH_REQUIRED;
+		return;
+	}
+
+	ret = nvme_auth_process_dhchap_challenge(ctrl, chap);
+	if (ret) {
+		/* Invalid challenge parameters */
+		goto fail2;
+	}
+
+	dev_dbg(ctrl->device, "%s: qid %d host response\n",
+		__func__, chap->qid);
+	ret = nvme_auth_dhchap_setup_host_response(ctrl, chap);
+	if (ret)
+		goto fail2;
+
+	/* DH-HMAC-CHAP Step 3: send reply */
+	dev_dbg(ctrl->device, "%s: qid %d send reply\n",
+		__func__, chap->qid);
+	ret = nvme_auth_set_dhchap_reply_data(ctrl, chap);
+	if (ret < 0)
+		goto fail2;
+
+	tl = ret;
+	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, tl, true);
+	if (ret)
+		goto fail2;
+
+	/* DH-HMAC-CHAP Step 4: receive success1 */
+	dev_dbg(ctrl->device, "%s: qid %d receive success1\n",
+		__func__, chap->qid);
+
+	memset(chap->buf, 0, chap->buf_size);
+	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, chap->buf_size, false);
+	if (ret) {
+		dev_warn(ctrl->device,
+			 "qid %d failed to receive success1, %s %d\n",
+			 chap->qid, ret < 0 ? "error" : "nvme status", ret);
+		chap->error = ret;
+		return;
+	}
+	ret = nvme_auth_receive_validate(ctrl, chap->qid,
+					 chap->buf, chap->transaction,
+					 NVME_AUTH_DHCHAP_MESSAGE_SUCCESS1);
+	if (ret) {
+		chap->status = ret;
+		chap->error = NVME_SC_AUTH_REQUIRED;
+		return;
+	}
+
+	if (ctrl->opts->dhchap_ctrl_secret) {
+		dev_dbg(ctrl->device,
+			"%s: qid %d controller response\n",
+			__func__, chap->qid);
+		ret = nvme_auth_dhchap_setup_ctrl_response(ctrl, chap);
+		if (ret) {
+			chap->error = ret;
+			goto fail2;
+		}
+	}
+
+	ret = nvme_auth_process_dhchap_success1(ctrl, chap);
+	if (ret) {
+		/* Controller authentication failed */
+		chap->error = NVME_SC_AUTH_REQUIRED;
+		goto fail2;
+	}
+
+	if (ctrl->opts->dhchap_ctrl_secret) {
+		/* DH-HMAC-CHAP Step 5: send success2 */
+		dev_dbg(ctrl->device, "%s: qid %d send success2\n",
+			__func__, chap->qid);
+		tl = nvme_auth_set_dhchap_success2_data(ctrl, chap);
+		ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, tl, true);
+	}
+	if (!ret) {
+		chap->error = 0;
+		return;
+	}
+
+fail2:
+	dev_dbg(ctrl->device, "%s: qid %d send failure2, status %x\n",
+		__func__, chap->qid, chap->status);
+	tl = nvme_auth_set_dhchap_failure2_data(ctrl, chap);
+	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, tl, true);
+	/*
+	 * only update error if send failure2 failed and no other
+	 * error had been set during authentication.
+	 */
+	if (ret && !chap->error)
+		chap->error = ret;
+}
+
+int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid)
+{
+	struct nvme_dhchap_queue_context *chap;
+
+	if (!ctrl->host_key) {
+		dev_warn(ctrl->device, "qid %d: no key\n", qid);
+		return -ENOKEY;
+	}
+
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	/* Check if the context is already queued */
+	list_for_each_entry(chap, &ctrl->dhchap_auth_list, entry) {
+		WARN_ON(!chap->buf);
+		if (chap->qid == qid) {
+			dev_dbg(ctrl->device, "qid %d: re-using context\n", qid);
+			mutex_unlock(&ctrl->dhchap_auth_mutex);
+			flush_work(&chap->auth_work);
+			__nvme_auth_reset(chap);
+			queue_work(nvme_wq, &chap->auth_work);
+			return 0;
+		}
+	}
+	chap = kzalloc(sizeof(*chap), GFP_KERNEL);
+	if (!chap) {
+		mutex_unlock(&ctrl->dhchap_auth_mutex);
+		return -ENOMEM;
+	}
+	chap->qid = (qid == NVME_QID_ANY) ? 0 : qid;
+	chap->ctrl = ctrl;
+
+	/*
+	 * Allocate a large enough buffer for the entire negotiation:
+	 * 4k should be enough to ffdhe8192.
+	 */
+	chap->buf_size = 4096;
+	chap->buf = kzalloc(chap->buf_size, GFP_KERNEL);
+	if (!chap->buf) {
+		mutex_unlock(&ctrl->dhchap_auth_mutex);
+		kfree(chap);
+		return -ENOMEM;
+	}
+
+	INIT_WORK(&chap->auth_work, __nvme_auth_work);
+	list_add(&chap->entry, &ctrl->dhchap_auth_list);
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+	queue_work(nvme_wq, &chap->auth_work);
+	return 0;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_negotiate);
+
+int nvme_auth_wait(struct nvme_ctrl *ctrl, int qid)
+{
+	struct nvme_dhchap_queue_context *chap;
+	int ret;
+
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	list_for_each_entry(chap, &ctrl->dhchap_auth_list, entry) {
+		if (chap->qid != qid)
+			continue;
+		mutex_unlock(&ctrl->dhchap_auth_mutex);
+		flush_work(&chap->auth_work);
+		ret = chap->error;
+		return ret;
+	}
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+	return -ENXIO;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_wait);
+
+void nvme_auth_reset(struct nvme_ctrl *ctrl)
+{
+	struct nvme_dhchap_queue_context *chap;
+
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	list_for_each_entry(chap, &ctrl->dhchap_auth_list, entry) {
+		mutex_unlock(&ctrl->dhchap_auth_mutex);
+		flush_work(&chap->auth_work);
+		__nvme_auth_reset(chap);
+	}
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_reset);
+
+static void nvme_dhchap_auth_work(struct work_struct *work)
+{
+	struct nvme_ctrl *ctrl =
+		container_of(work, struct nvme_ctrl, dhchap_auth_work);
+	int ret, q;
+
+	/* Authenticate admin queue first */
+	ret = nvme_auth_negotiate(ctrl, 0);
+	if (ret) {
+		dev_warn(ctrl->device,
+			 "qid 0: error %d setting up authentication\n", ret);
+		return;
+	}
+	ret = nvme_auth_wait(ctrl, 0);
+	if (ret) {
+		dev_warn(ctrl->device,
+			 "qid 0: authentication failed\n");
+		return;
+	}
+
+	for (q = 1; q < ctrl->queue_count; q++) {
+		ret = nvme_auth_negotiate(ctrl, q);
+		if (ret) {
+			dev_warn(ctrl->device,
+				 "qid %d: error %d setting up authentication\n",
+				 q, ret);
+			break;
+		}
+	}
+
+	/*
+	 * Failure is a soft-state; credentials remain valid until
+	 * the controller terminates the connection.
+	 */
+}
+
+void nvme_auth_init_ctrl(struct nvme_ctrl *ctrl)
+{
+	INIT_LIST_HEAD(&ctrl->dhchap_auth_list);
+	INIT_WORK(&ctrl->dhchap_auth_work, nvme_dhchap_auth_work);
+	mutex_init(&ctrl->dhchap_auth_mutex);
+	if (!ctrl->opts)
+		return;
+	nvme_auth_generate_key(ctrl->opts->dhchap_secret, &ctrl->host_key);
+	nvme_auth_generate_key(ctrl->opts->dhchap_ctrl_secret, &ctrl->ctrl_key);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_init_ctrl);
+
+void nvme_auth_stop(struct nvme_ctrl *ctrl)
+{
+	struct nvme_dhchap_queue_context *chap = NULL, *tmp;
+
+	cancel_work_sync(&ctrl->dhchap_auth_work);
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	list_for_each_entry_safe(chap, tmp, &ctrl->dhchap_auth_list, entry)
+		cancel_work_sync(&chap->auth_work);
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_stop);
+
+void nvme_auth_free(struct nvme_ctrl *ctrl)
+{
+	struct nvme_dhchap_queue_context *chap = NULL, *tmp;
+
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	list_for_each_entry_safe(chap, tmp, &ctrl->dhchap_auth_list, entry) {
+		list_del_init(&chap->entry);
+		flush_work(&chap->auth_work);
+		__nvme_auth_free(chap);
+	}
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+	if (ctrl->host_key) {
+		nvme_auth_free_key(ctrl->host_key);
+		ctrl->host_key = NULL;
+	}
+	if (ctrl->ctrl_key) {
+		nvme_auth_free_key(ctrl->ctrl_key);
+		ctrl->ctrl_key = NULL;
+	}
+}
+EXPORT_SYMBOL_GPL(nvme_auth_free);
diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 3ab2cfd254a4..8bf23791a5a6 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -24,6 +24,7 @@
 
 #include "nvme.h"
 #include "fabrics.h"
+#include <linux/nvme-auth.h>
 
 #define CREATE_TRACE_POINTS
 #include "trace.h"
@@ -330,6 +331,7 @@ enum nvme_disposition {
 	COMPLETE,
 	RETRY,
 	FAILOVER,
+	AUTHENTICATE,
 };
 
 static inline enum nvme_disposition nvme_decide_disposition(struct request *req)
@@ -337,6 +339,9 @@ static inline enum nvme_disposition nvme_decide_disposition(struct request *req)
 	if (likely(nvme_req(req)->status == 0))
 		return COMPLETE;
 
+	if ((nvme_req(req)->status & 0x7ff) == NVME_SC_AUTH_REQUIRED)
+		return AUTHENTICATE;
+
 	if (blk_noretry_request(req) ||
 	    (nvme_req(req)->status & NVME_SC_DNR) ||
 	    nvme_req(req)->retries >= nvme_max_retries)
@@ -375,11 +380,13 @@ static inline void nvme_end_req(struct request *req)
 
 void nvme_complete_rq(struct request *req)
 {
+	struct nvme_ctrl *ctrl = nvme_req(req)->ctrl;
+
 	trace_nvme_complete_rq(req);
 	nvme_cleanup_cmd(req);
 
-	if (nvme_req(req)->ctrl->kas)
-		nvme_req(req)->ctrl->comp_seen = true;
+	if (ctrl->kas)
+		ctrl->comp_seen = true;
 
 	switch (nvme_decide_disposition(req)) {
 	case COMPLETE:
@@ -391,6 +398,14 @@ void nvme_complete_rq(struct request *req)
 	case FAILOVER:
 		nvme_failover_req(req);
 		return;
+	case AUTHENTICATE:
+#ifdef CONFIG_NVME_AUTH
+		queue_work(nvme_wq, &ctrl->dhchap_auth_work);
+		nvme_retry_req(req);
+#else
+		nvme_end_req(req);
+#endif
+		return;
 	}
 }
 EXPORT_SYMBOL_GPL(nvme_complete_rq);
@@ -702,7 +717,9 @@ bool __nvme_check_ready(struct nvme_ctrl *ctrl, struct request *rq,
 		switch (ctrl->state) {
 		case NVME_CTRL_CONNECTING:
 			if (blk_rq_is_passthrough(rq) && nvme_is_fabrics(req->cmd) &&
-			    req->cmd->fabrics.fctype == nvme_fabrics_type_connect)
+			    (req->cmd->fabrics.fctype == nvme_fabrics_type_connect ||
+			     req->cmd->fabrics.fctype == nvme_fabrics_type_auth_send ||
+			     req->cmd->fabrics.fctype == nvme_fabrics_type_auth_receive))
 				return true;
 			break;
 		default:
@@ -3599,6 +3616,108 @@ static ssize_t dctype_show(struct device *dev,
 }
 static DEVICE_ATTR_RO(dctype);
 
+#ifdef CONFIG_NVME_AUTH
+static ssize_t nvme_ctrl_dhchap_secret_show(struct device *dev,
+		struct device_attribute *attr, char *buf)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	struct nvmf_ctrl_options *opts = ctrl->opts;
+
+	if (!opts->dhchap_secret)
+		return sysfs_emit(buf, "none\n");
+	return sysfs_emit(buf, "%s\n", opts->dhchap_secret);
+}
+
+static ssize_t nvme_ctrl_dhchap_secret_store(struct device *dev,
+		struct device_attribute *attr, const char *buf, size_t count)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	struct nvmf_ctrl_options *opts = ctrl->opts;
+	char *dhchap_secret;
+
+	if (!ctrl->opts->dhchap_secret)
+		return -EINVAL;
+	if (count < 7)
+		return -EINVAL;
+	if (memcmp(buf, "DHHC-1:", 7))
+		return -EINVAL;
+
+	dhchap_secret = kzalloc(count + 1, GFP_KERNEL);
+	if (!dhchap_secret)
+		return -ENOMEM;
+	memcpy(dhchap_secret, buf, count);
+	nvme_auth_stop(ctrl);
+	if (strcmp(dhchap_secret, opts->dhchap_secret)) {
+		int ret;
+
+		ret = nvme_auth_generate_key(dhchap_secret, &ctrl->host_key);
+		if (ret)
+			return ret;
+		kfree(opts->dhchap_secret);
+		opts->dhchap_secret = dhchap_secret;
+		/* Key has changed; re-authentication with new key */
+		nvme_auth_reset(ctrl);
+	}
+	/* Start re-authentication */
+	dev_info(ctrl->device, "re-authenticating controller\n");
+	queue_work(nvme_wq, &ctrl->dhchap_auth_work);
+
+	return count;
+}
+static DEVICE_ATTR(dhchap_secret, S_IRUGO | S_IWUSR,
+	nvme_ctrl_dhchap_secret_show, nvme_ctrl_dhchap_secret_store);
+
+static ssize_t nvme_ctrl_dhchap_ctrl_secret_show(struct device *dev,
+		struct device_attribute *attr, char *buf)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	struct nvmf_ctrl_options *opts = ctrl->opts;
+
+	if (!opts->dhchap_ctrl_secret)
+		return sysfs_emit(buf, "none\n");
+	return sysfs_emit(buf, "%s\n", opts->dhchap_ctrl_secret);
+}
+
+static ssize_t nvme_ctrl_dhchap_ctrl_secret_store(struct device *dev,
+		struct device_attribute *attr, const char *buf, size_t count)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	struct nvmf_ctrl_options *opts = ctrl->opts;
+	char *dhchap_secret;
+
+	if (!ctrl->opts->dhchap_ctrl_secret)
+		return -EINVAL;
+	if (count < 7)
+		return -EINVAL;
+	if (memcmp(buf, "DHHC-1:", 7))
+		return -EINVAL;
+
+	dhchap_secret = kzalloc(count + 1, GFP_KERNEL);
+	if (!dhchap_secret)
+		return -ENOMEM;
+	memcpy(dhchap_secret, buf, count);
+	nvme_auth_stop(ctrl);
+	if (strcmp(dhchap_secret, opts->dhchap_ctrl_secret)) {
+		int ret;
+
+		ret = nvme_auth_generate_key(dhchap_secret, &ctrl->ctrl_key);
+		if (ret)
+			return ret;
+		kfree(opts->dhchap_ctrl_secret);
+		opts->dhchap_ctrl_secret = dhchap_secret;
+		/* Key has changed; re-authentication with new key */
+		nvme_auth_reset(ctrl);
+	}
+	/* Start re-authentication */
+	dev_info(ctrl->device, "re-authenticating controller\n");
+	queue_work(nvme_wq, &ctrl->dhchap_auth_work);
+
+	return count;
+}
+static DEVICE_ATTR(dhchap_ctrl_secret, S_IRUGO | S_IWUSR,
+	nvme_ctrl_dhchap_ctrl_secret_show, nvme_ctrl_dhchap_ctrl_secret_store);
+#endif
+
 static struct attribute *nvme_dev_attrs[] = {
 	&dev_attr_reset_controller.attr,
 	&dev_attr_rescan_controller.attr,
@@ -3622,6 +3741,10 @@ static struct attribute *nvme_dev_attrs[] = {
 	&dev_attr_kato.attr,
 	&dev_attr_cntrltype.attr,
 	&dev_attr_dctype.attr,
+#ifdef CONFIG_NVME_AUTH
+	&dev_attr_dhchap_secret.attr,
+	&dev_attr_dhchap_ctrl_secret.attr,
+#endif
 	NULL
 };
 
@@ -3645,6 +3768,12 @@ static umode_t nvme_dev_attrs_are_visible(struct kobject *kobj,
 		return 0;
 	if (a == &dev_attr_fast_io_fail_tmo.attr && !ctrl->opts)
 		return 0;
+#ifdef CONFIG_NVME_AUTH
+	if (a == &dev_attr_dhchap_secret.attr && !ctrl->opts)
+		return 0;
+	if (a == &dev_attr_dhchap_ctrl_secret.attr && !ctrl->opts)
+		return 0;
+#endif
 
 	return a->mode;
 }
@@ -4529,8 +4658,10 @@ static void nvme_handle_aen_notice(struct nvme_ctrl *ctrl, u32 result)
 		 * recovery actions from interfering with the controller's
 		 * firmware activation.
 		 */
-		if (nvme_change_ctrl_state(ctrl, NVME_CTRL_RESETTING))
+		if (nvme_change_ctrl_state(ctrl, NVME_CTRL_RESETTING)) {
+			nvme_auth_stop(ctrl);
 			queue_work(nvme_wq, &ctrl->fw_act_work);
+		}
 		break;
 #ifdef CONFIG_NVME_MULTIPATH
 	case NVME_AER_NOTICE_ANA:
@@ -4577,6 +4708,7 @@ EXPORT_SYMBOL_GPL(nvme_complete_async_event);
 void nvme_stop_ctrl(struct nvme_ctrl *ctrl)
 {
 	nvme_mpath_stop(ctrl);
+	nvme_auth_stop(ctrl);
 	nvme_stop_keep_alive(ctrl);
 	nvme_stop_failfast_work(ctrl);
 	flush_work(&ctrl->async_event_work);
@@ -4634,6 +4766,8 @@ static void nvme_free_ctrl(struct device *dev)
 
 	nvme_free_cels(ctrl);
 	nvme_mpath_uninit(ctrl);
+	nvme_auth_stop(ctrl);
+	nvme_auth_free(ctrl);
 	__free_page(ctrl->discard_page);
 
 	if (subsys) {
@@ -4724,6 +4858,7 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev,
 
 	nvme_fault_inject_init(&ctrl->fault_inject, dev_name(ctrl->device));
 	nvme_mpath_init_ctrl(ctrl);
+	nvme_auth_init_ctrl(ctrl);
 
 	return 0;
 out_free_name:
diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index 953e3076aab1..ac10bb426a59 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -369,6 +369,7 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)
 	union nvme_result res;
 	struct nvmf_connect_data *data;
 	int ret;
+	u32 result;
 
 	cmd.connect.opcode = nvme_fabrics_command;
 	cmd.connect.fctype = nvme_fabrics_type_connect;
@@ -401,8 +402,25 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)
 		goto out_free_data;
 	}
 
-	ctrl->cntlid = le16_to_cpu(res.u16);
-
+	result = le32_to_cpu(res.u32);
+	ctrl->cntlid = result & 0xFFFF;
+	if ((result >> 16) & 0x3) {
+		/* Authentication required */
+		ret = nvme_auth_negotiate(ctrl, 0);
+		if (ret) {
+			dev_warn(ctrl->device,
+				 "qid 0: authentication setup failed\n");
+			ret = NVME_SC_AUTH_REQUIRED;
+			goto out_free_data;
+		}
+		ret = nvme_auth_wait(ctrl, 0);
+		if (ret)
+			dev_warn(ctrl->device,
+				 "qid 0: authentication failed\n");
+		else
+			dev_info(ctrl->device,
+				 "qid 0: authenticated\n");
+	}
 out_free_data:
 	kfree(data);
 	return ret;
@@ -435,6 +453,7 @@ int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, u16 qid)
 	struct nvmf_connect_data *data;
 	union nvme_result res;
 	int ret;
+	u32 result;
 
 	cmd.connect.opcode = nvme_fabrics_command;
 	cmd.connect.fctype = nvme_fabrics_type_connect;
@@ -460,6 +479,21 @@ int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, u16 qid)
 		nvmf_log_connect_error(ctrl, ret, le32_to_cpu(res.u32),
 				       &cmd, data);
 	}
+	result = le32_to_cpu(res.u32);
+	if ((result >> 16) & 2) {
+		/* Authentication required */
+		ret = nvme_auth_negotiate(ctrl, qid);
+		if (ret) {
+			dev_warn(ctrl->device,
+				 "qid %d: authentication setup failed\n", qid);
+			ret = NVME_SC_AUTH_REQUIRED;
+		} else {
+			ret = nvme_auth_wait(ctrl, qid);
+			if (ret)
+				dev_warn(ctrl->device,
+					 "qid %u: authentication failed\n", qid);
+		}
+	}
 	kfree(data);
 	return ret;
 }
@@ -552,6 +586,8 @@ static const match_table_t opt_tokens = {
 	{ NVMF_OPT_TOS,			"tos=%d"		},
 	{ NVMF_OPT_FAIL_FAST_TMO,	"fast_io_fail_tmo=%d"	},
 	{ NVMF_OPT_DISCOVERY,		"discovery"		},
+	{ NVMF_OPT_DHCHAP_SECRET,	"dhchap_secret=%s"	},
+	{ NVMF_OPT_DHCHAP_CTRL_SECRET,	"dhchap_ctrl_secret=%s"	},
 	{ NVMF_OPT_ERR,			NULL			}
 };
 
@@ -833,6 +869,34 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
 		case NVMF_OPT_DISCOVERY:
 			opts->discovery_nqn = true;
 			break;
+		case NVMF_OPT_DHCHAP_SECRET:
+			p = match_strdup(args);
+			if (!p) {
+				ret = -ENOMEM;
+				goto out;
+			}
+			if (strlen(p) < 11 || strncmp(p, "DHHC-1:", 7)) {
+				pr_err("Invalid DH-CHAP secret %s\n", p);
+				ret = -EINVAL;
+				goto out;
+			}
+			kfree(opts->dhchap_secret);
+			opts->dhchap_secret = p;
+			break;
+		case NVMF_OPT_DHCHAP_CTRL_SECRET:
+			p = match_strdup(args);
+			if (!p) {
+				ret = -ENOMEM;
+				goto out;
+			}
+			if (strlen(p) < 11 || strncmp(p, "DHHC-1:", 7)) {
+				pr_err("Invalid DH-CHAP secret %s\n", p);
+				ret = -EINVAL;
+				goto out;
+			}
+			kfree(opts->dhchap_ctrl_secret);
+			opts->dhchap_ctrl_secret = p;
+			break;
 		default:
 			pr_warn("unknown parameter or missing value '%s' in ctrl creation request\n",
 				p);
@@ -951,6 +1015,7 @@ void nvmf_free_options(struct nvmf_ctrl_options *opts)
 	kfree(opts->subsysnqn);
 	kfree(opts->host_traddr);
 	kfree(opts->host_iface);
+	kfree(opts->dhchap_secret);
 	kfree(opts);
 }
 EXPORT_SYMBOL_GPL(nvmf_free_options);
@@ -960,7 +1025,8 @@ EXPORT_SYMBOL_GPL(nvmf_free_options);
 				 NVMF_OPT_KATO | NVMF_OPT_HOSTNQN | \
 				 NVMF_OPT_HOST_ID | NVMF_OPT_DUP_CONNECT |\
 				 NVMF_OPT_DISABLE_SQFLOW | NVMF_OPT_DISCOVERY |\
-				 NVMF_OPT_FAIL_FAST_TMO)
+				 NVMF_OPT_FAIL_FAST_TMO | NVMF_OPT_DHCHAP_SECRET |\
+				 NVMF_OPT_DHCHAP_CTRL_SECRET)
 
 static struct nvme_ctrl *
 nvmf_create_ctrl(struct device *dev, const char *buf)
@@ -1196,7 +1262,14 @@ static void __exit nvmf_exit(void)
 	BUILD_BUG_ON(sizeof(struct nvmf_connect_command) != 64);
 	BUILD_BUG_ON(sizeof(struct nvmf_property_get_command) != 64);
 	BUILD_BUG_ON(sizeof(struct nvmf_property_set_command) != 64);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_send_command) != 64);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_receive_command) != 64);
 	BUILD_BUG_ON(sizeof(struct nvmf_connect_data) != 1024);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_negotiate_data) != 8);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_challenge_data) != 16);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_reply_data) != 16);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_success1_data) != 16);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_success2_data) != 16);
 }
 
 MODULE_LICENSE("GPL v2");
diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h
index 46d6e194ac2b..a6e22116e139 100644
--- a/drivers/nvme/host/fabrics.h
+++ b/drivers/nvme/host/fabrics.h
@@ -68,6 +68,8 @@ enum {
 	NVMF_OPT_FAIL_FAST_TMO	= 1 << 20,
 	NVMF_OPT_HOST_IFACE	= 1 << 21,
 	NVMF_OPT_DISCOVERY	= 1 << 22,
+	NVMF_OPT_DHCHAP_SECRET	= 1 << 23,
+	NVMF_OPT_DHCHAP_CTRL_SECRET = 1 << 24,
 };
 
 /**
@@ -97,6 +99,9 @@ enum {
  * @max_reconnects: maximum number of allowed reconnect attempts before removing
  *              the controller, (-1) means reconnect forever, zero means remove
  *              immediately;
+ * @dhchap_secret: DH-HMAC-CHAP secret
+ * @dhchap_ctrl_secret: DH-HMAC-CHAP controller secret for bi-directional
+ *              authentication
  * @disable_sqflow: disable controller sq flow control
  * @hdr_digest: generate/verify header digest (TCP)
  * @data_digest: generate/verify data digest (TCP)
@@ -121,6 +126,8 @@ struct nvmf_ctrl_options {
 	unsigned int		kato;
 	struct nvmf_host	*host;
 	int			max_reconnects;
+	char			*dhchap_secret;
+	char			*dhchap_ctrl_secret;
 	bool			disable_sqflow;
 	bool			hdr_digest;
 	bool			data_digest;
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 0da94b233fed..798ce748f244 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -328,6 +328,15 @@ struct nvme_ctrl {
 	struct work_struct ana_work;
 #endif
 
+#ifdef CONFIG_NVME_AUTH
+	struct work_struct dhchap_auth_work;
+	struct list_head dhchap_auth_list;
+	struct mutex dhchap_auth_mutex;
+	struct nvme_dhchap_key *host_key;
+	struct nvme_dhchap_key *ctrl_key;
+	u16 transaction;
+#endif
+
 	/* Power saving configuration */
 	u64 ps_max_latency_us;
 	bool apst_enabled;
@@ -991,6 +1000,27 @@ static inline bool nvme_ctrl_sgl_supported(struct nvme_ctrl *ctrl)
 	return ctrl->sgls & ((1 << 0) | (1 << 1));
 }
 
+#ifdef CONFIG_NVME_AUTH
+void nvme_auth_init_ctrl(struct nvme_ctrl *ctrl);
+void nvme_auth_stop(struct nvme_ctrl *ctrl);
+int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid);
+int nvme_auth_wait(struct nvme_ctrl *ctrl, int qid);
+void nvme_auth_reset(struct nvme_ctrl *ctrl);
+void nvme_auth_free(struct nvme_ctrl *ctrl);
+#else
+static inline void nvme_auth_init_ctrl(struct nvme_ctrl *ctrl) {};
+static inline void nvme_auth_stop(struct nvme_ctrl *ctrl) {};
+static inline int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid)
+{
+	return -EPROTONOSUPPORT;
+}
+static inline int nvme_auth_wait(struct nvme_ctrl *ctrl, int qid)
+{
+	return NVME_SC_AUTH_REQUIRED;
+}
+static inline void nvme_auth_free(struct nvme_ctrl *ctrl) {};
+#endif
+
 u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 			 u8 opcode);
 int nvme_execute_passthru_rq(struct request *rq);
diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index f2a5e1ea508a..84ce3347d158 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -1197,6 +1197,7 @@ static void nvme_rdma_error_recovery_work(struct work_struct *work)
 	struct nvme_rdma_ctrl *ctrl = container_of(work,
 			struct nvme_rdma_ctrl, err_work);
 
+	nvme_auth_stop(&ctrl->ctrl);
 	nvme_stop_keep_alive(&ctrl->ctrl);
 	flush_work(&ctrl->ctrl.async_event_work);
 	nvme_rdma_teardown_io_queues(ctrl, false);
diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index bb67538d241b..a7848e430a5c 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -2174,6 +2174,7 @@ static void nvme_tcp_error_recovery_work(struct work_struct *work)
 				struct nvme_tcp_ctrl, err_work);
 	struct nvme_ctrl *ctrl = &tcp_ctrl->ctrl;
 
+	nvme_auth_stop(ctrl);
 	nvme_stop_keep_alive(ctrl);
 	flush_work(&ctrl->async_event_work);
 	nvme_tcp_teardown_io_queues(ctrl, false);
diff --git a/drivers/nvme/host/trace.c b/drivers/nvme/host/trace.c
index 2a89c5aa0790..1c36fcedea20 100644
--- a/drivers/nvme/host/trace.c
+++ b/drivers/nvme/host/trace.c
@@ -287,6 +287,34 @@ static const char *nvme_trace_fabrics_property_get(struct trace_seq *p, u8 *spc)
 	return ret;
 }
 
+static const char *nvme_trace_fabrics_auth_send(struct trace_seq *p, u8 *spc)
+{
+	const char *ret = trace_seq_buffer_ptr(p);
+	u8 spsp0 = spc[1];
+	u8 spsp1 = spc[2];
+	u8 secp = spc[3];
+	u32 tl = get_unaligned_le32(spc + 4);
+
+	trace_seq_printf(p, "spsp0=%02x, spsp1=%02x, secp=%02x, tl=%u",
+			 spsp0, spsp1, secp, tl);
+	trace_seq_putc(p, 0);
+	return ret;
+}
+
+static const char *nvme_trace_fabrics_auth_receive(struct trace_seq *p, u8 *spc)
+{
+	const char *ret = trace_seq_buffer_ptr(p);
+	u8 spsp0 = spc[1];
+	u8 spsp1 = spc[2];
+	u8 secp = spc[3];
+	u32 al = get_unaligned_le32(spc + 4);
+
+	trace_seq_printf(p, "spsp0=%02x, spsp1=%02x, secp=%02x, al=%u",
+			 spsp0, spsp1, secp, al);
+	trace_seq_putc(p, 0);
+	return ret;
+}
+
 static const char *nvme_trace_fabrics_common(struct trace_seq *p, u8 *spc)
 {
 	const char *ret = trace_seq_buffer_ptr(p);
@@ -306,6 +334,10 @@ const char *nvme_trace_parse_fabrics_cmd(struct trace_seq *p,
 		return nvme_trace_fabrics_connect(p, spc);
 	case nvme_fabrics_type_property_get:
 		return nvme_trace_fabrics_property_get(p, spc);
+	case nvme_fabrics_type_auth_send:
+		return nvme_trace_fabrics_auth_send(p, spc);
+	case nvme_fabrics_type_auth_receive:
+		return nvme_trace_fabrics_auth_receive(p, spc);
 	default:
 		return nvme_trace_fabrics_common(p, spc);
 	}
diff --git a/include/linux/nvme-auth.h b/include/linux/nvme-auth.h
new file mode 100644
index 000000000000..354456826221
--- /dev/null
+++ b/include/linux/nvme-auth.h
@@ -0,0 +1,33 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2021 Hannes Reinecke, SUSE Software Solutions
+ */
+
+#ifndef _NVME_AUTH_H
+#define _NVME_AUTH_H
+
+#include <crypto/kpp.h>
+
+struct nvme_dhchap_key {
+	u8 *key;
+	size_t len;
+	u8 hash;
+};
+
+u32 nvme_auth_get_seqnum(void);
+const char *nvme_auth_dhgroup_name(u8 dhgroup_id);
+const char *nvme_auth_dhgroup_kpp(u8 dhgroup_id);
+u8 nvme_auth_dhgroup_id(const char *dhgroup_name);
+
+const char *nvme_auth_hmac_name(u8 hmac_id);
+const char *nvme_auth_digest_name(u8 hmac_id);
+size_t nvme_auth_hmac_hash_len(u8 hmac_id);
+u8 nvme_auth_hmac_id(const char *hmac_name);
+
+struct nvme_dhchap_key *nvme_auth_extract_key(unsigned char *secret,
+					      u8 key_hash);
+void nvme_auth_free_key(struct nvme_dhchap_key *key);
+u8 *nvme_auth_transform_key(struct nvme_dhchap_key *key, char *nqn);
+int nvme_auth_generate_key(u8 *secret, struct nvme_dhchap_key **ret_key);
+
+#endif /* _NVME_AUTH_H */
-- 
2.29.2

Re: [PATCH 06/11] nvme: Implement In-Band authentication

[Hannes Reinecke]

On 6/13/22 20:12, Christoph Hellwig wrote:
> On Wed, Jun 08, 2022 at 04:45:13PM +0200, Hannes Reinecke wrote:
>> Implement NVMe-oF In-Band authentication according to NVMe TPAR 8006.
>> This patch adds two new fabric options 'dhchap_secret' to specify the
>> pre-shared key (in ASCII respresentation according to NVMe 2.0 section
>> 8.13.5.8 'Secret representation') and 'dhchap_ctrl_secret' to specify
>> the pre-shared controller key for bi-directional authentication of both
>> the host and the controller.
>> Re-authentication can be triggered by writing the PSK into the new
>> controller sysfs attribute 'dhchap_secret' or 'dhchap_ctrl_secret'.
> 
> The Kconfig doesn't apply due to the new nvme-apple driver.
> 
> Also when I fix it up manually Kconfig and sparse are both not
> happy:
> 
> WARNING: unmet direct dependencies detected for CRYPTO_DH_RFC7919_GROUPS
>    Depends on [n]: CRYPTO [=y] && CRYPTO_DH [=n]
>    Selected by [y]:
>    - NVME_AUTH [=y] && NVME_CORE [=y]
> 
> i.e. when you select something you also need to select its dependencies
> 
> sparse also isn't exactly happy with the series:
> 
> rivers/nvme/host/core.c:3667:1: warning: symbol 'dev_attr_dhchap_secret' was not declared. Should it be static?
> drivers/nvme/host/core.c:3717:1: warning: symbol 'dev_attr_dhchap_ctrl_secret' was not declared. Should it be static?
> 
> drivers/nvme/target/fabrics-cmd-auth.c:33:30: warning: invalid assignment: |=
> drivers/nvme/target/fabrics-cmd-auth.c:33:30:    left side has type restricted __le32
> drivers/nvme/target/fabrics-cmd-auth.c:33:30:    right side has type int
> drivers/nvme/target/fabrics-cmd-auth.c:118:22: warning: cast to restricted __le32
> drivers/nvme/target/fabrics-cmd-auth.c:118:22: warning: cast from restricted __le16
> drivers/nvme/target/fabrics-cmd-auth.c:389:30: warning: incorrect type in assignment (different base types)
> drivers/nvme/target/fabrics-cmd-auth.c:389:30:    expected restricted __le16 [usertype] dhvlen
> drivers/nvme/target/fabrics-cmd-auth.c:389:30:    got restricted __le32 [usertype]

Right. Will be fixing it up.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman

Re: [PATCH 06/11] nvme: Implement In-Band authentication

[Christoph Hellwig]

On Wed, Jun 08, 2022 at 04:45:13PM +0200, Hannes Reinecke wrote:
> Implement NVMe-oF In-Band authentication according to NVMe TPAR 8006.
> This patch adds two new fabric options 'dhchap_secret' to specify the
> pre-shared key (in ASCII respresentation according to NVMe 2.0 section
> 8.13.5.8 'Secret representation') and 'dhchap_ctrl_secret' to specify
> the pre-shared controller key for bi-directional authentication of both
> the host and the controller.
> Re-authentication can be triggered by writing the PSK into the new
> controller sysfs attribute 'dhchap_secret' or 'dhchap_ctrl_secret'.

The Kconfig doesn't apply due to the new nvme-apple driver.

Also when I fix it up manually Kconfig and sparse are both not
happy:

WARNING: unmet direct dependencies detected for CRYPTO_DH_RFC7919_GROUPS
  Depends on [n]: CRYPTO [=y] && CRYPTO_DH [=n]
  Selected by [y]:
  - NVME_AUTH [=y] && NVME_CORE [=y]

i.e. when you select something you also need to select its dependencies

sparse also isn't exactly happy with the series:

rivers/nvme/host/core.c:3667:1: warning: symbol 'dev_attr_dhchap_secret' was not declared. Should it be static?
drivers/nvme/host/core.c:3717:1: warning: symbol 'dev_attr_dhchap_ctrl_secret' was not declared. Should it be static?

drivers/nvme/target/fabrics-cmd-auth.c:33:30: warning: invalid assignment: |=
drivers/nvme/target/fabrics-cmd-auth.c:33:30:    left side has type restricted __le32
drivers/nvme/target/fabrics-cmd-auth.c:33:30:    right side has type int
drivers/nvme/target/fabrics-cmd-auth.c:118:22: warning: cast to restricted __le32
drivers/nvme/target/fabrics-cmd-auth.c:118:22: warning: cast from restricted __le16
drivers/nvme/target/fabrics-cmd-auth.c:389:30: warning: incorrect type in assignment (different base types)
drivers/nvme/target/fabrics-cmd-auth.c:389:30:    expected restricted __le16 [usertype] dhvlen
drivers/nvme/target/fabrics-cmd-auth.c:389:30:    got restricted __le32 [usertype]

[PATCH 06/11] nvme: Implement In-Band authentication

[Hannes Reinecke]

Implement NVMe-oF In-Band authentication according to NVMe TPAR 8006.
This patch adds two new fabric options 'dhchap_secret' to specify the
pre-shared key (in ASCII respresentation according to NVMe 2.0 section
8.13.5.8 'Secret representation') and 'dhchap_ctrl_secret' to specify
the pre-shared controller key for bi-directional authentication of both
the host and the controller.
Re-authentication can be triggered by writing the PSK into the new
controller sysfs attribute 'dhchap_secret' or 'dhchap_ctrl_secret'.

Signed-off-by: Hannes Reinecke <hare@suse.de>
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
---
 drivers/nvme/Kconfig         |   1 +
 drivers/nvme/Makefile        |   1 +
 drivers/nvme/common/Kconfig  |   4 +
 drivers/nvme/common/Makefile |   7 +
 drivers/nvme/common/auth.c   | 328 ++++++++++++++
 drivers/nvme/host/Kconfig    |  12 +
 drivers/nvme/host/Makefile   |   1 +
 drivers/nvme/host/auth.c     | 813 +++++++++++++++++++++++++++++++++++
 drivers/nvme/host/core.c     | 143 +++++-
 drivers/nvme/host/fabrics.c  |  79 +++-
 drivers/nvme/host/fabrics.h  |   7 +
 drivers/nvme/host/nvme.h     |  30 ++
 drivers/nvme/host/rdma.c     |   1 +
 drivers/nvme/host/tcp.c      |   1 +
 drivers/nvme/host/trace.c    |  32 ++
 include/linux/nvme-auth.h    |  33 ++
 16 files changed, 1486 insertions(+), 7 deletions(-)
 create mode 100644 drivers/nvme/common/Kconfig
 create mode 100644 drivers/nvme/common/Makefile
 create mode 100644 drivers/nvme/common/auth.c
 create mode 100644 drivers/nvme/host/auth.c
 create mode 100644 include/linux/nvme-auth.h

diff --git a/drivers/nvme/Kconfig b/drivers/nvme/Kconfig
index 87ae409a32b9..656e46d938da 100644
--- a/drivers/nvme/Kconfig
+++ b/drivers/nvme/Kconfig
@@ -1,6 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0-only
 menu "NVME Support"
 
+source "drivers/nvme/common/Kconfig"
 source "drivers/nvme/host/Kconfig"
 source "drivers/nvme/target/Kconfig"
 
diff --git a/drivers/nvme/Makefile b/drivers/nvme/Makefile
index fb42c44609a8..eedca8c72098 100644
--- a/drivers/nvme/Makefile
+++ b/drivers/nvme/Makefile
@@ -1,4 +1,5 @@
 # SPDX-License-Identifier: GPL-2.0-only
 
+obj-$(CONFIG_NVME_COMMON)		+= common/
 obj-y		+= host/
 obj-y		+= target/
diff --git a/drivers/nvme/common/Kconfig b/drivers/nvme/common/Kconfig
new file mode 100644
index 000000000000..4514f44362dd
--- /dev/null
+++ b/drivers/nvme/common/Kconfig
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: GPL-2.0-only
+
+config NVME_COMMON
+       tristate
diff --git a/drivers/nvme/common/Makefile b/drivers/nvme/common/Makefile
new file mode 100644
index 000000000000..720c625b8a52
--- /dev/null
+++ b/drivers/nvme/common/Makefile
@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: GPL-2.0
+
+ccflags-y			+= -I$(src)
+
+obj-$(CONFIG_NVME_COMMON)	+= nvme-common.o
+
+nvme-common-y			+= auth.o
diff --git a/drivers/nvme/common/auth.c b/drivers/nvme/common/auth.c
new file mode 100644
index 000000000000..1cccaf6afdd2
--- /dev/null
+++ b/drivers/nvme/common/auth.c
@@ -0,0 +1,328 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2020 Hannes Reinecke, SUSE Linux
+ */
+
+#include <linux/module.h>
+#include <linux/crc32.h>
+#include <linux/base64.h>
+#include <linux/prandom.h>
+#include <linux/scatterlist.h>
+#include <asm/unaligned.h>
+#include <crypto/hash.h>
+#include <crypto/dh.h>
+#include <linux/nvme.h>
+#include <linux/nvme-auth.h>
+
+static u32 nvme_dhchap_seqnum;
+static DEFINE_MUTEX(nvme_dhchap_mutex);
+
+u32 nvme_auth_get_seqnum(void)
+{
+	u32 seqnum;
+
+	mutex_lock(&nvme_dhchap_mutex);
+	if (!nvme_dhchap_seqnum)
+		nvme_dhchap_seqnum = prandom_u32();
+	else {
+		nvme_dhchap_seqnum++;
+		if (!nvme_dhchap_seqnum)
+			nvme_dhchap_seqnum++;
+	}
+	seqnum = nvme_dhchap_seqnum;
+	mutex_unlock(&nvme_dhchap_mutex);
+	return seqnum;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_get_seqnum);
+
+static struct nvme_auth_dhgroup_map {
+	const char name[16];
+	const char kpp[16];
+} dhgroup_map[] = {
+	[NVME_AUTH_DHGROUP_NULL] = {
+		.name = "null", .kpp = "null" },
+	[NVME_AUTH_DHGROUP_2048] = {
+		.name = "ffdhe2048", .kpp = "ffdhe2048(dh)" },
+	[NVME_AUTH_DHGROUP_3072] = {
+		.name = "ffdhe3072", .kpp = "ffdhe3072(dh)" },
+	[NVME_AUTH_DHGROUP_4096] = {
+		.name = "ffdhe4096", .kpp = "ffdhe4096(dh)" },
+	[NVME_AUTH_DHGROUP_6144] = {
+		.name = "ffdhe6144", .kpp = "ffdhe6144(dh)" },
+	[NVME_AUTH_DHGROUP_8192] = {
+		.name = "ffdhe8192", .kpp = "ffdhe8192(dh)" },
+};
+
+const char *nvme_auth_dhgroup_name(u8 dhgroup_id)
+{
+	if ((dhgroup_id > ARRAY_SIZE(dhgroup_map)) ||
+	    !dhgroup_map[dhgroup_id].name ||
+	    !strlen(dhgroup_map[dhgroup_id].name))
+		return NULL;
+	return dhgroup_map[dhgroup_id].name;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_dhgroup_name);
+
+const char *nvme_auth_dhgroup_kpp(u8 dhgroup_id)
+{
+	if ((dhgroup_id > ARRAY_SIZE(dhgroup_map)) ||
+	    !dhgroup_map[dhgroup_id].kpp ||
+	    !strlen(dhgroup_map[dhgroup_id].kpp))
+		return NULL;
+	return dhgroup_map[dhgroup_id].kpp;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_dhgroup_kpp);
+
+u8 nvme_auth_dhgroup_id(const char *dhgroup_name)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(dhgroup_map); i++) {
+		if (!dhgroup_map[i].name ||
+		    !strlen(dhgroup_map[i].name))
+			continue;
+		if (!strncmp(dhgroup_map[i].name, dhgroup_name,
+			     strlen(dhgroup_map[i].name)))
+			return i;
+	}
+	return NVME_AUTH_DHGROUP_INVALID;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_dhgroup_id);
+
+static struct nvme_dhchap_hash_map {
+	int len;
+	const char hmac[15];
+	const char digest[8];
+} hash_map[] = {
+	[NVME_AUTH_HASH_SHA256] = {
+		.len = 32,
+		.hmac = "hmac(sha256)",
+		.digest = "sha256",
+	},
+	[NVME_AUTH_HASH_SHA384] = {
+		.len = 48,
+		.hmac = "hmac(sha384)",
+		.digest = "sha384",
+	},
+	[NVME_AUTH_HASH_SHA512] = {
+		.len = 64,
+		.hmac = "hmac(sha512)",
+		.digest = "sha512",
+	},
+};
+
+const char *nvme_auth_hmac_name(u8 hmac_id)
+{
+	if ((hmac_id > ARRAY_SIZE(hash_map)) ||
+	    !hash_map[hmac_id].hmac ||
+	    !strlen(hash_map[hmac_id].hmac))
+		return NULL;
+	return hash_map[hmac_id].hmac;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_hmac_name);
+
+const char *nvme_auth_digest_name(u8 hmac_id)
+{
+	if ((hmac_id > ARRAY_SIZE(hash_map)) ||
+	    !hash_map[hmac_id].digest ||
+	    !strlen(hash_map[hmac_id].digest))
+		return NULL;
+	return hash_map[hmac_id].digest;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_digest_name);
+
+u8 nvme_auth_hmac_id(const char *hmac_name)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
+		if (!hash_map[i].hmac || !strlen(hash_map[i].hmac))
+			continue;
+		if (!strncmp(hash_map[i].hmac, hmac_name,
+			     strlen(hash_map[i].hmac)))
+			return i;
+	}
+	return NVME_AUTH_HASH_INVALID;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_hmac_id);
+
+size_t nvme_auth_hmac_hash_len(u8 hmac_id)
+{
+	if ((hmac_id > ARRAY_SIZE(hash_map)) ||
+	    !hash_map[hmac_id].hmac ||
+	    !strlen(hash_map[hmac_id].hmac))
+		return 0;
+	return hash_map[hmac_id].len;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_hmac_hash_len);
+
+struct nvme_dhchap_key *nvme_auth_extract_key(unsigned char *secret,
+					      u8 key_hash)
+{
+	struct nvme_dhchap_key *key;
+	unsigned char *p;
+	u32 crc;
+	int ret, key_len;
+	size_t allocated_len = strlen(secret);
+
+	/* Secret might be affixed with a ':' */
+	p = strrchr(secret, ':');
+	if (p)
+		allocated_len = p - secret;
+	key = kzalloc(sizeof(*key), GFP_KERNEL);
+	if (!key)
+		return ERR_PTR(-ENOMEM);
+	key->key = kzalloc(allocated_len, GFP_KERNEL);
+	if (!key->key) {
+		ret = -ENOMEM;
+		goto out_free_key;
+	}
+
+	key_len = base64_decode(secret, allocated_len, key->key);
+	if (key_len < 0) {
+		pr_debug("base64 key decoding error %d\n",
+			 key_len);
+		ret = key_len;
+		goto out_free_secret;
+	}
+
+	if (key_len != 36 && key_len != 52 &&
+	    key_len != 68) {
+		pr_err("Invalid key len %d\n", key_len);
+		ret = -EINVAL;
+		goto out_free_secret;
+	}
+
+	if (key_hash > 0 &&
+	    (key_len - 4) != nvme_auth_hmac_hash_len(key_hash)) {
+		pr_err("Mismatched key len %d for %s\n", key_len,
+		       nvme_auth_hmac_name(key_hash));
+		ret = -EINVAL;
+		goto out_free_secret;
+	}
+
+	/* The last four bytes is the CRC in little-endian format */
+	key_len -= 4;
+	/*
+	 * The linux implementation doesn't do pre- and post-increments,
+	 * so we have to do it manually.
+	 */
+	crc = ~crc32(~0, key->key, key_len);
+
+	if (get_unaligned_le32(key->key + key_len) != crc) {
+		pr_err("key crc mismatch (key %08x, crc %08x)\n",
+		       get_unaligned_le32(key->key + key_len), crc);
+		ret = -EKEYREJECTED;
+		goto out_free_secret;
+	}
+	key->len = key_len;
+	key->hash = key_hash;
+	return key;
+out_free_secret:
+	kfree_sensitive(key->key);
+out_free_key:
+	kfree(key);
+	return ERR_PTR(ret);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_extract_key);
+
+void nvme_auth_free_key(struct nvme_dhchap_key *key)
+{
+	if (!key)
+		return;
+	kfree_sensitive(key->key);
+	kfree(key);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_free_key);
+
+u8 *nvme_auth_transform_key(struct nvme_dhchap_key *key, char *nqn)
+{
+	const char *hmac_name = nvme_auth_hmac_name(key->hash);
+	struct crypto_shash *key_tfm;
+	struct shash_desc *shash;
+	u8 *transformed_key;
+	int ret;
+
+	if (key->hash == 0) {
+		transformed_key = kmemdup(key->key, key->len, GFP_KERNEL);
+		return transformed_key ? transformed_key : ERR_PTR(-ENOMEM);
+	}
+
+	if (!key || !key->key) {
+		pr_warn("No key specified\n");
+		return ERR_PTR(-ENOKEY);
+	}
+	if (!hmac_name) {
+		pr_warn("Invalid key hash id %d\n", key->hash);
+		return ERR_PTR(-EINVAL);
+	}
+
+	key_tfm = crypto_alloc_shash(hmac_name, 0, 0);
+	if (IS_ERR(key_tfm))
+		return (u8 *)key_tfm;
+
+	shash = kmalloc(sizeof(struct shash_desc) +
+			crypto_shash_descsize(key_tfm),
+			GFP_KERNEL);
+	if (!shash) {
+		ret = -ENOMEM;
+		goto out_free_key;
+	}
+
+	transformed_key = kzalloc(crypto_shash_digestsize(key_tfm), GFP_KERNEL);
+	if (!transformed_key) {
+		ret = -ENOMEM;
+		goto out_free_shash;
+	}
+
+	shash->tfm = key_tfm;
+	ret = crypto_shash_setkey(key_tfm, key->key, key->len);
+	if (ret < 0)
+		goto out_free_shash;
+	ret = crypto_shash_init(shash);
+	if (ret < 0)
+		goto out_free_shash;
+	ret = crypto_shash_update(shash, nqn, strlen(nqn));
+	if (ret < 0)
+		goto out_free_shash;
+	ret = crypto_shash_update(shash, "NVMe-over-Fabrics", 17);
+	if (ret < 0)
+		goto out_free_shash;
+	ret = crypto_shash_final(shash, transformed_key);
+out_free_shash:
+	kfree(shash);
+out_free_key:
+	crypto_free_shash(key_tfm);
+	if (ret < 0) {
+		kfree_sensitive(transformed_key);
+		return ERR_PTR(ret);
+	}
+	return transformed_key;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_transform_key);
+
+int nvme_auth_generate_key(u8 *secret, struct nvme_dhchap_key **ret_key)
+{
+	struct nvme_dhchap_key *key;
+	u8 key_hash;
+
+	if (!secret) {
+		*ret_key = NULL;
+		return 0;
+	}
+
+	if (sscanf(secret, "DHHC-1:%hhd:%*s:", &key_hash) != 1)
+		return -EINVAL;
+
+	/* Pass in the secret without the 'DHHC-1:XX:' prefix */
+	key = nvme_auth_extract_key(secret + 10, key_hash);
+	if (IS_ERR(key)) {
+		return PTR_ERR(key);
+	}
+
+	*ret_key = key;
+	return 0;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_generate_key);
+
+MODULE_LICENSE("GPL v2");
diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig
index d6d056963c06..abed8f59d3f2 100644
--- a/drivers/nvme/host/Kconfig
+++ b/drivers/nvme/host/Kconfig
@@ -91,3 +91,15 @@ config NVME_TCP
 	  from https://github.com/linux-nvme/nvme-cli.
 
 	  If unsure, say N.
+
+config NVME_AUTH
+	bool "NVM Express over Fabrics In-Band Authentication"
+	depends on NVME_CORE
+	select NVME_COMMON
+	select CRYPTO_HMAC
+	select CRYPTO_SHA256
+	select CRYPTO_SHA512
+	help
+	  This provides support for NVMe over Fabrics In-Band Authentication.
+
+	  If unsure, say N.
diff --git a/drivers/nvme/host/Makefile b/drivers/nvme/host/Makefile
index 476c5c988496..7755f5e3b281 100644
--- a/drivers/nvme/host/Makefile
+++ b/drivers/nvme/host/Makefile
@@ -15,6 +15,7 @@ nvme-core-$(CONFIG_NVME_MULTIPATH)	+= multipath.o
 nvme-core-$(CONFIG_BLK_DEV_ZONED)	+= zns.o
 nvme-core-$(CONFIG_FAULT_INJECTION_DEBUG_FS)	+= fault_inject.o
 nvme-core-$(CONFIG_NVME_HWMON)		+= hwmon.o
+nvme-core-$(CONFIG_NVME_AUTH)		+= auth.o
 
 nvme-y					+= pci.o
 
diff --git a/drivers/nvme/host/auth.c b/drivers/nvme/host/auth.c
new file mode 100644
index 000000000000..c9f861b928da
--- /dev/null
+++ b/drivers/nvme/host/auth.c
@@ -0,0 +1,813 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2020 Hannes Reinecke, SUSE Linux
+ */
+
+#include <linux/crc32.h>
+#include <linux/base64.h>
+#include <linux/prandom.h>
+#include <asm/unaligned.h>
+#include <crypto/hash.h>
+#include <crypto/dh.h>
+#include "nvme.h"
+#include "fabrics.h"
+#include <linux/nvme-auth.h>
+
+struct nvme_dhchap_queue_context {
+	struct list_head entry;
+	struct work_struct auth_work;
+	struct nvme_ctrl *ctrl;
+	struct crypto_shash *shash_tfm;
+	void *buf;
+	size_t buf_size;
+	int qid;
+	int error;
+	u32 s1;
+	u32 s2;
+	u16 transaction;
+	u8 status;
+	u8 hash_id;
+	size_t hash_len;
+	u8 dhgroup_id;
+	u8 c1[64];
+	u8 c2[64];
+	u8 response[64];
+	u8 *host_response;
+};
+
+#define nvme_auth_flags_from_qid(qid) \
+	(qid == 0) ? 0 : BLK_MQ_REQ_NOWAIT | BLK_MQ_REQ_RESERVED
+#define nvme_auth_queue_from_qid(ctrl, qid) \
+	(qid == 0) ? (ctrl)->fabrics_q : (ctrl)->connect_q
+
+static int nvme_auth_submit(struct nvme_ctrl *ctrl, int qid,
+			    void *data, size_t data_len, bool auth_send)
+{
+	struct nvme_command cmd = {};
+	blk_mq_req_flags_t flags = nvme_auth_flags_from_qid(qid);
+	struct request_queue *q = nvme_auth_queue_from_qid(ctrl, qid);
+	int ret;
+
+	cmd.auth_common.opcode = nvme_fabrics_command;
+	cmd.auth_common.secp = NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER;
+	cmd.auth_common.spsp0 = 0x01;
+	cmd.auth_common.spsp1 = 0x01;
+	if (auth_send) {
+		cmd.auth_send.fctype = nvme_fabrics_type_auth_send;
+		cmd.auth_send.tl = cpu_to_le32(data_len);
+	} else {
+		cmd.auth_receive.fctype = nvme_fabrics_type_auth_receive;
+		cmd.auth_receive.al = cpu_to_le32(data_len);
+	}
+
+	ret = __nvme_submit_sync_cmd(q, &cmd, NULL, data, data_len, 0,
+				     qid == 0 ? NVME_QID_ANY : qid,
+				     0, flags);
+	if (ret > 0)
+		dev_warn(ctrl->device,
+			"qid %d auth_send failed with status %d\n", qid, ret);
+	else if (ret < 0)
+		dev_err(ctrl->device,
+			"qid %d auth_send failed with error %d\n", qid, ret);
+	return ret;
+}
+
+static int nvme_auth_receive_validate(struct nvme_ctrl *ctrl, int qid,
+		struct nvmf_auth_dhchap_failure_data *data,
+		u16 transaction, u8 expected_msg)
+{
+	dev_dbg(ctrl->device, "%s: qid %d auth_type %d auth_id %x\n",
+		__func__, qid, data->auth_type, data->auth_id);
+
+	if (data->auth_type == NVME_AUTH_COMMON_MESSAGES &&
+	    data->auth_id == NVME_AUTH_DHCHAP_MESSAGE_FAILURE1) {
+		return data->rescode_exp;
+	}
+	if (data->auth_type != NVME_AUTH_DHCHAP_MESSAGES ||
+	    data->auth_id != expected_msg) {
+		dev_warn(ctrl->device,
+			 "qid %d invalid message %02x/%02x\n",
+			 qid, data->auth_type, data->auth_id);
+		return NVME_AUTH_DHCHAP_FAILURE_INCORRECT_MESSAGE;
+	}
+	if (le16_to_cpu(data->t_id) != transaction) {
+		dev_warn(ctrl->device,
+			 "qid %d invalid transaction ID %d\n",
+			 qid, le16_to_cpu(data->t_id));
+		return NVME_AUTH_DHCHAP_FAILURE_INCORRECT_MESSAGE;
+	}
+	return 0;
+}
+
+static int nvme_auth_set_dhchap_negotiate_data(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_negotiate_data *data = chap->buf;
+	size_t size = sizeof(*data) + sizeof(union nvmf_auth_protocol);
+
+	if (chap->buf_size < size) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return -EINVAL;
+	}
+	memset((u8 *)chap->buf, 0, size);
+	data->auth_type = NVME_AUTH_COMMON_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE;
+	data->t_id = cpu_to_le16(chap->transaction);
+	data->sc_c = 0; /* No secure channel concatenation */
+	data->napd = 1;
+	data->auth_protocol[0].dhchap.authid = NVME_AUTH_DHCHAP_AUTH_ID;
+	data->auth_protocol[0].dhchap.halen = 3;
+	data->auth_protocol[0].dhchap.dhlen = 6;
+	data->auth_protocol[0].dhchap.idlist[0] = NVME_AUTH_HASH_SHA256;
+	data->auth_protocol[0].dhchap.idlist[1] = NVME_AUTH_HASH_SHA384;
+	data->auth_protocol[0].dhchap.idlist[2] = NVME_AUTH_HASH_SHA512;
+	data->auth_protocol[0].dhchap.idlist[30] = NVME_AUTH_DHGROUP_NULL;
+	data->auth_protocol[0].dhchap.idlist[31] = NVME_AUTH_DHGROUP_2048;
+	data->auth_protocol[0].dhchap.idlist[32] = NVME_AUTH_DHGROUP_3072;
+	data->auth_protocol[0].dhchap.idlist[33] = NVME_AUTH_DHGROUP_4096;
+	data->auth_protocol[0].dhchap.idlist[34] = NVME_AUTH_DHGROUP_6144;
+	data->auth_protocol[0].dhchap.idlist[35] = NVME_AUTH_DHGROUP_8192;
+
+	return size;
+}
+
+static int nvme_auth_process_dhchap_challenge(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_challenge_data *data = chap->buf;
+	u16 dhvlen = le16_to_cpu(data->dhvlen);
+	size_t size = sizeof(*data) + data->hl + dhvlen;
+	const char *hmac_name, *kpp_name;
+
+	if (chap->buf_size < size) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return NVME_SC_INVALID_FIELD;
+	}
+
+	hmac_name = nvme_auth_hmac_name(data->hashid);
+	if (!hmac_name) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid HASH ID %d\n",
+			 chap->qid, data->hashid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
+		return NVME_SC_INVALID_FIELD;
+	}
+
+	if (chap->hash_id == data->hashid && chap->shash_tfm &&
+	    !strcmp(crypto_shash_alg_name(chap->shash_tfm), hmac_name) &&
+	    crypto_shash_digestsize(chap->shash_tfm) == data->hl) {
+		dev_dbg(ctrl->device,
+			"qid %d: reuse existing hash %s\n",
+			chap->qid, hmac_name);
+		goto select_kpp;
+	}
+
+	/* Reset if hash cannot be reused */
+	if (chap->shash_tfm) {
+		crypto_free_shash(chap->shash_tfm);
+		chap->hash_id = 0;
+		chap->hash_len = 0;
+	}
+	chap->shash_tfm = crypto_alloc_shash(hmac_name, 0,
+					     CRYPTO_ALG_ALLOCATES_MEMORY);
+	if (IS_ERR(chap->shash_tfm)) {
+		dev_warn(ctrl->device,
+			 "qid %d: failed to allocate hash %s, error %ld\n",
+			 chap->qid, hmac_name, PTR_ERR(chap->shash_tfm));
+		chap->shash_tfm = NULL;
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_FAILED;
+		return NVME_SC_AUTH_REQUIRED;
+	}
+
+	if (crypto_shash_digestsize(chap->shash_tfm) != data->hl) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid hash length %d\n",
+			 chap->qid, data->hl);
+		crypto_free_shash(chap->shash_tfm);
+		chap->shash_tfm = NULL;
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
+		return NVME_SC_AUTH_REQUIRED;
+	}
+
+	/* Reset host response if the hash had been changed */
+	if (chap->hash_id != data->hashid) {
+		kfree(chap->host_response);
+		chap->host_response = NULL;
+	}
+
+	chap->hash_id = data->hashid;
+	chap->hash_len = data->hl;
+	dev_dbg(ctrl->device, "qid %d: selected hash %s\n",
+		chap->qid, hmac_name);
+
+select_kpp:
+	kpp_name = nvme_auth_dhgroup_kpp(data->dhgid);
+	if (!kpp_name) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid DH group id %d\n",
+			 chap->qid, data->dhgid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
+		return NVME_SC_AUTH_REQUIRED;
+	}
+
+	if (data->dhgid != NVME_AUTH_DHGROUP_NULL) {
+		dev_warn(ctrl->device,
+			 "qid %d: unsupported DH group %s\n",
+			 chap->qid, kpp_name);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
+		return NVME_SC_AUTH_REQUIRED;
+	} else if (dhvlen != 0) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid DH value for NULL DH\n",
+			 chap->qid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return NVME_SC_INVALID_FIELD;
+	}
+	chap->dhgroup_id = data->dhgid;
+
+	chap->s1 = le32_to_cpu(data->seqnum);
+	memcpy(chap->c1, data->cval, chap->hash_len);
+
+	return 0;
+}
+
+static int nvme_auth_set_dhchap_reply_data(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_reply_data *data = chap->buf;
+	size_t size = sizeof(*data);
+
+	size += 2 * chap->hash_len;
+
+	if (chap->buf_size < size) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return -EINVAL;
+	}
+
+	memset(chap->buf, 0, size);
+	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_REPLY;
+	data->t_id = cpu_to_le16(chap->transaction);
+	data->hl = chap->hash_len;
+	data->dhvlen = 0;
+	memcpy(data->rval, chap->response, chap->hash_len);
+	if (ctrl->opts->dhchap_ctrl_secret) {
+		get_random_bytes(chap->c2, chap->hash_len);
+		data->cvalid = 1;
+		chap->s2 = nvme_auth_get_seqnum();
+		memcpy(data->rval + chap->hash_len, chap->c2,
+		       chap->hash_len);
+		dev_dbg(ctrl->device, "%s: qid %d ctrl challenge %*ph\n",
+			__func__, chap->qid, (int)chap->hash_len, chap->c2);
+	} else {
+		memset(chap->c2, 0, chap->hash_len);
+		chap->s2 = 0;
+	}
+	data->seqnum = cpu_to_le32(chap->s2);
+	return size;
+}
+
+static int nvme_auth_process_dhchap_success1(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_success1_data *data = chap->buf;
+	size_t size = sizeof(*data);
+
+	if (ctrl->opts->dhchap_ctrl_secret)
+		size += chap->hash_len;
+
+	if (chap->buf_size < size) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return NVME_SC_INVALID_FIELD;
+	}
+
+	if (data->hl != chap->hash_len) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid hash length %u\n",
+			 chap->qid, data->hl);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
+		return NVME_SC_INVALID_FIELD;
+	}
+
+	/* Just print out information for the admin queue */
+	if (chap->qid == 0)
+		dev_info(ctrl->device,
+			 "qid 0: authenticated with hash %s dhgroup %s\n",
+			 nvme_auth_hmac_name(chap->hash_id),
+			 nvme_auth_dhgroup_name(chap->dhgroup_id));
+
+	if (!data->rvalid)
+		return 0;
+
+	/* Validate controller response */
+	if (memcmp(chap->response, data->rval, data->hl)) {
+		dev_dbg(ctrl->device, "%s: qid %d ctrl response %*ph\n",
+			__func__, chap->qid, (int)chap->hash_len, data->rval);
+		dev_dbg(ctrl->device, "%s: qid %d host response %*ph\n",
+			__func__, chap->qid, (int)chap->hash_len,
+			chap->response);
+		dev_warn(ctrl->device,
+			 "qid %d: controller authentication failed\n",
+			 chap->qid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_FAILED;
+		return NVME_SC_AUTH_REQUIRED;
+	}
+
+	/* Just print out information for the admin queue */
+	if (chap->qid == 0)
+		dev_info(ctrl->device,
+			 "qid 0: controller authenticated\n");
+	return 0;
+}
+
+static int nvme_auth_set_dhchap_success2_data(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_success2_data *data = chap->buf;
+	size_t size = sizeof(*data);
+
+	memset(chap->buf, 0, size);
+	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_SUCCESS2;
+	data->t_id = cpu_to_le16(chap->transaction);
+
+	return size;
+}
+
+static int nvme_auth_set_dhchap_failure2_data(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_failure_data *data = chap->buf;
+	size_t size = sizeof(*data);
+
+	memset(chap->buf, 0, size);
+	data->auth_type = NVME_AUTH_COMMON_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_FAILURE2;
+	data->t_id = cpu_to_le16(chap->transaction);
+	data->rescode = NVME_AUTH_DHCHAP_FAILURE_REASON_FAILED;
+	data->rescode_exp = chap->status;
+
+	return size;
+}
+
+static int nvme_auth_dhchap_setup_host_response(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
+	u8 buf[4], *challenge = chap->c1;
+	int ret;
+
+	dev_dbg(ctrl->device, "%s: qid %d host response seq %u transaction %d\n",
+		__func__, chap->qid, chap->s1, chap->transaction);
+
+	if (!chap->host_response) {
+		chap->host_response = nvme_auth_transform_key(ctrl->host_key,
+						ctrl->opts->host->nqn);
+		if (IS_ERR(chap->host_response)) {
+			ret = PTR_ERR(chap->host_response);
+			chap->host_response = NULL;
+			return ret;
+		}
+	} else {
+		dev_dbg(ctrl->device, "%s: qid %d re-using host response\n",
+			__func__, chap->qid);
+	}
+
+	ret = crypto_shash_setkey(chap->shash_tfm,
+			chap->host_response, ctrl->host_key->len);
+	if (ret) {
+		dev_warn(ctrl->device, "qid %d: failed to set key, error %d\n",
+			 chap->qid, ret);
+		goto out;
+	}
+
+	shash->tfm = chap->shash_tfm;
+	ret = crypto_shash_init(shash);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, challenge, chap->hash_len);
+	if (ret)
+		goto out;
+	put_unaligned_le32(chap->s1, buf);
+	ret = crypto_shash_update(shash, buf, 4);
+	if (ret)
+		goto out;
+	put_unaligned_le16(chap->transaction, buf);
+	ret = crypto_shash_update(shash, buf, 2);
+	if (ret)
+		goto out;
+	memset(buf, 0, sizeof(buf));
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, "HostHost", 8);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
+				  strlen(ctrl->opts->host->nqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
+			    strlen(ctrl->opts->subsysnqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_final(shash, chap->response);
+out:
+	if (challenge != chap->c1)
+		kfree(challenge);
+	return ret;
+}
+
+static int nvme_auth_dhchap_setup_ctrl_response(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
+	u8 *ctrl_response;
+	u8 buf[4], *challenge = chap->c2;
+	int ret;
+
+	ctrl_response = nvme_auth_transform_key(ctrl->ctrl_key,
+				ctrl->opts->subsysnqn);
+	if (IS_ERR(ctrl_response)) {
+		ret = PTR_ERR(ctrl_response);
+		return ret;
+	}
+	ret = crypto_shash_setkey(chap->shash_tfm,
+			ctrl_response, ctrl->ctrl_key->len);
+	if (ret) {
+		dev_warn(ctrl->device, "qid %d: failed to set key, error %d\n",
+			 chap->qid, ret);
+		goto out;
+	}
+
+	dev_dbg(ctrl->device, "%s: qid %d ctrl response seq %u transaction %d\n",
+		__func__, chap->qid, chap->s2, chap->transaction);
+	dev_dbg(ctrl->device, "%s: qid %d challenge %*ph\n",
+		__func__, chap->qid, (int)chap->hash_len, challenge);
+	dev_dbg(ctrl->device, "%s: qid %d subsysnqn %s\n",
+		__func__, chap->qid, ctrl->opts->subsysnqn);
+	dev_dbg(ctrl->device, "%s: qid %d hostnqn %s\n",
+		__func__, chap->qid, ctrl->opts->host->nqn);
+	shash->tfm = chap->shash_tfm;
+	ret = crypto_shash_init(shash);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, challenge, chap->hash_len);
+	if (ret)
+		goto out;
+	put_unaligned_le32(chap->s2, buf);
+	ret = crypto_shash_update(shash, buf, 4);
+	if (ret)
+		goto out;
+	put_unaligned_le16(chap->transaction, buf);
+	ret = crypto_shash_update(shash, buf, 2);
+	if (ret)
+		goto out;
+	memset(buf, 0, 4);
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, "Controller", 10);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
+				  strlen(ctrl->opts->subsysnqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
+				  strlen(ctrl->opts->host->nqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_final(shash, chap->response);
+out:
+	if (challenge != chap->c2)
+		kfree(challenge);
+	return ret;
+}
+
+static void __nvme_auth_reset(struct nvme_dhchap_queue_context *chap)
+{
+	chap->status = 0;
+	chap->error = 0;
+	chap->s1 = 0;
+	chap->s2 = 0;
+	chap->transaction = 0;
+	memset(chap->c1, 0, sizeof(chap->c1));
+	memset(chap->c2, 0, sizeof(chap->c2));
+}
+
+static void __nvme_auth_free(struct nvme_dhchap_queue_context *chap)
+{
+	__nvme_auth_reset(chap);
+	if (chap->shash_tfm)
+		crypto_free_shash(chap->shash_tfm);
+	kfree_sensitive(chap->host_response);
+	kfree(chap->buf);
+	kfree(chap);
+}
+
+static void __nvme_auth_work(struct work_struct *work)
+{
+	struct nvme_dhchap_queue_context *chap =
+		container_of(work, struct nvme_dhchap_queue_context, auth_work);
+	struct nvme_ctrl *ctrl = chap->ctrl;
+	size_t tl;
+	int ret = 0;
+
+	chap->transaction = ctrl->transaction++;
+
+	/* DH-HMAC-CHAP Step 1: send negotiate */
+	dev_dbg(ctrl->device, "%s: qid %d send negotiate\n",
+		__func__, chap->qid);
+	ret = nvme_auth_set_dhchap_negotiate_data(ctrl, chap);
+	if (ret < 0) {
+		chap->error = ret;
+		return;
+	}
+	tl = ret;
+	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, tl, true);
+	if (ret) {
+		chap->error = ret;
+		return;
+	}
+
+	/* DH-HMAC-CHAP Step 2: receive challenge */
+	dev_dbg(ctrl->device, "%s: qid %d receive challenge\n",
+		__func__, chap->qid);
+
+	memset(chap->buf, 0, chap->buf_size);
+	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, chap->buf_size, false);
+	if (ret) {
+		dev_warn(ctrl->device,
+			 "qid %d failed to receive challenge, %s %d\n",
+			 chap->qid, ret < 0 ? "error" : "nvme status", ret);
+		chap->error = ret;
+		return;
+	}
+	ret = nvme_auth_receive_validate(ctrl, chap->qid, chap->buf, chap->transaction,
+					 NVME_AUTH_DHCHAP_MESSAGE_CHALLENGE);
+	if (ret) {
+		chap->status = ret;
+		chap->error = NVME_SC_AUTH_REQUIRED;
+		return;
+	}
+
+	ret = nvme_auth_process_dhchap_challenge(ctrl, chap);
+	if (ret) {
+		/* Invalid challenge parameters */
+		goto fail2;
+	}
+
+	dev_dbg(ctrl->device, "%s: qid %d host response\n",
+		__func__, chap->qid);
+	ret = nvme_auth_dhchap_setup_host_response(ctrl, chap);
+	if (ret)
+		goto fail2;
+
+	/* DH-HMAC-CHAP Step 3: send reply */
+	dev_dbg(ctrl->device, "%s: qid %d send reply\n",
+		__func__, chap->qid);
+	ret = nvme_auth_set_dhchap_reply_data(ctrl, chap);
+	if (ret < 0)
+		goto fail2;
+
+	tl = ret;
+	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, tl, true);
+	if (ret)
+		goto fail2;
+
+	/* DH-HMAC-CHAP Step 4: receive success1 */
+	dev_dbg(ctrl->device, "%s: qid %d receive success1\n",
+		__func__, chap->qid);
+
+	memset(chap->buf, 0, chap->buf_size);
+	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, chap->buf_size, false);
+	if (ret) {
+		dev_warn(ctrl->device,
+			 "qid %d failed to receive success1, %s %d\n",
+			 chap->qid, ret < 0 ? "error" : "nvme status", ret);
+		chap->error = ret;
+		return;
+	}
+	ret = nvme_auth_receive_validate(ctrl, chap->qid,
+					 chap->buf, chap->transaction,
+					 NVME_AUTH_DHCHAP_MESSAGE_SUCCESS1);
+	if (ret) {
+		chap->status = ret;
+		chap->error = NVME_SC_AUTH_REQUIRED;
+		return;
+	}
+
+	if (ctrl->opts->dhchap_ctrl_secret) {
+		dev_dbg(ctrl->device,
+			"%s: qid %d controller response\n",
+			__func__, chap->qid);
+		ret = nvme_auth_dhchap_setup_ctrl_response(ctrl, chap);
+		if (ret) {
+			chap->error = ret;
+			goto fail2;
+		}
+	}
+
+	ret = nvme_auth_process_dhchap_success1(ctrl, chap);
+	if (ret) {
+		/* Controller authentication failed */
+		chap->error = NVME_SC_AUTH_REQUIRED;
+		goto fail2;
+	}
+
+	if (ctrl->opts->dhchap_ctrl_secret) {
+		/* DH-HMAC-CHAP Step 5: send success2 */
+		dev_dbg(ctrl->device, "%s: qid %d send success2\n",
+			__func__, chap->qid);
+		tl = nvme_auth_set_dhchap_success2_data(ctrl, chap);
+		ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, tl, true);
+	}
+	if (!ret) {
+		chap->error = 0;
+		return;
+	}
+
+fail2:
+	dev_dbg(ctrl->device, "%s: qid %d send failure2, status %x\n",
+		__func__, chap->qid, chap->status);
+	tl = nvme_auth_set_dhchap_failure2_data(ctrl, chap);
+	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, tl, true);
+	/*
+	 * only update error if send failure2 failed and no other
+	 * error had been set during authentication.
+	 */
+	if (ret && !chap->error)
+		chap->error = ret;
+}
+
+int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid)
+{
+	struct nvme_dhchap_queue_context *chap;
+
+	if (!ctrl->host_key) {
+		dev_warn(ctrl->device, "qid %d: no key\n", qid);
+		return -ENOKEY;
+	}
+
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	/* Check if the context is already queued */
+	list_for_each_entry(chap, &ctrl->dhchap_auth_list, entry) {
+		WARN_ON(!chap->buf);
+		if (chap->qid == qid) {
+			dev_dbg(ctrl->device, "qid %d: re-using context\n", qid);
+			mutex_unlock(&ctrl->dhchap_auth_mutex);
+			flush_work(&chap->auth_work);
+			__nvme_auth_reset(chap);
+			queue_work(nvme_wq, &chap->auth_work);
+			return 0;
+		}
+	}
+	chap = kzalloc(sizeof(*chap), GFP_KERNEL);
+	if (!chap) {
+		mutex_unlock(&ctrl->dhchap_auth_mutex);
+		return -ENOMEM;
+	}
+	chap->qid = (qid == NVME_QID_ANY) ? 0 : qid;
+	chap->ctrl = ctrl;
+
+	/*
+	 * Allocate a large enough buffer for the entire negotiation:
+	 * 4k should be enough to ffdhe8192.
+	 */
+	chap->buf_size = 4096;
+	chap->buf = kzalloc(chap->buf_size, GFP_KERNEL);
+	if (!chap->buf) {
+		mutex_unlock(&ctrl->dhchap_auth_mutex);
+		kfree(chap);
+		return -ENOMEM;
+	}
+
+	INIT_WORK(&chap->auth_work, __nvme_auth_work);
+	list_add(&chap->entry, &ctrl->dhchap_auth_list);
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+	queue_work(nvme_wq, &chap->auth_work);
+	return 0;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_negotiate);
+
+int nvme_auth_wait(struct nvme_ctrl *ctrl, int qid)
+{
+	struct nvme_dhchap_queue_context *chap;
+	int ret;
+
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	list_for_each_entry(chap, &ctrl->dhchap_auth_list, entry) {
+		if (chap->qid != qid)
+			continue;
+		mutex_unlock(&ctrl->dhchap_auth_mutex);
+		flush_work(&chap->auth_work);
+		ret = chap->error;
+		return ret;
+	}
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+	return -ENXIO;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_wait);
+
+void nvme_auth_reset(struct nvme_ctrl *ctrl)
+{
+	struct nvme_dhchap_queue_context *chap;
+
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	list_for_each_entry(chap, &ctrl->dhchap_auth_list, entry) {
+		mutex_unlock(&ctrl->dhchap_auth_mutex);
+		flush_work(&chap->auth_work);
+		__nvme_auth_reset(chap);
+	}
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_reset);
+
+static void nvme_dhchap_auth_work(struct work_struct *work)
+{
+	struct nvme_ctrl *ctrl =
+		container_of(work, struct nvme_ctrl, dhchap_auth_work);
+	int ret, q;
+
+	/* Authenticate admin queue first */
+	ret = nvme_auth_negotiate(ctrl, 0);
+	if (ret) {
+		dev_warn(ctrl->device,
+			 "qid 0: error %d setting up authentication\n", ret);
+		return;
+	}
+	ret = nvme_auth_wait(ctrl, 0);
+	if (ret) {
+		dev_warn(ctrl->device,
+			 "qid 0: authentication failed\n");
+		return;
+	}
+
+	for (q = 1; q < ctrl->queue_count; q++) {
+		ret = nvme_auth_negotiate(ctrl, q);
+		if (ret) {
+			dev_warn(ctrl->device,
+				 "qid %d: error %d setting up authentication\n",
+				 q, ret);
+			break;
+		}
+	}
+
+	/*
+	 * Failure is a soft-state; credentials remain valid until
+	 * the controller terminates the connection.
+	 */
+}
+
+void nvme_auth_init_ctrl(struct nvme_ctrl *ctrl)
+{
+	INIT_LIST_HEAD(&ctrl->dhchap_auth_list);
+	INIT_WORK(&ctrl->dhchap_auth_work, nvme_dhchap_auth_work);
+	mutex_init(&ctrl->dhchap_auth_mutex);
+	if (!ctrl->opts)
+		return;
+	nvme_auth_generate_key(ctrl->opts->dhchap_secret, &ctrl->host_key);
+	nvme_auth_generate_key(ctrl->opts->dhchap_ctrl_secret, &ctrl->ctrl_key);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_init_ctrl);
+
+void nvme_auth_stop(struct nvme_ctrl *ctrl)
+{
+	struct nvme_dhchap_queue_context *chap = NULL, *tmp;
+
+	cancel_work_sync(&ctrl->dhchap_auth_work);
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	list_for_each_entry_safe(chap, tmp, &ctrl->dhchap_auth_list, entry)
+		cancel_work_sync(&chap->auth_work);
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_stop);
+
+void nvme_auth_free(struct nvme_ctrl *ctrl)
+{
+	struct nvme_dhchap_queue_context *chap = NULL, *tmp;
+
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	list_for_each_entry_safe(chap, tmp, &ctrl->dhchap_auth_list, entry) {
+		list_del_init(&chap->entry);
+		flush_work(&chap->auth_work);
+		__nvme_auth_free(chap);
+	}
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+	if (ctrl->host_key) {
+		nvme_auth_free_key(ctrl->host_key);
+		ctrl->host_key = NULL;
+	}
+	if (ctrl->ctrl_key) {
+		nvme_auth_free_key(ctrl->ctrl_key);
+		ctrl->ctrl_key = NULL;
+	}
+}
+EXPORT_SYMBOL_GPL(nvme_auth_free);
diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index e1846d04817f..a8deaeed8a23 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -24,6 +24,7 @@
 
 #include "nvme.h"
 #include "fabrics.h"
+#include <linux/nvme-auth.h>
 
 #define CREATE_TRACE_POINTS
 #include "trace.h"
@@ -330,6 +331,7 @@ enum nvme_disposition {
 	COMPLETE,
 	RETRY,
 	FAILOVER,
+	AUTHENTICATE,
 };
 
 static inline enum nvme_disposition nvme_decide_disposition(struct request *req)
@@ -337,6 +339,9 @@ static inline enum nvme_disposition nvme_decide_disposition(struct request *req)
 	if (likely(nvme_req(req)->status == 0))
 		return COMPLETE;
 
+	if ((nvme_req(req)->status & 0x7ff) == NVME_SC_AUTH_REQUIRED)
+		return AUTHENTICATE;
+
 	if (blk_noretry_request(req) ||
 	    (nvme_req(req)->status & NVME_SC_DNR) ||
 	    nvme_req(req)->retries >= nvme_max_retries)
@@ -375,11 +380,13 @@ static inline void nvme_end_req(struct request *req)
 
 void nvme_complete_rq(struct request *req)
 {
+	struct nvme_ctrl *ctrl = nvme_req(req)->ctrl;
+
 	trace_nvme_complete_rq(req);
 	nvme_cleanup_cmd(req);
 
-	if (nvme_req(req)->ctrl->kas)
-		nvme_req(req)->ctrl->comp_seen = true;
+	if (ctrl->kas)
+		ctrl->comp_seen = true;
 
 	switch (nvme_decide_disposition(req)) {
 	case COMPLETE:
@@ -391,6 +398,14 @@ void nvme_complete_rq(struct request *req)
 	case FAILOVER:
 		nvme_failover_req(req);
 		return;
+	case AUTHENTICATE:
+#ifdef CONFIG_NVME_AUTH
+		queue_work(nvme_wq, &ctrl->dhchap_auth_work);
+		nvme_retry_req(req);
+#else
+		nvme_end_req(req);
+#endif
+		return;
 	}
 }
 EXPORT_SYMBOL_GPL(nvme_complete_rq);
@@ -702,7 +717,9 @@ bool __nvme_check_ready(struct nvme_ctrl *ctrl, struct request *rq,
 		switch (ctrl->state) {
 		case NVME_CTRL_CONNECTING:
 			if (blk_rq_is_passthrough(rq) && nvme_is_fabrics(req->cmd) &&
-			    req->cmd->fabrics.fctype == nvme_fabrics_type_connect)
+			    (req->cmd->fabrics.fctype == nvme_fabrics_type_connect ||
+			     req->cmd->fabrics.fctype == nvme_fabrics_type_auth_send ||
+			     req->cmd->fabrics.fctype == nvme_fabrics_type_auth_receive))
 				return true;
 			break;
 		default:
@@ -3544,6 +3561,108 @@ static ssize_t dctype_show(struct device *dev,
 }
 static DEVICE_ATTR_RO(dctype);
 
+#ifdef CONFIG_NVME_AUTH
+static ssize_t nvme_ctrl_dhchap_secret_show(struct device *dev,
+		struct device_attribute *attr, char *buf)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	struct nvmf_ctrl_options *opts = ctrl->opts;
+
+	if (!opts->dhchap_secret)
+		return sysfs_emit(buf, "none\n");
+	return sysfs_emit(buf, "%s\n", opts->dhchap_secret);
+}
+
+static ssize_t nvme_ctrl_dhchap_secret_store(struct device *dev,
+		struct device_attribute *attr, const char *buf, size_t count)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	struct nvmf_ctrl_options *opts = ctrl->opts;
+	char *dhchap_secret;
+
+	if (!ctrl->opts->dhchap_secret)
+		return -EINVAL;
+	if (count < 7)
+		return -EINVAL;
+	if (memcmp(buf, "DHHC-1:", 7))
+		return -EINVAL;
+
+	dhchap_secret = kzalloc(count + 1, GFP_KERNEL);
+	if (!dhchap_secret)
+		return -ENOMEM;
+	memcpy(dhchap_secret, buf, count);
+	nvme_auth_stop(ctrl);
+	if (strcmp(dhchap_secret, opts->dhchap_secret)) {
+		int ret;
+
+		ret = nvme_auth_generate_key(dhchap_secret, &ctrl->host_key);
+		if (ret)
+			return ret;
+		kfree(opts->dhchap_secret);
+		opts->dhchap_secret = dhchap_secret;
+		/* Key has changed; re-authentication with new key */
+		nvme_auth_reset(ctrl);
+	}
+	/* Start re-authentication */
+	dev_info(ctrl->device, "re-authenticating controller\n");
+	queue_work(nvme_wq, &ctrl->dhchap_auth_work);
+
+	return count;
+}
+DEVICE_ATTR(dhchap_secret, S_IRUGO | S_IWUSR,
+	nvme_ctrl_dhchap_secret_show, nvme_ctrl_dhchap_secret_store);
+
+static ssize_t nvme_ctrl_dhchap_ctrl_secret_show(struct device *dev,
+		struct device_attribute *attr, char *buf)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	struct nvmf_ctrl_options *opts = ctrl->opts;
+
+	if (!opts->dhchap_ctrl_secret)
+		return sysfs_emit(buf, "none\n");
+	return sysfs_emit(buf, "%s\n", opts->dhchap_ctrl_secret);
+}
+
+static ssize_t nvme_ctrl_dhchap_ctrl_secret_store(struct device *dev,
+		struct device_attribute *attr, const char *buf, size_t count)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	struct nvmf_ctrl_options *opts = ctrl->opts;
+	char *dhchap_secret;
+
+	if (!ctrl->opts->dhchap_ctrl_secret)
+		return -EINVAL;
+	if (count < 7)
+		return -EINVAL;
+	if (memcmp(buf, "DHHC-1:", 7))
+		return -EINVAL;
+
+	dhchap_secret = kzalloc(count + 1, GFP_KERNEL);
+	if (!dhchap_secret)
+		return -ENOMEM;
+	memcpy(dhchap_secret, buf, count);
+	nvme_auth_stop(ctrl);
+	if (strcmp(dhchap_secret, opts->dhchap_ctrl_secret)) {
+		int ret;
+
+		ret = nvme_auth_generate_key(dhchap_secret, &ctrl->ctrl_key);
+		if (ret)
+			return ret;
+		kfree(opts->dhchap_ctrl_secret);
+		opts->dhchap_ctrl_secret = dhchap_secret;
+		/* Key has changed; re-authentication with new key */
+		nvme_auth_reset(ctrl);
+	}
+	/* Start re-authentication */
+	dev_info(ctrl->device, "re-authenticating controller\n");
+	queue_work(nvme_wq, &ctrl->dhchap_auth_work);
+
+	return count;
+}
+DEVICE_ATTR(dhchap_ctrl_secret, S_IRUGO | S_IWUSR,
+	nvme_ctrl_dhchap_ctrl_secret_show, nvme_ctrl_dhchap_ctrl_secret_store);
+#endif
+
 static struct attribute *nvme_dev_attrs[] = {
 	&dev_attr_reset_controller.attr,
 	&dev_attr_rescan_controller.attr,
@@ -3567,6 +3686,10 @@ static struct attribute *nvme_dev_attrs[] = {
 	&dev_attr_kato.attr,
 	&dev_attr_cntrltype.attr,
 	&dev_attr_dctype.attr,
+#ifdef CONFIG_NVME_AUTH
+	&dev_attr_dhchap_secret.attr,
+	&dev_attr_dhchap_ctrl_secret.attr,
+#endif
 	NULL
 };
 
@@ -3590,6 +3713,12 @@ static umode_t nvme_dev_attrs_are_visible(struct kobject *kobj,
 		return 0;
 	if (a == &dev_attr_fast_io_fail_tmo.attr && !ctrl->opts)
 		return 0;
+#ifdef CONFIG_NVME_AUTH
+	if (a == &dev_attr_dhchap_secret.attr && !ctrl->opts)
+		return 0;
+	if (a == &dev_attr_dhchap_ctrl_secret.attr && !ctrl->opts)
+		return 0;
+#endif
 
 	return a->mode;
 }
@@ -4442,8 +4571,10 @@ static void nvme_handle_aen_notice(struct nvme_ctrl *ctrl, u32 result)
 		 * recovery actions from interfering with the controller's
 		 * firmware activation.
 		 */
-		if (nvme_change_ctrl_state(ctrl, NVME_CTRL_RESETTING))
+		if (nvme_change_ctrl_state(ctrl, NVME_CTRL_RESETTING)) {
+			nvme_auth_stop(ctrl);
 			queue_work(nvme_wq, &ctrl->fw_act_work);
+		}
 		break;
 #ifdef CONFIG_NVME_MULTIPATH
 	case NVME_AER_NOTICE_ANA:
@@ -4490,6 +4621,7 @@ EXPORT_SYMBOL_GPL(nvme_complete_async_event);
 void nvme_stop_ctrl(struct nvme_ctrl *ctrl)
 {
 	nvme_mpath_stop(ctrl);
+	nvme_auth_stop(ctrl);
 	nvme_stop_keep_alive(ctrl);
 	nvme_stop_failfast_work(ctrl);
 	flush_work(&ctrl->async_event_work);
@@ -4547,6 +4679,8 @@ static void nvme_free_ctrl(struct device *dev)
 
 	nvme_free_cels(ctrl);
 	nvme_mpath_uninit(ctrl);
+	nvme_auth_stop(ctrl);
+	nvme_auth_free(ctrl);
 	__free_page(ctrl->discard_page);
 
 	if (subsys) {
@@ -4637,6 +4771,7 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev,
 
 	nvme_fault_inject_init(&ctrl->fault_inject, dev_name(ctrl->device));
 	nvme_mpath_init_ctrl(ctrl);
+	nvme_auth_init_ctrl(ctrl);
 
 	return 0;
 out_free_name:
diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index 953e3076aab1..f5786a60931c 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -369,6 +369,7 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)
 	union nvme_result res;
 	struct nvmf_connect_data *data;
 	int ret;
+	u32 result;
 
 	cmd.connect.opcode = nvme_fabrics_command;
 	cmd.connect.fctype = nvme_fabrics_type_connect;
@@ -401,8 +402,25 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)
 		goto out_free_data;
 	}
 
-	ctrl->cntlid = le16_to_cpu(res.u16);
-
+	result = le32_to_cpu(res.u32);
+	ctrl->cntlid = result & 0xFFFF;
+	if ((result >> 16) & 2) {
+		/* Authentication required */
+		ret = nvme_auth_negotiate(ctrl, 0);
+		if (ret) {
+			dev_warn(ctrl->device,
+				 "qid 0: authentication setup failed\n");
+			ret = NVME_SC_AUTH_REQUIRED;
+			goto out_free_data;
+		}
+		ret = nvme_auth_wait(ctrl, 0);
+		if (ret)
+			dev_warn(ctrl->device,
+				 "qid 0: authentication failed\n");
+		else
+			dev_info(ctrl->device,
+				 "qid 0: authenticated\n");
+	}
 out_free_data:
 	kfree(data);
 	return ret;
@@ -435,6 +453,7 @@ int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, u16 qid)
 	struct nvmf_connect_data *data;
 	union nvme_result res;
 	int ret;
+	u32 result;
 
 	cmd.connect.opcode = nvme_fabrics_command;
 	cmd.connect.fctype = nvme_fabrics_type_connect;
@@ -460,6 +479,21 @@ int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, u16 qid)
 		nvmf_log_connect_error(ctrl, ret, le32_to_cpu(res.u32),
 				       &cmd, data);
 	}
+	result = le32_to_cpu(res.u32);
+	if ((result >> 16) & 2) {
+		/* Authentication required */
+		ret = nvme_auth_negotiate(ctrl, qid);
+		if (ret) {
+			dev_warn(ctrl->device,
+				 "qid %d: authentication setup failed\n", qid);
+			ret = NVME_SC_AUTH_REQUIRED;
+		} else {
+			ret = nvme_auth_wait(ctrl, qid);
+			if (ret)
+				dev_warn(ctrl->device,
+					 "qid %u: authentication failed\n", qid);
+		}
+	}
 	kfree(data);
 	return ret;
 }
@@ -552,6 +586,8 @@ static const match_table_t opt_tokens = {
 	{ NVMF_OPT_TOS,			"tos=%d"		},
 	{ NVMF_OPT_FAIL_FAST_TMO,	"fast_io_fail_tmo=%d"	},
 	{ NVMF_OPT_DISCOVERY,		"discovery"		},
+	{ NVMF_OPT_DHCHAP_SECRET,	"dhchap_secret=%s"	},
+	{ NVMF_OPT_DHCHAP_CTRL_SECRET,	"dhchap_ctrl_secret=%s"	},
 	{ NVMF_OPT_ERR,			NULL			}
 };
 
@@ -833,6 +869,34 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
 		case NVMF_OPT_DISCOVERY:
 			opts->discovery_nqn = true;
 			break;
+		case NVMF_OPT_DHCHAP_SECRET:
+			p = match_strdup(args);
+			if (!p) {
+				ret = -ENOMEM;
+				goto out;
+			}
+			if (strlen(p) < 11 || strncmp(p, "DHHC-1:", 7)) {
+				pr_err("Invalid DH-CHAP secret %s\n", p);
+				ret = -EINVAL;
+				goto out;
+			}
+			kfree(opts->dhchap_secret);
+			opts->dhchap_secret = p;
+			break;
+		case NVMF_OPT_DHCHAP_CTRL_SECRET:
+			p = match_strdup(args);
+			if (!p) {
+				ret = -ENOMEM;
+				goto out;
+			}
+			if (strlen(p) < 11 || strncmp(p, "DHHC-1:", 7)) {
+				pr_err("Invalid DH-CHAP secret %s\n", p);
+				ret = -EINVAL;
+				goto out;
+			}
+			kfree(opts->dhchap_ctrl_secret);
+			opts->dhchap_ctrl_secret = p;
+			break;
 		default:
 			pr_warn("unknown parameter or missing value '%s' in ctrl creation request\n",
 				p);
@@ -951,6 +1015,7 @@ void nvmf_free_options(struct nvmf_ctrl_options *opts)
 	kfree(opts->subsysnqn);
 	kfree(opts->host_traddr);
 	kfree(opts->host_iface);
+	kfree(opts->dhchap_secret);
 	kfree(opts);
 }
 EXPORT_SYMBOL_GPL(nvmf_free_options);
@@ -960,7 +1025,8 @@ EXPORT_SYMBOL_GPL(nvmf_free_options);
 				 NVMF_OPT_KATO | NVMF_OPT_HOSTNQN | \
 				 NVMF_OPT_HOST_ID | NVMF_OPT_DUP_CONNECT |\
 				 NVMF_OPT_DISABLE_SQFLOW | NVMF_OPT_DISCOVERY |\
-				 NVMF_OPT_FAIL_FAST_TMO)
+				 NVMF_OPT_FAIL_FAST_TMO | NVMF_OPT_DHCHAP_SECRET |\
+				 NVMF_OPT_DHCHAP_CTRL_SECRET)
 
 static struct nvme_ctrl *
 nvmf_create_ctrl(struct device *dev, const char *buf)
@@ -1196,7 +1262,14 @@ static void __exit nvmf_exit(void)
 	BUILD_BUG_ON(sizeof(struct nvmf_connect_command) != 64);
 	BUILD_BUG_ON(sizeof(struct nvmf_property_get_command) != 64);
 	BUILD_BUG_ON(sizeof(struct nvmf_property_set_command) != 64);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_send_command) != 64);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_receive_command) != 64);
 	BUILD_BUG_ON(sizeof(struct nvmf_connect_data) != 1024);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_negotiate_data) != 8);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_challenge_data) != 16);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_reply_data) != 16);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_success1_data) != 16);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_success2_data) != 16);
 }
 
 MODULE_LICENSE("GPL v2");
diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h
index 1e3a09cad961..15c142b277ab 100644
--- a/drivers/nvme/host/fabrics.h
+++ b/drivers/nvme/host/fabrics.h
@@ -68,6 +68,8 @@ enum {
 	NVMF_OPT_FAIL_FAST_TMO	= 1 << 20,
 	NVMF_OPT_HOST_IFACE	= 1 << 21,
 	NVMF_OPT_DISCOVERY	= 1 << 22,
+	NVMF_OPT_DHCHAP_SECRET	= 1 << 23,
+	NVMF_OPT_DHCHAP_CTRL_SECRET = 1 << 24,
 };
 
 /**
@@ -97,6 +99,9 @@ enum {
  * @max_reconnects: maximum number of allowed reconnect attempts before removing
  *              the controller, (-1) means reconnect forever, zero means remove
  *              immediately;
+ * @dhchap_secret: DH-HMAC-CHAP secret
+ * @dhchap_ctrl_secret: DH-HMAC-CHAP controller secret for bi-directional
+ *              authentication
  * @disable_sqflow: disable controller sq flow control
  * @hdr_digest: generate/verify header digest (TCP)
  * @data_digest: generate/verify data digest (TCP)
@@ -121,6 +126,8 @@ struct nvmf_ctrl_options {
 	unsigned int		kato;
 	struct nvmf_host	*host;
 	int			max_reconnects;
+	char			*dhchap_secret;
+	char			*dhchap_ctrl_secret;
 	bool			disable_sqflow;
 	bool			hdr_digest;
 	bool			data_digest;
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index a2b53ca63335..f46b752db02c 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -327,6 +327,15 @@ struct nvme_ctrl {
 	struct work_struct ana_work;
 #endif
 
+#ifdef CONFIG_NVME_AUTH
+	struct work_struct dhchap_auth_work;
+	struct list_head dhchap_auth_list;
+	struct mutex dhchap_auth_mutex;
+	struct nvme_dhchap_key *host_key;
+	struct nvme_dhchap_key *ctrl_key;
+	u16 transaction;
+#endif
+
 	/* Power saving configuration */
 	u64 ps_max_latency_us;
 	bool apst_enabled;
@@ -957,6 +966,27 @@ static inline bool nvme_ctrl_sgl_supported(struct nvme_ctrl *ctrl)
 	return ctrl->sgls & ((1 << 0) | (1 << 1));
 }
 
+#ifdef CONFIG_NVME_AUTH
+void nvme_auth_init_ctrl(struct nvme_ctrl *ctrl);
+void nvme_auth_stop(struct nvme_ctrl *ctrl);
+int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid);
+int nvme_auth_wait(struct nvme_ctrl *ctrl, int qid);
+void nvme_auth_reset(struct nvme_ctrl *ctrl);
+void nvme_auth_free(struct nvme_ctrl *ctrl);
+#else
+static inline void nvme_auth_init_ctrl(struct nvme_ctrl *ctrl) {};
+static inline void nvme_auth_stop(struct nvme_ctrl *ctrl) {};
+static inline int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid)
+{
+	return -EPROTONOSUPPORT;
+}
+static inline int nvme_auth_wait(struct nvme_ctrl *ctrl, int qid)
+{
+	return NVME_SC_AUTH_REQUIRED;
+}
+static inline void nvme_auth_free(struct nvme_ctrl *ctrl) {};
+#endif
+
 u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 			 u8 opcode);
 int nvme_execute_passthru_rq(struct request *rq);
diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index d9f19d901313..bd63568c0068 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -1197,6 +1197,7 @@ static void nvme_rdma_error_recovery_work(struct work_struct *work)
 	struct nvme_rdma_ctrl *ctrl = container_of(work,
 			struct nvme_rdma_ctrl, err_work);
 
+	nvme_auth_stop(&ctrl->ctrl);
 	nvme_stop_keep_alive(&ctrl->ctrl);
 	flush_work(&ctrl->ctrl.async_event_work);
 	nvme_rdma_teardown_io_queues(ctrl, false);
diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index ad3a2bf2f1e9..be52902b5f3b 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -2174,6 +2174,7 @@ static void nvme_tcp_error_recovery_work(struct work_struct *work)
 				struct nvme_tcp_ctrl, err_work);
 	struct nvme_ctrl *ctrl = &tcp_ctrl->ctrl;
 
+	nvme_auth_stop(ctrl);
 	nvme_stop_keep_alive(ctrl);
 	flush_work(&ctrl->async_event_work);
 	nvme_tcp_teardown_io_queues(ctrl, false);
diff --git a/drivers/nvme/host/trace.c b/drivers/nvme/host/trace.c
index 2a89c5aa0790..1c36fcedea20 100644
--- a/drivers/nvme/host/trace.c
+++ b/drivers/nvme/host/trace.c
@@ -287,6 +287,34 @@ static const char *nvme_trace_fabrics_property_get(struct trace_seq *p, u8 *spc)
 	return ret;
 }
 
+static const char *nvme_trace_fabrics_auth_send(struct trace_seq *p, u8 *spc)
+{
+	const char *ret = trace_seq_buffer_ptr(p);
+	u8 spsp0 = spc[1];
+	u8 spsp1 = spc[2];
+	u8 secp = spc[3];
+	u32 tl = get_unaligned_le32(spc + 4);
+
+	trace_seq_printf(p, "spsp0=%02x, spsp1=%02x, secp=%02x, tl=%u",
+			 spsp0, spsp1, secp, tl);
+	trace_seq_putc(p, 0);
+	return ret;
+}
+
+static const char *nvme_trace_fabrics_auth_receive(struct trace_seq *p, u8 *spc)
+{
+	const char *ret = trace_seq_buffer_ptr(p);
+	u8 spsp0 = spc[1];
+	u8 spsp1 = spc[2];
+	u8 secp = spc[3];
+	u32 al = get_unaligned_le32(spc + 4);
+
+	trace_seq_printf(p, "spsp0=%02x, spsp1=%02x, secp=%02x, al=%u",
+			 spsp0, spsp1, secp, al);
+	trace_seq_putc(p, 0);
+	return ret;
+}
+
 static const char *nvme_trace_fabrics_common(struct trace_seq *p, u8 *spc)
 {
 	const char *ret = trace_seq_buffer_ptr(p);
@@ -306,6 +334,10 @@ const char *nvme_trace_parse_fabrics_cmd(struct trace_seq *p,
 		return nvme_trace_fabrics_connect(p, spc);
 	case nvme_fabrics_type_property_get:
 		return nvme_trace_fabrics_property_get(p, spc);
+	case nvme_fabrics_type_auth_send:
+		return nvme_trace_fabrics_auth_send(p, spc);
+	case nvme_fabrics_type_auth_receive:
+		return nvme_trace_fabrics_auth_receive(p, spc);
 	default:
 		return nvme_trace_fabrics_common(p, spc);
 	}
diff --git a/include/linux/nvme-auth.h b/include/linux/nvme-auth.h
new file mode 100644
index 000000000000..354456826221
--- /dev/null
+++ b/include/linux/nvme-auth.h
@@ -0,0 +1,33 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2021 Hannes Reinecke, SUSE Software Solutions
+ */
+
+#ifndef _NVME_AUTH_H
+#define _NVME_AUTH_H
+
+#include <crypto/kpp.h>
+
+struct nvme_dhchap_key {
+	u8 *key;
+	size_t len;
+	u8 hash;
+};
+
+u32 nvme_auth_get_seqnum(void);
+const char *nvme_auth_dhgroup_name(u8 dhgroup_id);
+const char *nvme_auth_dhgroup_kpp(u8 dhgroup_id);
+u8 nvme_auth_dhgroup_id(const char *dhgroup_name);
+
+const char *nvme_auth_hmac_name(u8 hmac_id);
+const char *nvme_auth_digest_name(u8 hmac_id);
+size_t nvme_auth_hmac_hash_len(u8 hmac_id);
+u8 nvme_auth_hmac_id(const char *hmac_name);
+
+struct nvme_dhchap_key *nvme_auth_extract_key(unsigned char *secret,
+					      u8 key_hash);
+void nvme_auth_free_key(struct nvme_dhchap_key *key);
+u8 *nvme_auth_transform_key(struct nvme_dhchap_key *key, char *nqn);
+int nvme_auth_generate_key(u8 *secret, struct nvme_dhchap_key **ret_key);
+
+#endif /* _NVME_AUTH_H */
-- 
2.29.2

[PATCH 06/11] nvme: Implement In-Band authentication

[Hannes Reinecke]

Implement NVMe-oF In-Band authentication according to NVMe TPAR 8006.
This patch adds two new fabric options 'dhchap_secret' to specify the
pre-shared key (in ASCII respresentation according to NVMe 2.0 section
8.13.5.8 'Secret representation') and 'dhchap_ctrl_secret' to specify
the pre-shared controller key for bi-directional authentication of both
the host and the controller.
Re-authentication can be triggered by writing the PSK into the new
controller sysfs attribute 'dhchap_secret' or 'dhchap_ctrl_secret'.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/nvme/host/Kconfig   |   11 +
 drivers/nvme/host/Makefile  |    1 +
 drivers/nvme/host/auth.c    | 1124 +++++++++++++++++++++++++++++++++++
 drivers/nvme/host/auth.h    |   32 +
 drivers/nvme/host/core.c    |  141 ++++-
 drivers/nvme/host/fabrics.c |   79 ++-
 drivers/nvme/host/fabrics.h |    7 +
 drivers/nvme/host/nvme.h    |   31 +
 drivers/nvme/host/rdma.c    |    1 +
 drivers/nvme/host/tcp.c     |    1 +
 drivers/nvme/host/trace.c   |   32 +
 11 files changed, 1453 insertions(+), 7 deletions(-)
 create mode 100644 drivers/nvme/host/auth.c
 create mode 100644 drivers/nvme/host/auth.h

diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig
index d6d056963c06..dd0e91fb0615 100644
--- a/drivers/nvme/host/Kconfig
+++ b/drivers/nvme/host/Kconfig
@@ -91,3 +91,14 @@ config NVME_TCP
 	  from https://github.com/linux-nvme/nvme-cli.
 
 	  If unsure, say N.
+
+config NVME_AUTH
+	bool "NVM Express over Fabrics In-Band Authentication"
+	depends on NVME_CORE
+	select CRYPTO_HMAC
+	select CRYPTO_SHA256
+	select CRYPTO_SHA512
+	help
+	  This provides support for NVMe over Fabrics In-Band Authentication.
+
+	  If unsure, say N.
diff --git a/drivers/nvme/host/Makefile b/drivers/nvme/host/Makefile
index 476c5c988496..7755f5e3b281 100644
--- a/drivers/nvme/host/Makefile
+++ b/drivers/nvme/host/Makefile
@@ -15,6 +15,7 @@ nvme-core-$(CONFIG_NVME_MULTIPATH)	+= multipath.o
 nvme-core-$(CONFIG_BLK_DEV_ZONED)	+= zns.o
 nvme-core-$(CONFIG_FAULT_INJECTION_DEBUG_FS)	+= fault_inject.o
 nvme-core-$(CONFIG_NVME_HWMON)		+= hwmon.o
+nvme-core-$(CONFIG_NVME_AUTH)		+= auth.o
 
 nvme-y					+= pci.o
 
diff --git a/drivers/nvme/host/auth.c b/drivers/nvme/host/auth.c
new file mode 100644
index 000000000000..abe5c2fe8479
--- /dev/null
+++ b/drivers/nvme/host/auth.c
@@ -0,0 +1,1124 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2020 Hannes Reinecke, SUSE Linux
+ */
+
+#include <linux/crc32.h>
+#include <linux/base64.h>
+#include <linux/prandom.h>
+#include <asm/unaligned.h>
+#include <crypto/hash.h>
+#include <crypto/dh.h>
+#include "nvme.h"
+#include "fabrics.h"
+#include "auth.h"
+
+static u32 nvme_dhchap_seqnum;
+static DEFINE_MUTEX(nvme_dhchap_mutex);
+
+struct nvme_dhchap_queue_context {
+	struct list_head entry;
+	struct work_struct auth_work;
+	struct nvme_ctrl *ctrl;
+	struct crypto_shash *shash_tfm;
+	void *buf;
+	size_t buf_size;
+	int qid;
+	int error;
+	u32 s1;
+	u32 s2;
+	u16 transaction;
+	u8 status;
+	u8 hash_id;
+	size_t hash_len;
+	u8 dhgroup_id;
+	u8 c1[64];
+	u8 c2[64];
+	u8 response[64];
+	u8 *host_response;
+};
+
+u32 nvme_auth_get_seqnum(void)
+{
+	u32 seqnum;
+
+	mutex_lock(&nvme_dhchap_mutex);
+	if (!nvme_dhchap_seqnum)
+		nvme_dhchap_seqnum = prandom_u32();
+	else {
+		nvme_dhchap_seqnum++;
+		if (!nvme_dhchap_seqnum)
+			nvme_dhchap_seqnum++;
+	}
+	seqnum = nvme_dhchap_seqnum;
+	mutex_unlock(&nvme_dhchap_mutex);
+	return seqnum;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_get_seqnum);
+
+static struct nvme_auth_dhgroup_map {
+	const char name[16];
+	const char kpp[16];
+} dhgroup_map[] = {
+	[NVME_AUTH_DHGROUP_NULL] = {
+		.name = "null", .kpp = "null" },
+	[NVME_AUTH_DHGROUP_2048] = {
+		.name = "ffdhe2048", .kpp = "ffdhe2048(dh)" },
+	[NVME_AUTH_DHGROUP_3072] = {
+		.name = "ffdhe3072", .kpp = "ffdhe3072(dh)" },
+	[NVME_AUTH_DHGROUP_4096] = {
+		.name = "ffdhe4096", .kpp = "ffdhe4096(dh)" },
+	[NVME_AUTH_DHGROUP_6144] = {
+		.name = "ffdhe6144", .kpp = "ffdhe6144(dh)" },
+	[NVME_AUTH_DHGROUP_8192] = {
+		.name = "ffdhe8192", .kpp = "ffdhe8192(dh)" },
+};
+
+const char *nvme_auth_dhgroup_name(u8 dhgroup_id)
+{
+	if ((dhgroup_id > ARRAY_SIZE(dhgroup_map)) ||
+	    !dhgroup_map[dhgroup_id].name ||
+	    !strlen(dhgroup_map[dhgroup_id].name))
+		return NULL;
+	return dhgroup_map[dhgroup_id].name;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_dhgroup_name);
+
+const char *nvme_auth_dhgroup_kpp(u8 dhgroup_id)
+{
+	if ((dhgroup_id > ARRAY_SIZE(dhgroup_map)) ||
+	    !dhgroup_map[dhgroup_id].kpp ||
+	    !strlen(dhgroup_map[dhgroup_id].kpp))
+		return NULL;
+	return dhgroup_map[dhgroup_id].kpp;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_dhgroup_kpp);
+
+u8 nvme_auth_dhgroup_id(const char *dhgroup_name)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(dhgroup_map); i++) {
+		if (!dhgroup_map[i].name ||
+		    !strlen(dhgroup_map[i].name))
+			continue;
+		if (!strncmp(dhgroup_map[i].name, dhgroup_name,
+			     strlen(dhgroup_map[i].name)))
+			return i;
+	}
+	return NVME_AUTH_DHGROUP_INVALID;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_dhgroup_id);
+
+static struct nvme_dhchap_hash_map {
+	int len;
+	const char hmac[15];
+	const char digest[8];
+} hash_map[] = {
+	[NVME_AUTH_HASH_SHA256] = {
+		.len = 32,
+		.hmac = "hmac(sha256)",
+		.digest = "sha256",
+	},
+	[NVME_AUTH_HASH_SHA384] = {
+		.len = 48,
+		.hmac = "hmac(sha384)",
+		.digest = "sha384",
+	},
+	[NVME_AUTH_HASH_SHA512] = {
+		.len = 64,
+		.hmac = "hmac(sha512)",
+		.digest = "sha512",
+	},
+};
+
+const char *nvme_auth_hmac_name(u8 hmac_id)
+{
+	if ((hmac_id > ARRAY_SIZE(hash_map)) ||
+	    !hash_map[hmac_id].hmac ||
+	    !strlen(hash_map[hmac_id].hmac))
+		return NULL;
+	return hash_map[hmac_id].hmac;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_hmac_name);
+
+const char *nvme_auth_digest_name(u8 hmac_id)
+{
+	if ((hmac_id > ARRAY_SIZE(hash_map)) ||
+	    !hash_map[hmac_id].digest ||
+	    !strlen(hash_map[hmac_id].digest))
+		return NULL;
+	return hash_map[hmac_id].digest;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_digest_name);
+
+u8 nvme_auth_hmac_id(const char *hmac_name)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
+		if (!hash_map[i].hmac || !strlen(hash_map[i].hmac))
+			continue;
+		if (!strncmp(hash_map[i].hmac, hmac_name,
+			     strlen(hash_map[i].hmac)))
+			return i;
+	}
+	return NVME_AUTH_HASH_INVALID;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_hmac_id);
+
+size_t nvme_auth_hmac_hash_len(u8 hmac_id)
+{
+	if ((hmac_id > ARRAY_SIZE(hash_map)) ||
+	    !hash_map[hmac_id].hmac ||
+	    !strlen(hash_map[hmac_id].hmac))
+		return 0;
+	return hash_map[hmac_id].len;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_hmac_hash_len);
+
+struct nvme_dhchap_key *nvme_auth_extract_key(unsigned char *secret,
+					      u8 key_hash)
+{
+	struct nvme_dhchap_key *key;
+	unsigned char *p;
+	u32 crc;
+	int ret, key_len;
+	size_t allocated_len = strlen(secret);
+
+	/* Secret might be affixed with a ':' */
+	p = strrchr(secret, ':');
+	if (p)
+		allocated_len = p - secret;
+	key = kzalloc(sizeof(*key), GFP_KERNEL);
+	if (!key)
+		return ERR_PTR(-ENOMEM);
+	key->key = kzalloc(allocated_len, GFP_KERNEL);
+	if (!key->key) {
+		ret = -ENOMEM;
+		goto out_free_key;
+	}
+
+	key_len = base64_decode(secret, allocated_len, key->key);
+	if (key_len < 0) {
+		pr_debug("base64 key decoding error %d\n",
+			 key_len);
+		ret = key_len;
+		goto out_free_secret;
+	}
+
+	if (key_len != 36 && key_len != 52 &&
+	    key_len != 68) {
+		pr_err("Invalid key len %d\n", key_len);
+		ret = -EINVAL;
+		goto out_free_secret;
+	}
+
+	if (key_hash > 0 &&
+	    (key_len - 4) != nvme_auth_hmac_hash_len(key_hash)) {
+		pr_err("Mismatched key len %d for %s\n", key_len,
+		       nvme_auth_hmac_name(key_hash));
+		ret = -EINVAL;
+		goto out_free_secret;
+	}
+
+	/* The last four bytes is the CRC in little-endian format */
+	key_len -= 4;
+	/*
+	 * The linux implementation doesn't do pre- and post-increments,
+	 * so we have to do it manually.
+	 */
+	crc = ~crc32(~0, key->key, key_len);
+
+	if (get_unaligned_le32(key->key + key_len) != crc) {
+		pr_err("key crc mismatch (key %08x, crc %08x)\n",
+		       get_unaligned_le32(key->key + key_len), crc);
+		ret = -EKEYREJECTED;
+		goto out_free_secret;
+	}
+	key->len = key_len;
+	key->hash = key_hash;
+	return key;
+out_free_secret:
+	kfree_sensitive(key->key);
+out_free_key:
+	kfree(key);
+	return ERR_PTR(ret);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_extract_key);
+
+void nvme_auth_free_key(struct nvme_dhchap_key *key)
+{
+	if (!key)
+		return;
+	kfree_sensitive(key->key);
+	kfree(key);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_free_key);
+
+u8 *nvme_auth_transform_key(struct nvme_dhchap_key *key, char *nqn)
+{
+	const char *hmac_name = nvme_auth_hmac_name(key->hash);
+	struct crypto_shash *key_tfm;
+	struct shash_desc *shash;
+	u8 *transformed_key;
+	int ret;
+
+	if (key->hash == 0) {
+		transformed_key = kmemdup(key->key, key->len, GFP_KERNEL);
+		return transformed_key ? transformed_key : ERR_PTR(-ENOMEM);
+	}
+
+	if (!key || !key->key) {
+		pr_warn("No key specified\n");
+		return ERR_PTR(-ENOKEY);
+	}
+	if (!hmac_name) {
+		pr_warn("Invalid key hash id %d\n", key->hash);
+		return ERR_PTR(-EINVAL);
+	}
+
+	key_tfm = crypto_alloc_shash(hmac_name, 0, 0);
+	if (IS_ERR(key_tfm))
+		return (u8 *)key_tfm;
+
+	shash = kmalloc(sizeof(struct shash_desc) +
+			crypto_shash_descsize(key_tfm),
+			GFP_KERNEL);
+	if (!shash) {
+		ret = -ENOMEM;
+		goto out_free_key;
+	}
+
+	transformed_key = kzalloc(crypto_shash_digestsize(key_tfm), GFP_KERNEL);
+	if (!transformed_key) {
+		ret = -ENOMEM;
+		goto out_free_shash;
+	}
+
+	shash->tfm = key_tfm;
+	ret = crypto_shash_setkey(key_tfm, key->key, key->len);
+	if (ret < 0)
+		goto out_free_shash;
+	ret = crypto_shash_init(shash);
+	if (ret < 0)
+		goto out_free_shash;
+	ret = crypto_shash_update(shash, nqn, strlen(nqn));
+	if (ret < 0)
+		goto out_free_shash;
+	ret = crypto_shash_update(shash, "NVMe-over-Fabrics", 17);
+	if (ret < 0)
+		goto out_free_shash;
+	ret = crypto_shash_final(shash, transformed_key);
+out_free_shash:
+	kfree(shash);
+out_free_key:
+	crypto_free_shash(key_tfm);
+	if (ret < 0) {
+		kfree_sensitive(transformed_key);
+		return ERR_PTR(ret);
+	}
+	return transformed_key;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_transform_key);
+
+#define nvme_auth_flags_from_qid(qid) \
+	(qid == 0) ? 0 : BLK_MQ_REQ_NOWAIT | BLK_MQ_REQ_RESERVED
+#define nvme_auth_queue_from_qid(ctrl, qid) \
+	(qid == 0) ? (ctrl)->fabrics_q : (ctrl)->connect_q
+
+static int nvme_auth_submit(struct nvme_ctrl *ctrl, int qid,
+			    void *data, size_t data_len, bool auth_send)
+{
+	struct nvme_command cmd = {};
+	blk_mq_req_flags_t flags = nvme_auth_flags_from_qid(qid);
+	struct request_queue *q = nvme_auth_queue_from_qid(ctrl, qid);
+	int ret;
+
+	cmd.auth_common.opcode = nvme_fabrics_command;
+	cmd.auth_common.secp = NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER;
+	cmd.auth_common.spsp0 = 0x01;
+	cmd.auth_common.spsp1 = 0x01;
+	if (auth_send) {
+		cmd.auth_send.fctype = nvme_fabrics_type_auth_send;
+		cmd.auth_send.tl = cpu_to_le32(data_len);
+	} else {
+		cmd.auth_receive.fctype = nvme_fabrics_type_auth_receive;
+		cmd.auth_receive.al = cpu_to_le32(data_len);
+	}
+
+	ret = __nvme_submit_sync_cmd(q, &cmd, NULL, data, data_len, 0,
+				     qid == 0 ? NVME_QID_ANY : qid,
+				     0, flags);
+	if (ret > 0)
+		dev_warn(ctrl->device,
+			"qid %d auth_send failed with status %d\n", qid, ret);
+	else if (ret < 0)
+		dev_err(ctrl->device,
+			"qid %d auth_send failed with error %d\n", qid, ret);
+	return ret;
+}
+
+static int nvme_auth_receive_validate(struct nvme_ctrl *ctrl, int qid,
+		struct nvmf_auth_dhchap_failure_data *data,
+		u16 transaction, u8 expected_msg)
+{
+	dev_dbg(ctrl->device, "%s: qid %d auth_type %d auth_id %x\n",
+		__func__, qid, data->auth_type, data->auth_id);
+
+	if (data->auth_type == NVME_AUTH_COMMON_MESSAGES &&
+	    data->auth_id == NVME_AUTH_DHCHAP_MESSAGE_FAILURE1) {
+		return data->rescode_exp;
+	}
+	if (data->auth_type != NVME_AUTH_DHCHAP_MESSAGES ||
+	    data->auth_id != expected_msg) {
+		dev_warn(ctrl->device,
+			 "qid %d invalid message %02x/%02x\n",
+			 qid, data->auth_type, data->auth_id);
+		return NVME_AUTH_DHCHAP_FAILURE_INCORRECT_MESSAGE;
+	}
+	if (le16_to_cpu(data->t_id) != transaction) {
+		dev_warn(ctrl->device,
+			 "qid %d invalid transaction ID %d\n",
+			 qid, le16_to_cpu(data->t_id));
+		return NVME_AUTH_DHCHAP_FAILURE_INCORRECT_MESSAGE;
+	}
+	return 0;
+}
+
+static int nvme_auth_set_dhchap_negotiate_data(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_negotiate_data *data = chap->buf;
+	size_t size = sizeof(*data) + sizeof(union nvmf_auth_protocol);
+
+	if (chap->buf_size < size) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return -EINVAL;
+	}
+	memset((u8 *)chap->buf, 0, size);
+	data->auth_type = NVME_AUTH_COMMON_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE;
+	data->t_id = cpu_to_le16(chap->transaction);
+	data->sc_c = 0; /* No secure channel concatenation */
+	data->napd = 1;
+	data->auth_protocol[0].dhchap.authid = NVME_AUTH_DHCHAP_AUTH_ID;
+	data->auth_protocol[0].dhchap.halen = 3;
+	data->auth_protocol[0].dhchap.dhlen = 6;
+	data->auth_protocol[0].dhchap.idlist[0] = NVME_AUTH_HASH_SHA256;
+	data->auth_protocol[0].dhchap.idlist[1] = NVME_AUTH_HASH_SHA384;
+	data->auth_protocol[0].dhchap.idlist[2] = NVME_AUTH_HASH_SHA512;
+	data->auth_protocol[0].dhchap.idlist[30] = NVME_AUTH_DHGROUP_NULL;
+	data->auth_protocol[0].dhchap.idlist[31] = NVME_AUTH_DHGROUP_2048;
+	data->auth_protocol[0].dhchap.idlist[32] = NVME_AUTH_DHGROUP_3072;
+	data->auth_protocol[0].dhchap.idlist[33] = NVME_AUTH_DHGROUP_4096;
+	data->auth_protocol[0].dhchap.idlist[34] = NVME_AUTH_DHGROUP_6144;
+	data->auth_protocol[0].dhchap.idlist[35] = NVME_AUTH_DHGROUP_8192;
+
+	return size;
+}
+
+static int nvme_auth_process_dhchap_challenge(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_challenge_data *data = chap->buf;
+	u16 dhvlen = le16_to_cpu(data->dhvlen);
+	size_t size = sizeof(*data) + data->hl + dhvlen;
+	const char *hmac_name, *kpp_name;
+
+	if (chap->buf_size < size) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return NVME_SC_INVALID_FIELD;
+	}
+
+	hmac_name = nvme_auth_hmac_name(data->hashid);
+	if (!hmac_name) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid HASH ID %d\n",
+			 chap->qid, data->hashid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
+		return NVME_SC_INVALID_FIELD;
+	}
+
+	if (chap->hash_id == data->hashid && chap->shash_tfm &&
+	    !strcmp(crypto_shash_alg_name(chap->shash_tfm), hmac_name) &&
+	    crypto_shash_digestsize(chap->shash_tfm) == data->hl) {
+		dev_dbg(ctrl->device,
+			"qid %d: reuse existing hash %s\n",
+			chap->qid, hmac_name);
+		goto select_kpp;
+	}
+
+	/* Reset if hash cannot be reused */
+	if (chap->shash_tfm) {
+		crypto_free_shash(chap->shash_tfm);
+		chap->hash_id = 0;
+		chap->hash_len = 0;
+	}
+	chap->shash_tfm = crypto_alloc_shash(hmac_name, 0,
+					     CRYPTO_ALG_ALLOCATES_MEMORY);
+	if (IS_ERR(chap->shash_tfm)) {
+		dev_warn(ctrl->device,
+			 "qid %d: failed to allocate hash %s, error %ld\n",
+			 chap->qid, hmac_name, PTR_ERR(chap->shash_tfm));
+		chap->shash_tfm = NULL;
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_FAILED;
+		return NVME_SC_AUTH_REQUIRED;
+	}
+
+	if (crypto_shash_digestsize(chap->shash_tfm) != data->hl) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid hash length %d\n",
+			 chap->qid, data->hl);
+		crypto_free_shash(chap->shash_tfm);
+		chap->shash_tfm = NULL;
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
+		return NVME_SC_AUTH_REQUIRED;
+	}
+
+	/* Reset host response if the hash had been changed */
+	if (chap->hash_id != data->hashid) {
+		kfree(chap->host_response);
+		chap->host_response = NULL;
+	}
+
+	chap->hash_id = data->hashid;
+	chap->hash_len = data->hl;
+	dev_dbg(ctrl->device, "qid %d: selected hash %s\n",
+		chap->qid, hmac_name);
+
+select_kpp:
+	kpp_name = nvme_auth_dhgroup_kpp(data->dhgid);
+	if (!kpp_name) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid DH group id %d\n",
+			 chap->qid, data->dhgid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
+		return NVME_SC_AUTH_REQUIRED;
+	}
+
+	if (data->dhgid != NVME_AUTH_DHGROUP_NULL) {
+		dev_warn(ctrl->device,
+			 "qid %d: unsupported DH group %s\n",
+			 chap->qid, kpp_name);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
+		return NVME_SC_AUTH_REQUIRED;
+	} else if (dhvlen != 0) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid DH value for NULL DH\n",
+			 chap->qid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return NVME_SC_INVALID_FIELD;
+	}
+	chap->dhgroup_id = data->dhgid;
+
+	chap->s1 = le32_to_cpu(data->seqnum);
+	memcpy(chap->c1, data->cval, chap->hash_len);
+
+	return 0;
+}
+
+static int nvme_auth_set_dhchap_reply_data(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_reply_data *data = chap->buf;
+	size_t size = sizeof(*data);
+
+	size += 2 * chap->hash_len;
+
+	if (chap->buf_size < size) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return -EINVAL;
+	}
+
+	memset(chap->buf, 0, size);
+	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_REPLY;
+	data->t_id = cpu_to_le16(chap->transaction);
+	data->hl = chap->hash_len;
+	data->dhvlen = 0;
+	memcpy(data->rval, chap->response, chap->hash_len);
+	if (ctrl->opts->dhchap_ctrl_secret) {
+		get_random_bytes(chap->c2, chap->hash_len);
+		data->cvalid = 1;
+		chap->s2 = nvme_auth_get_seqnum();
+		memcpy(data->rval + chap->hash_len, chap->c2,
+		       chap->hash_len);
+		dev_dbg(ctrl->device, "%s: qid %d ctrl challenge %*ph\n",
+			__func__, chap->qid, (int)chap->hash_len, chap->c2);
+	} else {
+		memset(chap->c2, 0, chap->hash_len);
+		chap->s2 = 0;
+	}
+	data->seqnum = cpu_to_le32(chap->s2);
+	return size;
+}
+
+static int nvme_auth_process_dhchap_success1(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_success1_data *data = chap->buf;
+	size_t size = sizeof(*data);
+
+	if (ctrl->opts->dhchap_ctrl_secret)
+		size += chap->hash_len;
+
+	if (chap->buf_size < size) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return NVME_SC_INVALID_FIELD;
+	}
+
+	if (data->hl != chap->hash_len) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid hash length %u\n",
+			 chap->qid, data->hl);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
+		return NVME_SC_INVALID_FIELD;
+	}
+
+	/* Just print out information for the admin queue */
+	if (chap->qid == 0)
+		dev_info(ctrl->device,
+			 "qid 0: authenticated with hash %s dhgroup %s\n",
+			 nvme_auth_hmac_name(chap->hash_id),
+			 nvme_auth_dhgroup_name(chap->dhgroup_id));
+
+	if (!data->rvalid)
+		return 0;
+
+	/* Validate controller response */
+	if (memcmp(chap->response, data->rval, data->hl)) {
+		dev_dbg(ctrl->device, "%s: qid %d ctrl response %*ph\n",
+			__func__, chap->qid, (int)chap->hash_len, data->rval);
+		dev_dbg(ctrl->device, "%s: qid %d host response %*ph\n",
+			__func__, chap->qid, (int)chap->hash_len,
+			chap->response);
+		dev_warn(ctrl->device,
+			 "qid %d: controller authentication failed\n",
+			 chap->qid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_FAILED;
+		return NVME_SC_AUTH_REQUIRED;
+	}
+
+	/* Just print out information for the admin queue */
+	if (chap->qid == 0)
+		dev_info(ctrl->device,
+			 "qid 0: controller authenticated\n");
+	return 0;
+}
+
+static int nvme_auth_set_dhchap_success2_data(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_success2_data *data = chap->buf;
+	size_t size = sizeof(*data);
+
+	memset(chap->buf, 0, size);
+	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_SUCCESS2;
+	data->t_id = cpu_to_le16(chap->transaction);
+
+	return size;
+}
+
+static int nvme_auth_set_dhchap_failure2_data(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_failure_data *data = chap->buf;
+	size_t size = sizeof(*data);
+
+	memset(chap->buf, 0, size);
+	data->auth_type = NVME_AUTH_COMMON_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_FAILURE2;
+	data->t_id = cpu_to_le16(chap->transaction);
+	data->rescode = NVME_AUTH_DHCHAP_FAILURE_REASON_FAILED;
+	data->rescode_exp = chap->status;
+
+	return size;
+}
+
+static int nvme_auth_dhchap_setup_host_response(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
+	u8 buf[4], *challenge = chap->c1;
+	int ret;
+
+	dev_dbg(ctrl->device, "%s: qid %d host response seq %u transaction %d\n",
+		__func__, chap->qid, chap->s1, chap->transaction);
+
+	if (!chap->host_response) {
+		chap->host_response = nvme_auth_transform_key(ctrl->host_key,
+						ctrl->opts->host->nqn);
+		if (IS_ERR(chap->host_response)) {
+			ret = PTR_ERR(chap->host_response);
+			chap->host_response = NULL;
+			return ret;
+		}
+	} else {
+		dev_dbg(ctrl->device, "%s: qid %d re-using host response\n",
+			__func__, chap->qid);
+	}
+
+	ret = crypto_shash_setkey(chap->shash_tfm,
+			chap->host_response, ctrl->host_key->len);
+	if (ret) {
+		dev_warn(ctrl->device, "qid %d: failed to set key, error %d\n",
+			 chap->qid, ret);
+		goto out;
+	}
+
+	shash->tfm = chap->shash_tfm;
+	ret = crypto_shash_init(shash);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, challenge, chap->hash_len);
+	if (ret)
+		goto out;
+	put_unaligned_le32(chap->s1, buf);
+	ret = crypto_shash_update(shash, buf, 4);
+	if (ret)
+		goto out;
+	put_unaligned_le16(chap->transaction, buf);
+	ret = crypto_shash_update(shash, buf, 2);
+	if (ret)
+		goto out;
+	memset(buf, 0, sizeof(buf));
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, "HostHost", 8);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
+				  strlen(ctrl->opts->host->nqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
+			    strlen(ctrl->opts->subsysnqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_final(shash, chap->response);
+out:
+	if (challenge != chap->c1)
+		kfree(challenge);
+	return ret;
+}
+
+static int nvme_auth_dhchap_setup_ctrl_response(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
+	u8 *ctrl_response;
+	u8 buf[4], *challenge = chap->c2;
+	int ret;
+
+	ctrl_response = nvme_auth_transform_key(ctrl->ctrl_key,
+				ctrl->opts->subsysnqn);
+	if (IS_ERR(ctrl_response)) {
+		ret = PTR_ERR(ctrl_response);
+		return ret;
+	}
+	ret = crypto_shash_setkey(chap->shash_tfm,
+			ctrl_response, ctrl->ctrl_key->len);
+	if (ret) {
+		dev_warn(ctrl->device, "qid %d: failed to set key, error %d\n",
+			 chap->qid, ret);
+		goto out;
+	}
+
+	dev_dbg(ctrl->device, "%s: qid %d ctrl response seq %u transaction %d\n",
+		__func__, chap->qid, chap->s2, chap->transaction);
+	dev_dbg(ctrl->device, "%s: qid %d challenge %*ph\n",
+		__func__, chap->qid, (int)chap->hash_len, challenge);
+	dev_dbg(ctrl->device, "%s: qid %d subsysnqn %s\n",
+		__func__, chap->qid, ctrl->opts->subsysnqn);
+	dev_dbg(ctrl->device, "%s: qid %d hostnqn %s\n",
+		__func__, chap->qid, ctrl->opts->host->nqn);
+	shash->tfm = chap->shash_tfm;
+	ret = crypto_shash_init(shash);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, challenge, chap->hash_len);
+	if (ret)
+		goto out;
+	put_unaligned_le32(chap->s2, buf);
+	ret = crypto_shash_update(shash, buf, 4);
+	if (ret)
+		goto out;
+	put_unaligned_le16(chap->transaction, buf);
+	ret = crypto_shash_update(shash, buf, 2);
+	if (ret)
+		goto out;
+	memset(buf, 0, 4);
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, "Controller", 10);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
+				  strlen(ctrl->opts->subsysnqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
+				  strlen(ctrl->opts->host->nqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_final(shash, chap->response);
+out:
+	if (challenge != chap->c2)
+		kfree(challenge);
+	return ret;
+}
+
+int nvme_auth_generate_key(u8 *secret, struct nvme_dhchap_key **ret_key)
+{
+	struct nvme_dhchap_key *key;
+	u8 key_hash;
+
+	if (!secret) {
+		*ret_key = NULL;
+		return 0;
+	}
+
+	if (sscanf(secret, "DHHC-1:%hhd:%*s:", &key_hash) != 1)
+		return -EINVAL;
+
+	/* Pass in the secret without the 'DHHC-1:XX:' prefix */
+	key = nvme_auth_extract_key(secret + 10, key_hash);
+	if (IS_ERR(key)) {
+		return PTR_ERR(key);
+	}
+
+	*ret_key = key;
+	return 0;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_generate_key);
+
+static void __nvme_auth_reset(struct nvme_dhchap_queue_context *chap)
+{
+	chap->status = 0;
+	chap->error = 0;
+	chap->s1 = 0;
+	chap->s2 = 0;
+	chap->transaction = 0;
+	memset(chap->c1, 0, sizeof(chap->c1));
+	memset(chap->c2, 0, sizeof(chap->c2));
+}
+
+static void __nvme_auth_free(struct nvme_dhchap_queue_context *chap)
+{
+	__nvme_auth_reset(chap);
+	if (chap->shash_tfm)
+		crypto_free_shash(chap->shash_tfm);
+	kfree_sensitive(chap->host_response);
+	kfree(chap->buf);
+	kfree(chap);
+}
+
+static void __nvme_auth_work(struct work_struct *work)
+{
+	struct nvme_dhchap_queue_context *chap =
+		container_of(work, struct nvme_dhchap_queue_context, auth_work);
+	struct nvme_ctrl *ctrl = chap->ctrl;
+	size_t tl;
+	int ret = 0;
+
+	chap->transaction = ctrl->transaction++;
+
+	/* DH-HMAC-CHAP Step 1: send negotiate */
+	dev_dbg(ctrl->device, "%s: qid %d send negotiate\n",
+		__func__, chap->qid);
+	ret = nvme_auth_set_dhchap_negotiate_data(ctrl, chap);
+	if (ret < 0) {
+		chap->error = ret;
+		return;
+	}
+	tl = ret;
+	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, tl, true);
+	if (ret) {
+		chap->error = ret;
+		return;
+	}
+
+	/* DH-HMAC-CHAP Step 2: receive challenge */
+	dev_dbg(ctrl->device, "%s: qid %d receive challenge\n",
+		__func__, chap->qid);
+
+	memset(chap->buf, 0, chap->buf_size);
+	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, chap->buf_size, false);
+	if (ret) {
+		dev_warn(ctrl->device,
+			 "qid %d failed to receive challenge, %s %d\n",
+			 chap->qid, ret < 0 ? "error" : "nvme status", ret);
+		chap->error = ret;
+		return;
+	}
+	ret = nvme_auth_receive_validate(ctrl, chap->qid, chap->buf, chap->transaction,
+					 NVME_AUTH_DHCHAP_MESSAGE_CHALLENGE);
+	if (ret) {
+		chap->status = ret;
+		chap->error = NVME_SC_AUTH_REQUIRED;
+		return;
+	}
+
+	ret = nvme_auth_process_dhchap_challenge(ctrl, chap);
+	if (ret) {
+		/* Invalid challenge parameters */
+		goto fail2;
+	}
+
+	dev_dbg(ctrl->device, "%s: qid %d host response\n",
+		__func__, chap->qid);
+	ret = nvme_auth_dhchap_setup_host_response(ctrl, chap);
+	if (ret)
+		goto fail2;
+
+	/* DH-HMAC-CHAP Step 3: send reply */
+	dev_dbg(ctrl->device, "%s: qid %d send reply\n",
+		__func__, chap->qid);
+	ret = nvme_auth_set_dhchap_reply_data(ctrl, chap);
+	if (ret < 0)
+		goto fail2;
+
+	tl = ret;
+	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, tl, true);
+	if (ret)
+		goto fail2;
+
+	/* DH-HMAC-CHAP Step 4: receive success1 */
+	dev_dbg(ctrl->device, "%s: qid %d receive success1\n",
+		__func__, chap->qid);
+
+	memset(chap->buf, 0, chap->buf_size);
+	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, chap->buf_size, false);
+	if (ret) {
+		dev_warn(ctrl->device,
+			 "qid %d failed to receive success1, %s %d\n",
+			 chap->qid, ret < 0 ? "error" : "nvme status", ret);
+		chap->error = ret;
+		return;
+	}
+	ret = nvme_auth_receive_validate(ctrl, chap->qid,
+					 chap->buf, chap->transaction,
+					 NVME_AUTH_DHCHAP_MESSAGE_SUCCESS1);
+	if (ret) {
+		chap->status = ret;
+		chap->error = NVME_SC_AUTH_REQUIRED;
+		return;
+	}
+
+	if (ctrl->opts->dhchap_ctrl_secret) {
+		dev_dbg(ctrl->device,
+			"%s: qid %d controller response\n",
+			__func__, chap->qid);
+		ret = nvme_auth_dhchap_setup_ctrl_response(ctrl, chap);
+		if (ret) {
+			chap->error = ret;
+			goto fail2;
+		}
+	}
+
+	ret = nvme_auth_process_dhchap_success1(ctrl, chap);
+	if (ret) {
+		/* Controller authentication failed */
+		chap->error = NVME_SC_AUTH_REQUIRED;
+		goto fail2;
+	}
+
+	if (ctrl->opts->dhchap_ctrl_secret) {
+		/* DH-HMAC-CHAP Step 5: send success2 */
+		dev_dbg(ctrl->device, "%s: qid %d send success2\n",
+			__func__, chap->qid);
+		tl = nvme_auth_set_dhchap_success2_data(ctrl, chap);
+		ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, tl, true);
+	}
+	if (!ret) {
+		chap->error = 0;
+		return;
+	}
+
+fail2:
+	dev_dbg(ctrl->device, "%s: qid %d send failure2, status %x\n",
+		__func__, chap->qid, chap->status);
+	tl = nvme_auth_set_dhchap_failure2_data(ctrl, chap);
+	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, tl, true);
+	/*
+	 * only update error if send failure2 failed and no other
+	 * error had been set during authentication.
+	 */
+	if (ret && !chap->error)
+		chap->error = ret;
+}
+
+int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid)
+{
+	struct nvme_dhchap_queue_context *chap;
+
+	if (!ctrl->host_key) {
+		dev_warn(ctrl->device, "qid %d: no key\n", qid);
+		return -ENOKEY;
+	}
+
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	/* Check if the context is already queued */
+	list_for_each_entry(chap, &ctrl->dhchap_auth_list, entry) {
+		WARN_ON(!chap->buf);
+		if (chap->qid == qid) {
+			dev_dbg(ctrl->device, "qid %d: re-using context\n", qid);
+			mutex_unlock(&ctrl->dhchap_auth_mutex);
+			flush_work(&chap->auth_work);
+			__nvme_auth_reset(chap);
+			queue_work(nvme_wq, &chap->auth_work);
+			return 0;
+		}
+	}
+	chap = kzalloc(sizeof(*chap), GFP_KERNEL);
+	if (!chap) {
+		mutex_unlock(&ctrl->dhchap_auth_mutex);
+		return -ENOMEM;
+	}
+	chap->qid = (qid == NVME_QID_ANY) ? 0 : qid;
+	chap->ctrl = ctrl;
+
+	/*
+	 * Allocate a large enough buffer for the entire negotiation:
+	 * 4k should be enough to ffdhe8192.
+	 */
+	chap->buf_size = 4096;
+	chap->buf = kzalloc(chap->buf_size, GFP_KERNEL);
+	if (!chap->buf) {
+		mutex_unlock(&ctrl->dhchap_auth_mutex);
+		kfree(chap);
+		return -ENOMEM;
+	}
+
+	INIT_WORK(&chap->auth_work, __nvme_auth_work);
+	list_add(&chap->entry, &ctrl->dhchap_auth_list);
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+	queue_work(nvme_wq, &chap->auth_work);
+	return 0;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_negotiate);
+
+int nvme_auth_wait(struct nvme_ctrl *ctrl, int qid)
+{
+	struct nvme_dhchap_queue_context *chap;
+	int ret;
+
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	list_for_each_entry(chap, &ctrl->dhchap_auth_list, entry) {
+		if (chap->qid != qid)
+			continue;
+		mutex_unlock(&ctrl->dhchap_auth_mutex);
+		flush_work(&chap->auth_work);
+		ret = chap->error;
+		return ret;
+	}
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+	return -ENXIO;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_wait);
+
+void nvme_auth_reset(struct nvme_ctrl *ctrl)
+{
+	struct nvme_dhchap_queue_context *chap;
+
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	list_for_each_entry(chap, &ctrl->dhchap_auth_list, entry) {
+		mutex_unlock(&ctrl->dhchap_auth_mutex);
+		flush_work(&chap->auth_work);
+		__nvme_auth_reset(chap);
+	}
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_reset);
+
+static void nvme_dhchap_auth_work(struct work_struct *work)
+{
+	struct nvme_ctrl *ctrl =
+		container_of(work, struct nvme_ctrl, dhchap_auth_work);
+	int ret, q;
+
+	/* Authenticate admin queue first */
+	ret = nvme_auth_negotiate(ctrl, 0);
+	if (ret) {
+		dev_warn(ctrl->device,
+			 "qid 0: error %d setting up authentication\n", ret);
+		return;
+	}
+	ret = nvme_auth_wait(ctrl, 0);
+	if (ret) {
+		dev_warn(ctrl->device,
+			 "qid 0: authentication failed\n");
+		return;
+	}
+
+	for (q = 1; q < ctrl->queue_count; q++) {
+		ret = nvme_auth_negotiate(ctrl, q);
+		if (ret) {
+			dev_warn(ctrl->device,
+				 "qid %d: error %d setting up authentication\n",
+				 q, ret);
+			break;
+		}
+	}
+
+	/*
+	 * Failure is a soft-state; credentials remain valid until
+	 * the controller terminates the connection.
+	 */
+}
+
+void nvme_auth_init_ctrl(struct nvme_ctrl *ctrl)
+{
+	INIT_LIST_HEAD(&ctrl->dhchap_auth_list);
+	INIT_WORK(&ctrl->dhchap_auth_work, nvme_dhchap_auth_work);
+	mutex_init(&ctrl->dhchap_auth_mutex);
+	if (!ctrl->opts)
+		return;
+	nvme_auth_generate_key(ctrl->opts->dhchap_secret, &ctrl->host_key);
+	nvme_auth_generate_key(ctrl->opts->dhchap_ctrl_secret, &ctrl->ctrl_key);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_init_ctrl);
+
+void nvme_auth_stop(struct nvme_ctrl *ctrl)
+{
+	struct nvme_dhchap_queue_context *chap = NULL, *tmp;
+
+	cancel_work_sync(&ctrl->dhchap_auth_work);
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	list_for_each_entry_safe(chap, tmp, &ctrl->dhchap_auth_list, entry)
+		cancel_work_sync(&chap->auth_work);
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_stop);
+
+void nvme_auth_free(struct nvme_ctrl *ctrl)
+{
+	struct nvme_dhchap_queue_context *chap = NULL, *tmp;
+
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	list_for_each_entry_safe(chap, tmp, &ctrl->dhchap_auth_list, entry) {
+		list_del_init(&chap->entry);
+		flush_work(&chap->auth_work);
+		__nvme_auth_free(chap);
+	}
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+	if (ctrl->host_key) {
+		nvme_auth_free_key(ctrl->host_key);
+		ctrl->host_key = NULL;
+	}
+	if (ctrl->ctrl_key) {
+		nvme_auth_free_key(ctrl->ctrl_key);
+		ctrl->ctrl_key = NULL;
+	}
+}
+EXPORT_SYMBOL_GPL(nvme_auth_free);
diff --git a/drivers/nvme/host/auth.h b/drivers/nvme/host/auth.h
new file mode 100644
index 000000000000..73d6ec63a5c8
--- /dev/null
+++ b/drivers/nvme/host/auth.h
@@ -0,0 +1,32 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2021 Hannes Reinecke, SUSE Software Solutions
+ */
+
+#ifndef _NVME_AUTH_H
+#define _NVME_AUTH_H
+
+#include <crypto/kpp.h>
+
+struct nvme_dhchap_key {
+	u8 *key;
+	size_t len;
+	u8 hash;
+};
+
+u32 nvme_auth_get_seqnum(void);
+const char *nvme_auth_dhgroup_name(u8 dhgroup_id);
+const char *nvme_auth_dhgroup_kpp(u8 dhgroup_id);
+u8 nvme_auth_dhgroup_id(const char *dhgroup_name);
+
+const char *nvme_auth_hmac_name(u8 hmac_id);
+const char *nvme_auth_digest_name(u8 hmac_id);
+size_t nvme_auth_hmac_hash_len(u8 hmac_id);
+u8 nvme_auth_hmac_id(const char *hmac_name);
+
+struct nvme_dhchap_key *nvme_auth_extract_key(unsigned char *secret,
+					      u8 key_hash);
+void nvme_auth_free_key(struct nvme_dhchap_key *key);
+u8 *nvme_auth_transform_key(struct nvme_dhchap_key *key, char *nqn);
+
+#endif /* _NVME_AUTH_H */
diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 42f9772abc4d..979350909794 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -24,6 +24,7 @@
 
 #include "nvme.h"
 #include "fabrics.h"
+#include "auth.h"
 
 #define CREATE_TRACE_POINTS
 #include "trace.h"
@@ -330,6 +331,7 @@ enum nvme_disposition {
 	COMPLETE,
 	RETRY,
 	FAILOVER,
+	AUTHENTICATE,
 };
 
 static inline enum nvme_disposition nvme_decide_disposition(struct request *req)
@@ -337,6 +339,9 @@ static inline enum nvme_disposition nvme_decide_disposition(struct request *req)
 	if (likely(nvme_req(req)->status == 0))
 		return COMPLETE;
 
+	if ((nvme_req(req)->status & 0x7ff) == NVME_SC_AUTH_REQUIRED)
+		return AUTHENTICATE;
+
 	if (blk_noretry_request(req) ||
 	    (nvme_req(req)->status & NVME_SC_DNR) ||
 	    nvme_req(req)->retries >= nvme_max_retries)
@@ -375,11 +380,13 @@ static inline void nvme_end_req(struct request *req)
 
 void nvme_complete_rq(struct request *req)
 {
+	struct nvme_ctrl *ctrl = nvme_req(req)->ctrl;
+
 	trace_nvme_complete_rq(req);
 	nvme_cleanup_cmd(req);
 
-	if (nvme_req(req)->ctrl->kas)
-		nvme_req(req)->ctrl->comp_seen = true;
+	if (ctrl->kas)
+		ctrl->comp_seen = true;
 
 	switch (nvme_decide_disposition(req)) {
 	case COMPLETE:
@@ -391,6 +398,14 @@ void nvme_complete_rq(struct request *req)
 	case FAILOVER:
 		nvme_failover_req(req);
 		return;
+	case AUTHENTICATE:
+#ifdef CONFIG_NVME_AUTH
+		queue_work(nvme_wq, &ctrl->dhchap_auth_work);
+		nvme_retry_req(req);
+#else
+		nvme_end_req(req);
+#endif
+		return;
 	}
 }
 EXPORT_SYMBOL_GPL(nvme_complete_rq);
@@ -702,7 +717,9 @@ bool __nvme_check_ready(struct nvme_ctrl *ctrl, struct request *rq,
 		switch (ctrl->state) {
 		case NVME_CTRL_CONNECTING:
 			if (blk_rq_is_passthrough(rq) && nvme_is_fabrics(req->cmd) &&
-			    req->cmd->fabrics.fctype == nvme_fabrics_type_connect)
+			    (req->cmd->fabrics.fctype == nvme_fabrics_type_connect ||
+			     req->cmd->fabrics.fctype == nvme_fabrics_type_auth_send ||
+			     req->cmd->fabrics.fctype == nvme_fabrics_type_auth_receive))
 				return true;
 			break;
 		default:
@@ -3546,6 +3563,108 @@ static ssize_t dctype_show(struct device *dev,
 }
 static DEVICE_ATTR_RO(dctype);
 
+#ifdef CONFIG_NVME_AUTH
+static ssize_t nvme_ctrl_dhchap_secret_show(struct device *dev,
+		struct device_attribute *attr, char *buf)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	struct nvmf_ctrl_options *opts = ctrl->opts;
+
+	if (!opts->dhchap_secret)
+		return sysfs_emit(buf, "none\n");
+	return sysfs_emit(buf, "%s\n", opts->dhchap_secret);
+}
+
+static ssize_t nvme_ctrl_dhchap_secret_store(struct device *dev,
+		struct device_attribute *attr, const char *buf, size_t count)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	struct nvmf_ctrl_options *opts = ctrl->opts;
+	char *dhchap_secret;
+
+	if (!ctrl->opts->dhchap_secret)
+		return -EINVAL;
+	if (count < 7)
+		return -EINVAL;
+	if (memcmp(buf, "DHHC-1:", 7))
+		return -EINVAL;
+
+	dhchap_secret = kzalloc(count + 1, GFP_KERNEL);
+	if (!dhchap_secret)
+		return -ENOMEM;
+	memcpy(dhchap_secret, buf, count);
+	nvme_auth_stop(ctrl);
+	if (strcmp(dhchap_secret, opts->dhchap_secret)) {
+		int ret;
+
+		ret = nvme_auth_generate_key(dhchap_secret, &ctrl->host_key);
+		if (ret)
+			return ret;
+		kfree(opts->dhchap_secret);
+		opts->dhchap_secret = dhchap_secret;
+		/* Key has changed; re-authentication with new key */
+		nvme_auth_reset(ctrl);
+	}
+	/* Start re-authentication */
+	dev_info(ctrl->device, "re-authenticating controller\n");
+	queue_work(nvme_wq, &ctrl->dhchap_auth_work);
+
+	return count;
+}
+DEVICE_ATTR(dhchap_secret, S_IRUGO | S_IWUSR,
+	nvme_ctrl_dhchap_secret_show, nvme_ctrl_dhchap_secret_store);
+
+static ssize_t nvme_ctrl_dhchap_ctrl_secret_show(struct device *dev,
+		struct device_attribute *attr, char *buf)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	struct nvmf_ctrl_options *opts = ctrl->opts;
+
+	if (!opts->dhchap_ctrl_secret)
+		return sysfs_emit(buf, "none\n");
+	return sysfs_emit(buf, "%s\n", opts->dhchap_ctrl_secret);
+}
+
+static ssize_t nvme_ctrl_dhchap_ctrl_secret_store(struct device *dev,
+		struct device_attribute *attr, const char *buf, size_t count)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	struct nvmf_ctrl_options *opts = ctrl->opts;
+	char *dhchap_secret;
+
+	if (!ctrl->opts->dhchap_ctrl_secret)
+		return -EINVAL;
+	if (count < 7)
+		return -EINVAL;
+	if (memcmp(buf, "DHHC-1:", 7))
+		return -EINVAL;
+
+	dhchap_secret = kzalloc(count + 1, GFP_KERNEL);
+	if (!dhchap_secret)
+		return -ENOMEM;
+	memcpy(dhchap_secret, buf, count);
+	nvme_auth_stop(ctrl);
+	if (strcmp(dhchap_secret, opts->dhchap_ctrl_secret)) {
+		int ret;
+
+		ret = nvme_auth_generate_key(dhchap_secret, &ctrl->ctrl_key);
+		if (ret)
+			return ret;
+		kfree(opts->dhchap_ctrl_secret);
+		opts->dhchap_ctrl_secret = dhchap_secret;
+		/* Key has changed; re-authentication with new key */
+		nvme_auth_reset(ctrl);
+	}
+	/* Start re-authentication */
+	dev_info(ctrl->device, "re-authenticating controller\n");
+	queue_work(nvme_wq, &ctrl->dhchap_auth_work);
+
+	return count;
+}
+DEVICE_ATTR(dhchap_ctrl_secret, S_IRUGO | S_IWUSR,
+	nvme_ctrl_dhchap_ctrl_secret_show, nvme_ctrl_dhchap_ctrl_secret_store);
+#endif
+
 static struct attribute *nvme_dev_attrs[] = {
 	&dev_attr_reset_controller.attr,
 	&dev_attr_rescan_controller.attr,
@@ -3569,6 +3688,10 @@ static struct attribute *nvme_dev_attrs[] = {
 	&dev_attr_kato.attr,
 	&dev_attr_cntrltype.attr,
 	&dev_attr_dctype.attr,
+#ifdef CONFIG_NVME_AUTH
+	&dev_attr_dhchap_secret.attr,
+	&dev_attr_dhchap_ctrl_secret.attr,
+#endif
 	NULL
 };
 
@@ -3592,6 +3715,10 @@ static umode_t nvme_dev_attrs_are_visible(struct kobject *kobj,
 		return 0;
 	if (a == &dev_attr_fast_io_fail_tmo.attr && !ctrl->opts)
 		return 0;
+#ifdef CONFIG_NVME_AUTH
+	if (a == &dev_attr_dhchap_secret.attr && !ctrl->opts)
+		return 0;
+#endif
 
 	return a->mode;
 }
@@ -4444,8 +4571,10 @@ static void nvme_handle_aen_notice(struct nvme_ctrl *ctrl, u32 result)
 		 * recovery actions from interfering with the controller's
 		 * firmware activation.
 		 */
-		if (nvme_change_ctrl_state(ctrl, NVME_CTRL_RESETTING))
+		if (nvme_change_ctrl_state(ctrl, NVME_CTRL_RESETTING)) {
+			nvme_auth_stop(ctrl);
 			queue_work(nvme_wq, &ctrl->fw_act_work);
+		}
 		break;
 #ifdef CONFIG_NVME_MULTIPATH
 	case NVME_AER_NOTICE_ANA:
@@ -4492,6 +4621,7 @@ EXPORT_SYMBOL_GPL(nvme_complete_async_event);
 void nvme_stop_ctrl(struct nvme_ctrl *ctrl)
 {
 	nvme_mpath_stop(ctrl);
+	nvme_auth_stop(ctrl);
 	nvme_stop_keep_alive(ctrl);
 	nvme_stop_failfast_work(ctrl);
 	flush_work(&ctrl->async_event_work);
@@ -4549,6 +4679,8 @@ static void nvme_free_ctrl(struct device *dev)
 
 	nvme_free_cels(ctrl);
 	nvme_mpath_uninit(ctrl);
+	nvme_auth_stop(ctrl);
+	nvme_auth_free(ctrl);
 	__free_page(ctrl->discard_page);
 
 	if (subsys) {
@@ -4639,6 +4771,7 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev,
 
 	nvme_fault_inject_init(&ctrl->fault_inject, dev_name(ctrl->device));
 	nvme_mpath_init_ctrl(ctrl);
+	nvme_auth_init_ctrl(ctrl);
 
 	return 0;
 out_free_name:
diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index 953e3076aab1..f5786a60931c 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -369,6 +369,7 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)
 	union nvme_result res;
 	struct nvmf_connect_data *data;
 	int ret;
+	u32 result;
 
 	cmd.connect.opcode = nvme_fabrics_command;
 	cmd.connect.fctype = nvme_fabrics_type_connect;
@@ -401,8 +402,25 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)
 		goto out_free_data;
 	}
 
-	ctrl->cntlid = le16_to_cpu(res.u16);
-
+	result = le32_to_cpu(res.u32);
+	ctrl->cntlid = result & 0xFFFF;
+	if ((result >> 16) & 2) {
+		/* Authentication required */
+		ret = nvme_auth_negotiate(ctrl, 0);
+		if (ret) {
+			dev_warn(ctrl->device,
+				 "qid 0: authentication setup failed\n");
+			ret = NVME_SC_AUTH_REQUIRED;
+			goto out_free_data;
+		}
+		ret = nvme_auth_wait(ctrl, 0);
+		if (ret)
+			dev_warn(ctrl->device,
+				 "qid 0: authentication failed\n");
+		else
+			dev_info(ctrl->device,
+				 "qid 0: authenticated\n");
+	}
 out_free_data:
 	kfree(data);
 	return ret;
@@ -435,6 +453,7 @@ int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, u16 qid)
 	struct nvmf_connect_data *data;
 	union nvme_result res;
 	int ret;
+	u32 result;
 
 	cmd.connect.opcode = nvme_fabrics_command;
 	cmd.connect.fctype = nvme_fabrics_type_connect;
@@ -460,6 +479,21 @@ int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, u16 qid)
 		nvmf_log_connect_error(ctrl, ret, le32_to_cpu(res.u32),
 				       &cmd, data);
 	}
+	result = le32_to_cpu(res.u32);
+	if ((result >> 16) & 2) {
+		/* Authentication required */
+		ret = nvme_auth_negotiate(ctrl, qid);
+		if (ret) {
+			dev_warn(ctrl->device,
+				 "qid %d: authentication setup failed\n", qid);
+			ret = NVME_SC_AUTH_REQUIRED;
+		} else {
+			ret = nvme_auth_wait(ctrl, qid);
+			if (ret)
+				dev_warn(ctrl->device,
+					 "qid %u: authentication failed\n", qid);
+		}
+	}
 	kfree(data);
 	return ret;
 }
@@ -552,6 +586,8 @@ static const match_table_t opt_tokens = {
 	{ NVMF_OPT_TOS,			"tos=%d"		},
 	{ NVMF_OPT_FAIL_FAST_TMO,	"fast_io_fail_tmo=%d"	},
 	{ NVMF_OPT_DISCOVERY,		"discovery"		},
+	{ NVMF_OPT_DHCHAP_SECRET,	"dhchap_secret=%s"	},
+	{ NVMF_OPT_DHCHAP_CTRL_SECRET,	"dhchap_ctrl_secret=%s"	},
 	{ NVMF_OPT_ERR,			NULL			}
 };
 
@@ -833,6 +869,34 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
 		case NVMF_OPT_DISCOVERY:
 			opts->discovery_nqn = true;
 			break;
+		case NVMF_OPT_DHCHAP_SECRET:
+			p = match_strdup(args);
+			if (!p) {
+				ret = -ENOMEM;
+				goto out;
+			}
+			if (strlen(p) < 11 || strncmp(p, "DHHC-1:", 7)) {
+				pr_err("Invalid DH-CHAP secret %s\n", p);
+				ret = -EINVAL;
+				goto out;
+			}
+			kfree(opts->dhchap_secret);
+			opts->dhchap_secret = p;
+			break;
+		case NVMF_OPT_DHCHAP_CTRL_SECRET:
+			p = match_strdup(args);
+			if (!p) {
+				ret = -ENOMEM;
+				goto out;
+			}
+			if (strlen(p) < 11 || strncmp(p, "DHHC-1:", 7)) {
+				pr_err("Invalid DH-CHAP secret %s\n", p);
+				ret = -EINVAL;
+				goto out;
+			}
+			kfree(opts->dhchap_ctrl_secret);
+			opts->dhchap_ctrl_secret = p;
+			break;
 		default:
 			pr_warn("unknown parameter or missing value '%s' in ctrl creation request\n",
 				p);
@@ -951,6 +1015,7 @@ void nvmf_free_options(struct nvmf_ctrl_options *opts)
 	kfree(opts->subsysnqn);
 	kfree(opts->host_traddr);
 	kfree(opts->host_iface);
+	kfree(opts->dhchap_secret);
 	kfree(opts);
 }
 EXPORT_SYMBOL_GPL(nvmf_free_options);
@@ -960,7 +1025,8 @@ EXPORT_SYMBOL_GPL(nvmf_free_options);
 				 NVMF_OPT_KATO | NVMF_OPT_HOSTNQN | \
 				 NVMF_OPT_HOST_ID | NVMF_OPT_DUP_CONNECT |\
 				 NVMF_OPT_DISABLE_SQFLOW | NVMF_OPT_DISCOVERY |\
-				 NVMF_OPT_FAIL_FAST_TMO)
+				 NVMF_OPT_FAIL_FAST_TMO | NVMF_OPT_DHCHAP_SECRET |\
+				 NVMF_OPT_DHCHAP_CTRL_SECRET)
 
 static struct nvme_ctrl *
 nvmf_create_ctrl(struct device *dev, const char *buf)
@@ -1196,7 +1262,14 @@ static void __exit nvmf_exit(void)
 	BUILD_BUG_ON(sizeof(struct nvmf_connect_command) != 64);
 	BUILD_BUG_ON(sizeof(struct nvmf_property_get_command) != 64);
 	BUILD_BUG_ON(sizeof(struct nvmf_property_set_command) != 64);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_send_command) != 64);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_receive_command) != 64);
 	BUILD_BUG_ON(sizeof(struct nvmf_connect_data) != 1024);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_negotiate_data) != 8);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_challenge_data) != 16);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_reply_data) != 16);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_success1_data) != 16);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_success2_data) != 16);
 }
 
 MODULE_LICENSE("GPL v2");
diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h
index 46d6e194ac2b..a6e22116e139 100644
--- a/drivers/nvme/host/fabrics.h
+++ b/drivers/nvme/host/fabrics.h
@@ -68,6 +68,8 @@ enum {
 	NVMF_OPT_FAIL_FAST_TMO	= 1 << 20,
 	NVMF_OPT_HOST_IFACE	= 1 << 21,
 	NVMF_OPT_DISCOVERY	= 1 << 22,
+	NVMF_OPT_DHCHAP_SECRET	= 1 << 23,
+	NVMF_OPT_DHCHAP_CTRL_SECRET = 1 << 24,
 };
 
 /**
@@ -97,6 +99,9 @@ enum {
  * @max_reconnects: maximum number of allowed reconnect attempts before removing
  *              the controller, (-1) means reconnect forever, zero means remove
  *              immediately;
+ * @dhchap_secret: DH-HMAC-CHAP secret
+ * @dhchap_ctrl_secret: DH-HMAC-CHAP controller secret for bi-directional
+ *              authentication
  * @disable_sqflow: disable controller sq flow control
  * @hdr_digest: generate/verify header digest (TCP)
  * @data_digest: generate/verify data digest (TCP)
@@ -121,6 +126,8 @@ struct nvmf_ctrl_options {
 	unsigned int		kato;
 	struct nvmf_host	*host;
 	int			max_reconnects;
+	char			*dhchap_secret;
+	char			*dhchap_ctrl_secret;
 	bool			disable_sqflow;
 	bool			hdr_digest;
 	bool			data_digest;
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 81c4f5379c0c..004f6c118c7b 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -328,6 +328,15 @@ struct nvme_ctrl {
 	struct work_struct ana_work;
 #endif
 
+#ifdef CONFIG_NVME_AUTH
+	struct work_struct dhchap_auth_work;
+	struct list_head dhchap_auth_list;
+	struct mutex dhchap_auth_mutex;
+	struct nvme_dhchap_key *host_key;
+	struct nvme_dhchap_key *ctrl_key;
+	u16 transaction;
+#endif
+
 	/* Power saving configuration */
 	u64 ps_max_latency_us;
 	bool apst_enabled;
@@ -958,6 +967,28 @@ static inline bool nvme_ctrl_sgl_supported(struct nvme_ctrl *ctrl)
 	return ctrl->sgls & ((1 << 0) | (1 << 1));
 }
 
+#ifdef CONFIG_NVME_AUTH
+void nvme_auth_init_ctrl(struct nvme_ctrl *ctrl);
+void nvme_auth_stop(struct nvme_ctrl *ctrl);
+int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid);
+int nvme_auth_wait(struct nvme_ctrl *ctrl, int qid);
+void nvme_auth_reset(struct nvme_ctrl *ctrl);
+void nvme_auth_free(struct nvme_ctrl *ctrl);
+int nvme_auth_generate_key(u8 *secret, struct nvme_dhchap_key **ret_key);
+#else
+static inline void nvme_auth_init_ctrl(struct nvme_ctrl *ctrl) {};
+static inline void nvme_auth_stop(struct nvme_ctrl *ctrl) {};
+static inline int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid)
+{
+	return -EPROTONOSUPPORT;
+}
+static inline int nvme_auth_wait(struct nvme_ctrl *ctrl, int qid)
+{
+	return NVME_SC_AUTH_REQUIRED;
+}
+static inline void nvme_auth_free(struct nvme_ctrl *ctrl) {};
+#endif
+
 u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 			 u8 opcode);
 int nvme_execute_passthru_rq(struct request *rq);
diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index b87c8ae41d9b..bd4c25af74d3 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -1197,6 +1197,7 @@ static void nvme_rdma_error_recovery_work(struct work_struct *work)
 	struct nvme_rdma_ctrl *ctrl = container_of(work,
 			struct nvme_rdma_ctrl, err_work);
 
+	nvme_auth_stop(&ctrl->ctrl);
 	nvme_stop_keep_alive(&ctrl->ctrl);
 	flush_work(&ctrl->ctrl.async_event_work);
 	nvme_rdma_teardown_io_queues(ctrl, false);
diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index bb67538d241b..a7848e430a5c 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -2174,6 +2174,7 @@ static void nvme_tcp_error_recovery_work(struct work_struct *work)
 				struct nvme_tcp_ctrl, err_work);
 	struct nvme_ctrl *ctrl = &tcp_ctrl->ctrl;
 
+	nvme_auth_stop(ctrl);
 	nvme_stop_keep_alive(ctrl);
 	flush_work(&ctrl->async_event_work);
 	nvme_tcp_teardown_io_queues(ctrl, false);
diff --git a/drivers/nvme/host/trace.c b/drivers/nvme/host/trace.c
index 2a89c5aa0790..1c36fcedea20 100644
--- a/drivers/nvme/host/trace.c
+++ b/drivers/nvme/host/trace.c
@@ -287,6 +287,34 @@ static const char *nvme_trace_fabrics_property_get(struct trace_seq *p, u8 *spc)
 	return ret;
 }
 
+static const char *nvme_trace_fabrics_auth_send(struct trace_seq *p, u8 *spc)
+{
+	const char *ret = trace_seq_buffer_ptr(p);
+	u8 spsp0 = spc[1];
+	u8 spsp1 = spc[2];
+	u8 secp = spc[3];
+	u32 tl = get_unaligned_le32(spc + 4);
+
+	trace_seq_printf(p, "spsp0=%02x, spsp1=%02x, secp=%02x, tl=%u",
+			 spsp0, spsp1, secp, tl);
+	trace_seq_putc(p, 0);
+	return ret;
+}
+
+static const char *nvme_trace_fabrics_auth_receive(struct trace_seq *p, u8 *spc)
+{
+	const char *ret = trace_seq_buffer_ptr(p);
+	u8 spsp0 = spc[1];
+	u8 spsp1 = spc[2];
+	u8 secp = spc[3];
+	u32 al = get_unaligned_le32(spc + 4);
+
+	trace_seq_printf(p, "spsp0=%02x, spsp1=%02x, secp=%02x, al=%u",
+			 spsp0, spsp1, secp, al);
+	trace_seq_putc(p, 0);
+	return ret;
+}
+
 static const char *nvme_trace_fabrics_common(struct trace_seq *p, u8 *spc)
 {
 	const char *ret = trace_seq_buffer_ptr(p);
@@ -306,6 +334,10 @@ const char *nvme_trace_parse_fabrics_cmd(struct trace_seq *p,
 		return nvme_trace_fabrics_connect(p, spc);
 	case nvme_fabrics_type_property_get:
 		return nvme_trace_fabrics_property_get(p, spc);
+	case nvme_fabrics_type_auth_send:
+		return nvme_trace_fabrics_auth_send(p, spc);
+	case nvme_fabrics_type_auth_receive:
+		return nvme_trace_fabrics_auth_receive(p, spc);
 	default:
 		return nvme_trace_fabrics_common(p, spc);
 	}
-- 
2.29.2

Re: [PATCH 06/11] nvme: Implement In-Band authentication

[Hannes Reinecke]

On 4/27/22 19:59, Nayak, Prashanth wrote:
> Hi Hannes
> 
>> On Mar 28, 2022, at 9:39 AM, Hannes Reinecke <hare@suse.de> wrote:
>>
>> Implement NVMe-oF In-Band authentication according to NVMe TPAR 8006.
>> This patch adds two new fabric options 'dhchap_secret' to specify the
>> pre-shared key (in ASCII respresentation according to NVMe 2.0 section
>> 8.13.5.8 'Secret representation') and 'dhchap_ctrl_secret' to specify
>> the pre-shared controller key for bi-directional authentication of both
>> the host and the controller.
>> Re-authentication can be triggered by writing the PSK into the new
>> controller sysfs attribute 'dhchap_secret' or 'dhchap_ctrl_secret'.
>>
>> Signed-off-by: Hannes Reinecke <hare@suse.de>
>> ---
>> drivers/nvme/host/Kconfig   |   11 +
>> drivers/nvme/host/Makefile  |    1 +
>> drivers/nvme/host/auth.c    | 1123 +++++++++++++++++++++++++++++++++++
>> drivers/nvme/host/auth.h    |   32 +
>> drivers/nvme/host/core.c    |  141 ++++-
>> drivers/nvme/host/fabrics.c |   79 ++-
>> drivers/nvme/host/fabrics.h |    7 +
>> drivers/nvme/host/nvme.h    |   31 +
>> drivers/nvme/host/rdma.c    |    1 +
>> drivers/nvme/host/tcp.c     |    1 +
>> drivers/nvme/host/trace.c   |   32 +
>> 11 files changed, 1452 insertions(+), 7 deletions(-)
>> create mode 100644 drivers/nvme/host/auth.c
>> create mode 100644 drivers/nvme/host/auth.h
>>
[ .. ]
>> +static int nvme_auth_set_dhchap_failure2_data(struct nvme_ctrl *ctrl,
>> +		struct nvme_dhchap_queue_context *chap)
>> +{
>> +	struct nvmf_auth_dhchap_failure_data *data = chap->buf;
>> +	size_t size = sizeof(*data);
>> +
>> +	memset(chap->buf, 0, size);
>> +	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
> 
> Auth_type should be NVME_AUTH_COMMON_MESSAGES.
> 

Right.

>> +	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_FAILURE2;
>> +	data->t_id = cpu_to_le16(chap->transaction);
>> +	data->rescode = NVME_AUTH_DHCHAP_FAILURE_REASON_FAILED;
>> +	data->rescode_exp = chap->status;
>> +
>> +	return size;
>> +}
>> +
>> +static int nvme_auth_dhchap_setup_host_response(struct nvme_ctrl *ctrl,
>> +		struct nvme_dhchap_queue_context *chap)
>> +{
>> +	SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
>> +	u8 buf[4], *challenge = chap->c1;
>> +	int ret;
>> +
>> +	dev_dbg(ctrl->device, "%s: qid %d host response seq %u transaction %d\n",
>> +		__func__, chap->qid, chap->s1, chap->transaction);
>> +
>> +	if (!chap->host_response) {
>> +		chap->host_response = nvme_auth_transform_key(ctrl->host_key,
>> +						ctrl->opts->host->nqn);
>> +		if (IS_ERR(chap->host_response)) {
>> +			ret = PTR_ERR(chap->host_response);
>> +			chap->host_response = NULL;
>> +			return ret;
>> +		}
>> +	} else {
>> +		dev_dbg(ctrl->device, "%s: qid %d re-using host response\n",
>> +			__func__, chap->qid);
>> +	}
>> +
>> +	ret = crypto_shash_setkey(chap->shash_tfm,
>> +			chap->host_response, ctrl->host_key->len);
>> +	if (ret) {
>> +		dev_warn(ctrl->device, "qid %d: failed to set key, error %d\n",
>> +			 chap->qid, ret);
>> +		goto out;
>> +	}
>> +
>> +	shash->tfm = chap->shash_tfm;
>> +	ret = crypto_shash_init(shash);
>> +	if (ret)
>> +		goto out;
>> +	ret = crypto_shash_update(shash, challenge, chap->hash_len);
>> +	if (ret)
>> +		goto out;
>> +	put_unaligned_le32(chap->s1, buf);
>> +	ret = crypto_shash_update(shash, buf, 4);
>> +	if (ret)
>> +		goto out;
>> +	put_unaligned_le16(chap->transaction, buf);
>> +	ret = crypto_shash_update(shash, buf, 2);
>> +	if (ret)
>> +		goto out;
>> +	memset(buf, 0, sizeof(buf));
>> +	ret = crypto_shash_update(shash, buf, 1);
>> +	if (ret)
>> +		goto out;
>> +	ret = crypto_shash_update(shash, "HostHost", 8);
>> +	if (ret)
>> +		goto out;
>> +	ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
>> +				  strlen(ctrl->opts->host->nqn));
>> +	if (ret)
>> +		goto out;
>> +	ret = crypto_shash_update(shash, buf, 1);
>> +	if (ret)
>> +		goto out;
>> +	ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
>> +			    strlen(ctrl->opts->subsysnqn));
>> +	if (ret)
>> +		goto out;
>> +	ret = crypto_shash_final(shash, chap->response);
>> +out:
>> +	if (challenge != chap->c1)
>> +		kfree(challenge);
>> +	return ret;
>> +}
>> +
>> +static int nvme_auth_dhchap_setup_ctrl_response(struct nvme_ctrl *ctrl,
>> +		struct nvme_dhchap_queue_context *chap)
>> +{
>> +	SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
>> +	u8 *ctrl_response;
>> +	u8 buf[4], *challenge = chap->c2;
>> +	int ret;
>> +
>> +	ctrl_response = nvme_auth_transform_key(ctrl->ctrl_key,
>> +				ctrl->opts->subsysnqn);
>> +	if (IS_ERR(ctrl_response)) {
>> +		ret = PTR_ERR(ctrl_response);
>> +		return ret;
>> +	}
>> +	ret = crypto_shash_setkey(chap->shash_tfm,
>> +			ctrl_response, ctrl->ctrl_key->len);
>> +	if (ret) {
>> +		dev_warn(ctrl->device, "qid %d: failed to set key, error %d\n",
>> +			 chap->qid, ret);
>> +		goto out;
>> +	}
>> +
>> +	dev_dbg(ctrl->device, "%s: qid %d ctrl response seq %u transaction %d\n",
>> +		__func__, chap->qid, chap->s2, chap->transaction);
>> +	dev_dbg(ctrl->device, "%s: qid %d challenge %*ph\n",
>> +		__func__, chap->qid, (int)chap->hash_len, challenge);
>> +	dev_dbg(ctrl->device, "%s: qid %d subsysnqn %s\n",
>> +		__func__, chap->qid, ctrl->opts->subsysnqn);
>> +	dev_dbg(ctrl->device, "%s: qid %d hostnqn %s\n",
>> +		__func__, chap->qid, ctrl->opts->host->nqn);
>> +	shash->tfm = chap->shash_tfm;
>> +	ret = crypto_shash_init(shash);
>> +	if (ret)
>> +		goto out;
>> +	ret = crypto_shash_update(shash, challenge, chap->hash_len);
>> +	if (ret)
>> +		goto out;
>> +	put_unaligned_le32(chap->s2, buf);
>> +	ret = crypto_shash_update(shash, buf, 4);
>> +	if (ret)
>> +		goto out;
>> +	put_unaligned_le16(chap->transaction, buf);
>> +	ret = crypto_shash_update(shash, buf, 2);
>> +	if (ret)
>> +		goto out;
>> +	memset(buf, 0, 4);
>> +	ret = crypto_shash_update(shash, buf, 1);
>> +	if (ret)
>> +		goto out;
>> +	ret = crypto_shash_update(shash, "Controller", 10);
>> +	if (ret)
>> +		goto out;
>> +	ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
>> +				  strlen(ctrl->opts->subsysnqn));
>> +	if (ret)
>> +		goto out;
>> +	ret = crypto_shash_update(shash, buf, 1);
>> +	if (ret)
>> +		goto out;
>> +	ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
>> +				  strlen(ctrl->opts->host->nqn));
>> +	if (ret)
>> +		goto out;
>> +	ret = crypto_shash_final(shash, chap->response);
>> +out:
>> +	if (challenge != chap->c2)
>> +		kfree(challenge);
>> +	return ret;
>> +}
>> +
>> +int nvme_auth_generate_key(u8 *secret, struct nvme_dhchap_key **ret_key)
>> +{
>> +	struct nvme_dhchap_key *key;
>> +	u8 key_hash;
>> +
>> +	if (!secret) {
>> +		*ret_key = NULL;
>> +		return 0;
>> +	}
>> +
>> +	if (sscanf(secret, "DHHC-1:%hhd:%*s:", &key_hash) != 1)
>> +		return -EINVAL;
>> +
>> +	/* Pass in the secret without the 'DHHC-1:XX:' prefix */
>> +	key = nvme_auth_extract_key(secret + 10, key_hash);
>> +	if (IS_ERR(key)) {
>> +		return PTR_ERR(key);
>> +	}
>> +
>> +	*ret_key = key;
>> +	return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(nvme_auth_generate_key);
>> +
>> +static void __nvme_auth_reset(struct nvme_dhchap_queue_context *chap)
>> +{
>> +	chap->status = 0;
>> +	chap->error = 0;
>> +	chap->s1 = 0;
>> +	chap->s2 = 0;
>> +	chap->transaction = 0;
>> +	memset(chap->c1, 0, sizeof(chap->c1));
>> +	memset(chap->c2, 0, sizeof(chap->c2));
>> +}
>> +
>> +static void __nvme_auth_free(struct nvme_dhchap_queue_context *chap)
>> +{
>> +	__nvme_auth_reset(chap);
>> +	if (chap->shash_tfm)
>> +		crypto_free_shash(chap->shash_tfm);
>> +	kfree_sensitive(chap->host_response);
>> +	kfree(chap->buf);
>> +	kfree(chap);
>> +}
>> +
>> +static void __nvme_auth_work(struct work_struct *work)
>> +{
>> +	struct nvme_dhchap_queue_context *chap =
>> +		container_of(work, struct nvme_dhchap_queue_context, auth_work);
>> +	struct nvme_ctrl *ctrl = chap->ctrl;
>> +	size_t tl;
>> +	int ret = 0;
>> +
>> +	chap->transaction = ctrl->transaction++;
>> +
>> +	/* DH-HMAC-CHAP Step 1: send negotiate */
>> +	dev_dbg(ctrl->device, "%s: qid %d send negotiate\n",
>> +		__func__, chap->qid);
>> +	ret = nvme_auth_set_dhchap_negotiate_data(ctrl, chap);
>> +	if (ret < 0) {
>> +		chap->error = ret;
>> +		return;
>> +	}
>> +	tl = ret;
>> +	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, tl, true);
>> +	if (ret) {
>> +		chap->error = ret;
>> +		return;
>> +	}
>> +
>> +	/* DH-HMAC-CHAP Step 2: receive challenge */
>> +	dev_dbg(ctrl->device, "%s: qid %d receive challenge\n",
>> +		__func__, chap->qid);
>> +
>> +	memset(chap->buf, 0, chap->buf_size);
>> +	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, chap->buf_size, false);
>> +	if (ret) {
>> +		dev_warn(ctrl->device,
>> +			 "qid %d failed to receive challenge, %s %d\n",
>> +			 chap->qid, ret < 0 ? "error" : "nvme status", ret);
>> +		chap->error = ret;
>> +		return;
>> +	}
>> +	ret = nvme_auth_receive_validate(ctrl, chap->qid, chap->buf, chap->transaction,
>> +					 NVME_AUTH_DHCHAP_MESSAGE_CHALLENGE);
>> +	if (ret) {
>> +		chap->status = ret;
>> +		chap->error = NVME_SC_AUTH_REQUIRED;
>> +		return;
>> +	}
>> +
>> +	ret = nvme_auth_process_dhchap_challenge(ctrl, chap);
>> +	if (ret) {
>> +		/* Invalid challenge parameters */
>> +		goto fail2;
>> +	}
>> +
>> +	dev_dbg(ctrl->device, "%s: qid %d host response\n",
>> +		__func__, chap->qid);
>> +	ret = nvme_auth_dhchap_setup_host_response(ctrl, chap);
>> +	if (ret)
>> +		goto fail2;
>> +
>> +	/* DH-HMAC-CHAP Step 3: send reply */
>> +	dev_dbg(ctrl->device, "%s: qid %d send reply\n",
>> +		__func__, chap->qid);
>> +	ret = nvme_auth_set_dhchap_reply_data(ctrl, chap);
>> +	if (ret < 0)
>> +		goto fail2;
>> +
>> +	tl = ret;
>> +	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, tl, true);
>> +	if (ret)
>> +		goto fail2;
>> +
>> +	/* DH-HMAC-CHAP Step 4: receive success1 */
>> +	dev_dbg(ctrl->device, "%s: qid %d receive success1\n",
>> +		__func__, chap->qid);
>> +
>> +	memset(chap->buf, 0, chap->buf_size);
>> +	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, chap->buf_size, false);
>> +	if (ret) {
>> +		dev_warn(ctrl->device,
>> +			 "qid %d failed to receive success1, %s %d\n",
>> +			 chap->qid, ret < 0 ? "error" : "nvme status", ret);
>> +		chap->error = ret;
>> +		return;
>> +	}
>> +	ret = nvme_auth_receive_validate(ctrl, chap->qid,
>> +					 chap->buf, chap->transaction,
>> +					 NVME_AUTH_DHCHAP_MESSAGE_SUCCESS1);
>> +	if (ret) {
>> +		chap->status = ret;
>> +		chap->error = NVME_SC_AUTH_REQUIRED;
>> +		return;
>> +	}
>> +
>> +	if (ctrl->opts->dhchap_ctrl_secret) {
>> +		dev_dbg(ctrl->device,
>> +			"%s: qid %d controller response\n",
>> +			__func__, chap->qid);
>> +		ret = nvme_auth_dhchap_setup_ctrl_response(ctrl, chap);
>> +		if (ret) {
>> +			chap->error = ret;
>> +			goto fail2;
>> +		}
>> +	}
>> +
>> +	ret = nvme_auth_process_dhchap_success1(ctrl, chap);
>> +	if (ret) {
>> +		/* Controller authentication failed */
>> +		chap->error = NVME_SC_AUTH_REQUIRED;
>> +		goto fail2;
>> +	}
>> +
>> +	/* DH-HMAC-CHAP Step 5: send success2 */
>> +	dev_dbg(ctrl->device, "%s: qid %d send success2\n",
>> +		__func__, chap->qid);
>> +	tl = nvme_auth_set_dhchap_success2_data(ctrl, chap);
> 
> Host should send DH-HMAC-CHAP_Success2 message only if it requested
> bidirectional authentication and that authentication succeeded.
> 
Indeed. Will be fixing it in the next submission.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer

Re: [PATCH 06/11] nvme: Implement In-Band authentication

[Nayak, Prashanth]

Hi Hannes

> On Mar 28, 2022, at 9:39 AM, Hannes Reinecke <hare@suse.de> wrote:
> 
> Implement NVMe-oF In-Band authentication according to NVMe TPAR 8006.
> This patch adds two new fabric options 'dhchap_secret' to specify the
> pre-shared key (in ASCII respresentation according to NVMe 2.0 section
> 8.13.5.8 'Secret representation') and 'dhchap_ctrl_secret' to specify
> the pre-shared controller key for bi-directional authentication of both
> the host and the controller.
> Re-authentication can be triggered by writing the PSK into the new
> controller sysfs attribute 'dhchap_secret' or 'dhchap_ctrl_secret'.
> 
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
> drivers/nvme/host/Kconfig   |   11 +
> drivers/nvme/host/Makefile  |    1 +
> drivers/nvme/host/auth.c    | 1123 +++++++++++++++++++++++++++++++++++
> drivers/nvme/host/auth.h    |   32 +
> drivers/nvme/host/core.c    |  141 ++++-
> drivers/nvme/host/fabrics.c |   79 ++-
> drivers/nvme/host/fabrics.h |    7 +
> drivers/nvme/host/nvme.h    |   31 +
> drivers/nvme/host/rdma.c    |    1 +
> drivers/nvme/host/tcp.c     |    1 +
> drivers/nvme/host/trace.c   |   32 +
> 11 files changed, 1452 insertions(+), 7 deletions(-)
> create mode 100644 drivers/nvme/host/auth.c
> create mode 100644 drivers/nvme/host/auth.h
> 
> diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig
> index d6d056963c06..dd0e91fb0615 100644
> --- a/drivers/nvme/host/Kconfig
> +++ b/drivers/nvme/host/Kconfig
> @@ -91,3 +91,14 @@ config NVME_TCP
> 	  from https://github.com/linux-nvme/nvme-cli.
> 
> 	  If unsure, say N.
> +
> +config NVME_AUTH
> +	bool "NVM Express over Fabrics In-Band Authentication"
> +	depends on NVME_CORE
> +	select CRYPTO_HMAC
> +	select CRYPTO_SHA256
> +	select CRYPTO_SHA512
> +	help
> +	  This provides support for NVMe over Fabrics In-Band Authentication.
> +
> +	  If unsure, say N.
> diff --git a/drivers/nvme/host/Makefile b/drivers/nvme/host/Makefile
> index 476c5c988496..7755f5e3b281 100644
> --- a/drivers/nvme/host/Makefile
> +++ b/drivers/nvme/host/Makefile
> @@ -15,6 +15,7 @@ nvme-core-$(CONFIG_NVME_MULTIPATH)	+= multipath.o
> nvme-core-$(CONFIG_BLK_DEV_ZONED)	+= zns.o
> nvme-core-$(CONFIG_FAULT_INJECTION_DEBUG_FS)	+= fault_inject.o
> nvme-core-$(CONFIG_NVME_HWMON)		+= hwmon.o
> +nvme-core-$(CONFIG_NVME_AUTH)		+= auth.o
> 
> nvme-y					+= pci.o
> 
> diff --git a/drivers/nvme/host/auth.c b/drivers/nvme/host/auth.c
> new file mode 100644
> index 000000000000..f15754d1bf45
> --- /dev/null
> +++ b/drivers/nvme/host/auth.c
> @@ -0,0 +1,1123 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Copyright (c) 2020 Hannes Reinecke, SUSE Linux
> + */
> +
> +#include <linux/crc32.h>
> +#include <linux/base64.h>
> +#include <linux/prandom.h>
> +#include <asm/unaligned.h>
> +#include <crypto/hash.h>
> +#include <crypto/dh.h>
> +#include "nvme.h"
> +#include "fabrics.h"
> +#include "auth.h"
> +
> +static u32 nvme_dhchap_seqnum;
> +static DEFINE_MUTEX(nvme_dhchap_mutex);
> +
> +struct nvme_dhchap_queue_context {
> +	struct list_head entry;
> +	struct work_struct auth_work;
> +	struct nvme_ctrl *ctrl;
> +	struct crypto_shash *shash_tfm;
> +	void *buf;
> +	size_t buf_size;
> +	int qid;
> +	int error;
> +	u32 s1;
> +	u32 s2;
> +	u16 transaction;
> +	u8 status;
> +	u8 hash_id;
> +	size_t hash_len;
> +	u8 dhgroup_id;
> +	u8 c1[64];
> +	u8 c2[64];
> +	u8 response[64];
> +	u8 *host_response;
> +};
> +
> +u32 nvme_auth_get_seqnum(void)
> +{
> +	u32 seqnum;
> +
> +	mutex_lock(&nvme_dhchap_mutex);
> +	if (!nvme_dhchap_seqnum)
> +		nvme_dhchap_seqnum = prandom_u32();
> +	else {
> +		nvme_dhchap_seqnum++;
> +		if (!nvme_dhchap_seqnum)
> +			nvme_dhchap_seqnum++;
> +	}
> +	seqnum = nvme_dhchap_seqnum;
> +	mutex_unlock(&nvme_dhchap_mutex);
> +	return seqnum;
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_get_seqnum);
> +
> +static struct nvme_auth_dhgroup_map {
> +	const char name[16];
> +	const char kpp[16];
> +} dhgroup_map[] = {
> +	[NVME_AUTH_DHGROUP_NULL] = {
> +		.name = "null", .kpp = "null" },
> +	[NVME_AUTH_DHGROUP_2048] = {
> +		.name = "ffdhe2048", .kpp = "ffdhe2048(dh)" },
> +	[NVME_AUTH_DHGROUP_3072] = {
> +		.name = "ffdhe3072", .kpp = "ffdhe3072(dh)" },
> +	[NVME_AUTH_DHGROUP_4096] = {
> +		.name = "ffdhe4096", .kpp = "ffdhe4096(dh)" },
> +	[NVME_AUTH_DHGROUP_6144] = {
> +		.name = "ffdhe6144", .kpp = "ffdhe6144(dh)" },
> +	[NVME_AUTH_DHGROUP_8192] = {
> +		.name = "ffdhe8192", .kpp = "ffdhe8192(dh)" },
> +};
> +
> +const char *nvme_auth_dhgroup_name(u8 dhgroup_id)
> +{
> +	if ((dhgroup_id > ARRAY_SIZE(dhgroup_map)) ||
> +	    !dhgroup_map[dhgroup_id].name ||
> +	    !strlen(dhgroup_map[dhgroup_id].name))
> +		return NULL;
> +	return dhgroup_map[dhgroup_id].name;
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_dhgroup_name);
> +
> +const char *nvme_auth_dhgroup_kpp(u8 dhgroup_id)
> +{
> +	if ((dhgroup_id > ARRAY_SIZE(dhgroup_map)) ||
> +	    !dhgroup_map[dhgroup_id].kpp ||
> +	    !strlen(dhgroup_map[dhgroup_id].kpp))
> +		return NULL;
> +	return dhgroup_map[dhgroup_id].kpp;
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_dhgroup_kpp);
> +
> +u8 nvme_auth_dhgroup_id(const char *dhgroup_name)
> +{
> +	int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(dhgroup_map); i++) {
> +		if (!dhgroup_map[i].name ||
> +		    !strlen(dhgroup_map[i].name))
> +			continue;
> +		if (!strncmp(dhgroup_map[i].name, dhgroup_name,
> +			     strlen(dhgroup_map[i].name)))
> +			return i;
> +	}
> +	return NVME_AUTH_DHGROUP_INVALID;
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_dhgroup_id);
> +
> +static struct nvme_dhchap_hash_map {
> +	int len;
> +	const char hmac[15];
> +	const char digest[8];
> +} hash_map[] = {
> +	[NVME_AUTH_HASH_SHA256] = {
> +		.len = 32,
> +		.hmac = "hmac(sha256)",
> +		.digest = "sha256",
> +	},
> +	[NVME_AUTH_HASH_SHA384] = {
> +		.len = 48,
> +		.hmac = "hmac(sha384)",
> +		.digest = "sha384",
> +	},
> +	[NVME_AUTH_HASH_SHA512] = {
> +		.len = 64,
> +		.hmac = "hmac(sha512)",
> +		.digest = "sha512",
> +	},
> +};
> +
> +const char *nvme_auth_hmac_name(u8 hmac_id)
> +{
> +	if ((hmac_id > ARRAY_SIZE(hash_map)) ||
> +	    !hash_map[hmac_id].hmac ||
> +	    !strlen(hash_map[hmac_id].hmac))
> +		return NULL;
> +	return hash_map[hmac_id].hmac;
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_hmac_name);
> +
> +const char *nvme_auth_digest_name(u8 hmac_id)
> +{
> +	if ((hmac_id > ARRAY_SIZE(hash_map)) ||
> +	    !hash_map[hmac_id].digest ||
> +	    !strlen(hash_map[hmac_id].digest))
> +		return NULL;
> +	return hash_map[hmac_id].digest;
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_digest_name);
> +
> +u8 nvme_auth_hmac_id(const char *hmac_name)
> +{
> +	int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
> +		if (!hash_map[i].hmac || !strlen(hash_map[i].hmac))
> +			continue;
> +		if (!strncmp(hash_map[i].hmac, hmac_name,
> +			     strlen(hash_map[i].hmac)))
> +			return i;
> +	}
> +	return NVME_AUTH_HASH_INVALID;
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_hmac_id);
> +
> +size_t nvme_auth_hmac_hash_len(u8 hmac_id)
> +{
> +	if ((hmac_id > ARRAY_SIZE(hash_map)) ||
> +	    !hash_map[hmac_id].hmac ||
> +	    !strlen(hash_map[hmac_id].hmac))
> +		return 0;
> +	return hash_map[hmac_id].len;
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_hmac_hash_len);
> +
> +struct nvme_dhchap_key *nvme_auth_extract_key(unsigned char *secret,
> +					      u8 key_hash)
> +{
> +	struct nvme_dhchap_key *key;
> +	unsigned char *p;
> +	u32 crc;
> +	int ret, key_len;
> +	size_t allocated_len = strlen(secret);
> +
> +	/* Secret might be affixed with a ':' */
> +	p = strrchr(secret, ':');
> +	if (p)
> +		allocated_len = p - secret;
> +	key = kzalloc(sizeof(*key), GFP_KERNEL);
> +	if (!key)
> +		return ERR_PTR(-ENOMEM);
> +	key->key = kzalloc(allocated_len, GFP_KERNEL);
> +	if (!key->key) {
> +		ret = -ENOMEM;
> +		goto out_free_key;
> +	}
> +
> +	key_len = base64_decode(secret, allocated_len, key->key);
> +	if (key_len < 0) {
> +		pr_debug("base64 key decoding error %d\n",
> +			 key_len);
> +		ret = key_len;
> +		goto out_free_secret;
> +	}
> +
> +	if (key_len != 36 && key_len != 52 &&
> +	    key_len != 68) {
> +		pr_err("Invalid DH-HMAC-CHAP key len %d\n",
> +		       key_len);
> +		ret = -EINVAL;
> +		goto out_free_secret;
> +	}
> +
> +	if (key_hash > 0 &&
> +	    (key_len - 4) != nvme_auth_hmac_hash_len(key_hash)) {
> +		pr_err("Invalid DH-HMAC-CHAP key len %d for %s\n", key_len,
> +		       nvme_auth_hmac_name(key_hash));
> +		ret = -EINVAL;
> +		goto out_free_secret;
> +	}
> +
> +	/* The last four bytes is the CRC in little-endian format */
> +	key_len -= 4;
> +	/*
> +	 * The linux implementation doesn't do pre- and post-increments,
> +	 * so we have to do it manually.
> +	 */
> +	crc = ~crc32(~0, key->key, key_len);
> +
> +	if (get_unaligned_le32(key->key + key_len) != crc) {
> +		pr_err("DH-HMAC-CHAP key crc mismatch (key %08x, crc %08x)\n",
> +		       get_unaligned_le32(key->key + key_len), crc);
> +		ret = -EKEYREJECTED;
> +		goto out_free_secret;
> +	}
> +	key->len = key_len;
> +	key->hash = key_hash;
> +	return key;
> +out_free_secret:
> +	kfree_sensitive(key->key);
> +out_free_key:
> +	kfree(key);
> +	return ERR_PTR(ret);
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_extract_key);
> +
> +void nvme_auth_free_key(struct nvme_dhchap_key *key)
> +{
> +	if (!key)
> +		return;
> +	kfree_sensitive(key->key);
> +	kfree(key);
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_free_key);
> +
> +u8 *nvme_auth_transform_key(struct nvme_dhchap_key *key, char *nqn)
> +{
> +	const char *hmac_name = nvme_auth_hmac_name(key->hash);
> +	struct crypto_shash *key_tfm;
> +	struct shash_desc *shash;
> +	u8 *transformed_key;
> +	int ret;
> +
> +	if (key->hash == 0) {
> +		transformed_key = kmemdup(key->key, key->len, GFP_KERNEL);
> +		return transformed_key ? transformed_key : ERR_PTR(-ENOMEM);
> +	}
> +
> +	if (!key || !key->key) {
> +		pr_warn("No key specified\n");
> +		return ERR_PTR(-ENOKEY);
> +	}
> +	if (!hmac_name) {
> +		pr_warn("Invalid key hash id %d\n", key->hash);
> +		return ERR_PTR(-EINVAL);
> +	}
> +
> +	key_tfm = crypto_alloc_shash(hmac_name, 0, 0);
> +	if (IS_ERR(key_tfm))
> +		return (u8 *)key_tfm;
> +
> +	shash = kmalloc(sizeof(struct shash_desc) +
> +			crypto_shash_descsize(key_tfm),
> +			GFP_KERNEL);
> +	if (!shash) {
> +		ret = -ENOMEM;
> +		goto out_free_key;
> +	}
> +
> +	transformed_key = kzalloc(crypto_shash_digestsize(key_tfm), GFP_KERNEL);
> +	if (!transformed_key) {
> +		ret = -ENOMEM;
> +		goto out_free_shash;
> +	}
> +
> +	shash->tfm = key_tfm;
> +	ret = crypto_shash_setkey(key_tfm, key->key, key->len);
> +	if (ret < 0)
> +		goto out_free_shash;
> +	ret = crypto_shash_init(shash);
> +	if (ret < 0)
> +		goto out_free_shash;
> +	ret = crypto_shash_update(shash, nqn, strlen(nqn));
> +	if (ret < 0)
> +		goto out_free_shash;
> +	ret = crypto_shash_update(shash, "NVMe-over-Fabrics", 17);
> +	if (ret < 0)
> +		goto out_free_shash;
> +	ret = crypto_shash_final(shash, transformed_key);
> +out_free_shash:
> +	kfree(shash);
> +out_free_key:
> +	crypto_free_shash(key_tfm);
> +	if (ret < 0) {
> +		kfree_sensitive(transformed_key);
> +		return ERR_PTR(ret);
> +	}
> +	return transformed_key;
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_transform_key);
> +
> +#define nvme_auth_flags_from_qid(qid) \
> +	(qid == 0) ? 0 : BLK_MQ_REQ_NOWAIT | BLK_MQ_REQ_RESERVED
> +#define nvme_auth_queue_from_qid(ctrl, qid) \
> +	(qid == 0) ? (ctrl)->fabrics_q : (ctrl)->connect_q
> +
> +static int nvme_auth_submit(struct nvme_ctrl *ctrl, int qid,
> +			    void *data, size_t data_len, bool auth_send)
> +{
> +	struct nvme_command cmd = {};
> +	blk_mq_req_flags_t flags = nvme_auth_flags_from_qid(qid);
> +	struct request_queue *q = nvme_auth_queue_from_qid(ctrl, qid);
> +	int ret;
> +
> +	cmd.auth_common.opcode = nvme_fabrics_command;
> +	cmd.auth_common.secp = NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER;
> +	cmd.auth_common.spsp0 = 0x01;
> +	cmd.auth_common.spsp1 = 0x01;
> +	if (auth_send) {
> +		cmd.auth_send.fctype = nvme_fabrics_type_auth_send;
> +		cmd.auth_send.tl = cpu_to_le32(data_len);
> +	} else {
> +		cmd.auth_receive.fctype = nvme_fabrics_type_auth_receive;
> +		cmd.auth_receive.al = cpu_to_le32(data_len);
> +	}
> +
> +	ret = __nvme_submit_sync_cmd(q, &cmd, NULL, data, data_len, 0,
> +				     qid == 0 ? NVME_QID_ANY : qid,
> +				     0, flags);
> +	if (ret > 0)
> +		dev_warn(ctrl->device,
> +			"qid %d auth_send failed with status %d\n", qid, ret);
> +	else if (ret < 0)
> +		dev_err(ctrl->device,
> +			"qid %d auth_send failed with error %d\n", qid, ret);
> +	return ret;
> +}
> +
> +static int nvme_auth_receive_validate(struct nvme_ctrl *ctrl, int qid,
> +		struct nvmf_auth_dhchap_failure_data *data,
> +		u16 transaction, u8 expected_msg)
> +{
> +	dev_dbg(ctrl->device, "%s: qid %d auth_type %d auth_id %x\n",
> +		__func__, qid, data->auth_type, data->auth_id);
> +
> +	if (data->auth_type == NVME_AUTH_COMMON_MESSAGES &&
> +	    data->auth_id == NVME_AUTH_DHCHAP_MESSAGE_FAILURE1) {
> +		return data->rescode_exp;
> +	}
> +	if (data->auth_type != NVME_AUTH_DHCHAP_MESSAGES ||
> +	    data->auth_id != expected_msg) {
> +		dev_warn(ctrl->device,
> +			 "qid %d invalid message %02x/%02x\n",
> +			 qid, data->auth_type, data->auth_id);
> +		return NVME_AUTH_DHCHAP_FAILURE_INCORRECT_MESSAGE;
> +	}
> +	if (le16_to_cpu(data->t_id) != transaction) {
> +		dev_warn(ctrl->device,
> +			 "qid %d invalid transaction ID %d\n",
> +			 qid, le16_to_cpu(data->t_id));
> +		return NVME_AUTH_DHCHAP_FAILURE_INCORRECT_MESSAGE;
> +	}
> +	return 0;
> +}
> +
> +static int nvme_auth_set_dhchap_negotiate_data(struct nvme_ctrl *ctrl,
> +		struct nvme_dhchap_queue_context *chap)
> +{
> +	struct nvmf_auth_dhchap_negotiate_data *data = chap->buf;
> +	size_t size = sizeof(*data) + sizeof(union nvmf_auth_protocol);
> +
> +	if (chap->buf_size < size) {
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
> +		return -EINVAL;
> +	}
> +	memset((u8 *)chap->buf, 0, size);
> +	data->auth_type = NVME_AUTH_COMMON_MESSAGES;
> +	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE;
> +	data->t_id = cpu_to_le16(chap->transaction);
> +	data->sc_c = 0; /* No secure channel concatenation */
> +	data->napd = 1;
> +	data->auth_protocol[0].dhchap.authid = NVME_AUTH_DHCHAP_AUTH_ID;
> +	data->auth_protocol[0].dhchap.halen = 3;
> +	data->auth_protocol[0].dhchap.dhlen = 6;
> +	data->auth_protocol[0].dhchap.idlist[0] = NVME_AUTH_HASH_SHA256;
> +	data->auth_protocol[0].dhchap.idlist[1] = NVME_AUTH_HASH_SHA384;
> +	data->auth_protocol[0].dhchap.idlist[2] = NVME_AUTH_HASH_SHA512;
> +	data->auth_protocol[0].dhchap.idlist[30] = NVME_AUTH_DHGROUP_NULL;
> +	data->auth_protocol[0].dhchap.idlist[31] = NVME_AUTH_DHGROUP_2048;
> +	data->auth_protocol[0].dhchap.idlist[32] = NVME_AUTH_DHGROUP_3072;
> +	data->auth_protocol[0].dhchap.idlist[33] = NVME_AUTH_DHGROUP_4096;
> +	data->auth_protocol[0].dhchap.idlist[34] = NVME_AUTH_DHGROUP_6144;
> +	data->auth_protocol[0].dhchap.idlist[35] = NVME_AUTH_DHGROUP_8192;
> +
> +	return size;
> +}
> +
> +static int nvme_auth_process_dhchap_challenge(struct nvme_ctrl *ctrl,
> +		struct nvme_dhchap_queue_context *chap)
> +{
> +	struct nvmf_auth_dhchap_challenge_data *data = chap->buf;
> +	u16 dhvlen = le16_to_cpu(data->dhvlen);
> +	size_t size = sizeof(*data) + data->hl + dhvlen;
> +	const char *hmac_name, *kpp_name;
> +
> +	if (chap->buf_size < size) {
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
> +		return NVME_SC_INVALID_FIELD;
> +	}
> +
> +	hmac_name = nvme_auth_hmac_name(data->hashid);
> +	if (!hmac_name) {
> +		dev_warn(ctrl->device,
> +			 "qid %d: invalid HASH ID %d\n",
> +			 chap->qid, data->hashid);
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
> +		return NVME_SC_INVALID_FIELD;
> +	}
> +
> +	if (chap->hash_id == data->hashid && chap->shash_tfm &&
> +	    !strcmp(crypto_shash_alg_name(chap->shash_tfm), hmac_name) &&
> +	    crypto_shash_digestsize(chap->shash_tfm) == data->hl) {
> +		dev_dbg(ctrl->device,
> +			"qid %d: reuse existing hash %s\n",
> +			chap->qid, hmac_name);
> +		goto select_kpp;
> +	}
> +
> +	/* Reset if hash cannot be reused */
> +	if (chap->shash_tfm) {
> +		crypto_free_shash(chap->shash_tfm);
> +		chap->hash_id = 0;
> +		chap->hash_len = 0;
> +	}
> +	chap->shash_tfm = crypto_alloc_shash(hmac_name, 0,
> +					     CRYPTO_ALG_ALLOCATES_MEMORY);
> +	if (IS_ERR(chap->shash_tfm)) {
> +		dev_warn(ctrl->device,
> +			 "qid %d: failed to allocate hash %s, error %ld\n",
> +			 chap->qid, hmac_name, PTR_ERR(chap->shash_tfm));
> +		chap->shash_tfm = NULL;
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_FAILED;
> +		return NVME_SC_AUTH_REQUIRED;
> +	}
> +
> +	if (crypto_shash_digestsize(chap->shash_tfm) != data->hl) {
> +		dev_warn(ctrl->device,
> +			 "qid %d: invalid hash length %d\n",
> +			 chap->qid, data->hl);
> +		crypto_free_shash(chap->shash_tfm);
> +		chap->shash_tfm = NULL;
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
> +		return NVME_SC_AUTH_REQUIRED;
> +	}
> +
> +	/* Reset host response if the hash had been changed */
> +	if (chap->hash_id != data->hashid) {
> +		kfree(chap->host_response);
> +		chap->host_response = NULL;
> +	}
> +
> +	chap->hash_id = data->hashid;
> +	chap->hash_len = data->hl;
> +	dev_dbg(ctrl->device, "qid %d: selected hash %s\n",
> +		chap->qid, hmac_name);
> +
> +select_kpp:
> +	kpp_name = nvme_auth_dhgroup_kpp(data->dhgid);
> +	if (!kpp_name) {
> +		dev_warn(ctrl->device,
> +			 "qid %d: invalid DH group id %d\n",
> +			 chap->qid, data->dhgid);
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
> +		return NVME_SC_AUTH_REQUIRED;
> +	}
> +
> +	if (data->dhgid != NVME_AUTH_DHGROUP_NULL) {
> +		dev_warn(ctrl->device,
> +			 "qid %d: unsupported DH group %s\n",
> +			 chap->qid, kpp_name);
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
> +		return NVME_SC_AUTH_REQUIRED;
> +	} else if (dhvlen != 0) {
> +		dev_warn(ctrl->device,
> +			 "qid %d: invalid DH value for NULL DH\n",
> +			 chap->qid);
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
> +		return NVME_SC_INVALID_FIELD;
> +	}
> +	chap->dhgroup_id = data->dhgid;
> +
> +	chap->s1 = le32_to_cpu(data->seqnum);
> +	memcpy(chap->c1, data->cval, chap->hash_len);
> +
> +	return 0;
> +}
> +
> +static int nvme_auth_set_dhchap_reply_data(struct nvme_ctrl *ctrl,
> +		struct nvme_dhchap_queue_context *chap)
> +{
> +	struct nvmf_auth_dhchap_reply_data *data = chap->buf;
> +	size_t size = sizeof(*data);
> +
> +	size += 2 * chap->hash_len;
> +
> +	if (chap->buf_size < size) {
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
> +		return -EINVAL;
> +	}
> +
> +	memset(chap->buf, 0, size);
> +	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
> +	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_REPLY;
> +	data->t_id = cpu_to_le16(chap->transaction);
> +	data->hl = chap->hash_len;
> +	data->dhvlen = 0;
> +	memcpy(data->rval, chap->response, chap->hash_len);
> +	if (ctrl->opts->dhchap_ctrl_secret) {
> +		get_random_bytes(chap->c2, chap->hash_len);
> +		data->cvalid = 1;
> +		chap->s2 = nvme_auth_get_seqnum();
> +		memcpy(data->rval + chap->hash_len, chap->c2,
> +		       chap->hash_len);
> +		dev_dbg(ctrl->device, "%s: qid %d ctrl challenge %*ph\n",
> +			__func__, chap->qid, (int)chap->hash_len, chap->c2);
> +	} else {
> +		memset(chap->c2, 0, chap->hash_len);
> +		chap->s2 = 0;
> +	}
> +	data->seqnum = cpu_to_le32(chap->s2);
> +	return size;
> +}
> +
> +static int nvme_auth_process_dhchap_success1(struct nvme_ctrl *ctrl,
> +		struct nvme_dhchap_queue_context *chap)
> +{
> +	struct nvmf_auth_dhchap_success1_data *data = chap->buf;
> +	size_t size = sizeof(*data);
> +
> +	if (ctrl->opts->dhchap_ctrl_secret)
> +		size += chap->hash_len;
> +
> +	if (chap->buf_size < size) {
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
> +		return NVME_SC_INVALID_FIELD;
> +	}
> +
> +	if (data->hl != chap->hash_len) {
> +		dev_warn(ctrl->device,
> +			 "qid %d: invalid hash length %u\n",
> +			 chap->qid, data->hl);
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
> +		return NVME_SC_INVALID_FIELD;
> +	}
> +
> +	/* Just print out information for the admin queue */
> +	if (chap->qid == 0)
> +		dev_info(ctrl->device,
> +			 "qid 0: authenticated with hash %s dhgroup %s\n",
> +			 nvme_auth_hmac_name(chap->hash_id),
> +			 nvme_auth_dhgroup_name(chap->dhgroup_id));
> +
> +	if (!data->rvalid)
> +		return 0;
> +
> +	/* Validate controller response */
> +	if (memcmp(chap->response, data->rval, data->hl)) {
> +		dev_dbg(ctrl->device, "%s: qid %d ctrl response %*ph\n",
> +			__func__, chap->qid, (int)chap->hash_len, data->rval);
> +		dev_dbg(ctrl->device, "%s: qid %d host response %*ph\n",
> +			__func__, chap->qid, (int)chap->hash_len,
> +			chap->response);
> +		dev_warn(ctrl->device,
> +			 "qid %d: controller authentication failed\n",
> +			 chap->qid);
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_FAILED;
> +		return NVME_SC_AUTH_REQUIRED;
> +	}
> +
> +	/* Just print out information for the admin queue */
> +	if (chap->qid == 0)
> +		dev_info(ctrl->device,
> +			 "qid 0: controller authenticated\n");
> +	return 0;
> +}
> +
> +static int nvme_auth_set_dhchap_success2_data(struct nvme_ctrl *ctrl,
> +		struct nvme_dhchap_queue_context *chap)
> +{
> +	struct nvmf_auth_dhchap_success2_data *data = chap->buf;
> +	size_t size = sizeof(*data);
> +
> +	memset(chap->buf, 0, size);
> +	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
> +	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_SUCCESS2;
> +	data->t_id = cpu_to_le16(chap->transaction);
> +
> +	return size;
> +}
> +
> +static int nvme_auth_set_dhchap_failure2_data(struct nvme_ctrl *ctrl,
> +		struct nvme_dhchap_queue_context *chap)
> +{
> +	struct nvmf_auth_dhchap_failure_data *data = chap->buf;
> +	size_t size = sizeof(*data);
> +
> +	memset(chap->buf, 0, size);
> +	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;

Auth_type should be NVME_AUTH_COMMON_MESSAGES. 

> +	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_FAILURE2;
> +	data->t_id = cpu_to_le16(chap->transaction);
> +	data->rescode = NVME_AUTH_DHCHAP_FAILURE_REASON_FAILED;
> +	data->rescode_exp = chap->status;
> +
> +	return size;
> +}
> +
> +static int nvme_auth_dhchap_setup_host_response(struct nvme_ctrl *ctrl,
> +		struct nvme_dhchap_queue_context *chap)
> +{
> +	SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
> +	u8 buf[4], *challenge = chap->c1;
> +	int ret;
> +
> +	dev_dbg(ctrl->device, "%s: qid %d host response seq %u transaction %d\n",
> +		__func__, chap->qid, chap->s1, chap->transaction);
> +
> +	if (!chap->host_response) {
> +		chap->host_response = nvme_auth_transform_key(ctrl->host_key,
> +						ctrl->opts->host->nqn);
> +		if (IS_ERR(chap->host_response)) {
> +			ret = PTR_ERR(chap->host_response);
> +			chap->host_response = NULL;
> +			return ret;
> +		}
> +	} else {
> +		dev_dbg(ctrl->device, "%s: qid %d re-using host response\n",
> +			__func__, chap->qid);
> +	}
> +
> +	ret = crypto_shash_setkey(chap->shash_tfm,
> +			chap->host_response, ctrl->host_key->len);
> +	if (ret) {
> +		dev_warn(ctrl->device, "qid %d: failed to set key, error %d\n",
> +			 chap->qid, ret);
> +		goto out;
> +	}
> +
> +	shash->tfm = chap->shash_tfm;
> +	ret = crypto_shash_init(shash);
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, challenge, chap->hash_len);
> +	if (ret)
> +		goto out;
> +	put_unaligned_le32(chap->s1, buf);
> +	ret = crypto_shash_update(shash, buf, 4);
> +	if (ret)
> +		goto out;
> +	put_unaligned_le16(chap->transaction, buf);
> +	ret = crypto_shash_update(shash, buf, 2);
> +	if (ret)
> +		goto out;
> +	memset(buf, 0, sizeof(buf));
> +	ret = crypto_shash_update(shash, buf, 1);
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, "HostHost", 8);
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
> +				  strlen(ctrl->opts->host->nqn));
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, buf, 1);
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
> +			    strlen(ctrl->opts->subsysnqn));
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_final(shash, chap->response);
> +out:
> +	if (challenge != chap->c1)
> +		kfree(challenge);
> +	return ret;
> +}
> +
> +static int nvme_auth_dhchap_setup_ctrl_response(struct nvme_ctrl *ctrl,
> +		struct nvme_dhchap_queue_context *chap)
> +{
> +	SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
> +	u8 *ctrl_response;
> +	u8 buf[4], *challenge = chap->c2;
> +	int ret;
> +
> +	ctrl_response = nvme_auth_transform_key(ctrl->ctrl_key,
> +				ctrl->opts->subsysnqn);
> +	if (IS_ERR(ctrl_response)) {
> +		ret = PTR_ERR(ctrl_response);
> +		return ret;
> +	}
> +	ret = crypto_shash_setkey(chap->shash_tfm,
> +			ctrl_response, ctrl->ctrl_key->len);
> +	if (ret) {
> +		dev_warn(ctrl->device, "qid %d: failed to set key, error %d\n",
> +			 chap->qid, ret);
> +		goto out;
> +	}
> +
> +	dev_dbg(ctrl->device, "%s: qid %d ctrl response seq %u transaction %d\n",
> +		__func__, chap->qid, chap->s2, chap->transaction);
> +	dev_dbg(ctrl->device, "%s: qid %d challenge %*ph\n",
> +		__func__, chap->qid, (int)chap->hash_len, challenge);
> +	dev_dbg(ctrl->device, "%s: qid %d subsysnqn %s\n",
> +		__func__, chap->qid, ctrl->opts->subsysnqn);
> +	dev_dbg(ctrl->device, "%s: qid %d hostnqn %s\n",
> +		__func__, chap->qid, ctrl->opts->host->nqn);
> +	shash->tfm = chap->shash_tfm;
> +	ret = crypto_shash_init(shash);
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, challenge, chap->hash_len);
> +	if (ret)
> +		goto out;
> +	put_unaligned_le32(chap->s2, buf);
> +	ret = crypto_shash_update(shash, buf, 4);
> +	if (ret)
> +		goto out;
> +	put_unaligned_le16(chap->transaction, buf);
> +	ret = crypto_shash_update(shash, buf, 2);
> +	if (ret)
> +		goto out;
> +	memset(buf, 0, 4);
> +	ret = crypto_shash_update(shash, buf, 1);
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, "Controller", 10);
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
> +				  strlen(ctrl->opts->subsysnqn));
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, buf, 1);
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
> +				  strlen(ctrl->opts->host->nqn));
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_final(shash, chap->response);
> +out:
> +	if (challenge != chap->c2)
> +		kfree(challenge);
> +	return ret;
> +}
> +
> +int nvme_auth_generate_key(u8 *secret, struct nvme_dhchap_key **ret_key)
> +{
> +	struct nvme_dhchap_key *key;
> +	u8 key_hash;
> +
> +	if (!secret) {
> +		*ret_key = NULL;
> +		return 0;
> +	}
> +
> +	if (sscanf(secret, "DHHC-1:%hhd:%*s:", &key_hash) != 1)
> +		return -EINVAL;
> +
> +	/* Pass in the secret without the 'DHHC-1:XX:' prefix */
> +	key = nvme_auth_extract_key(secret + 10, key_hash);
> +	if (IS_ERR(key)) {
> +		return PTR_ERR(key);
> +	}
> +
> +	*ret_key = key;
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_generate_key);
> +
> +static void __nvme_auth_reset(struct nvme_dhchap_queue_context *chap)
> +{
> +	chap->status = 0;
> +	chap->error = 0;
> +	chap->s1 = 0;
> +	chap->s2 = 0;
> +	chap->transaction = 0;
> +	memset(chap->c1, 0, sizeof(chap->c1));
> +	memset(chap->c2, 0, sizeof(chap->c2));
> +}
> +
> +static void __nvme_auth_free(struct nvme_dhchap_queue_context *chap)
> +{
> +	__nvme_auth_reset(chap);
> +	if (chap->shash_tfm)
> +		crypto_free_shash(chap->shash_tfm);
> +	kfree_sensitive(chap->host_response);
> +	kfree(chap->buf);
> +	kfree(chap);
> +}
> +
> +static void __nvme_auth_work(struct work_struct *work)
> +{
> +	struct nvme_dhchap_queue_context *chap =
> +		container_of(work, struct nvme_dhchap_queue_context, auth_work);
> +	struct nvme_ctrl *ctrl = chap->ctrl;
> +	size_t tl;
> +	int ret = 0;
> +
> +	chap->transaction = ctrl->transaction++;
> +
> +	/* DH-HMAC-CHAP Step 1: send negotiate */
> +	dev_dbg(ctrl->device, "%s: qid %d send negotiate\n",
> +		__func__, chap->qid);
> +	ret = nvme_auth_set_dhchap_negotiate_data(ctrl, chap);
> +	if (ret < 0) {
> +		chap->error = ret;
> +		return;
> +	}
> +	tl = ret;
> +	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, tl, true);
> +	if (ret) {
> +		chap->error = ret;
> +		return;
> +	}
> +
> +	/* DH-HMAC-CHAP Step 2: receive challenge */
> +	dev_dbg(ctrl->device, "%s: qid %d receive challenge\n",
> +		__func__, chap->qid);
> +
> +	memset(chap->buf, 0, chap->buf_size);
> +	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, chap->buf_size, false);
> +	if (ret) {
> +		dev_warn(ctrl->device,
> +			 "qid %d failed to receive challenge, %s %d\n",
> +			 chap->qid, ret < 0 ? "error" : "nvme status", ret);
> +		chap->error = ret;
> +		return;
> +	}
> +	ret = nvme_auth_receive_validate(ctrl, chap->qid, chap->buf, chap->transaction,
> +					 NVME_AUTH_DHCHAP_MESSAGE_CHALLENGE);
> +	if (ret) {
> +		chap->status = ret;
> +		chap->error = NVME_SC_AUTH_REQUIRED;
> +		return;
> +	}
> +
> +	ret = nvme_auth_process_dhchap_challenge(ctrl, chap);
> +	if (ret) {
> +		/* Invalid challenge parameters */
> +		goto fail2;
> +	}
> +
> +	dev_dbg(ctrl->device, "%s: qid %d host response\n",
> +		__func__, chap->qid);
> +	ret = nvme_auth_dhchap_setup_host_response(ctrl, chap);
> +	if (ret)
> +		goto fail2;
> +
> +	/* DH-HMAC-CHAP Step 3: send reply */
> +	dev_dbg(ctrl->device, "%s: qid %d send reply\n",
> +		__func__, chap->qid);
> +	ret = nvme_auth_set_dhchap_reply_data(ctrl, chap);
> +	if (ret < 0)
> +		goto fail2;
> +
> +	tl = ret;
> +	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, tl, true);
> +	if (ret)
> +		goto fail2;
> +
> +	/* DH-HMAC-CHAP Step 4: receive success1 */
> +	dev_dbg(ctrl->device, "%s: qid %d receive success1\n",
> +		__func__, chap->qid);
> +
> +	memset(chap->buf, 0, chap->buf_size);
> +	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, chap->buf_size, false);
> +	if (ret) {
> +		dev_warn(ctrl->device,
> +			 "qid %d failed to receive success1, %s %d\n",
> +			 chap->qid, ret < 0 ? "error" : "nvme status", ret);
> +		chap->error = ret;
> +		return;
> +	}
> +	ret = nvme_auth_receive_validate(ctrl, chap->qid,
> +					 chap->buf, chap->transaction,
> +					 NVME_AUTH_DHCHAP_MESSAGE_SUCCESS1);
> +	if (ret) {
> +		chap->status = ret;
> +		chap->error = NVME_SC_AUTH_REQUIRED;
> +		return;
> +	}
> +
> +	if (ctrl->opts->dhchap_ctrl_secret) {
> +		dev_dbg(ctrl->device,
> +			"%s: qid %d controller response\n",
> +			__func__, chap->qid);
> +		ret = nvme_auth_dhchap_setup_ctrl_response(ctrl, chap);
> +		if (ret) {
> +			chap->error = ret;
> +			goto fail2;
> +		}
> +	}
> +
> +	ret = nvme_auth_process_dhchap_success1(ctrl, chap);
> +	if (ret) {
> +		/* Controller authentication failed */
> +		chap->error = NVME_SC_AUTH_REQUIRED;
> +		goto fail2;
> +	}
> +
> +	/* DH-HMAC-CHAP Step 5: send success2 */
> +	dev_dbg(ctrl->device, "%s: qid %d send success2\n",
> +		__func__, chap->qid);
> +	tl = nvme_auth_set_dhchap_success2_data(ctrl, chap);

Host should send DH-HMAC-CHAP_Success2 message only if it requested bidirectional authentication and that authentication succeeded.

> +	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, tl, true);
> +	if (!ret) {
> +		chap->error = 0;
> +		return;
> +	}
> +
> +fail2:
> +	dev_dbg(ctrl->device, "%s: qid %d send failure2, status %x\n",
> +		__func__, chap->qid, chap->status);
> +	tl = nvme_auth_set_dhchap_failure2_data(ctrl, chap);
> +	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, tl, true);
> +	/*
> +	 * only update error if send failure2 failed and no other
> +	 * error had been set during authentication.
> +	 */
> +	if (ret && !chap->error)
> +		chap->error = ret;
> +}
> +
> +int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid)
> +{
> +	struct nvme_dhchap_queue_context *chap;
> +
> +	if (!ctrl->host_key) {
> +		dev_warn(ctrl->device, "qid %d: no key\n", qid);
> +		return -ENOKEY;
> +	}
> +
> +	mutex_lock(&ctrl->dhchap_auth_mutex);
> +	/* Check if the context is already queued */
> +	list_for_each_entry(chap, &ctrl->dhchap_auth_list, entry) {
> +		WARN_ON(!chap->buf);
> +		if (chap->qid == qid) {
> +			dev_dbg(ctrl->device, "qid %d: re-using context\n", qid);
> +			mutex_unlock(&ctrl->dhchap_auth_mutex);
> +			flush_work(&chap->auth_work);
> +			__nvme_auth_reset(chap);
> +			queue_work(nvme_wq, &chap->auth_work);
> +			return 0;
> +		}
> +	}
> +	chap = kzalloc(sizeof(*chap), GFP_KERNEL);
> +	if (!chap) {
> +		mutex_unlock(&ctrl->dhchap_auth_mutex);
> +		return -ENOMEM;
> +	}
> +	chap->qid = (qid == NVME_QID_ANY) ? 0 : qid;
> +	chap->ctrl = ctrl;
> +
> +	/*
> +	 * Allocate a large enough buffer for the entire negotiation:
> +	 * 4k should be enough to ffdhe8192.
> +	 */
> +	chap->buf_size = 4096;
> +	chap->buf = kzalloc(chap->buf_size, GFP_KERNEL);
> +	if (!chap->buf) {
> +		mutex_unlock(&ctrl->dhchap_auth_mutex);
> +		kfree(chap);
> +		return -ENOMEM;
> +	}
> +
> +	INIT_WORK(&chap->auth_work, __nvme_auth_work);
> +	list_add(&chap->entry, &ctrl->dhchap_auth_list);
> +	mutex_unlock(&ctrl->dhchap_auth_mutex);
> +	queue_work(nvme_wq, &chap->auth_work);
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_negotiate);
> +
> +int nvme_auth_wait(struct nvme_ctrl *ctrl, int qid)
> +{
> +	struct nvme_dhchap_queue_context *chap;
> +	int ret;
> +
> +	mutex_lock(&ctrl->dhchap_auth_mutex);
> +	list_for_each_entry(chap, &ctrl->dhchap_auth_list, entry) {
> +		if (chap->qid != qid)
> +			continue;
> +		mutex_unlock(&ctrl->dhchap_auth_mutex);
> +		flush_work(&chap->auth_work);
> +		ret = chap->error;
> +		return ret;
> +	}
> +	mutex_unlock(&ctrl->dhchap_auth_mutex);
> +	return -ENXIO;
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_wait);
> +
> +void nvme_auth_reset(struct nvme_ctrl *ctrl)
> +{
> +	struct nvme_dhchap_queue_context *chap;
> +
> +	mutex_lock(&ctrl->dhchap_auth_mutex);
> +	list_for_each_entry(chap, &ctrl->dhchap_auth_list, entry) {
> +		mutex_unlock(&ctrl->dhchap_auth_mutex);
> +		flush_work(&chap->auth_work);
> +		__nvme_auth_reset(chap);
> +	}
> +	mutex_unlock(&ctrl->dhchap_auth_mutex);
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_reset);
> +
> +static void nvme_dhchap_auth_work(struct work_struct *work)
> +{
> +	struct nvme_ctrl *ctrl =
> +		container_of(work, struct nvme_ctrl, dhchap_auth_work);
> +	int ret, q;
> +
> +	/* Authenticate admin queue first */
> +	ret = nvme_auth_negotiate(ctrl, 0);
> +	if (ret) {
> +		dev_warn(ctrl->device,
> +			 "qid 0: error %d setting up authentication\n", ret);
> +		return;
> +	}
> +	ret = nvme_auth_wait(ctrl, 0);
> +	if (ret) {
> +		dev_warn(ctrl->device,
> +			 "qid 0: authentication failed\n");
> +		return;
> +	}
> +
> +	for (q = 1; q < ctrl->queue_count; q++) {
> +		ret = nvme_auth_negotiate(ctrl, q);
> +		if (ret) {
> +			dev_warn(ctrl->device,
> +				 "qid %d: error %d setting up authentication\n",
> +				 q, ret);
> +			break;
> +		}
> +	}
> +
> +	/*
> +	 * Failure is a soft-state; credentials remain valid until
> +	 * the controller terminates the connection.
> +	 */
> +}
> +
> +void nvme_auth_init_ctrl(struct nvme_ctrl *ctrl)
> +{
> +	INIT_LIST_HEAD(&ctrl->dhchap_auth_list);
> +	INIT_WORK(&ctrl->dhchap_auth_work, nvme_dhchap_auth_work);
> +	mutex_init(&ctrl->dhchap_auth_mutex);
> +	if (!ctrl->opts)
> +		return;
> +	nvme_auth_generate_key(ctrl->opts->dhchap_secret, &ctrl->host_key);
> +	nvme_auth_generate_key(ctrl->opts->dhchap_ctrl_secret, &ctrl->ctrl_key);
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_init_ctrl);
> +
> +void nvme_auth_stop(struct nvme_ctrl *ctrl)
> +{
> +	struct nvme_dhchap_queue_context *chap = NULL, *tmp;
> +
> +	cancel_work_sync(&ctrl->dhchap_auth_work);
> +	mutex_lock(&ctrl->dhchap_auth_mutex);
> +	list_for_each_entry_safe(chap, tmp, &ctrl->dhchap_auth_list, entry)
> +		cancel_work_sync(&chap->auth_work);
> +	mutex_unlock(&ctrl->dhchap_auth_mutex);
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_stop);
> +
> +void nvme_auth_free(struct nvme_ctrl *ctrl)
> +{
> +	struct nvme_dhchap_queue_context *chap = NULL, *tmp;
> +
> +	mutex_lock(&ctrl->dhchap_auth_mutex);
> +	list_for_each_entry_safe(chap, tmp, &ctrl->dhchap_auth_list, entry) {
> +		list_del_init(&chap->entry);
> +		flush_work(&chap->auth_work);
> +		__nvme_auth_free(chap);
> +	}
> +	mutex_unlock(&ctrl->dhchap_auth_mutex);
> +	if (ctrl->host_key) {
> +		nvme_auth_free_key(ctrl->host_key);
> +		ctrl->host_key = NULL;
> +	}
> +	if (ctrl->ctrl_key) {
> +		nvme_auth_free_key(ctrl->ctrl_key);
> +		ctrl->ctrl_key = NULL;
> +	}
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_free);
> diff --git a/drivers/nvme/host/auth.h b/drivers/nvme/host/auth.h
> new file mode 100644
> index 000000000000..73d6ec63a5c8
> --- /dev/null
> +++ b/drivers/nvme/host/auth.h
> @@ -0,0 +1,32 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Copyright (c) 2021 Hannes Reinecke, SUSE Software Solutions
> + */
> +
> +#ifndef _NVME_AUTH_H
> +#define _NVME_AUTH_H
> +
> +#include <crypto/kpp.h>
> +
> +struct nvme_dhchap_key {
> +	u8 *key;
> +	size_t len;
> +	u8 hash;
> +};
> +
> +u32 nvme_auth_get_seqnum(void);
> +const char *nvme_auth_dhgroup_name(u8 dhgroup_id);
> +const char *nvme_auth_dhgroup_kpp(u8 dhgroup_id);
> +u8 nvme_auth_dhgroup_id(const char *dhgroup_name);
> +
> +const char *nvme_auth_hmac_name(u8 hmac_id);
> +const char *nvme_auth_digest_name(u8 hmac_id);
> +size_t nvme_auth_hmac_hash_len(u8 hmac_id);
> +u8 nvme_auth_hmac_id(const char *hmac_name);
> +
> +struct nvme_dhchap_key *nvme_auth_extract_key(unsigned char *secret,
> +					      u8 key_hash);
> +void nvme_auth_free_key(struct nvme_dhchap_key *key);
> +u8 *nvme_auth_transform_key(struct nvme_dhchap_key *key, char *nqn);
> +
> +#endif /* _NVME_AUTH_H */
> diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
> index cd6eac8e3dd6..dc10ba7cfc6c 100644
> --- a/drivers/nvme/host/core.c
> +++ b/drivers/nvme/host/core.c
> @@ -24,6 +24,7 @@
> 
> #include "nvme.h"
> #include "fabrics.h"
> +#include "auth.h"
> 
> #define CREATE_TRACE_POINTS
> #include "trace.h"
> @@ -334,6 +335,7 @@ enum nvme_disposition {
> 	COMPLETE,
> 	RETRY,
> 	FAILOVER,
> +	AUTHENTICATE,
> };
> 
> static inline enum nvme_disposition nvme_decide_disposition(struct request *req)
> @@ -341,6 +343,9 @@ static inline enum nvme_disposition nvme_decide_disposition(struct request *req)
> 	if (likely(nvme_req(req)->status == 0))
> 		return COMPLETE;
> 
> +	if ((nvme_req(req)->status & 0x7ff) == NVME_SC_AUTH_REQUIRED)
> +		return AUTHENTICATE;
> +
> 	if (blk_noretry_request(req) ||
> 	    (nvme_req(req)->status & NVME_SC_DNR) ||
> 	    nvme_req(req)->retries >= nvme_max_retries)
> @@ -379,11 +384,13 @@ static inline void nvme_end_req(struct request *req)
> 
> void nvme_complete_rq(struct request *req)
> {
> +	struct nvme_ctrl *ctrl = nvme_req(req)->ctrl;
> +
> 	trace_nvme_complete_rq(req);
> 	nvme_cleanup_cmd(req);
> 
> -	if (nvme_req(req)->ctrl->kas)
> -		nvme_req(req)->ctrl->comp_seen = true;
> +	if (ctrl->kas)
> +		ctrl->comp_seen = true;
> 
> 	switch (nvme_decide_disposition(req)) {
> 	case COMPLETE:
> @@ -395,6 +402,14 @@ void nvme_complete_rq(struct request *req)
> 	case FAILOVER:
> 		nvme_failover_req(req);
> 		return;
> +	case AUTHENTICATE:
> +#ifdef CONFIG_NVME_AUTH
> +		queue_work(nvme_wq, &ctrl->dhchap_auth_work);
> +		nvme_retry_req(req);
> +#else
> +		nvme_end_req(req);
> +#endif
> +		return;
> 	}
> }
> EXPORT_SYMBOL_GPL(nvme_complete_rq);
> @@ -706,7 +721,9 @@ bool __nvme_check_ready(struct nvme_ctrl *ctrl, struct request *rq,
> 		switch (ctrl->state) {
> 		case NVME_CTRL_CONNECTING:
> 			if (blk_rq_is_passthrough(rq) && nvme_is_fabrics(req->cmd) &&
> -			    req->cmd->fabrics.fctype == nvme_fabrics_type_connect)
> +			    (req->cmd->fabrics.fctype == nvme_fabrics_type_connect ||
> +			     req->cmd->fabrics.fctype == nvme_fabrics_type_auth_send ||
> +			     req->cmd->fabrics.fctype == nvme_fabrics_type_auth_receive))
> 				return true;
> 			break;
> 		default:
> @@ -3563,6 +3580,108 @@ static ssize_t dctype_show(struct device *dev,
> }
> static DEVICE_ATTR_RO(dctype);
> 
> +#ifdef CONFIG_NVME_AUTH
> +static ssize_t nvme_ctrl_dhchap_secret_show(struct device *dev,
> +		struct device_attribute *attr, char *buf)
> +{
> +	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
> +	struct nvmf_ctrl_options *opts = ctrl->opts;
> +
> +	if (!opts->dhchap_secret)
> +		return sysfs_emit(buf, "none\n");
> +	return sysfs_emit(buf, "%s\n", opts->dhchap_secret);
> +}
> +
> +static ssize_t nvme_ctrl_dhchap_secret_store(struct device *dev,
> +		struct device_attribute *attr, const char *buf, size_t count)
> +{
> +	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
> +	struct nvmf_ctrl_options *opts = ctrl->opts;
> +	char *dhchap_secret;
> +
> +	if (!ctrl->opts->dhchap_secret)
> +		return -EINVAL;
> +	if (count < 7)
> +		return -EINVAL;
> +	if (memcmp(buf, "DHHC-1:", 7))
> +		return -EINVAL;
> +
> +	dhchap_secret = kzalloc(count + 1, GFP_KERNEL);
> +	if (!dhchap_secret)
> +		return -ENOMEM;
> +	memcpy(dhchap_secret, buf, count);
> +	nvme_auth_stop(ctrl);
> +	if (strcmp(dhchap_secret, opts->dhchap_secret)) {
> +		int ret;
> +
> +		ret = nvme_auth_generate_key(dhchap_secret, &ctrl->host_key);
> +		if (ret)
> +			return ret;
> +		kfree(opts->dhchap_secret);
> +		opts->dhchap_secret = dhchap_secret;
> +		/* Key has changed; re-authentication with new key */
> +		nvme_auth_reset(ctrl);
> +	}
> +	/* Start re-authentication */
> +	dev_info(ctrl->device, "re-authenticating controller\n");
> +	queue_work(nvme_wq, &ctrl->dhchap_auth_work);
> +
> +	return count;
> +}
> +DEVICE_ATTR(dhchap_secret, S_IRUGO | S_IWUSR,
> +	nvme_ctrl_dhchap_secret_show, nvme_ctrl_dhchap_secret_store);
> +
> +static ssize_t nvme_ctrl_dhchap_ctrl_secret_show(struct device *dev,
> +		struct device_attribute *attr, char *buf)
> +{
> +	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
> +	struct nvmf_ctrl_options *opts = ctrl->opts;
> +
> +	if (!opts->dhchap_ctrl_secret)
> +		return sysfs_emit(buf, "none\n");
> +	return sysfs_emit(buf, "%s\n", opts->dhchap_ctrl_secret);
> +}
> +
> +static ssize_t nvme_ctrl_dhchap_ctrl_secret_store(struct device *dev,
> +		struct device_attribute *attr, const char *buf, size_t count)
> +{
> +	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
> +	struct nvmf_ctrl_options *opts = ctrl->opts;
> +	char *dhchap_secret;
> +
> +	if (!ctrl->opts->dhchap_ctrl_secret)
> +		return -EINVAL;
> +	if (count < 7)
> +		return -EINVAL;
> +	if (memcmp(buf, "DHHC-1:", 7))
> +		return -EINVAL;
> +
> +	dhchap_secret = kzalloc(count + 1, GFP_KERNEL);
> +	if (!dhchap_secret)
> +		return -ENOMEM;
> +	memcpy(dhchap_secret, buf, count);
> +	nvme_auth_stop(ctrl);
> +	if (strcmp(dhchap_secret, opts->dhchap_ctrl_secret)) {
> +		int ret;
> +
> +		ret = nvme_auth_generate_key(dhchap_secret, &ctrl->ctrl_key);
> +		if (ret)
> +			return ret;
> +		kfree(opts->dhchap_ctrl_secret);
> +		opts->dhchap_ctrl_secret = dhchap_secret;
> +		/* Key has changed; re-authentication with new key */
> +		nvme_auth_reset(ctrl);
> +	}
> +	/* Start re-authentication */
> +	dev_info(ctrl->device, "re-authenticating controller\n");
> +	queue_work(nvme_wq, &ctrl->dhchap_auth_work);
> +
> +	return count;
> +}
> +DEVICE_ATTR(dhchap_ctrl_secret, S_IRUGO | S_IWUSR,
> +	nvme_ctrl_dhchap_ctrl_secret_show, nvme_ctrl_dhchap_ctrl_secret_store);
> +#endif
> +
> static struct attribute *nvme_dev_attrs[] = {
> 	&dev_attr_reset_controller.attr,
> 	&dev_attr_rescan_controller.attr,
> @@ -3586,6 +3705,10 @@ static struct attribute *nvme_dev_attrs[] = {
> 	&dev_attr_kato.attr,
> 	&dev_attr_cntrltype.attr,
> 	&dev_attr_dctype.attr,
> +#ifdef CONFIG_NVME_AUTH
> +	&dev_attr_dhchap_secret.attr,
> +	&dev_attr_dhchap_ctrl_secret.attr,
> +#endif
> 	NULL
> };
> 
> @@ -3609,6 +3732,10 @@ static umode_t nvme_dev_attrs_are_visible(struct kobject *kobj,
> 		return 0;
> 	if (a == &dev_attr_fast_io_fail_tmo.attr && !ctrl->opts)
> 		return 0;
> +#ifdef CONFIG_NVME_AUTH
> +	if (a == &dev_attr_dhchap_secret.attr && !ctrl->opts)
> +		return 0;
> +#endif
> 
> 	return a->mode;
> }
> @@ -4450,8 +4577,10 @@ static void nvme_handle_aen_notice(struct nvme_ctrl *ctrl, u32 result)
> 		 * recovery actions from interfering with the controller's
> 		 * firmware activation.
> 		 */
> -		if (nvme_change_ctrl_state(ctrl, NVME_CTRL_RESETTING))
> +		if (nvme_change_ctrl_state(ctrl, NVME_CTRL_RESETTING)) {
> +			nvme_auth_stop(ctrl);
> 			queue_work(nvme_wq, &ctrl->fw_act_work);
> +		}
> 		break;
> #ifdef CONFIG_NVME_MULTIPATH
> 	case NVME_AER_NOTICE_ANA:
> @@ -4498,6 +4627,7 @@ EXPORT_SYMBOL_GPL(nvme_complete_async_event);
> void nvme_stop_ctrl(struct nvme_ctrl *ctrl)
> {
> 	nvme_mpath_stop(ctrl);
> +	nvme_auth_stop(ctrl);
> 	nvme_stop_keep_alive(ctrl);
> 	nvme_stop_failfast_work(ctrl);
> 	flush_work(&ctrl->async_event_work);
> @@ -4554,6 +4684,8 @@ static void nvme_free_ctrl(struct device *dev)
> 
> 	nvme_free_cels(ctrl);
> 	nvme_mpath_uninit(ctrl);
> +	nvme_auth_stop(ctrl);
> +	nvme_auth_free(ctrl);
> 	__free_page(ctrl->discard_page);
> 
> 	if (subsys) {
> @@ -4644,6 +4776,7 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev,
> 
> 	nvme_fault_inject_init(&ctrl->fault_inject, dev_name(ctrl->device));
> 	nvme_mpath_init_ctrl(ctrl);
> +	nvme_auth_init_ctrl(ctrl);
> 
> 	return 0;
> out_free_name:
> diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
> index 953e3076aab1..f5786a60931c 100644
> --- a/drivers/nvme/host/fabrics.c
> +++ b/drivers/nvme/host/fabrics.c
> @@ -369,6 +369,7 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)
> 	union nvme_result res;
> 	struct nvmf_connect_data *data;
> 	int ret;
> +	u32 result;
> 
> 	cmd.connect.opcode = nvme_fabrics_command;
> 	cmd.connect.fctype = nvme_fabrics_type_connect;
> @@ -401,8 +402,25 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)
> 		goto out_free_data;
> 	}
> 
> -	ctrl->cntlid = le16_to_cpu(res.u16);
> -
> +	result = le32_to_cpu(res.u32);
> +	ctrl->cntlid = result & 0xFFFF;
> +	if ((result >> 16) & 2) {
> +		/* Authentication required */
> +		ret = nvme_auth_negotiate(ctrl, 0);
> +		if (ret) {
> +			dev_warn(ctrl->device,
> +				 "qid 0: authentication setup failed\n");
> +			ret = NVME_SC_AUTH_REQUIRED;
> +			goto out_free_data;
> +		}
> +		ret = nvme_auth_wait(ctrl, 0);
> +		if (ret)
> +			dev_warn(ctrl->device,
> +				 "qid 0: authentication failed\n");
> +		else
> +			dev_info(ctrl->device,
> +				 "qid 0: authenticated\n");
> +	}
> out_free_data:
> 	kfree(data);
> 	return ret;
> @@ -435,6 +453,7 @@ int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, u16 qid)
> 	struct nvmf_connect_data *data;
> 	union nvme_result res;
> 	int ret;
> +	u32 result;
> 
> 	cmd.connect.opcode = nvme_fabrics_command;
> 	cmd.connect.fctype = nvme_fabrics_type_connect;
> @@ -460,6 +479,21 @@ int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, u16 qid)
> 		nvmf_log_connect_error(ctrl, ret, le32_to_cpu(res.u32),
> 				       &cmd, data);
> 	}
> +	result = le32_to_cpu(res.u32);
> +	if ((result >> 16) & 2) {
> +		/* Authentication required */
> +		ret = nvme_auth_negotiate(ctrl, qid);
> +		if (ret) {
> +			dev_warn(ctrl->device,
> +				 "qid %d: authentication setup failed\n", qid);
> +			ret = NVME_SC_AUTH_REQUIRED;
> +		} else {
> +			ret = nvme_auth_wait(ctrl, qid);
> +			if (ret)
> +				dev_warn(ctrl->device,
> +					 "qid %u: authentication failed\n", qid);
> +		}
> +	}
> 	kfree(data);
> 	return ret;
> }
> @@ -552,6 +586,8 @@ static const match_table_t opt_tokens = {
> 	{ NVMF_OPT_TOS,			"tos=%d"		},
> 	{ NVMF_OPT_FAIL_FAST_TMO,	"fast_io_fail_tmo=%d"	},
> 	{ NVMF_OPT_DISCOVERY,		"discovery"		},
> +	{ NVMF_OPT_DHCHAP_SECRET,	"dhchap_secret=%s"	},
> +	{ NVMF_OPT_DHCHAP_CTRL_SECRET,	"dhchap_ctrl_secret=%s"	},
> 	{ NVMF_OPT_ERR,			NULL			}
> };
> 
> @@ -833,6 +869,34 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
> 		case NVMF_OPT_DISCOVERY:
> 			opts->discovery_nqn = true;
> 			break;
> +		case NVMF_OPT_DHCHAP_SECRET:
> +			p = match_strdup(args);
> +			if (!p) {
> +				ret = -ENOMEM;
> +				goto out;
> +			}
> +			if (strlen(p) < 11 || strncmp(p, "DHHC-1:", 7)) {
> +				pr_err("Invalid DH-CHAP secret %s\n", p);
> +				ret = -EINVAL;
> +				goto out;
> +			}
> +			kfree(opts->dhchap_secret);
> +			opts->dhchap_secret = p;
> +			break;
> +		case NVMF_OPT_DHCHAP_CTRL_SECRET:
> +			p = match_strdup(args);
> +			if (!p) {
> +				ret = -ENOMEM;
> +				goto out;
> +			}
> +			if (strlen(p) < 11 || strncmp(p, "DHHC-1:", 7)) {
> +				pr_err("Invalid DH-CHAP secret %s\n", p);
> +				ret = -EINVAL;
> +				goto out;
> +			}
> +			kfree(opts->dhchap_ctrl_secret);
> +			opts->dhchap_ctrl_secret = p;
> +			break;
> 		default:
> 			pr_warn("unknown parameter or missing value '%s' in ctrl creation request\n",
> 				p);
> @@ -951,6 +1015,7 @@ void nvmf_free_options(struct nvmf_ctrl_options *opts)
> 	kfree(opts->subsysnqn);
> 	kfree(opts->host_traddr);
> 	kfree(opts->host_iface);
> +	kfree(opts->dhchap_secret);
> 	kfree(opts);
> }
> EXPORT_SYMBOL_GPL(nvmf_free_options);
> @@ -960,7 +1025,8 @@ EXPORT_SYMBOL_GPL(nvmf_free_options);
> 				 NVMF_OPT_KATO | NVMF_OPT_HOSTNQN | \
> 				 NVMF_OPT_HOST_ID | NVMF_OPT_DUP_CONNECT |\
> 				 NVMF_OPT_DISABLE_SQFLOW | NVMF_OPT_DISCOVERY |\
> -				 NVMF_OPT_FAIL_FAST_TMO)
> +				 NVMF_OPT_FAIL_FAST_TMO | NVMF_OPT_DHCHAP_SECRET |\
> +				 NVMF_OPT_DHCHAP_CTRL_SECRET)
> 
> static struct nvme_ctrl *
> nvmf_create_ctrl(struct device *dev, const char *buf)
> @@ -1196,7 +1262,14 @@ static void __exit nvmf_exit(void)
> 	BUILD_BUG_ON(sizeof(struct nvmf_connect_command) != 64);
> 	BUILD_BUG_ON(sizeof(struct nvmf_property_get_command) != 64);
> 	BUILD_BUG_ON(sizeof(struct nvmf_property_set_command) != 64);
> +	BUILD_BUG_ON(sizeof(struct nvmf_auth_send_command) != 64);
> +	BUILD_BUG_ON(sizeof(struct nvmf_auth_receive_command) != 64);
> 	BUILD_BUG_ON(sizeof(struct nvmf_connect_data) != 1024);
> +	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_negotiate_data) != 8);
> +	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_challenge_data) != 16);
> +	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_reply_data) != 16);
> +	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_success1_data) != 16);
> +	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_success2_data) != 16);
> }
> 
> MODULE_LICENSE("GPL v2");
> diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h
> index 1e3a09cad961..15c142b277ab 100644
> --- a/drivers/nvme/host/fabrics.h
> +++ b/drivers/nvme/host/fabrics.h
> @@ -68,6 +68,8 @@ enum {
> 	NVMF_OPT_FAIL_FAST_TMO	= 1 << 20,
> 	NVMF_OPT_HOST_IFACE	= 1 << 21,
> 	NVMF_OPT_DISCOVERY	= 1 << 22,
> +	NVMF_OPT_DHCHAP_SECRET	= 1 << 23,
> +	NVMF_OPT_DHCHAP_CTRL_SECRET = 1 << 24,
> };
> 
> /**
> @@ -97,6 +99,9 @@ enum {
>  * @max_reconnects: maximum number of allowed reconnect attempts before removing
>  *              the controller, (-1) means reconnect forever, zero means remove
>  *              immediately;
> + * @dhchap_secret: DH-HMAC-CHAP secret
> + * @dhchap_ctrl_secret: DH-HMAC-CHAP controller secret for bi-directional
> + *              authentication
>  * @disable_sqflow: disable controller sq flow control
>  * @hdr_digest: generate/verify header digest (TCP)
>  * @data_digest: generate/verify data digest (TCP)
> @@ -121,6 +126,8 @@ struct nvmf_ctrl_options {
> 	unsigned int		kato;
> 	struct nvmf_host	*host;
> 	int			max_reconnects;
> +	char			*dhchap_secret;
> +	char			*dhchap_ctrl_secret;
> 	bool			disable_sqflow;
> 	bool			hdr_digest;
> 	bool			data_digest;
> diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
> index 1ea908d43e17..f110437a3190 100644
> --- a/drivers/nvme/host/nvme.h
> +++ b/drivers/nvme/host/nvme.h
> @@ -323,6 +323,15 @@ struct nvme_ctrl {
> 	struct work_struct ana_work;
> #endif
> 
> +#ifdef CONFIG_NVME_AUTH
> +	struct work_struct dhchap_auth_work;
> +	struct list_head dhchap_auth_list;
> +	struct mutex dhchap_auth_mutex;
> +	struct nvme_dhchap_key *host_key;
> +	struct nvme_dhchap_key *ctrl_key;
> +	u16 transaction;
> +#endif
> +
> 	/* Power saving configuration */
> 	u64 ps_max_latency_us;
> 	bool apst_enabled;
> @@ -928,6 +937,28 @@ static inline bool nvme_ctrl_sgl_supported(struct nvme_ctrl *ctrl)
> 	return ctrl->sgls & ((1 << 0) | (1 << 1));
> }
> 
> +#ifdef CONFIG_NVME_AUTH
> +void nvme_auth_init_ctrl(struct nvme_ctrl *ctrl);
> +void nvme_auth_stop(struct nvme_ctrl *ctrl);
> +int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid);
> +int nvme_auth_wait(struct nvme_ctrl *ctrl, int qid);
> +void nvme_auth_reset(struct nvme_ctrl *ctrl);
> +void nvme_auth_free(struct nvme_ctrl *ctrl);
> +int nvme_auth_generate_key(u8 *secret, struct nvme_dhchap_key **ret_key);
> +#else
> +static inline void nvme_auth_init_ctrl(struct nvme_ctrl *ctrl) {};
> +static inline void nvme_auth_stop(struct nvme_ctrl *ctrl) {};
> +static inline int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid)
> +{
> +	return -EPROTONOSUPPORT;
> +}
> +static inline int nvme_auth_wait(struct nvme_ctrl *ctrl, int qid)
> +{
> +	return NVME_SC_AUTH_REQUIRED;
> +}
> +static inline void nvme_auth_free(struct nvme_ctrl *ctrl) {};
> +#endif
> +
> u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
> 			 u8 opcode);
> int nvme_execute_passthru_rq(struct request *rq);
> diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
> index d9f19d901313..bd63568c0068 100644
> --- a/drivers/nvme/host/rdma.c
> +++ b/drivers/nvme/host/rdma.c
> @@ -1197,6 +1197,7 @@ static void nvme_rdma_error_recovery_work(struct work_struct *work)
> 	struct nvme_rdma_ctrl *ctrl = container_of(work,
> 			struct nvme_rdma_ctrl, err_work);
> 
> +	nvme_auth_stop(&ctrl->ctrl);
> 	nvme_stop_keep_alive(&ctrl->ctrl);
> 	flush_work(&ctrl->ctrl.async_event_work);
> 	nvme_rdma_teardown_io_queues(ctrl, false);
> diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
> index ad3a2bf2f1e9..be52902b5f3b 100644
> --- a/drivers/nvme/host/tcp.c
> +++ b/drivers/nvme/host/tcp.c
> @@ -2174,6 +2174,7 @@ static void nvme_tcp_error_recovery_work(struct work_struct *work)
> 				struct nvme_tcp_ctrl, err_work);
> 	struct nvme_ctrl *ctrl = &tcp_ctrl->ctrl;
> 
> +	nvme_auth_stop(ctrl);
> 	nvme_stop_keep_alive(ctrl);
> 	flush_work(&ctrl->async_event_work);
> 	nvme_tcp_teardown_io_queues(ctrl, false);
> diff --git a/drivers/nvme/host/trace.c b/drivers/nvme/host/trace.c
> index 2a89c5aa0790..1c36fcedea20 100644
> --- a/drivers/nvme/host/trace.c
> +++ b/drivers/nvme/host/trace.c
> @@ -287,6 +287,34 @@ static const char *nvme_trace_fabrics_property_get(struct trace_seq *p, u8 *spc)
> 	return ret;
> }
> 
> +static const char *nvme_trace_fabrics_auth_send(struct trace_seq *p, u8 *spc)
> +{
> +	const char *ret = trace_seq_buffer_ptr(p);
> +	u8 spsp0 = spc[1];
> +	u8 spsp1 = spc[2];
> +	u8 secp = spc[3];
> +	u32 tl = get_unaligned_le32(spc + 4);
> +
> +	trace_seq_printf(p, "spsp0=%02x, spsp1=%02x, secp=%02x, tl=%u",
> +			 spsp0, spsp1, secp, tl);
> +	trace_seq_putc(p, 0);
> +	return ret;
> +}
> +
> +static const char *nvme_trace_fabrics_auth_receive(struct trace_seq *p, u8 *spc)
> +{
> +	const char *ret = trace_seq_buffer_ptr(p);
> +	u8 spsp0 = spc[1];
> +	u8 spsp1 = spc[2];
> +	u8 secp = spc[3];
> +	u32 al = get_unaligned_le32(spc + 4);
> +
> +	trace_seq_printf(p, "spsp0=%02x, spsp1=%02x, secp=%02x, al=%u",
> +			 spsp0, spsp1, secp, al);
> +	trace_seq_putc(p, 0);
> +	return ret;
> +}
> +
> static const char *nvme_trace_fabrics_common(struct trace_seq *p, u8 *spc)
> {
> 	const char *ret = trace_seq_buffer_ptr(p);
> @@ -306,6 +334,10 @@ const char *nvme_trace_parse_fabrics_cmd(struct trace_seq *p,
> 		return nvme_trace_fabrics_connect(p, spc);
> 	case nvme_fabrics_type_property_get:
> 		return nvme_trace_fabrics_property_get(p, spc);
> +	case nvme_fabrics_type_auth_send:
> +		return nvme_trace_fabrics_auth_send(p, spc);
> +	case nvme_fabrics_type_auth_receive:
> +		return nvme_trace_fabrics_auth_receive(p, spc);
> 	default:
> 		return nvme_trace_fabrics_common(p, spc);
> 	}
> -- 
> 2.29.2
> 
> 
> 

Thanks
Prashanth

[PATCH 06/11] nvme: Implement In-Band authentication

[Hannes Reinecke]

Implement NVMe-oF In-Band authentication according to NVMe TPAR 8006.
This patch adds two new fabric options 'dhchap_secret' to specify the
pre-shared key (in ASCII respresentation according to NVMe 2.0 section
8.13.5.8 'Secret representation') and 'dhchap_ctrl_secret' to specify
the pre-shared controller key for bi-directional authentication of both
the host and the controller.
Re-authentication can be triggered by writing the PSK into the new
controller sysfs attribute 'dhchap_secret' or 'dhchap_ctrl_secret'.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/nvme/host/Kconfig   |   11 +
 drivers/nvme/host/Makefile  |    1 +
 drivers/nvme/host/auth.c    | 1123 +++++++++++++++++++++++++++++++++++
 drivers/nvme/host/auth.h    |   32 +
 drivers/nvme/host/core.c    |  141 ++++-
 drivers/nvme/host/fabrics.c |   79 ++-
 drivers/nvme/host/fabrics.h |    7 +
 drivers/nvme/host/nvme.h    |   31 +
 drivers/nvme/host/rdma.c    |    1 +
 drivers/nvme/host/tcp.c     |    1 +
 drivers/nvme/host/trace.c   |   32 +
 11 files changed, 1452 insertions(+), 7 deletions(-)
 create mode 100644 drivers/nvme/host/auth.c
 create mode 100644 drivers/nvme/host/auth.h

diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig
index d6d056963c06..dd0e91fb0615 100644
--- a/drivers/nvme/host/Kconfig
+++ b/drivers/nvme/host/Kconfig
@@ -91,3 +91,14 @@ config NVME_TCP
 	  from https://github.com/linux-nvme/nvme-cli.
 
 	  If unsure, say N.
+
+config NVME_AUTH
+	bool "NVM Express over Fabrics In-Band Authentication"
+	depends on NVME_CORE
+	select CRYPTO_HMAC
+	select CRYPTO_SHA256
+	select CRYPTO_SHA512
+	help
+	  This provides support for NVMe over Fabrics In-Band Authentication.
+
+	  If unsure, say N.
diff --git a/drivers/nvme/host/Makefile b/drivers/nvme/host/Makefile
index 476c5c988496..7755f5e3b281 100644
--- a/drivers/nvme/host/Makefile
+++ b/drivers/nvme/host/Makefile
@@ -15,6 +15,7 @@ nvme-core-$(CONFIG_NVME_MULTIPATH)	+= multipath.o
 nvme-core-$(CONFIG_BLK_DEV_ZONED)	+= zns.o
 nvme-core-$(CONFIG_FAULT_INJECTION_DEBUG_FS)	+= fault_inject.o
 nvme-core-$(CONFIG_NVME_HWMON)		+= hwmon.o
+nvme-core-$(CONFIG_NVME_AUTH)		+= auth.o
 
 nvme-y					+= pci.o
 
diff --git a/drivers/nvme/host/auth.c b/drivers/nvme/host/auth.c
new file mode 100644
index 000000000000..f15754d1bf45
--- /dev/null
+++ b/drivers/nvme/host/auth.c
@@ -0,0 +1,1123 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2020 Hannes Reinecke, SUSE Linux
+ */
+
+#include <linux/crc32.h>
+#include <linux/base64.h>
+#include <linux/prandom.h>
+#include <asm/unaligned.h>
+#include <crypto/hash.h>
+#include <crypto/dh.h>
+#include "nvme.h"
+#include "fabrics.h"
+#include "auth.h"
+
+static u32 nvme_dhchap_seqnum;
+static DEFINE_MUTEX(nvme_dhchap_mutex);
+
+struct nvme_dhchap_queue_context {
+	struct list_head entry;
+	struct work_struct auth_work;
+	struct nvme_ctrl *ctrl;
+	struct crypto_shash *shash_tfm;
+	void *buf;
+	size_t buf_size;
+	int qid;
+	int error;
+	u32 s1;
+	u32 s2;
+	u16 transaction;
+	u8 status;
+	u8 hash_id;
+	size_t hash_len;
+	u8 dhgroup_id;
+	u8 c1[64];
+	u8 c2[64];
+	u8 response[64];
+	u8 *host_response;
+};
+
+u32 nvme_auth_get_seqnum(void)
+{
+	u32 seqnum;
+
+	mutex_lock(&nvme_dhchap_mutex);
+	if (!nvme_dhchap_seqnum)
+		nvme_dhchap_seqnum = prandom_u32();
+	else {
+		nvme_dhchap_seqnum++;
+		if (!nvme_dhchap_seqnum)
+			nvme_dhchap_seqnum++;
+	}
+	seqnum = nvme_dhchap_seqnum;
+	mutex_unlock(&nvme_dhchap_mutex);
+	return seqnum;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_get_seqnum);
+
+static struct nvme_auth_dhgroup_map {
+	const char name[16];
+	const char kpp[16];
+} dhgroup_map[] = {
+	[NVME_AUTH_DHGROUP_NULL] = {
+		.name = "null", .kpp = "null" },
+	[NVME_AUTH_DHGROUP_2048] = {
+		.name = "ffdhe2048", .kpp = "ffdhe2048(dh)" },
+	[NVME_AUTH_DHGROUP_3072] = {
+		.name = "ffdhe3072", .kpp = "ffdhe3072(dh)" },
+	[NVME_AUTH_DHGROUP_4096] = {
+		.name = "ffdhe4096", .kpp = "ffdhe4096(dh)" },
+	[NVME_AUTH_DHGROUP_6144] = {
+		.name = "ffdhe6144", .kpp = "ffdhe6144(dh)" },
+	[NVME_AUTH_DHGROUP_8192] = {
+		.name = "ffdhe8192", .kpp = "ffdhe8192(dh)" },
+};
+
+const char *nvme_auth_dhgroup_name(u8 dhgroup_id)
+{
+	if ((dhgroup_id > ARRAY_SIZE(dhgroup_map)) ||
+	    !dhgroup_map[dhgroup_id].name ||
+	    !strlen(dhgroup_map[dhgroup_id].name))
+		return NULL;
+	return dhgroup_map[dhgroup_id].name;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_dhgroup_name);
+
+const char *nvme_auth_dhgroup_kpp(u8 dhgroup_id)
+{
+	if ((dhgroup_id > ARRAY_SIZE(dhgroup_map)) ||
+	    !dhgroup_map[dhgroup_id].kpp ||
+	    !strlen(dhgroup_map[dhgroup_id].kpp))
+		return NULL;
+	return dhgroup_map[dhgroup_id].kpp;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_dhgroup_kpp);
+
+u8 nvme_auth_dhgroup_id(const char *dhgroup_name)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(dhgroup_map); i++) {
+		if (!dhgroup_map[i].name ||
+		    !strlen(dhgroup_map[i].name))
+			continue;
+		if (!strncmp(dhgroup_map[i].name, dhgroup_name,
+			     strlen(dhgroup_map[i].name)))
+			return i;
+	}
+	return NVME_AUTH_DHGROUP_INVALID;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_dhgroup_id);
+
+static struct nvme_dhchap_hash_map {
+	int len;
+	const char hmac[15];
+	const char digest[8];
+} hash_map[] = {
+	[NVME_AUTH_HASH_SHA256] = {
+		.len = 32,
+		.hmac = "hmac(sha256)",
+		.digest = "sha256",
+	},
+	[NVME_AUTH_HASH_SHA384] = {
+		.len = 48,
+		.hmac = "hmac(sha384)",
+		.digest = "sha384",
+	},
+	[NVME_AUTH_HASH_SHA512] = {
+		.len = 64,
+		.hmac = "hmac(sha512)",
+		.digest = "sha512",
+	},
+};
+
+const char *nvme_auth_hmac_name(u8 hmac_id)
+{
+	if ((hmac_id > ARRAY_SIZE(hash_map)) ||
+	    !hash_map[hmac_id].hmac ||
+	    !strlen(hash_map[hmac_id].hmac))
+		return NULL;
+	return hash_map[hmac_id].hmac;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_hmac_name);
+
+const char *nvme_auth_digest_name(u8 hmac_id)
+{
+	if ((hmac_id > ARRAY_SIZE(hash_map)) ||
+	    !hash_map[hmac_id].digest ||
+	    !strlen(hash_map[hmac_id].digest))
+		return NULL;
+	return hash_map[hmac_id].digest;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_digest_name);
+
+u8 nvme_auth_hmac_id(const char *hmac_name)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
+		if (!hash_map[i].hmac || !strlen(hash_map[i].hmac))
+			continue;
+		if (!strncmp(hash_map[i].hmac, hmac_name,
+			     strlen(hash_map[i].hmac)))
+			return i;
+	}
+	return NVME_AUTH_HASH_INVALID;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_hmac_id);
+
+size_t nvme_auth_hmac_hash_len(u8 hmac_id)
+{
+	if ((hmac_id > ARRAY_SIZE(hash_map)) ||
+	    !hash_map[hmac_id].hmac ||
+	    !strlen(hash_map[hmac_id].hmac))
+		return 0;
+	return hash_map[hmac_id].len;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_hmac_hash_len);
+
+struct nvme_dhchap_key *nvme_auth_extract_key(unsigned char *secret,
+					      u8 key_hash)
+{
+	struct nvme_dhchap_key *key;
+	unsigned char *p;
+	u32 crc;
+	int ret, key_len;
+	size_t allocated_len = strlen(secret);
+
+	/* Secret might be affixed with a ':' */
+	p = strrchr(secret, ':');
+	if (p)
+		allocated_len = p - secret;
+	key = kzalloc(sizeof(*key), GFP_KERNEL);
+	if (!key)
+		return ERR_PTR(-ENOMEM);
+	key->key = kzalloc(allocated_len, GFP_KERNEL);
+	if (!key->key) {
+		ret = -ENOMEM;
+		goto out_free_key;
+	}
+
+	key_len = base64_decode(secret, allocated_len, key->key);
+	if (key_len < 0) {
+		pr_debug("base64 key decoding error %d\n",
+			 key_len);
+		ret = key_len;
+		goto out_free_secret;
+	}
+
+	if (key_len != 36 && key_len != 52 &&
+	    key_len != 68) {
+		pr_err("Invalid DH-HMAC-CHAP key len %d\n",
+		       key_len);
+		ret = -EINVAL;
+		goto out_free_secret;
+	}
+
+	if (key_hash > 0 &&
+	    (key_len - 4) != nvme_auth_hmac_hash_len(key_hash)) {
+		pr_err("Invalid DH-HMAC-CHAP key len %d for %s\n", key_len,
+		       nvme_auth_hmac_name(key_hash));
+		ret = -EINVAL;
+		goto out_free_secret;
+	}
+
+	/* The last four bytes is the CRC in little-endian format */
+	key_len -= 4;
+	/*
+	 * The linux implementation doesn't do pre- and post-increments,
+	 * so we have to do it manually.
+	 */
+	crc = ~crc32(~0, key->key, key_len);
+
+	if (get_unaligned_le32(key->key + key_len) != crc) {
+		pr_err("DH-HMAC-CHAP key crc mismatch (key %08x, crc %08x)\n",
+		       get_unaligned_le32(key->key + key_len), crc);
+		ret = -EKEYREJECTED;
+		goto out_free_secret;
+	}
+	key->len = key_len;
+	key->hash = key_hash;
+	return key;
+out_free_secret:
+	kfree_sensitive(key->key);
+out_free_key:
+	kfree(key);
+	return ERR_PTR(ret);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_extract_key);
+
+void nvme_auth_free_key(struct nvme_dhchap_key *key)
+{
+	if (!key)
+		return;
+	kfree_sensitive(key->key);
+	kfree(key);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_free_key);
+
+u8 *nvme_auth_transform_key(struct nvme_dhchap_key *key, char *nqn)
+{
+	const char *hmac_name = nvme_auth_hmac_name(key->hash);
+	struct crypto_shash *key_tfm;
+	struct shash_desc *shash;
+	u8 *transformed_key;
+	int ret;
+
+	if (key->hash == 0) {
+		transformed_key = kmemdup(key->key, key->len, GFP_KERNEL);
+		return transformed_key ? transformed_key : ERR_PTR(-ENOMEM);
+	}
+
+	if (!key || !key->key) {
+		pr_warn("No key specified\n");
+		return ERR_PTR(-ENOKEY);
+	}
+	if (!hmac_name) {
+		pr_warn("Invalid key hash id %d\n", key->hash);
+		return ERR_PTR(-EINVAL);
+	}
+
+	key_tfm = crypto_alloc_shash(hmac_name, 0, 0);
+	if (IS_ERR(key_tfm))
+		return (u8 *)key_tfm;
+
+	shash = kmalloc(sizeof(struct shash_desc) +
+			crypto_shash_descsize(key_tfm),
+			GFP_KERNEL);
+	if (!shash) {
+		ret = -ENOMEM;
+		goto out_free_key;
+	}
+
+	transformed_key = kzalloc(crypto_shash_digestsize(key_tfm), GFP_KERNEL);
+	if (!transformed_key) {
+		ret = -ENOMEM;
+		goto out_free_shash;
+	}
+
+	shash->tfm = key_tfm;
+	ret = crypto_shash_setkey(key_tfm, key->key, key->len);
+	if (ret < 0)
+		goto out_free_shash;
+	ret = crypto_shash_init(shash);
+	if (ret < 0)
+		goto out_free_shash;
+	ret = crypto_shash_update(shash, nqn, strlen(nqn));
+	if (ret < 0)
+		goto out_free_shash;
+	ret = crypto_shash_update(shash, "NVMe-over-Fabrics", 17);
+	if (ret < 0)
+		goto out_free_shash;
+	ret = crypto_shash_final(shash, transformed_key);
+out_free_shash:
+	kfree(shash);
+out_free_key:
+	crypto_free_shash(key_tfm);
+	if (ret < 0) {
+		kfree_sensitive(transformed_key);
+		return ERR_PTR(ret);
+	}
+	return transformed_key;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_transform_key);
+
+#define nvme_auth_flags_from_qid(qid) \
+	(qid == 0) ? 0 : BLK_MQ_REQ_NOWAIT | BLK_MQ_REQ_RESERVED
+#define nvme_auth_queue_from_qid(ctrl, qid) \
+	(qid == 0) ? (ctrl)->fabrics_q : (ctrl)->connect_q
+
+static int nvme_auth_submit(struct nvme_ctrl *ctrl, int qid,
+			    void *data, size_t data_len, bool auth_send)
+{
+	struct nvme_command cmd = {};
+	blk_mq_req_flags_t flags = nvme_auth_flags_from_qid(qid);
+	struct request_queue *q = nvme_auth_queue_from_qid(ctrl, qid);
+	int ret;
+
+	cmd.auth_common.opcode = nvme_fabrics_command;
+	cmd.auth_common.secp = NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER;
+	cmd.auth_common.spsp0 = 0x01;
+	cmd.auth_common.spsp1 = 0x01;
+	if (auth_send) {
+		cmd.auth_send.fctype = nvme_fabrics_type_auth_send;
+		cmd.auth_send.tl = cpu_to_le32(data_len);
+	} else {
+		cmd.auth_receive.fctype = nvme_fabrics_type_auth_receive;
+		cmd.auth_receive.al = cpu_to_le32(data_len);
+	}
+
+	ret = __nvme_submit_sync_cmd(q, &cmd, NULL, data, data_len, 0,
+				     qid == 0 ? NVME_QID_ANY : qid,
+				     0, flags);
+	if (ret > 0)
+		dev_warn(ctrl->device,
+			"qid %d auth_send failed with status %d\n", qid, ret);
+	else if (ret < 0)
+		dev_err(ctrl->device,
+			"qid %d auth_send failed with error %d\n", qid, ret);
+	return ret;
+}
+
+static int nvme_auth_receive_validate(struct nvme_ctrl *ctrl, int qid,
+		struct nvmf_auth_dhchap_failure_data *data,
+		u16 transaction, u8 expected_msg)
+{
+	dev_dbg(ctrl->device, "%s: qid %d auth_type %d auth_id %x\n",
+		__func__, qid, data->auth_type, data->auth_id);
+
+	if (data->auth_type == NVME_AUTH_COMMON_MESSAGES &&
+	    data->auth_id == NVME_AUTH_DHCHAP_MESSAGE_FAILURE1) {
+		return data->rescode_exp;
+	}
+	if (data->auth_type != NVME_AUTH_DHCHAP_MESSAGES ||
+	    data->auth_id != expected_msg) {
+		dev_warn(ctrl->device,
+			 "qid %d invalid message %02x/%02x\n",
+			 qid, data->auth_type, data->auth_id);
+		return NVME_AUTH_DHCHAP_FAILURE_INCORRECT_MESSAGE;
+	}
+	if (le16_to_cpu(data->t_id) != transaction) {
+		dev_warn(ctrl->device,
+			 "qid %d invalid transaction ID %d\n",
+			 qid, le16_to_cpu(data->t_id));
+		return NVME_AUTH_DHCHAP_FAILURE_INCORRECT_MESSAGE;
+	}
+	return 0;
+}
+
+static int nvme_auth_set_dhchap_negotiate_data(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_negotiate_data *data = chap->buf;
+	size_t size = sizeof(*data) + sizeof(union nvmf_auth_protocol);
+
+	if (chap->buf_size < size) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return -EINVAL;
+	}
+	memset((u8 *)chap->buf, 0, size);
+	data->auth_type = NVME_AUTH_COMMON_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE;
+	data->t_id = cpu_to_le16(chap->transaction);
+	data->sc_c = 0; /* No secure channel concatenation */
+	data->napd = 1;
+	data->auth_protocol[0].dhchap.authid = NVME_AUTH_DHCHAP_AUTH_ID;
+	data->auth_protocol[0].dhchap.halen = 3;
+	data->auth_protocol[0].dhchap.dhlen = 6;
+	data->auth_protocol[0].dhchap.idlist[0] = NVME_AUTH_HASH_SHA256;
+	data->auth_protocol[0].dhchap.idlist[1] = NVME_AUTH_HASH_SHA384;
+	data->auth_protocol[0].dhchap.idlist[2] = NVME_AUTH_HASH_SHA512;
+	data->auth_protocol[0].dhchap.idlist[30] = NVME_AUTH_DHGROUP_NULL;
+	data->auth_protocol[0].dhchap.idlist[31] = NVME_AUTH_DHGROUP_2048;
+	data->auth_protocol[0].dhchap.idlist[32] = NVME_AUTH_DHGROUP_3072;
+	data->auth_protocol[0].dhchap.idlist[33] = NVME_AUTH_DHGROUP_4096;
+	data->auth_protocol[0].dhchap.idlist[34] = NVME_AUTH_DHGROUP_6144;
+	data->auth_protocol[0].dhchap.idlist[35] = NVME_AUTH_DHGROUP_8192;
+
+	return size;
+}
+
+static int nvme_auth_process_dhchap_challenge(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_challenge_data *data = chap->buf;
+	u16 dhvlen = le16_to_cpu(data->dhvlen);
+	size_t size = sizeof(*data) + data->hl + dhvlen;
+	const char *hmac_name, *kpp_name;
+
+	if (chap->buf_size < size) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return NVME_SC_INVALID_FIELD;
+	}
+
+	hmac_name = nvme_auth_hmac_name(data->hashid);
+	if (!hmac_name) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid HASH ID %d\n",
+			 chap->qid, data->hashid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
+		return NVME_SC_INVALID_FIELD;
+	}
+
+	if (chap->hash_id == data->hashid && chap->shash_tfm &&
+	    !strcmp(crypto_shash_alg_name(chap->shash_tfm), hmac_name) &&
+	    crypto_shash_digestsize(chap->shash_tfm) == data->hl) {
+		dev_dbg(ctrl->device,
+			"qid %d: reuse existing hash %s\n",
+			chap->qid, hmac_name);
+		goto select_kpp;
+	}
+
+	/* Reset if hash cannot be reused */
+	if (chap->shash_tfm) {
+		crypto_free_shash(chap->shash_tfm);
+		chap->hash_id = 0;
+		chap->hash_len = 0;
+	}
+	chap->shash_tfm = crypto_alloc_shash(hmac_name, 0,
+					     CRYPTO_ALG_ALLOCATES_MEMORY);
+	if (IS_ERR(chap->shash_tfm)) {
+		dev_warn(ctrl->device,
+			 "qid %d: failed to allocate hash %s, error %ld\n",
+			 chap->qid, hmac_name, PTR_ERR(chap->shash_tfm));
+		chap->shash_tfm = NULL;
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_FAILED;
+		return NVME_SC_AUTH_REQUIRED;
+	}
+
+	if (crypto_shash_digestsize(chap->shash_tfm) != data->hl) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid hash length %d\n",
+			 chap->qid, data->hl);
+		crypto_free_shash(chap->shash_tfm);
+		chap->shash_tfm = NULL;
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
+		return NVME_SC_AUTH_REQUIRED;
+	}
+
+	/* Reset host response if the hash had been changed */
+	if (chap->hash_id != data->hashid) {
+		kfree(chap->host_response);
+		chap->host_response = NULL;
+	}
+
+	chap->hash_id = data->hashid;
+	chap->hash_len = data->hl;
+	dev_dbg(ctrl->device, "qid %d: selected hash %s\n",
+		chap->qid, hmac_name);
+
+select_kpp:
+	kpp_name = nvme_auth_dhgroup_kpp(data->dhgid);
+	if (!kpp_name) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid DH group id %d\n",
+			 chap->qid, data->dhgid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
+		return NVME_SC_AUTH_REQUIRED;
+	}
+
+	if (data->dhgid != NVME_AUTH_DHGROUP_NULL) {
+		dev_warn(ctrl->device,
+			 "qid %d: unsupported DH group %s\n",
+			 chap->qid, kpp_name);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
+		return NVME_SC_AUTH_REQUIRED;
+	} else if (dhvlen != 0) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid DH value for NULL DH\n",
+			 chap->qid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return NVME_SC_INVALID_FIELD;
+	}
+	chap->dhgroup_id = data->dhgid;
+
+	chap->s1 = le32_to_cpu(data->seqnum);
+	memcpy(chap->c1, data->cval, chap->hash_len);
+
+	return 0;
+}
+
+static int nvme_auth_set_dhchap_reply_data(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_reply_data *data = chap->buf;
+	size_t size = sizeof(*data);
+
+	size += 2 * chap->hash_len;
+
+	if (chap->buf_size < size) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return -EINVAL;
+	}
+
+	memset(chap->buf, 0, size);
+	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_REPLY;
+	data->t_id = cpu_to_le16(chap->transaction);
+	data->hl = chap->hash_len;
+	data->dhvlen = 0;
+	memcpy(data->rval, chap->response, chap->hash_len);
+	if (ctrl->opts->dhchap_ctrl_secret) {
+		get_random_bytes(chap->c2, chap->hash_len);
+		data->cvalid = 1;
+		chap->s2 = nvme_auth_get_seqnum();
+		memcpy(data->rval + chap->hash_len, chap->c2,
+		       chap->hash_len);
+		dev_dbg(ctrl->device, "%s: qid %d ctrl challenge %*ph\n",
+			__func__, chap->qid, (int)chap->hash_len, chap->c2);
+	} else {
+		memset(chap->c2, 0, chap->hash_len);
+		chap->s2 = 0;
+	}
+	data->seqnum = cpu_to_le32(chap->s2);
+	return size;
+}
+
+static int nvme_auth_process_dhchap_success1(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_success1_data *data = chap->buf;
+	size_t size = sizeof(*data);
+
+	if (ctrl->opts->dhchap_ctrl_secret)
+		size += chap->hash_len;
+
+	if (chap->buf_size < size) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return NVME_SC_INVALID_FIELD;
+	}
+
+	if (data->hl != chap->hash_len) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid hash length %u\n",
+			 chap->qid, data->hl);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
+		return NVME_SC_INVALID_FIELD;
+	}
+
+	/* Just print out information for the admin queue */
+	if (chap->qid == 0)
+		dev_info(ctrl->device,
+			 "qid 0: authenticated with hash %s dhgroup %s\n",
+			 nvme_auth_hmac_name(chap->hash_id),
+			 nvme_auth_dhgroup_name(chap->dhgroup_id));
+
+	if (!data->rvalid)
+		return 0;
+
+	/* Validate controller response */
+	if (memcmp(chap->response, data->rval, data->hl)) {
+		dev_dbg(ctrl->device, "%s: qid %d ctrl response %*ph\n",
+			__func__, chap->qid, (int)chap->hash_len, data->rval);
+		dev_dbg(ctrl->device, "%s: qid %d host response %*ph\n",
+			__func__, chap->qid, (int)chap->hash_len,
+			chap->response);
+		dev_warn(ctrl->device,
+			 "qid %d: controller authentication failed\n",
+			 chap->qid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_FAILED;
+		return NVME_SC_AUTH_REQUIRED;
+	}
+
+	/* Just print out information for the admin queue */
+	if (chap->qid == 0)
+		dev_info(ctrl->device,
+			 "qid 0: controller authenticated\n");
+	return 0;
+}
+
+static int nvme_auth_set_dhchap_success2_data(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_success2_data *data = chap->buf;
+	size_t size = sizeof(*data);
+
+	memset(chap->buf, 0, size);
+	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_SUCCESS2;
+	data->t_id = cpu_to_le16(chap->transaction);
+
+	return size;
+}
+
+static int nvme_auth_set_dhchap_failure2_data(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_failure_data *data = chap->buf;
+	size_t size = sizeof(*data);
+
+	memset(chap->buf, 0, size);
+	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_FAILURE2;
+	data->t_id = cpu_to_le16(chap->transaction);
+	data->rescode = NVME_AUTH_DHCHAP_FAILURE_REASON_FAILED;
+	data->rescode_exp = chap->status;
+
+	return size;
+}
+
+static int nvme_auth_dhchap_setup_host_response(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
+	u8 buf[4], *challenge = chap->c1;
+	int ret;
+
+	dev_dbg(ctrl->device, "%s: qid %d host response seq %u transaction %d\n",
+		__func__, chap->qid, chap->s1, chap->transaction);
+
+	if (!chap->host_response) {
+		chap->host_response = nvme_auth_transform_key(ctrl->host_key,
+						ctrl->opts->host->nqn);
+		if (IS_ERR(chap->host_response)) {
+			ret = PTR_ERR(chap->host_response);
+			chap->host_response = NULL;
+			return ret;
+		}
+	} else {
+		dev_dbg(ctrl->device, "%s: qid %d re-using host response\n",
+			__func__, chap->qid);
+	}
+
+	ret = crypto_shash_setkey(chap->shash_tfm,
+			chap->host_response, ctrl->host_key->len);
+	if (ret) {
+		dev_warn(ctrl->device, "qid %d: failed to set key, error %d\n",
+			 chap->qid, ret);
+		goto out;
+	}
+
+	shash->tfm = chap->shash_tfm;
+	ret = crypto_shash_init(shash);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, challenge, chap->hash_len);
+	if (ret)
+		goto out;
+	put_unaligned_le32(chap->s1, buf);
+	ret = crypto_shash_update(shash, buf, 4);
+	if (ret)
+		goto out;
+	put_unaligned_le16(chap->transaction, buf);
+	ret = crypto_shash_update(shash, buf, 2);
+	if (ret)
+		goto out;
+	memset(buf, 0, sizeof(buf));
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, "HostHost", 8);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
+				  strlen(ctrl->opts->host->nqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
+			    strlen(ctrl->opts->subsysnqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_final(shash, chap->response);
+out:
+	if (challenge != chap->c1)
+		kfree(challenge);
+	return ret;
+}
+
+static int nvme_auth_dhchap_setup_ctrl_response(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
+	u8 *ctrl_response;
+	u8 buf[4], *challenge = chap->c2;
+	int ret;
+
+	ctrl_response = nvme_auth_transform_key(ctrl->ctrl_key,
+				ctrl->opts->subsysnqn);
+	if (IS_ERR(ctrl_response)) {
+		ret = PTR_ERR(ctrl_response);
+		return ret;
+	}
+	ret = crypto_shash_setkey(chap->shash_tfm,
+			ctrl_response, ctrl->ctrl_key->len);
+	if (ret) {
+		dev_warn(ctrl->device, "qid %d: failed to set key, error %d\n",
+			 chap->qid, ret);
+		goto out;
+	}
+
+	dev_dbg(ctrl->device, "%s: qid %d ctrl response seq %u transaction %d\n",
+		__func__, chap->qid, chap->s2, chap->transaction);
+	dev_dbg(ctrl->device, "%s: qid %d challenge %*ph\n",
+		__func__, chap->qid, (int)chap->hash_len, challenge);
+	dev_dbg(ctrl->device, "%s: qid %d subsysnqn %s\n",
+		__func__, chap->qid, ctrl->opts->subsysnqn);
+	dev_dbg(ctrl->device, "%s: qid %d hostnqn %s\n",
+		__func__, chap->qid, ctrl->opts->host->nqn);
+	shash->tfm = chap->shash_tfm;
+	ret = crypto_shash_init(shash);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, challenge, chap->hash_len);
+	if (ret)
+		goto out;
+	put_unaligned_le32(chap->s2, buf);
+	ret = crypto_shash_update(shash, buf, 4);
+	if (ret)
+		goto out;
+	put_unaligned_le16(chap->transaction, buf);
+	ret = crypto_shash_update(shash, buf, 2);
+	if (ret)
+		goto out;
+	memset(buf, 0, 4);
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, "Controller", 10);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
+				  strlen(ctrl->opts->subsysnqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
+				  strlen(ctrl->opts->host->nqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_final(shash, chap->response);
+out:
+	if (challenge != chap->c2)
+		kfree(challenge);
+	return ret;
+}
+
+int nvme_auth_generate_key(u8 *secret, struct nvme_dhchap_key **ret_key)
+{
+	struct nvme_dhchap_key *key;
+	u8 key_hash;
+
+	if (!secret) {
+		*ret_key = NULL;
+		return 0;
+	}
+
+	if (sscanf(secret, "DHHC-1:%hhd:%*s:", &key_hash) != 1)
+		return -EINVAL;
+
+	/* Pass in the secret without the 'DHHC-1:XX:' prefix */
+	key = nvme_auth_extract_key(secret + 10, key_hash);
+	if (IS_ERR(key)) {
+		return PTR_ERR(key);
+	}
+
+	*ret_key = key;
+	return 0;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_generate_key);
+
+static void __nvme_auth_reset(struct nvme_dhchap_queue_context *chap)
+{
+	chap->status = 0;
+	chap->error = 0;
+	chap->s1 = 0;
+	chap->s2 = 0;
+	chap->transaction = 0;
+	memset(chap->c1, 0, sizeof(chap->c1));
+	memset(chap->c2, 0, sizeof(chap->c2));
+}
+
+static void __nvme_auth_free(struct nvme_dhchap_queue_context *chap)
+{
+	__nvme_auth_reset(chap);
+	if (chap->shash_tfm)
+		crypto_free_shash(chap->shash_tfm);
+	kfree_sensitive(chap->host_response);
+	kfree(chap->buf);
+	kfree(chap);
+}
+
+static void __nvme_auth_work(struct work_struct *work)
+{
+	struct nvme_dhchap_queue_context *chap =
+		container_of(work, struct nvme_dhchap_queue_context, auth_work);
+	struct nvme_ctrl *ctrl = chap->ctrl;
+	size_t tl;
+	int ret = 0;
+
+	chap->transaction = ctrl->transaction++;
+
+	/* DH-HMAC-CHAP Step 1: send negotiate */
+	dev_dbg(ctrl->device, "%s: qid %d send negotiate\n",
+		__func__, chap->qid);
+	ret = nvme_auth_set_dhchap_negotiate_data(ctrl, chap);
+	if (ret < 0) {
+		chap->error = ret;
+		return;
+	}
+	tl = ret;
+	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, tl, true);
+	if (ret) {
+		chap->error = ret;
+		return;
+	}
+
+	/* DH-HMAC-CHAP Step 2: receive challenge */
+	dev_dbg(ctrl->device, "%s: qid %d receive challenge\n",
+		__func__, chap->qid);
+
+	memset(chap->buf, 0, chap->buf_size);
+	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, chap->buf_size, false);
+	if (ret) {
+		dev_warn(ctrl->device,
+			 "qid %d failed to receive challenge, %s %d\n",
+			 chap->qid, ret < 0 ? "error" : "nvme status", ret);
+		chap->error = ret;
+		return;
+	}
+	ret = nvme_auth_receive_validate(ctrl, chap->qid, chap->buf, chap->transaction,
+					 NVME_AUTH_DHCHAP_MESSAGE_CHALLENGE);
+	if (ret) {
+		chap->status = ret;
+		chap->error = NVME_SC_AUTH_REQUIRED;
+		return;
+	}
+
+	ret = nvme_auth_process_dhchap_challenge(ctrl, chap);
+	if (ret) {
+		/* Invalid challenge parameters */
+		goto fail2;
+	}
+
+	dev_dbg(ctrl->device, "%s: qid %d host response\n",
+		__func__, chap->qid);
+	ret = nvme_auth_dhchap_setup_host_response(ctrl, chap);
+	if (ret)
+		goto fail2;
+
+	/* DH-HMAC-CHAP Step 3: send reply */
+	dev_dbg(ctrl->device, "%s: qid %d send reply\n",
+		__func__, chap->qid);
+	ret = nvme_auth_set_dhchap_reply_data(ctrl, chap);
+	if (ret < 0)
+		goto fail2;
+
+	tl = ret;
+	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, tl, true);
+	if (ret)
+		goto fail2;
+
+	/* DH-HMAC-CHAP Step 4: receive success1 */
+	dev_dbg(ctrl->device, "%s: qid %d receive success1\n",
+		__func__, chap->qid);
+
+	memset(chap->buf, 0, chap->buf_size);
+	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, chap->buf_size, false);
+	if (ret) {
+		dev_warn(ctrl->device,
+			 "qid %d failed to receive success1, %s %d\n",
+			 chap->qid, ret < 0 ? "error" : "nvme status", ret);
+		chap->error = ret;
+		return;
+	}
+	ret = nvme_auth_receive_validate(ctrl, chap->qid,
+					 chap->buf, chap->transaction,
+					 NVME_AUTH_DHCHAP_MESSAGE_SUCCESS1);
+	if (ret) {
+		chap->status = ret;
+		chap->error = NVME_SC_AUTH_REQUIRED;
+		return;
+	}
+
+	if (ctrl->opts->dhchap_ctrl_secret) {
+		dev_dbg(ctrl->device,
+			"%s: qid %d controller response\n",
+			__func__, chap->qid);
+		ret = nvme_auth_dhchap_setup_ctrl_response(ctrl, chap);
+		if (ret) {
+			chap->error = ret;
+			goto fail2;
+		}
+	}
+
+	ret = nvme_auth_process_dhchap_success1(ctrl, chap);
+	if (ret) {
+		/* Controller authentication failed */
+		chap->error = NVME_SC_AUTH_REQUIRED;
+		goto fail2;
+	}
+
+	/* DH-HMAC-CHAP Step 5: send success2 */
+	dev_dbg(ctrl->device, "%s: qid %d send success2\n",
+		__func__, chap->qid);
+	tl = nvme_auth_set_dhchap_success2_data(ctrl, chap);
+	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, tl, true);
+	if (!ret) {
+		chap->error = 0;
+		return;
+	}
+
+fail2:
+	dev_dbg(ctrl->device, "%s: qid %d send failure2, status %x\n",
+		__func__, chap->qid, chap->status);
+	tl = nvme_auth_set_dhchap_failure2_data(ctrl, chap);
+	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, tl, true);
+	/*
+	 * only update error if send failure2 failed and no other
+	 * error had been set during authentication.
+	 */
+	if (ret && !chap->error)
+		chap->error = ret;
+}
+
+int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid)
+{
+	struct nvme_dhchap_queue_context *chap;
+
+	if (!ctrl->host_key) {
+		dev_warn(ctrl->device, "qid %d: no key\n", qid);
+		return -ENOKEY;
+	}
+
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	/* Check if the context is already queued */
+	list_for_each_entry(chap, &ctrl->dhchap_auth_list, entry) {
+		WARN_ON(!chap->buf);
+		if (chap->qid == qid) {
+			dev_dbg(ctrl->device, "qid %d: re-using context\n", qid);
+			mutex_unlock(&ctrl->dhchap_auth_mutex);
+			flush_work(&chap->auth_work);
+			__nvme_auth_reset(chap);
+			queue_work(nvme_wq, &chap->auth_work);
+			return 0;
+		}
+	}
+	chap = kzalloc(sizeof(*chap), GFP_KERNEL);
+	if (!chap) {
+		mutex_unlock(&ctrl->dhchap_auth_mutex);
+		return -ENOMEM;
+	}
+	chap->qid = (qid == NVME_QID_ANY) ? 0 : qid;
+	chap->ctrl = ctrl;
+
+	/*
+	 * Allocate a large enough buffer for the entire negotiation:
+	 * 4k should be enough to ffdhe8192.
+	 */
+	chap->buf_size = 4096;
+	chap->buf = kzalloc(chap->buf_size, GFP_KERNEL);
+	if (!chap->buf) {
+		mutex_unlock(&ctrl->dhchap_auth_mutex);
+		kfree(chap);
+		return -ENOMEM;
+	}
+
+	INIT_WORK(&chap->auth_work, __nvme_auth_work);
+	list_add(&chap->entry, &ctrl->dhchap_auth_list);
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+	queue_work(nvme_wq, &chap->auth_work);
+	return 0;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_negotiate);
+
+int nvme_auth_wait(struct nvme_ctrl *ctrl, int qid)
+{
+	struct nvme_dhchap_queue_context *chap;
+	int ret;
+
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	list_for_each_entry(chap, &ctrl->dhchap_auth_list, entry) {
+		if (chap->qid != qid)
+			continue;
+		mutex_unlock(&ctrl->dhchap_auth_mutex);
+		flush_work(&chap->auth_work);
+		ret = chap->error;
+		return ret;
+	}
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+	return -ENXIO;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_wait);
+
+void nvme_auth_reset(struct nvme_ctrl *ctrl)
+{
+	struct nvme_dhchap_queue_context *chap;
+
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	list_for_each_entry(chap, &ctrl->dhchap_auth_list, entry) {
+		mutex_unlock(&ctrl->dhchap_auth_mutex);
+		flush_work(&chap->auth_work);
+		__nvme_auth_reset(chap);
+	}
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_reset);
+
+static void nvme_dhchap_auth_work(struct work_struct *work)
+{
+	struct nvme_ctrl *ctrl =
+		container_of(work, struct nvme_ctrl, dhchap_auth_work);
+	int ret, q;
+
+	/* Authenticate admin queue first */
+	ret = nvme_auth_negotiate(ctrl, 0);
+	if (ret) {
+		dev_warn(ctrl->device,
+			 "qid 0: error %d setting up authentication\n", ret);
+		return;
+	}
+	ret = nvme_auth_wait(ctrl, 0);
+	if (ret) {
+		dev_warn(ctrl->device,
+			 "qid 0: authentication failed\n");
+		return;
+	}
+
+	for (q = 1; q < ctrl->queue_count; q++) {
+		ret = nvme_auth_negotiate(ctrl, q);
+		if (ret) {
+			dev_warn(ctrl->device,
+				 "qid %d: error %d setting up authentication\n",
+				 q, ret);
+			break;
+		}
+	}
+
+	/*
+	 * Failure is a soft-state; credentials remain valid until
+	 * the controller terminates the connection.
+	 */
+}
+
+void nvme_auth_init_ctrl(struct nvme_ctrl *ctrl)
+{
+	INIT_LIST_HEAD(&ctrl->dhchap_auth_list);
+	INIT_WORK(&ctrl->dhchap_auth_work, nvme_dhchap_auth_work);
+	mutex_init(&ctrl->dhchap_auth_mutex);
+	if (!ctrl->opts)
+		return;
+	nvme_auth_generate_key(ctrl->opts->dhchap_secret, &ctrl->host_key);
+	nvme_auth_generate_key(ctrl->opts->dhchap_ctrl_secret, &ctrl->ctrl_key);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_init_ctrl);
+
+void nvme_auth_stop(struct nvme_ctrl *ctrl)
+{
+	struct nvme_dhchap_queue_context *chap = NULL, *tmp;
+
+	cancel_work_sync(&ctrl->dhchap_auth_work);
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	list_for_each_entry_safe(chap, tmp, &ctrl->dhchap_auth_list, entry)
+		cancel_work_sync(&chap->auth_work);
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_stop);
+
+void nvme_auth_free(struct nvme_ctrl *ctrl)
+{
+	struct nvme_dhchap_queue_context *chap = NULL, *tmp;
+
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	list_for_each_entry_safe(chap, tmp, &ctrl->dhchap_auth_list, entry) {
+		list_del_init(&chap->entry);
+		flush_work(&chap->auth_work);
+		__nvme_auth_free(chap);
+	}
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+	if (ctrl->host_key) {
+		nvme_auth_free_key(ctrl->host_key);
+		ctrl->host_key = NULL;
+	}
+	if (ctrl->ctrl_key) {
+		nvme_auth_free_key(ctrl->ctrl_key);
+		ctrl->ctrl_key = NULL;
+	}
+}
+EXPORT_SYMBOL_GPL(nvme_auth_free);
diff --git a/drivers/nvme/host/auth.h b/drivers/nvme/host/auth.h
new file mode 100644
index 000000000000..73d6ec63a5c8
--- /dev/null
+++ b/drivers/nvme/host/auth.h
@@ -0,0 +1,32 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2021 Hannes Reinecke, SUSE Software Solutions
+ */
+
+#ifndef _NVME_AUTH_H
+#define _NVME_AUTH_H
+
+#include <crypto/kpp.h>
+
+struct nvme_dhchap_key {
+	u8 *key;
+	size_t len;
+	u8 hash;
+};
+
+u32 nvme_auth_get_seqnum(void);
+const char *nvme_auth_dhgroup_name(u8 dhgroup_id);
+const char *nvme_auth_dhgroup_kpp(u8 dhgroup_id);
+u8 nvme_auth_dhgroup_id(const char *dhgroup_name);
+
+const char *nvme_auth_hmac_name(u8 hmac_id);
+const char *nvme_auth_digest_name(u8 hmac_id);
+size_t nvme_auth_hmac_hash_len(u8 hmac_id);
+u8 nvme_auth_hmac_id(const char *hmac_name);
+
+struct nvme_dhchap_key *nvme_auth_extract_key(unsigned char *secret,
+					      u8 key_hash);
+void nvme_auth_free_key(struct nvme_dhchap_key *key);
+u8 *nvme_auth_transform_key(struct nvme_dhchap_key *key, char *nqn);
+
+#endif /* _NVME_AUTH_H */
diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index cd6eac8e3dd6..dc10ba7cfc6c 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -24,6 +24,7 @@
 
 #include "nvme.h"
 #include "fabrics.h"
+#include "auth.h"
 
 #define CREATE_TRACE_POINTS
 #include "trace.h"
@@ -334,6 +335,7 @@ enum nvme_disposition {
 	COMPLETE,
 	RETRY,
 	FAILOVER,
+	AUTHENTICATE,
 };
 
 static inline enum nvme_disposition nvme_decide_disposition(struct request *req)
@@ -341,6 +343,9 @@ static inline enum nvme_disposition nvme_decide_disposition(struct request *req)
 	if (likely(nvme_req(req)->status == 0))
 		return COMPLETE;
 
+	if ((nvme_req(req)->status & 0x7ff) == NVME_SC_AUTH_REQUIRED)
+		return AUTHENTICATE;
+
 	if (blk_noretry_request(req) ||
 	    (nvme_req(req)->status & NVME_SC_DNR) ||
 	    nvme_req(req)->retries >= nvme_max_retries)
@@ -379,11 +384,13 @@ static inline void nvme_end_req(struct request *req)
 
 void nvme_complete_rq(struct request *req)
 {
+	struct nvme_ctrl *ctrl = nvme_req(req)->ctrl;
+
 	trace_nvme_complete_rq(req);
 	nvme_cleanup_cmd(req);
 
-	if (nvme_req(req)->ctrl->kas)
-		nvme_req(req)->ctrl->comp_seen = true;
+	if (ctrl->kas)
+		ctrl->comp_seen = true;
 
 	switch (nvme_decide_disposition(req)) {
 	case COMPLETE:
@@ -395,6 +402,14 @@ void nvme_complete_rq(struct request *req)
 	case FAILOVER:
 		nvme_failover_req(req);
 		return;
+	case AUTHENTICATE:
+#ifdef CONFIG_NVME_AUTH
+		queue_work(nvme_wq, &ctrl->dhchap_auth_work);
+		nvme_retry_req(req);
+#else
+		nvme_end_req(req);
+#endif
+		return;
 	}
 }
 EXPORT_SYMBOL_GPL(nvme_complete_rq);
@@ -706,7 +721,9 @@ bool __nvme_check_ready(struct nvme_ctrl *ctrl, struct request *rq,
 		switch (ctrl->state) {
 		case NVME_CTRL_CONNECTING:
 			if (blk_rq_is_passthrough(rq) && nvme_is_fabrics(req->cmd) &&
-			    req->cmd->fabrics.fctype == nvme_fabrics_type_connect)
+			    (req->cmd->fabrics.fctype == nvme_fabrics_type_connect ||
+			     req->cmd->fabrics.fctype == nvme_fabrics_type_auth_send ||
+			     req->cmd->fabrics.fctype == nvme_fabrics_type_auth_receive))
 				return true;
 			break;
 		default:
@@ -3563,6 +3580,108 @@ static ssize_t dctype_show(struct device *dev,
 }
 static DEVICE_ATTR_RO(dctype);
 
+#ifdef CONFIG_NVME_AUTH
+static ssize_t nvme_ctrl_dhchap_secret_show(struct device *dev,
+		struct device_attribute *attr, char *buf)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	struct nvmf_ctrl_options *opts = ctrl->opts;
+
+	if (!opts->dhchap_secret)
+		return sysfs_emit(buf, "none\n");
+	return sysfs_emit(buf, "%s\n", opts->dhchap_secret);
+}
+
+static ssize_t nvme_ctrl_dhchap_secret_store(struct device *dev,
+		struct device_attribute *attr, const char *buf, size_t count)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	struct nvmf_ctrl_options *opts = ctrl->opts;
+	char *dhchap_secret;
+
+	if (!ctrl->opts->dhchap_secret)
+		return -EINVAL;
+	if (count < 7)
+		return -EINVAL;
+	if (memcmp(buf, "DHHC-1:", 7))
+		return -EINVAL;
+
+	dhchap_secret = kzalloc(count + 1, GFP_KERNEL);
+	if (!dhchap_secret)
+		return -ENOMEM;
+	memcpy(dhchap_secret, buf, count);
+	nvme_auth_stop(ctrl);
+	if (strcmp(dhchap_secret, opts->dhchap_secret)) {
+		int ret;
+
+		ret = nvme_auth_generate_key(dhchap_secret, &ctrl->host_key);
+		if (ret)
+			return ret;
+		kfree(opts->dhchap_secret);
+		opts->dhchap_secret = dhchap_secret;
+		/* Key has changed; re-authentication with new key */
+		nvme_auth_reset(ctrl);
+	}
+	/* Start re-authentication */
+	dev_info(ctrl->device, "re-authenticating controller\n");
+	queue_work(nvme_wq, &ctrl->dhchap_auth_work);
+
+	return count;
+}
+DEVICE_ATTR(dhchap_secret, S_IRUGO | S_IWUSR,
+	nvme_ctrl_dhchap_secret_show, nvme_ctrl_dhchap_secret_store);
+
+static ssize_t nvme_ctrl_dhchap_ctrl_secret_show(struct device *dev,
+		struct device_attribute *attr, char *buf)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	struct nvmf_ctrl_options *opts = ctrl->opts;
+
+	if (!opts->dhchap_ctrl_secret)
+		return sysfs_emit(buf, "none\n");
+	return sysfs_emit(buf, "%s\n", opts->dhchap_ctrl_secret);
+}
+
+static ssize_t nvme_ctrl_dhchap_ctrl_secret_store(struct device *dev,
+		struct device_attribute *attr, const char *buf, size_t count)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	struct nvmf_ctrl_options *opts = ctrl->opts;
+	char *dhchap_secret;
+
+	if (!ctrl->opts->dhchap_ctrl_secret)
+		return -EINVAL;
+	if (count < 7)
+		return -EINVAL;
+	if (memcmp(buf, "DHHC-1:", 7))
+		return -EINVAL;
+
+	dhchap_secret = kzalloc(count + 1, GFP_KERNEL);
+	if (!dhchap_secret)
+		return -ENOMEM;
+	memcpy(dhchap_secret, buf, count);
+	nvme_auth_stop(ctrl);
+	if (strcmp(dhchap_secret, opts->dhchap_ctrl_secret)) {
+		int ret;
+
+		ret = nvme_auth_generate_key(dhchap_secret, &ctrl->ctrl_key);
+		if (ret)
+			return ret;
+		kfree(opts->dhchap_ctrl_secret);
+		opts->dhchap_ctrl_secret = dhchap_secret;
+		/* Key has changed; re-authentication with new key */
+		nvme_auth_reset(ctrl);
+	}
+	/* Start re-authentication */
+	dev_info(ctrl->device, "re-authenticating controller\n");
+	queue_work(nvme_wq, &ctrl->dhchap_auth_work);
+
+	return count;
+}
+DEVICE_ATTR(dhchap_ctrl_secret, S_IRUGO | S_IWUSR,
+	nvme_ctrl_dhchap_ctrl_secret_show, nvme_ctrl_dhchap_ctrl_secret_store);
+#endif
+
 static struct attribute *nvme_dev_attrs[] = {
 	&dev_attr_reset_controller.attr,
 	&dev_attr_rescan_controller.attr,
@@ -3586,6 +3705,10 @@ static struct attribute *nvme_dev_attrs[] = {
 	&dev_attr_kato.attr,
 	&dev_attr_cntrltype.attr,
 	&dev_attr_dctype.attr,
+#ifdef CONFIG_NVME_AUTH
+	&dev_attr_dhchap_secret.attr,
+	&dev_attr_dhchap_ctrl_secret.attr,
+#endif
 	NULL
 };
 
@@ -3609,6 +3732,10 @@ static umode_t nvme_dev_attrs_are_visible(struct kobject *kobj,
 		return 0;
 	if (a == &dev_attr_fast_io_fail_tmo.attr && !ctrl->opts)
 		return 0;
+#ifdef CONFIG_NVME_AUTH
+	if (a == &dev_attr_dhchap_secret.attr && !ctrl->opts)
+		return 0;
+#endif
 
 	return a->mode;
 }
@@ -4450,8 +4577,10 @@ static void nvme_handle_aen_notice(struct nvme_ctrl *ctrl, u32 result)
 		 * recovery actions from interfering with the controller's
 		 * firmware activation.
 		 */
-		if (nvme_change_ctrl_state(ctrl, NVME_CTRL_RESETTING))
+		if (nvme_change_ctrl_state(ctrl, NVME_CTRL_RESETTING)) {
+			nvme_auth_stop(ctrl);
 			queue_work(nvme_wq, &ctrl->fw_act_work);
+		}
 		break;
 #ifdef CONFIG_NVME_MULTIPATH
 	case NVME_AER_NOTICE_ANA:
@@ -4498,6 +4627,7 @@ EXPORT_SYMBOL_GPL(nvme_complete_async_event);
 void nvme_stop_ctrl(struct nvme_ctrl *ctrl)
 {
 	nvme_mpath_stop(ctrl);
+	nvme_auth_stop(ctrl);
 	nvme_stop_keep_alive(ctrl);
 	nvme_stop_failfast_work(ctrl);
 	flush_work(&ctrl->async_event_work);
@@ -4554,6 +4684,8 @@ static void nvme_free_ctrl(struct device *dev)
 
 	nvme_free_cels(ctrl);
 	nvme_mpath_uninit(ctrl);
+	nvme_auth_stop(ctrl);
+	nvme_auth_free(ctrl);
 	__free_page(ctrl->discard_page);
 
 	if (subsys) {
@@ -4644,6 +4776,7 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev,
 
 	nvme_fault_inject_init(&ctrl->fault_inject, dev_name(ctrl->device));
 	nvme_mpath_init_ctrl(ctrl);
+	nvme_auth_init_ctrl(ctrl);
 
 	return 0;
 out_free_name:
diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index 953e3076aab1..f5786a60931c 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -369,6 +369,7 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)
 	union nvme_result res;
 	struct nvmf_connect_data *data;
 	int ret;
+	u32 result;
 
 	cmd.connect.opcode = nvme_fabrics_command;
 	cmd.connect.fctype = nvme_fabrics_type_connect;
@@ -401,8 +402,25 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)
 		goto out_free_data;
 	}
 
-	ctrl->cntlid = le16_to_cpu(res.u16);
-
+	result = le32_to_cpu(res.u32);
+	ctrl->cntlid = result & 0xFFFF;
+	if ((result >> 16) & 2) {
+		/* Authentication required */
+		ret = nvme_auth_negotiate(ctrl, 0);
+		if (ret) {
+			dev_warn(ctrl->device,
+				 "qid 0: authentication setup failed\n");
+			ret = NVME_SC_AUTH_REQUIRED;
+			goto out_free_data;
+		}
+		ret = nvme_auth_wait(ctrl, 0);
+		if (ret)
+			dev_warn(ctrl->device,
+				 "qid 0: authentication failed\n");
+		else
+			dev_info(ctrl->device,
+				 "qid 0: authenticated\n");
+	}
 out_free_data:
 	kfree(data);
 	return ret;
@@ -435,6 +453,7 @@ int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, u16 qid)
 	struct nvmf_connect_data *data;
 	union nvme_result res;
 	int ret;
+	u32 result;
 
 	cmd.connect.opcode = nvme_fabrics_command;
 	cmd.connect.fctype = nvme_fabrics_type_connect;
@@ -460,6 +479,21 @@ int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, u16 qid)
 		nvmf_log_connect_error(ctrl, ret, le32_to_cpu(res.u32),
 				       &cmd, data);
 	}
+	result = le32_to_cpu(res.u32);
+	if ((result >> 16) & 2) {
+		/* Authentication required */
+		ret = nvme_auth_negotiate(ctrl, qid);
+		if (ret) {
+			dev_warn(ctrl->device,
+				 "qid %d: authentication setup failed\n", qid);
+			ret = NVME_SC_AUTH_REQUIRED;
+		} else {
+			ret = nvme_auth_wait(ctrl, qid);
+			if (ret)
+				dev_warn(ctrl->device,
+					 "qid %u: authentication failed\n", qid);
+		}
+	}
 	kfree(data);
 	return ret;
 }
@@ -552,6 +586,8 @@ static const match_table_t opt_tokens = {
 	{ NVMF_OPT_TOS,			"tos=%d"		},
 	{ NVMF_OPT_FAIL_FAST_TMO,	"fast_io_fail_tmo=%d"	},
 	{ NVMF_OPT_DISCOVERY,		"discovery"		},
+	{ NVMF_OPT_DHCHAP_SECRET,	"dhchap_secret=%s"	},
+	{ NVMF_OPT_DHCHAP_CTRL_SECRET,	"dhchap_ctrl_secret=%s"	},
 	{ NVMF_OPT_ERR,			NULL			}
 };
 
@@ -833,6 +869,34 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
 		case NVMF_OPT_DISCOVERY:
 			opts->discovery_nqn = true;
 			break;
+		case NVMF_OPT_DHCHAP_SECRET:
+			p = match_strdup(args);
+			if (!p) {
+				ret = -ENOMEM;
+				goto out;
+			}
+			if (strlen(p) < 11 || strncmp(p, "DHHC-1:", 7)) {
+				pr_err("Invalid DH-CHAP secret %s\n", p);
+				ret = -EINVAL;
+				goto out;
+			}
+			kfree(opts->dhchap_secret);
+			opts->dhchap_secret = p;
+			break;
+		case NVMF_OPT_DHCHAP_CTRL_SECRET:
+			p = match_strdup(args);
+			if (!p) {
+				ret = -ENOMEM;
+				goto out;
+			}
+			if (strlen(p) < 11 || strncmp(p, "DHHC-1:", 7)) {
+				pr_err("Invalid DH-CHAP secret %s\n", p);
+				ret = -EINVAL;
+				goto out;
+			}
+			kfree(opts->dhchap_ctrl_secret);
+			opts->dhchap_ctrl_secret = p;
+			break;
 		default:
 			pr_warn("unknown parameter or missing value '%s' in ctrl creation request\n",
 				p);
@@ -951,6 +1015,7 @@ void nvmf_free_options(struct nvmf_ctrl_options *opts)
 	kfree(opts->subsysnqn);
 	kfree(opts->host_traddr);
 	kfree(opts->host_iface);
+	kfree(opts->dhchap_secret);
 	kfree(opts);
 }
 EXPORT_SYMBOL_GPL(nvmf_free_options);
@@ -960,7 +1025,8 @@ EXPORT_SYMBOL_GPL(nvmf_free_options);
 				 NVMF_OPT_KATO | NVMF_OPT_HOSTNQN | \
 				 NVMF_OPT_HOST_ID | NVMF_OPT_DUP_CONNECT |\
 				 NVMF_OPT_DISABLE_SQFLOW | NVMF_OPT_DISCOVERY |\
-				 NVMF_OPT_FAIL_FAST_TMO)
+				 NVMF_OPT_FAIL_FAST_TMO | NVMF_OPT_DHCHAP_SECRET |\
+				 NVMF_OPT_DHCHAP_CTRL_SECRET)
 
 static struct nvme_ctrl *
 nvmf_create_ctrl(struct device *dev, const char *buf)
@@ -1196,7 +1262,14 @@ static void __exit nvmf_exit(void)
 	BUILD_BUG_ON(sizeof(struct nvmf_connect_command) != 64);
 	BUILD_BUG_ON(sizeof(struct nvmf_property_get_command) != 64);
 	BUILD_BUG_ON(sizeof(struct nvmf_property_set_command) != 64);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_send_command) != 64);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_receive_command) != 64);
 	BUILD_BUG_ON(sizeof(struct nvmf_connect_data) != 1024);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_negotiate_data) != 8);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_challenge_data) != 16);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_reply_data) != 16);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_success1_data) != 16);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_success2_data) != 16);
 }
 
 MODULE_LICENSE("GPL v2");
diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h
index 1e3a09cad961..15c142b277ab 100644
--- a/drivers/nvme/host/fabrics.h
+++ b/drivers/nvme/host/fabrics.h
@@ -68,6 +68,8 @@ enum {
 	NVMF_OPT_FAIL_FAST_TMO	= 1 << 20,
 	NVMF_OPT_HOST_IFACE	= 1 << 21,
 	NVMF_OPT_DISCOVERY	= 1 << 22,
+	NVMF_OPT_DHCHAP_SECRET	= 1 << 23,
+	NVMF_OPT_DHCHAP_CTRL_SECRET = 1 << 24,
 };
 
 /**
@@ -97,6 +99,9 @@ enum {
  * @max_reconnects: maximum number of allowed reconnect attempts before removing
  *              the controller, (-1) means reconnect forever, zero means remove
  *              immediately;
+ * @dhchap_secret: DH-HMAC-CHAP secret
+ * @dhchap_ctrl_secret: DH-HMAC-CHAP controller secret for bi-directional
+ *              authentication
  * @disable_sqflow: disable controller sq flow control
  * @hdr_digest: generate/verify header digest (TCP)
  * @data_digest: generate/verify data digest (TCP)
@@ -121,6 +126,8 @@ struct nvmf_ctrl_options {
 	unsigned int		kato;
 	struct nvmf_host	*host;
 	int			max_reconnects;
+	char			*dhchap_secret;
+	char			*dhchap_ctrl_secret;
 	bool			disable_sqflow;
 	bool			hdr_digest;
 	bool			data_digest;
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 1ea908d43e17..f110437a3190 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -323,6 +323,15 @@ struct nvme_ctrl {
 	struct work_struct ana_work;
 #endif
 
+#ifdef CONFIG_NVME_AUTH
+	struct work_struct dhchap_auth_work;
+	struct list_head dhchap_auth_list;
+	struct mutex dhchap_auth_mutex;
+	struct nvme_dhchap_key *host_key;
+	struct nvme_dhchap_key *ctrl_key;
+	u16 transaction;
+#endif
+
 	/* Power saving configuration */
 	u64 ps_max_latency_us;
 	bool apst_enabled;
@@ -928,6 +937,28 @@ static inline bool nvme_ctrl_sgl_supported(struct nvme_ctrl *ctrl)
 	return ctrl->sgls & ((1 << 0) | (1 << 1));
 }
 
+#ifdef CONFIG_NVME_AUTH
+void nvme_auth_init_ctrl(struct nvme_ctrl *ctrl);
+void nvme_auth_stop(struct nvme_ctrl *ctrl);
+int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid);
+int nvme_auth_wait(struct nvme_ctrl *ctrl, int qid);
+void nvme_auth_reset(struct nvme_ctrl *ctrl);
+void nvme_auth_free(struct nvme_ctrl *ctrl);
+int nvme_auth_generate_key(u8 *secret, struct nvme_dhchap_key **ret_key);
+#else
+static inline void nvme_auth_init_ctrl(struct nvme_ctrl *ctrl) {};
+static inline void nvme_auth_stop(struct nvme_ctrl *ctrl) {};
+static inline int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid)
+{
+	return -EPROTONOSUPPORT;
+}
+static inline int nvme_auth_wait(struct nvme_ctrl *ctrl, int qid)
+{
+	return NVME_SC_AUTH_REQUIRED;
+}
+static inline void nvme_auth_free(struct nvme_ctrl *ctrl) {};
+#endif
+
 u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 			 u8 opcode);
 int nvme_execute_passthru_rq(struct request *rq);
diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index d9f19d901313..bd63568c0068 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -1197,6 +1197,7 @@ static void nvme_rdma_error_recovery_work(struct work_struct *work)
 	struct nvme_rdma_ctrl *ctrl = container_of(work,
 			struct nvme_rdma_ctrl, err_work);
 
+	nvme_auth_stop(&ctrl->ctrl);
 	nvme_stop_keep_alive(&ctrl->ctrl);
 	flush_work(&ctrl->ctrl.async_event_work);
 	nvme_rdma_teardown_io_queues(ctrl, false);
diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index ad3a2bf2f1e9..be52902b5f3b 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -2174,6 +2174,7 @@ static void nvme_tcp_error_recovery_work(struct work_struct *work)
 				struct nvme_tcp_ctrl, err_work);
 	struct nvme_ctrl *ctrl = &tcp_ctrl->ctrl;
 
+	nvme_auth_stop(ctrl);
 	nvme_stop_keep_alive(ctrl);
 	flush_work(&ctrl->async_event_work);
 	nvme_tcp_teardown_io_queues(ctrl, false);
diff --git a/drivers/nvme/host/trace.c b/drivers/nvme/host/trace.c
index 2a89c5aa0790..1c36fcedea20 100644
--- a/drivers/nvme/host/trace.c
+++ b/drivers/nvme/host/trace.c
@@ -287,6 +287,34 @@ static const char *nvme_trace_fabrics_property_get(struct trace_seq *p, u8 *spc)
 	return ret;
 }
 
+static const char *nvme_trace_fabrics_auth_send(struct trace_seq *p, u8 *spc)
+{
+	const char *ret = trace_seq_buffer_ptr(p);
+	u8 spsp0 = spc[1];
+	u8 spsp1 = spc[2];
+	u8 secp = spc[3];
+	u32 tl = get_unaligned_le32(spc + 4);
+
+	trace_seq_printf(p, "spsp0=%02x, spsp1=%02x, secp=%02x, tl=%u",
+			 spsp0, spsp1, secp, tl);
+	trace_seq_putc(p, 0);
+	return ret;
+}
+
+static const char *nvme_trace_fabrics_auth_receive(struct trace_seq *p, u8 *spc)
+{
+	const char *ret = trace_seq_buffer_ptr(p);
+	u8 spsp0 = spc[1];
+	u8 spsp1 = spc[2];
+	u8 secp = spc[3];
+	u32 al = get_unaligned_le32(spc + 4);
+
+	trace_seq_printf(p, "spsp0=%02x, spsp1=%02x, secp=%02x, al=%u",
+			 spsp0, spsp1, secp, al);
+	trace_seq_putc(p, 0);
+	return ret;
+}
+
 static const char *nvme_trace_fabrics_common(struct trace_seq *p, u8 *spc)
 {
 	const char *ret = trace_seq_buffer_ptr(p);
@@ -306,6 +334,10 @@ const char *nvme_trace_parse_fabrics_cmd(struct trace_seq *p,
 		return nvme_trace_fabrics_connect(p, spc);
 	case nvme_fabrics_type_property_get:
 		return nvme_trace_fabrics_property_get(p, spc);
+	case nvme_fabrics_type_auth_send:
+		return nvme_trace_fabrics_auth_send(p, spc);
+	case nvme_fabrics_type_auth_receive:
+		return nvme_trace_fabrics_auth_receive(p, spc);
 	default:
 		return nvme_trace_fabrics_common(p, spc);
 	}
-- 
2.29.2

[PATCH 06/11] nvme: Implement In-Band authentication

[Hannes Reinecke]

Implement NVMe-oF In-Band authentication according to NVMe TPAR 8006.
This patch adds two new fabric options 'dhchap_secret' to specify the
pre-shared key (in ASCII respresentation according to NVMe 2.0 section
8.13.5.8 'Secret representation') and 'dhchap_ctrl_secret' to specify
the pre-shared controller key for bi-directional authentication of both
the host and the controller.
Re-authentication can be triggered by writing the PSK into the new
controller sysfs attribute 'dhchap_secret' or 'dhchap_ctrl_secret'.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/nvme/host/Kconfig   |   11 +
 drivers/nvme/host/Makefile  |    1 +
 drivers/nvme/host/auth.c    | 1117 +++++++++++++++++++++++++++++++++++
 drivers/nvme/host/auth.h    |   32 +
 drivers/nvme/host/core.c    |  141 ++++-
 drivers/nvme/host/fabrics.c |   79 ++-
 drivers/nvme/host/fabrics.h |    7 +
 drivers/nvme/host/nvme.h    |   31 +
 drivers/nvme/host/rdma.c    |    1 +
 drivers/nvme/host/tcp.c     |    1 +
 drivers/nvme/host/trace.c   |   32 +
 11 files changed, 1446 insertions(+), 7 deletions(-)
 create mode 100644 drivers/nvme/host/auth.c
 create mode 100644 drivers/nvme/host/auth.h

diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig
index d6d056963c06..dd0e91fb0615 100644
--- a/drivers/nvme/host/Kconfig
+++ b/drivers/nvme/host/Kconfig
@@ -91,3 +91,14 @@ config NVME_TCP
 	  from https://github.com/linux-nvme/nvme-cli.
 
 	  If unsure, say N.
+
+config NVME_AUTH
+	bool "NVM Express over Fabrics In-Band Authentication"
+	depends on NVME_CORE
+	select CRYPTO_HMAC
+	select CRYPTO_SHA256
+	select CRYPTO_SHA512
+	help
+	  This provides support for NVMe over Fabrics In-Band Authentication.
+
+	  If unsure, say N.
diff --git a/drivers/nvme/host/Makefile b/drivers/nvme/host/Makefile
index 476c5c988496..7755f5e3b281 100644
--- a/drivers/nvme/host/Makefile
+++ b/drivers/nvme/host/Makefile
@@ -15,6 +15,7 @@ nvme-core-$(CONFIG_NVME_MULTIPATH)	+= multipath.o
 nvme-core-$(CONFIG_BLK_DEV_ZONED)	+= zns.o
 nvme-core-$(CONFIG_FAULT_INJECTION_DEBUG_FS)	+= fault_inject.o
 nvme-core-$(CONFIG_NVME_HWMON)		+= hwmon.o
+nvme-core-$(CONFIG_NVME_AUTH)		+= auth.o
 
 nvme-y					+= pci.o
 
diff --git a/drivers/nvme/host/auth.c b/drivers/nvme/host/auth.c
new file mode 100644
index 000000000000..6fe0100def2c
--- /dev/null
+++ b/drivers/nvme/host/auth.c
@@ -0,0 +1,1117 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2020 Hannes Reinecke, SUSE Linux
+ */
+
+#include <linux/crc32.h>
+#include <linux/base64.h>
+#include <linux/prandom.h>
+#include <asm/unaligned.h>
+#include <crypto/hash.h>
+#include <crypto/dh.h>
+#include "nvme.h"
+#include "fabrics.h"
+#include "auth.h"
+
+static u32 nvme_dhchap_seqnum;
+static DEFINE_MUTEX(nvme_dhchap_mutex);
+
+struct nvme_dhchap_queue_context {
+	struct list_head entry;
+	struct work_struct auth_work;
+	struct nvme_ctrl *ctrl;
+	struct crypto_shash *shash_tfm;
+	void *buf;
+	size_t buf_size;
+	int qid;
+	int error;
+	u32 s1;
+	u32 s2;
+	u16 transaction;
+	u8 status;
+	u8 hash_id;
+	size_t hash_len;
+	u8 dhgroup_id;
+	u8 c1[64];
+	u8 c2[64];
+	u8 response[64];
+	u8 *host_response;
+};
+
+u32 nvme_auth_get_seqnum(void)
+{
+	u32 seqnum;
+
+	mutex_lock(&nvme_dhchap_mutex);
+	if (!nvme_dhchap_seqnum)
+		nvme_dhchap_seqnum = prandom_u32();
+	else {
+		nvme_dhchap_seqnum++;
+		if (!nvme_dhchap_seqnum)
+			nvme_dhchap_seqnum++;
+	}
+	seqnum = nvme_dhchap_seqnum;
+	mutex_unlock(&nvme_dhchap_mutex);
+	return seqnum;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_get_seqnum);
+
+static struct nvme_auth_dhgroup_map {
+	const char name[16];
+	const char kpp[16];
+} dhgroup_map[] = {
+	[NVME_AUTH_DHGROUP_NULL] = {
+		.name = "null", .kpp = "null" },
+	[NVME_AUTH_DHGROUP_2048] = {
+		.name = "ffdhe2048", .kpp = "ffdhe2048(dh)" },
+	[NVME_AUTH_DHGROUP_3072] = {
+		.name = "ffdhe3072", .kpp = "ffdhe3072(dh)" },
+	[NVME_AUTH_DHGROUP_4096] = {
+		.name = "ffdhe4096", .kpp = "ffdhe4096(dh)" },
+	[NVME_AUTH_DHGROUP_6144] = {
+		.name = "ffdhe6144", .kpp = "ffdhe6144(dh)" },
+	[NVME_AUTH_DHGROUP_8192] = {
+		.name = "ffdhe8192", .kpp = "ffdhe8192(dh)" },
+};
+
+const char *nvme_auth_dhgroup_name(u8 dhgroup_id)
+{
+	if ((dhgroup_id > ARRAY_SIZE(dhgroup_map)) ||
+	    !dhgroup_map[dhgroup_id].name ||
+	    !strlen(dhgroup_map[dhgroup_id].name))
+		return NULL;
+	return dhgroup_map[dhgroup_id].name;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_dhgroup_name);
+
+const char *nvme_auth_dhgroup_kpp(u8 dhgroup_id)
+{
+	if ((dhgroup_id > ARRAY_SIZE(dhgroup_map)) ||
+	    !dhgroup_map[dhgroup_id].kpp ||
+	    !strlen(dhgroup_map[dhgroup_id].kpp))
+		return NULL;
+	return dhgroup_map[dhgroup_id].kpp;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_dhgroup_kpp);
+
+u8 nvme_auth_dhgroup_id(const char *dhgroup_name)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(dhgroup_map); i++) {
+		if (!dhgroup_map[i].name ||
+		    !strlen(dhgroup_map[i].name))
+			continue;
+		if (!strncmp(dhgroup_map[i].name, dhgroup_name,
+			     strlen(dhgroup_map[i].name)))
+			return i;
+	}
+	return NVME_AUTH_DHGROUP_INVALID;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_dhgroup_id);
+
+static struct nvme_dhchap_hash_map {
+	int len;
+	const char hmac[15];
+	const char digest[8];
+} hash_map[] = {
+	[NVME_AUTH_HASH_SHA256] = {
+		.len = 32,
+		.hmac = "hmac(sha256)",
+		.digest = "sha256",
+	},
+	[NVME_AUTH_HASH_SHA384] = {
+		.len = 48,
+		.hmac = "hmac(sha384)",
+		.digest = "sha384",
+	},
+	[NVME_AUTH_HASH_SHA512] = {
+		.len = 64,
+		.hmac = "hmac(sha512)",
+		.digest = "sha512",
+	},
+};
+
+const char *nvme_auth_hmac_name(u8 hmac_id)
+{
+	if ((hmac_id > ARRAY_SIZE(hash_map)) ||
+	    !hash_map[hmac_id].hmac ||
+	    !strlen(hash_map[hmac_id].hmac))
+		return NULL;
+	return hash_map[hmac_id].hmac;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_hmac_name);
+
+const char *nvme_auth_digest_name(u8 hmac_id)
+{
+	if ((hmac_id > ARRAY_SIZE(hash_map)) ||
+	    !hash_map[hmac_id].digest ||
+	    !strlen(hash_map[hmac_id].digest))
+		return NULL;
+	return hash_map[hmac_id].digest;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_digest_name);
+
+u8 nvme_auth_hmac_id(const char *hmac_name)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
+		if (!hash_map[i].hmac || !strlen(hash_map[i].hmac))
+			continue;
+		if (!strncmp(hash_map[i].hmac, hmac_name,
+			     strlen(hash_map[i].hmac)))
+			return i;
+	}
+	return NVME_AUTH_HASH_INVALID;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_hmac_id);
+
+size_t nvme_auth_hmac_hash_len(u8 hmac_id)
+{
+	if ((hmac_id > ARRAY_SIZE(hash_map)) ||
+	    !hash_map[hmac_id].hmac ||
+	    !strlen(hash_map[hmac_id].hmac))
+		return 0;
+	return hash_map[hmac_id].len;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_hmac_hash_len);
+
+struct nvme_dhchap_key *nvme_auth_extract_key(unsigned char *secret,
+					      u8 key_hash)
+{
+	struct nvme_dhchap_key *key;
+	unsigned char *p;
+	u32 crc;
+	int ret, key_len;
+	size_t allocated_len = strlen(secret);
+
+	/* Secret might be affixed with a ':' */
+	p = strrchr(secret, ':');
+	if (p)
+		allocated_len = p - secret;
+	key = kzalloc(sizeof(*key), GFP_KERNEL);
+	if (!key)
+		return ERR_PTR(-ENOMEM);
+	key->key = kzalloc(allocated_len, GFP_KERNEL);
+	if (!key->key) {
+		ret = -ENOMEM;
+		goto out_free_key;
+	}
+
+	key_len = base64_decode(secret, allocated_len, key->key);
+	if (key_len < 0) {
+		pr_debug("base64 key decoding error %d\n",
+			 key_len);
+		ret = key_len;
+		goto out_free_secret;
+	}
+
+	if (key_len != 36 && key_len != 52 &&
+	    key_len != 68) {
+		pr_err("Invalid DH-HMAC-CHAP key len %d\n",
+		       key_len);
+		ret = -EINVAL;
+		goto out_free_secret;
+	}
+
+	if (key_hash > 0 &&
+	    (key_len - 4) != nvme_auth_hmac_hash_len(key_hash)) {
+		pr_err("Invalid DH-HMAC-CHAP key len %d for %s\n", key_len,
+		       nvme_auth_hmac_name(key_hash));
+		ret = -EINVAL;
+		goto out_free_secret;
+	}
+
+	/* The last four bytes is the CRC in little-endian format */
+	key_len -= 4;
+	/*
+	 * The linux implementation doesn't do pre- and post-increments,
+	 * so we have to do it manually.
+	 */
+	crc = ~crc32(~0, key->key, key_len);
+
+	if (get_unaligned_le32(key->key + key_len) != crc) {
+		pr_err("DH-HMAC-CHAP key crc mismatch (key %08x, crc %08x)\n",
+		       get_unaligned_le32(key->key + key_len), crc);
+		ret = -EKEYREJECTED;
+		goto out_free_secret;
+	}
+	key->len = key_len;
+	key->hash = key_hash;
+	return key;
+out_free_secret:
+	kfree_sensitive(key->key);
+out_free_key:
+	kfree(key);
+	return ERR_PTR(ret);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_extract_key);
+
+void nvme_auth_free_key(struct nvme_dhchap_key *key)
+{
+	if (!key)
+		return;
+	kfree_sensitive(key->key);
+	kfree(key);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_free_key);
+
+u8 *nvme_auth_transform_key(struct nvme_dhchap_key *key, char *nqn)
+{
+	const char *hmac_name = nvme_auth_hmac_name(key->hash);
+	struct crypto_shash *key_tfm;
+	struct shash_desc *shash;
+	u8 *transformed_key;
+	int ret;
+
+	if (key->hash == 0) {
+		transformed_key = kmemdup(key->key, key->len, GFP_KERNEL);
+		return transformed_key ? transformed_key : ERR_PTR(-ENOMEM);
+	}
+
+	if (!key || !key->key) {
+		pr_warn("No key specified\n");
+		return ERR_PTR(-ENOKEY);
+	}
+	if (!hmac_name) {
+		pr_warn("Invalid key hash id %d\n", key->hash);
+		return ERR_PTR(-EINVAL);
+	}
+
+	key_tfm = crypto_alloc_shash(hmac_name, 0, 0);
+	if (IS_ERR(key_tfm))
+		return (u8 *)key_tfm;
+
+	shash = kmalloc(sizeof(struct shash_desc) +
+			crypto_shash_descsize(key_tfm),
+			GFP_KERNEL);
+	if (!shash) {
+		ret = -ENOMEM;
+		goto out_free_key;
+	}
+
+	transformed_key = kzalloc(crypto_shash_digestsize(key_tfm), GFP_KERNEL);
+	if (!transformed_key) {
+		ret = -ENOMEM;
+		goto out_free_shash;
+	}
+
+	shash->tfm = key_tfm;
+	ret = crypto_shash_setkey(key_tfm, key->key, key->len);
+	if (ret < 0)
+		goto out_free_shash;
+	ret = crypto_shash_init(shash);
+	if (ret < 0)
+		goto out_free_shash;
+	ret = crypto_shash_update(shash, nqn, strlen(nqn));
+	if (ret < 0)
+		goto out_free_shash;
+	ret = crypto_shash_update(shash, "NVMe-over-Fabrics", 17);
+	if (ret < 0)
+		goto out_free_shash;
+	ret = crypto_shash_final(shash, transformed_key);
+out_free_shash:
+	kfree(shash);
+out_free_key:
+	crypto_free_shash(key_tfm);
+	if (ret < 0) {
+		kfree_sensitive(transformed_key);
+		return ERR_PTR(ret);
+	}
+	return transformed_key;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_transform_key);
+
+#define nvme_auth_flags_from_qid(qid) \
+	(qid == 0) ? 0 : BLK_MQ_REQ_NOWAIT | BLK_MQ_REQ_RESERVED
+#define nvme_auth_queue_from_qid(ctrl, qid) \
+	(qid == 0) ? (ctrl)->fabrics_q : (ctrl)->connect_q
+
+static int nvme_auth_submit(struct nvme_ctrl *ctrl, int qid,
+			    void *data, size_t data_len, bool auth_send)
+{
+	struct nvme_command cmd = {};
+	blk_mq_req_flags_t flags = nvme_auth_flags_from_qid(qid);
+	struct request_queue *q = nvme_auth_queue_from_qid(ctrl, qid);
+	int ret;
+
+	cmd.auth_common.opcode = nvme_fabrics_command;
+	cmd.auth_common.secp = NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER;
+	cmd.auth_common.spsp0 = 0x01;
+	cmd.auth_common.spsp1 = 0x01;
+	if (auth_send) {
+		cmd.auth_send.fctype = nvme_fabrics_type_auth_send;
+		cmd.auth_send.tl = cpu_to_le32(data_len);
+	} else {
+		cmd.auth_receive.fctype = nvme_fabrics_type_auth_receive;
+		cmd.auth_receive.al = cpu_to_le32(data_len);
+	}
+
+	ret = __nvme_submit_sync_cmd(q, &cmd, NULL, data, data_len, 0,
+				     qid == 0 ? NVME_QID_ANY : qid,
+				     0, flags);
+	if (ret > 0)
+		dev_warn(ctrl->device,
+			"qid %d auth_send failed with status %d\n", qid, ret);
+	else if (ret < 0)
+		dev_err(ctrl->device,
+			"qid %d auth_send failed with error %d\n", qid, ret);
+	return ret;
+}
+
+static int nvme_auth_receive_validate(struct nvme_ctrl *ctrl, int qid,
+		struct nvmf_auth_dhchap_failure_data *data,
+		u16 transaction, u8 expected_msg)
+{
+	dev_dbg(ctrl->device, "%s: qid %d auth_type %d auth_id %x\n",
+		__func__, qid, data->auth_type, data->auth_id);
+
+	if (data->auth_type == NVME_AUTH_COMMON_MESSAGES &&
+	    data->auth_id == NVME_AUTH_DHCHAP_MESSAGE_FAILURE1) {
+		return data->rescode_exp;
+	}
+	if (data->auth_type != NVME_AUTH_DHCHAP_MESSAGES ||
+	    data->auth_id != expected_msg) {
+		dev_warn(ctrl->device,
+			 "qid %d invalid message %02x/%02x\n",
+			 qid, data->auth_type, data->auth_id);
+		return NVME_AUTH_DHCHAP_FAILURE_INCORRECT_MESSAGE;
+	}
+	if (le16_to_cpu(data->t_id) != transaction) {
+		dev_warn(ctrl->device,
+			 "qid %d invalid transaction ID %d\n",
+			 qid, le16_to_cpu(data->t_id));
+		return NVME_AUTH_DHCHAP_FAILURE_INCORRECT_MESSAGE;
+	}
+	return 0;
+}
+
+static int nvme_auth_set_dhchap_negotiate_data(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_negotiate_data *data = chap->buf;
+	size_t size = sizeof(*data) + sizeof(union nvmf_auth_protocol);
+
+	if (chap->buf_size < size) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return -EINVAL;
+	}
+	memset((u8 *)chap->buf, 0, size);
+	data->auth_type = NVME_AUTH_COMMON_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE;
+	data->t_id = cpu_to_le16(chap->transaction);
+	data->sc_c = 0; /* No secure channel concatenation */
+	data->napd = 1;
+	data->auth_protocol[0].dhchap.authid = NVME_AUTH_DHCHAP_AUTH_ID;
+	data->auth_protocol[0].dhchap.halen = 3;
+	data->auth_protocol[0].dhchap.dhlen = 6;
+	data->auth_protocol[0].dhchap.idlist[0] = NVME_AUTH_HASH_SHA256;
+	data->auth_protocol[0].dhchap.idlist[1] = NVME_AUTH_HASH_SHA384;
+	data->auth_protocol[0].dhchap.idlist[2] = NVME_AUTH_HASH_SHA512;
+	data->auth_protocol[0].dhchap.idlist[30] = NVME_AUTH_DHGROUP_NULL;
+	data->auth_protocol[0].dhchap.idlist[31] = NVME_AUTH_DHGROUP_2048;
+	data->auth_protocol[0].dhchap.idlist[32] = NVME_AUTH_DHGROUP_3072;
+	data->auth_protocol[0].dhchap.idlist[33] = NVME_AUTH_DHGROUP_4096;
+	data->auth_protocol[0].dhchap.idlist[34] = NVME_AUTH_DHGROUP_6144;
+	data->auth_protocol[0].dhchap.idlist[35] = NVME_AUTH_DHGROUP_8192;
+
+	return size;
+}
+
+static int nvme_auth_process_dhchap_challenge(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_challenge_data *data = chap->buf;
+	u16 dhvlen = le16_to_cpu(data->dhvlen);
+	size_t size = sizeof(*data) + data->hl + dhvlen;
+	const char *hmac_name, *kpp_name;
+
+	if (chap->buf_size < size) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return NVME_SC_INVALID_FIELD;
+	}
+
+	hmac_name = nvme_auth_hmac_name(data->hashid);
+	if (!hmac_name) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid HASH ID %d\n",
+			 chap->qid, data->hashid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
+		return NVME_SC_INVALID_FIELD;
+	}
+
+	if (chap->hash_id == data->hashid && chap->shash_tfm &&
+	    !strcmp(crypto_shash_alg_name(chap->shash_tfm), hmac_name) &&
+	    crypto_shash_digestsize(chap->shash_tfm) == data->hl) {
+		dev_dbg(ctrl->device,
+			"qid %d: reuse existing hash %s\n",
+			chap->qid, hmac_name);
+		goto select_kpp;
+	}
+
+	/* Reset if hash cannot be reused */
+	if (chap->shash_tfm) {
+		crypto_free_shash(chap->shash_tfm);
+		chap->hash_id = 0;
+		chap->hash_len = 0;
+	}
+	chap->shash_tfm = crypto_alloc_shash(hmac_name, 0,
+					     CRYPTO_ALG_ALLOCATES_MEMORY);
+	if (IS_ERR(chap->shash_tfm)) {
+		dev_warn(ctrl->device,
+			 "qid %d: failed to allocate hash %s, error %ld\n",
+			 chap->qid, hmac_name, PTR_ERR(chap->shash_tfm));
+		chap->shash_tfm = NULL;
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_FAILED;
+		return NVME_SC_AUTH_REQUIRED;
+	}
+
+	if (crypto_shash_digestsize(chap->shash_tfm) != data->hl) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid hash length %d\n",
+			 chap->qid, data->hl);
+		crypto_free_shash(chap->shash_tfm);
+		chap->shash_tfm = NULL;
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
+		return NVME_SC_AUTH_REQUIRED;
+	}
+
+	/* Reset host response if the hash had been changed */
+	if (chap->hash_id != data->hashid) {
+		kfree(chap->host_response);
+		chap->host_response = NULL;
+	}
+
+	chap->hash_id = data->hashid;
+	chap->hash_len = data->hl;
+	dev_dbg(ctrl->device, "qid %d: selected hash %s\n",
+		chap->qid, hmac_name);
+
+select_kpp:
+	kpp_name = nvme_auth_dhgroup_kpp(data->dhgid);
+	if (!kpp_name) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid DH group id %d\n",
+			 chap->qid, data->dhgid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
+		return NVME_SC_AUTH_REQUIRED;
+	}
+
+	if (data->dhgid != NVME_AUTH_DHGROUP_NULL) {
+		dev_warn(ctrl->device,
+			 "qid %d: unsupported DH group %s\n",
+			 chap->qid, kpp_name);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
+		return NVME_SC_AUTH_REQUIRED;
+	} else if (dhvlen != 0) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid DH value for NULL DH\n",
+			 chap->qid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return NVME_SC_INVALID_FIELD;
+	}
+	chap->dhgroup_id = data->dhgid;
+
+	chap->s1 = le32_to_cpu(data->seqnum);
+	memcpy(chap->c1, data->cval, chap->hash_len);
+
+	return 0;
+}
+
+static int nvme_auth_set_dhchap_reply_data(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_reply_data *data = chap->buf;
+	size_t size = sizeof(*data);
+
+	size += 2 * chap->hash_len;
+
+	if (chap->buf_size < size) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return -EINVAL;
+	}
+
+	memset(chap->buf, 0, size);
+	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_REPLY;
+	data->t_id = cpu_to_le16(chap->transaction);
+	data->hl = chap->hash_len;
+	data->dhvlen = 0;
+	memcpy(data->rval, chap->response, chap->hash_len);
+	if (ctrl->opts->dhchap_ctrl_secret) {
+		get_random_bytes(chap->c2, chap->hash_len);
+		data->cvalid = 1;
+		chap->s2 = nvme_auth_get_seqnum();
+		memcpy(data->rval + chap->hash_len, chap->c2,
+		       chap->hash_len);
+		dev_dbg(ctrl->device, "%s: qid %d ctrl challenge %*ph\n",
+			__func__, chap->qid, (int)chap->hash_len, chap->c2);
+	} else {
+		memset(chap->c2, 0, chap->hash_len);
+		chap->s2 = 0;
+	}
+	data->seqnum = cpu_to_le32(chap->s2);
+	return size;
+}
+
+static int nvme_auth_process_dhchap_success1(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_success1_data *data = chap->buf;
+	size_t size = sizeof(*data);
+
+	if (ctrl->opts->dhchap_ctrl_secret)
+		size += chap->hash_len;
+
+	if (chap->buf_size < size) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return NVME_SC_INVALID_FIELD;
+	}
+
+	if (data->hl != chap->hash_len) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid hash length %u\n",
+			 chap->qid, data->hl);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
+		return NVME_SC_INVALID_FIELD;
+	}
+
+	/* Just print out information for the admin queue */
+	if (chap->qid == 0)
+		dev_info(ctrl->device,
+			 "qid 0: authenticated with hash %s dhgroup %s\n",
+			 nvme_auth_hmac_name(chap->hash_id),
+			 nvme_auth_dhgroup_name(chap->dhgroup_id));
+
+	if (!data->rvalid)
+		return 0;
+
+	/* Validate controller response */
+	if (memcmp(chap->response, data->rval, data->hl)) {
+		dev_dbg(ctrl->device, "%s: qid %d ctrl response %*ph\n",
+			__func__, chap->qid, (int)chap->hash_len, data->rval);
+		dev_dbg(ctrl->device, "%s: qid %d host response %*ph\n",
+			__func__, chap->qid, (int)chap->hash_len,
+			chap->response);
+		dev_warn(ctrl->device,
+			 "qid %d: controller authentication failed\n",
+			 chap->qid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_FAILED;
+		return NVME_SC_AUTH_REQUIRED;
+	}
+
+	/* Just print out information for the admin queue */
+	if (chap->qid == 0)
+		dev_info(ctrl->device,
+			 "qid 0: controller authenticated\n");
+	return 0;
+}
+
+static int nvme_auth_set_dhchap_success2_data(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_success2_data *data = chap->buf;
+	size_t size = sizeof(*data);
+
+	memset(chap->buf, 0, size);
+	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_SUCCESS2;
+	data->t_id = cpu_to_le16(chap->transaction);
+
+	return size;
+}
+
+static int nvme_auth_set_dhchap_failure2_data(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_failure_data *data = chap->buf;
+	size_t size = sizeof(*data);
+
+	memset(chap->buf, 0, size);
+	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_FAILURE2;
+	data->t_id = cpu_to_le16(chap->transaction);
+	data->rescode = NVME_AUTH_DHCHAP_FAILURE_REASON_FAILED;
+	data->rescode_exp = chap->status;
+
+	return size;
+}
+
+static int nvme_auth_dhchap_setup_host_response(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
+	u8 buf[4], *challenge = chap->c1;
+	int ret;
+
+	dev_dbg(ctrl->device, "%s: qid %d host response seq %u transaction %d\n",
+		__func__, chap->qid, chap->s1, chap->transaction);
+
+	if (!chap->host_response) {
+		chap->host_response = nvme_auth_transform_key(ctrl->host_key,
+						ctrl->opts->host->nqn);
+		if (IS_ERR(chap->host_response)) {
+			ret = PTR_ERR(chap->host_response);
+			chap->host_response = NULL;
+			return ret;
+		}
+	} else {
+		dev_dbg(ctrl->device, "%s: qid %d re-using host response\n",
+			__func__, chap->qid);
+	}
+
+	ret = crypto_shash_setkey(chap->shash_tfm,
+			chap->host_response, ctrl->host_key->len);
+	if (ret) {
+		dev_warn(ctrl->device, "qid %d: failed to set key, error %d\n",
+			 chap->qid, ret);
+		goto out;
+	}
+
+	shash->tfm = chap->shash_tfm;
+	ret = crypto_shash_init(shash);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, challenge, chap->hash_len);
+	if (ret)
+		goto out;
+	put_unaligned_le32(chap->s1, buf);
+	ret = crypto_shash_update(shash, buf, 4);
+	if (ret)
+		goto out;
+	put_unaligned_le16(chap->transaction, buf);
+	ret = crypto_shash_update(shash, buf, 2);
+	if (ret)
+		goto out;
+	memset(buf, 0, sizeof(buf));
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, "HostHost", 8);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
+				  strlen(ctrl->opts->host->nqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
+			    strlen(ctrl->opts->subsysnqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_final(shash, chap->response);
+out:
+	if (challenge != chap->c1)
+		kfree(challenge);
+	return ret;
+}
+
+static int nvme_auth_dhchap_setup_ctrl_response(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
+	u8 *ctrl_response;
+	u8 buf[4], *challenge = chap->c2;
+	int ret;
+
+	ctrl_response = nvme_auth_transform_key(ctrl->ctrl_key,
+				ctrl->opts->subsysnqn);
+	if (IS_ERR(ctrl_response)) {
+		ret = PTR_ERR(ctrl_response);
+		return ret;
+	}
+	ret = crypto_shash_setkey(chap->shash_tfm,
+			ctrl_response, ctrl->ctrl_key->len);
+	if (ret) {
+		dev_warn(ctrl->device, "qid %d: failed to set key, error %d\n",
+			 chap->qid, ret);
+		goto out;
+	}
+
+	dev_dbg(ctrl->device, "%s: qid %d ctrl response seq %u transaction %d\n",
+		__func__, chap->qid, chap->s2, chap->transaction);
+	dev_dbg(ctrl->device, "%s: qid %d challenge %*ph\n",
+		__func__, chap->qid, (int)chap->hash_len, challenge);
+	dev_dbg(ctrl->device, "%s: qid %d subsysnqn %s\n",
+		__func__, chap->qid, ctrl->opts->subsysnqn);
+	dev_dbg(ctrl->device, "%s: qid %d hostnqn %s\n",
+		__func__, chap->qid, ctrl->opts->host->nqn);
+	shash->tfm = chap->shash_tfm;
+	ret = crypto_shash_init(shash);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, challenge, chap->hash_len);
+	if (ret)
+		goto out;
+	put_unaligned_le32(chap->s2, buf);
+	ret = crypto_shash_update(shash, buf, 4);
+	if (ret)
+		goto out;
+	put_unaligned_le16(chap->transaction, buf);
+	ret = crypto_shash_update(shash, buf, 2);
+	if (ret)
+		goto out;
+	memset(buf, 0, 4);
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, "Controller", 10);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
+				  strlen(ctrl->opts->subsysnqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
+				  strlen(ctrl->opts->host->nqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_final(shash, chap->response);
+out:
+	if (challenge != chap->c2)
+		kfree(challenge);
+	return ret;
+}
+
+int nvme_auth_generate_key(u8 *secret, struct nvme_dhchap_key **ret_key)
+{
+	struct nvme_dhchap_key *key;
+	u8 key_hash;
+
+	if (!secret) {
+		*ret_key = NULL;
+		return 0;
+	}
+
+	if (sscanf(secret, "DHHC-1:%hhd:%*s:", &key_hash) != 1)
+		return -EINVAL;
+
+	/* Pass in the secret without the 'DHHC-1:XX:' prefix */
+	key = nvme_auth_extract_key(secret + 10, key_hash);
+	if (IS_ERR(key)) {
+		return PTR_ERR(key);
+	}
+
+	*ret_key = key;
+	return 0;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_generate_key);
+
+static void __nvme_auth_reset(struct nvme_dhchap_queue_context *chap)
+{
+	chap->status = 0;
+	chap->error = 0;
+	chap->s1 = 0;
+	chap->s2 = 0;
+	chap->transaction = 0;
+	memset(chap->c1, 0, sizeof(chap->c1));
+	memset(chap->c2, 0, sizeof(chap->c2));
+}
+
+static void __nvme_auth_free(struct nvme_dhchap_queue_context *chap)
+{
+	__nvme_auth_reset(chap);
+	if (chap->shash_tfm)
+		crypto_free_shash(chap->shash_tfm);
+	kfree_sensitive(chap->host_response);
+	kfree(chap->buf);
+	kfree(chap);
+}
+
+static void __nvme_auth_work(struct work_struct *work)
+{
+	struct nvme_dhchap_queue_context *chap =
+		container_of(work, struct nvme_dhchap_queue_context, auth_work);
+	struct nvme_ctrl *ctrl = chap->ctrl;
+	size_t tl;
+	int ret = 0;
+
+	chap->transaction = ctrl->transaction++;
+
+	/* DH-HMAC-CHAP Step 1: send negotiate */
+	dev_dbg(ctrl->device, "%s: qid %d send negotiate\n",
+		__func__, chap->qid);
+	ret = nvme_auth_set_dhchap_negotiate_data(ctrl, chap);
+	if (ret < 0) {
+		chap->error = ret;
+		return;
+	}
+	tl = ret;
+	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, tl, true);
+	if (ret) {
+		chap->error = ret;
+		return;
+	}
+
+	/* DH-HMAC-CHAP Step 2: receive challenge */
+	dev_dbg(ctrl->device, "%s: qid %d receive challenge\n",
+		__func__, chap->qid);
+
+	memset(chap->buf, 0, chap->buf_size);
+	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, chap->buf_size, false);
+	if (ret) {
+		dev_warn(ctrl->device,
+			 "qid %d failed to receive challenge, %s %d\n",
+			 chap->qid, ret < 0 ? "error" : "nvme status", ret);
+		chap->error = ret;
+		return;
+	}
+	ret = nvme_auth_receive_validate(ctrl, chap->qid, chap->buf, chap->transaction,
+					 NVME_AUTH_DHCHAP_MESSAGE_CHALLENGE);
+	if (ret) {
+		chap->status = ret;
+		chap->error = NVME_SC_AUTH_REQUIRED;
+		return;
+	}
+
+	ret = nvme_auth_process_dhchap_challenge(ctrl, chap);
+	if (ret) {
+		/* Invalid challenge parameters */
+		goto fail2;
+	}
+
+	dev_dbg(ctrl->device, "%s: qid %d host response\n",
+		__func__, chap->qid);
+	ret = nvme_auth_dhchap_setup_host_response(ctrl, chap);
+	if (ret)
+		goto fail2;
+
+	/* DH-HMAC-CHAP Step 3: send reply */
+	dev_dbg(ctrl->device, "%s: qid %d send reply\n",
+		__func__, chap->qid);
+	ret = nvme_auth_set_dhchap_reply_data(ctrl, chap);
+	if (ret < 0)
+		goto fail2;
+
+	tl = ret;
+	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, tl, true);
+	if (ret)
+		goto fail2;
+
+	/* DH-HMAC-CHAP Step 4: receive success1 */
+	dev_dbg(ctrl->device, "%s: qid %d receive success1\n",
+		__func__, chap->qid);
+
+	memset(chap->buf, 0, chap->buf_size);
+	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, chap->buf_size, false);
+	if (ret) {
+		dev_warn(ctrl->device,
+			 "qid %d failed to receive success1, %s %d\n",
+			 chap->qid, ret < 0 ? "error" : "nvme status", ret);
+		chap->error = ret;
+		return;
+	}
+	ret = nvme_auth_receive_validate(ctrl, chap->qid,
+					 chap->buf, chap->transaction,
+					 NVME_AUTH_DHCHAP_MESSAGE_SUCCESS1);
+	if (ret) {
+		chap->status = ret;
+		chap->error = NVME_SC_AUTH_REQUIRED;
+		return;
+	}
+
+	if (ctrl->opts->dhchap_ctrl_secret) {
+		dev_dbg(ctrl->device,
+			"%s: qid %d controller response\n",
+			__func__, chap->qid);
+		ret = nvme_auth_dhchap_setup_ctrl_response(ctrl, chap);
+		if (ret)
+			goto fail2;
+	}
+
+	ret = nvme_auth_process_dhchap_success1(ctrl, chap);
+	if (ret) {
+		/* Controller authentication failed */
+		goto fail2;
+	}
+
+	/* DH-HMAC-CHAP Step 5: send success2 */
+	dev_dbg(ctrl->device, "%s: qid %d send success2\n",
+		__func__, chap->qid);
+	tl = nvme_auth_set_dhchap_success2_data(ctrl, chap);
+	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, tl, true);
+	if (!ret) {
+		chap->error = 0;
+		return;
+	}
+
+fail2:
+	dev_dbg(ctrl->device, "%s: qid %d send failure2, status %x\n",
+		__func__, chap->qid, chap->status);
+	tl = nvme_auth_set_dhchap_failure2_data(ctrl, chap);
+	ret = nvme_auth_submit(ctrl, chap->qid, chap->buf, tl, true);
+	if (!ret)
+		ret = -EPROTO;
+	chap->error = ret;
+}
+
+int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid)
+{
+	struct nvme_dhchap_queue_context *chap;
+
+	if (!ctrl->host_key) {
+		dev_warn(ctrl->device, "qid %d: no key\n", qid);
+		return -ENOKEY;
+	}
+
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	/* Check if the context is already queued */
+	list_for_each_entry(chap, &ctrl->dhchap_auth_list, entry) {
+		WARN_ON(!chap->buf);
+		if (chap->qid == qid) {
+			dev_dbg(ctrl->device, "qid %d: re-using context\n", qid);
+			mutex_unlock(&ctrl->dhchap_auth_mutex);
+			flush_work(&chap->auth_work);
+			__nvme_auth_reset(chap);
+			queue_work(nvme_wq, &chap->auth_work);
+			return 0;
+		}
+	}
+	chap = kzalloc(sizeof(*chap), GFP_KERNEL);
+	if (!chap) {
+		mutex_unlock(&ctrl->dhchap_auth_mutex);
+		return -ENOMEM;
+	}
+	chap->qid = (qid == NVME_QID_ANY) ? 0 : qid;
+	chap->ctrl = ctrl;
+
+	/*
+	 * Allocate a large enough buffer for the entire negotiation:
+	 * 4k should be enough to ffdhe8192.
+	 */
+	chap->buf_size = 4096;
+	chap->buf = kzalloc(chap->buf_size, GFP_KERNEL);
+	if (!chap->buf) {
+		mutex_unlock(&ctrl->dhchap_auth_mutex);
+		kfree(chap);
+		return -ENOMEM;
+	}
+
+	INIT_WORK(&chap->auth_work, __nvme_auth_work);
+	list_add(&chap->entry, &ctrl->dhchap_auth_list);
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+	queue_work(nvme_wq, &chap->auth_work);
+	return 0;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_negotiate);
+
+int nvme_auth_wait(struct nvme_ctrl *ctrl, int qid)
+{
+	struct nvme_dhchap_queue_context *chap;
+	int ret;
+
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	list_for_each_entry(chap, &ctrl->dhchap_auth_list, entry) {
+		if (chap->qid != qid)
+			continue;
+		mutex_unlock(&ctrl->dhchap_auth_mutex);
+		flush_work(&chap->auth_work);
+		ret = chap->error;
+		return ret;
+	}
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+	return -ENXIO;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_wait);
+
+void nvme_auth_reset(struct nvme_ctrl *ctrl)
+{
+	struct nvme_dhchap_queue_context *chap;
+
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	list_for_each_entry(chap, &ctrl->dhchap_auth_list, entry) {
+		mutex_unlock(&ctrl->dhchap_auth_mutex);
+		flush_work(&chap->auth_work);
+		__nvme_auth_reset(chap);
+	}
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_reset);
+
+static void nvme_dhchap_auth_work(struct work_struct *work)
+{
+	struct nvme_ctrl *ctrl =
+		container_of(work, struct nvme_ctrl, dhchap_auth_work);
+	int ret, q;
+
+	/* Authenticate admin queue first */
+	ret = nvme_auth_negotiate(ctrl, 0);
+	if (ret) {
+		dev_warn(ctrl->device,
+			 "qid 0: error %d setting up authentication\n", ret);
+		return;
+	}
+	ret = nvme_auth_wait(ctrl, 0);
+	if (ret) {
+		dev_warn(ctrl->device,
+			 "qid 0: authentication failed\n");
+		return;
+	}
+
+	for (q = 1; q < ctrl->queue_count; q++) {
+		ret = nvme_auth_negotiate(ctrl, q);
+		if (ret) {
+			dev_warn(ctrl->device,
+				 "qid %d: error %d setting up authentication\n",
+				 q, ret);
+			break;
+		}
+	}
+
+	/*
+	 * Failure is a soft-state; credentials remain valid until
+	 * the controller terminates the connection.
+	 */
+}
+
+void nvme_auth_init_ctrl(struct nvme_ctrl *ctrl)
+{
+	INIT_LIST_HEAD(&ctrl->dhchap_auth_list);
+	INIT_WORK(&ctrl->dhchap_auth_work, nvme_dhchap_auth_work);
+	mutex_init(&ctrl->dhchap_auth_mutex);
+	if (!ctrl->opts)
+		return;
+	nvme_auth_generate_key(ctrl->opts->dhchap_secret, &ctrl->host_key);
+	nvme_auth_generate_key(ctrl->opts->dhchap_ctrl_secret, &ctrl->ctrl_key);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_init_ctrl);
+
+void nvme_auth_stop(struct nvme_ctrl *ctrl)
+{
+	struct nvme_dhchap_queue_context *chap = NULL, *tmp;
+
+	cancel_work_sync(&ctrl->dhchap_auth_work);
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	list_for_each_entry_safe(chap, tmp, &ctrl->dhchap_auth_list, entry)
+		cancel_work_sync(&chap->auth_work);
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_stop);
+
+void nvme_auth_free(struct nvme_ctrl *ctrl)
+{
+	struct nvme_dhchap_queue_context *chap = NULL, *tmp;
+
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	list_for_each_entry_safe(chap, tmp, &ctrl->dhchap_auth_list, entry) {
+		list_del_init(&chap->entry);
+		flush_work(&chap->auth_work);
+		__nvme_auth_free(chap);
+	}
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+	if (ctrl->host_key) {
+		nvme_auth_free_key(ctrl->host_key);
+		ctrl->host_key = NULL;
+	}
+	if (ctrl->ctrl_key) {
+		nvme_auth_free_key(ctrl->ctrl_key);
+		ctrl->ctrl_key = NULL;
+	}
+}
+EXPORT_SYMBOL_GPL(nvme_auth_free);
diff --git a/drivers/nvme/host/auth.h b/drivers/nvme/host/auth.h
new file mode 100644
index 000000000000..73d6ec63a5c8
--- /dev/null
+++ b/drivers/nvme/host/auth.h
@@ -0,0 +1,32 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2021 Hannes Reinecke, SUSE Software Solutions
+ */
+
+#ifndef _NVME_AUTH_H
+#define _NVME_AUTH_H
+
+#include <crypto/kpp.h>
+
+struct nvme_dhchap_key {
+	u8 *key;
+	size_t len;
+	u8 hash;
+};
+
+u32 nvme_auth_get_seqnum(void);
+const char *nvme_auth_dhgroup_name(u8 dhgroup_id);
+const char *nvme_auth_dhgroup_kpp(u8 dhgroup_id);
+u8 nvme_auth_dhgroup_id(const char *dhgroup_name);
+
+const char *nvme_auth_hmac_name(u8 hmac_id);
+const char *nvme_auth_digest_name(u8 hmac_id);
+size_t nvme_auth_hmac_hash_len(u8 hmac_id);
+u8 nvme_auth_hmac_id(const char *hmac_name);
+
+struct nvme_dhchap_key *nvme_auth_extract_key(unsigned char *secret,
+					      u8 key_hash);
+void nvme_auth_free_key(struct nvme_dhchap_key *key);
+u8 *nvme_auth_transform_key(struct nvme_dhchap_key *key, char *nqn);
+
+#endif /* _NVME_AUTH_H */
diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index cd6eac8e3dd6..dc10ba7cfc6c 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -24,6 +24,7 @@
 
 #include "nvme.h"
 #include "fabrics.h"
+#include "auth.h"
 
 #define CREATE_TRACE_POINTS
 #include "trace.h"
@@ -334,6 +335,7 @@ enum nvme_disposition {
 	COMPLETE,
 	RETRY,
 	FAILOVER,
+	AUTHENTICATE,
 };
 
 static inline enum nvme_disposition nvme_decide_disposition(struct request *req)
@@ -341,6 +343,9 @@ static inline enum nvme_disposition nvme_decide_disposition(struct request *req)
 	if (likely(nvme_req(req)->status == 0))
 		return COMPLETE;
 
+	if ((nvme_req(req)->status & 0x7ff) == NVME_SC_AUTH_REQUIRED)
+		return AUTHENTICATE;
+
 	if (blk_noretry_request(req) ||
 	    (nvme_req(req)->status & NVME_SC_DNR) ||
 	    nvme_req(req)->retries >= nvme_max_retries)
@@ -379,11 +384,13 @@ static inline void nvme_end_req(struct request *req)
 
 void nvme_complete_rq(struct request *req)
 {
+	struct nvme_ctrl *ctrl = nvme_req(req)->ctrl;
+
 	trace_nvme_complete_rq(req);
 	nvme_cleanup_cmd(req);
 
-	if (nvme_req(req)->ctrl->kas)
-		nvme_req(req)->ctrl->comp_seen = true;
+	if (ctrl->kas)
+		ctrl->comp_seen = true;
 
 	switch (nvme_decide_disposition(req)) {
 	case COMPLETE:
@@ -395,6 +402,14 @@ void nvme_complete_rq(struct request *req)
 	case FAILOVER:
 		nvme_failover_req(req);
 		return;
+	case AUTHENTICATE:
+#ifdef CONFIG_NVME_AUTH
+		queue_work(nvme_wq, &ctrl->dhchap_auth_work);
+		nvme_retry_req(req);
+#else
+		nvme_end_req(req);
+#endif
+		return;
 	}
 }
 EXPORT_SYMBOL_GPL(nvme_complete_rq);
@@ -706,7 +721,9 @@ bool __nvme_check_ready(struct nvme_ctrl *ctrl, struct request *rq,
 		switch (ctrl->state) {
 		case NVME_CTRL_CONNECTING:
 			if (blk_rq_is_passthrough(rq) && nvme_is_fabrics(req->cmd) &&
-			    req->cmd->fabrics.fctype == nvme_fabrics_type_connect)
+			    (req->cmd->fabrics.fctype == nvme_fabrics_type_connect ||
+			     req->cmd->fabrics.fctype == nvme_fabrics_type_auth_send ||
+			     req->cmd->fabrics.fctype == nvme_fabrics_type_auth_receive))
 				return true;
 			break;
 		default:
@@ -3563,6 +3580,108 @@ static ssize_t dctype_show(struct device *dev,
 }
 static DEVICE_ATTR_RO(dctype);
 
+#ifdef CONFIG_NVME_AUTH
+static ssize_t nvme_ctrl_dhchap_secret_show(struct device *dev,
+		struct device_attribute *attr, char *buf)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	struct nvmf_ctrl_options *opts = ctrl->opts;
+
+	if (!opts->dhchap_secret)
+		return sysfs_emit(buf, "none\n");
+	return sysfs_emit(buf, "%s\n", opts->dhchap_secret);
+}
+
+static ssize_t nvme_ctrl_dhchap_secret_store(struct device *dev,
+		struct device_attribute *attr, const char *buf, size_t count)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	struct nvmf_ctrl_options *opts = ctrl->opts;
+	char *dhchap_secret;
+
+	if (!ctrl->opts->dhchap_secret)
+		return -EINVAL;
+	if (count < 7)
+		return -EINVAL;
+	if (memcmp(buf, "DHHC-1:", 7))
+		return -EINVAL;
+
+	dhchap_secret = kzalloc(count + 1, GFP_KERNEL);
+	if (!dhchap_secret)
+		return -ENOMEM;
+	memcpy(dhchap_secret, buf, count);
+	nvme_auth_stop(ctrl);
+	if (strcmp(dhchap_secret, opts->dhchap_secret)) {
+		int ret;
+
+		ret = nvme_auth_generate_key(dhchap_secret, &ctrl->host_key);
+		if (ret)
+			return ret;
+		kfree(opts->dhchap_secret);
+		opts->dhchap_secret = dhchap_secret;
+		/* Key has changed; re-authentication with new key */
+		nvme_auth_reset(ctrl);
+	}
+	/* Start re-authentication */
+	dev_info(ctrl->device, "re-authenticating controller\n");
+	queue_work(nvme_wq, &ctrl->dhchap_auth_work);
+
+	return count;
+}
+DEVICE_ATTR(dhchap_secret, S_IRUGO | S_IWUSR,
+	nvme_ctrl_dhchap_secret_show, nvme_ctrl_dhchap_secret_store);
+
+static ssize_t nvme_ctrl_dhchap_ctrl_secret_show(struct device *dev,
+		struct device_attribute *attr, char *buf)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	struct nvmf_ctrl_options *opts = ctrl->opts;
+
+	if (!opts->dhchap_ctrl_secret)
+		return sysfs_emit(buf, "none\n");
+	return sysfs_emit(buf, "%s\n", opts->dhchap_ctrl_secret);
+}
+
+static ssize_t nvme_ctrl_dhchap_ctrl_secret_store(struct device *dev,
+		struct device_attribute *attr, const char *buf, size_t count)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	struct nvmf_ctrl_options *opts = ctrl->opts;
+	char *dhchap_secret;
+
+	if (!ctrl->opts->dhchap_ctrl_secret)
+		return -EINVAL;
+	if (count < 7)
+		return -EINVAL;
+	if (memcmp(buf, "DHHC-1:", 7))
+		return -EINVAL;
+
+	dhchap_secret = kzalloc(count + 1, GFP_KERNEL);
+	if (!dhchap_secret)
+		return -ENOMEM;
+	memcpy(dhchap_secret, buf, count);
+	nvme_auth_stop(ctrl);
+	if (strcmp(dhchap_secret, opts->dhchap_ctrl_secret)) {
+		int ret;
+
+		ret = nvme_auth_generate_key(dhchap_secret, &ctrl->ctrl_key);
+		if (ret)
+			return ret;
+		kfree(opts->dhchap_ctrl_secret);
+		opts->dhchap_ctrl_secret = dhchap_secret;
+		/* Key has changed; re-authentication with new key */
+		nvme_auth_reset(ctrl);
+	}
+	/* Start re-authentication */
+	dev_info(ctrl->device, "re-authenticating controller\n");
+	queue_work(nvme_wq, &ctrl->dhchap_auth_work);
+
+	return count;
+}
+DEVICE_ATTR(dhchap_ctrl_secret, S_IRUGO | S_IWUSR,
+	nvme_ctrl_dhchap_ctrl_secret_show, nvme_ctrl_dhchap_ctrl_secret_store);
+#endif
+
 static struct attribute *nvme_dev_attrs[] = {
 	&dev_attr_reset_controller.attr,
 	&dev_attr_rescan_controller.attr,
@@ -3586,6 +3705,10 @@ static struct attribute *nvme_dev_attrs[] = {
 	&dev_attr_kato.attr,
 	&dev_attr_cntrltype.attr,
 	&dev_attr_dctype.attr,
+#ifdef CONFIG_NVME_AUTH
+	&dev_attr_dhchap_secret.attr,
+	&dev_attr_dhchap_ctrl_secret.attr,
+#endif
 	NULL
 };
 
@@ -3609,6 +3732,10 @@ static umode_t nvme_dev_attrs_are_visible(struct kobject *kobj,
 		return 0;
 	if (a == &dev_attr_fast_io_fail_tmo.attr && !ctrl->opts)
 		return 0;
+#ifdef CONFIG_NVME_AUTH
+	if (a == &dev_attr_dhchap_secret.attr && !ctrl->opts)
+		return 0;
+#endif
 
 	return a->mode;
 }
@@ -4450,8 +4577,10 @@ static void nvme_handle_aen_notice(struct nvme_ctrl *ctrl, u32 result)
 		 * recovery actions from interfering with the controller's
 		 * firmware activation.
 		 */
-		if (nvme_change_ctrl_state(ctrl, NVME_CTRL_RESETTING))
+		if (nvme_change_ctrl_state(ctrl, NVME_CTRL_RESETTING)) {
+			nvme_auth_stop(ctrl);
 			queue_work(nvme_wq, &ctrl->fw_act_work);
+		}
 		break;
 #ifdef CONFIG_NVME_MULTIPATH
 	case NVME_AER_NOTICE_ANA:
@@ -4498,6 +4627,7 @@ EXPORT_SYMBOL_GPL(nvme_complete_async_event);
 void nvme_stop_ctrl(struct nvme_ctrl *ctrl)
 {
 	nvme_mpath_stop(ctrl);
+	nvme_auth_stop(ctrl);
 	nvme_stop_keep_alive(ctrl);
 	nvme_stop_failfast_work(ctrl);
 	flush_work(&ctrl->async_event_work);
@@ -4554,6 +4684,8 @@ static void nvme_free_ctrl(struct device *dev)
 
 	nvme_free_cels(ctrl);
 	nvme_mpath_uninit(ctrl);
+	nvme_auth_stop(ctrl);
+	nvme_auth_free(ctrl);
 	__free_page(ctrl->discard_page);
 
 	if (subsys) {
@@ -4644,6 +4776,7 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev,
 
 	nvme_fault_inject_init(&ctrl->fault_inject, dev_name(ctrl->device));
 	nvme_mpath_init_ctrl(ctrl);
+	nvme_auth_init_ctrl(ctrl);
 
 	return 0;
 out_free_name:
diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index 953e3076aab1..f5786a60931c 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -369,6 +369,7 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)
 	union nvme_result res;
 	struct nvmf_connect_data *data;
 	int ret;
+	u32 result;
 
 	cmd.connect.opcode = nvme_fabrics_command;
 	cmd.connect.fctype = nvme_fabrics_type_connect;
@@ -401,8 +402,25 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)
 		goto out_free_data;
 	}
 
-	ctrl->cntlid = le16_to_cpu(res.u16);
-
+	result = le32_to_cpu(res.u32);
+	ctrl->cntlid = result & 0xFFFF;
+	if ((result >> 16) & 2) {
+		/* Authentication required */
+		ret = nvme_auth_negotiate(ctrl, 0);
+		if (ret) {
+			dev_warn(ctrl->device,
+				 "qid 0: authentication setup failed\n");
+			ret = NVME_SC_AUTH_REQUIRED;
+			goto out_free_data;
+		}
+		ret = nvme_auth_wait(ctrl, 0);
+		if (ret)
+			dev_warn(ctrl->device,
+				 "qid 0: authentication failed\n");
+		else
+			dev_info(ctrl->device,
+				 "qid 0: authenticated\n");
+	}
 out_free_data:
 	kfree(data);
 	return ret;
@@ -435,6 +453,7 @@ int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, u16 qid)
 	struct nvmf_connect_data *data;
 	union nvme_result res;
 	int ret;
+	u32 result;
 
 	cmd.connect.opcode = nvme_fabrics_command;
 	cmd.connect.fctype = nvme_fabrics_type_connect;
@@ -460,6 +479,21 @@ int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, u16 qid)
 		nvmf_log_connect_error(ctrl, ret, le32_to_cpu(res.u32),
 				       &cmd, data);
 	}
+	result = le32_to_cpu(res.u32);
+	if ((result >> 16) & 2) {
+		/* Authentication required */
+		ret = nvme_auth_negotiate(ctrl, qid);
+		if (ret) {
+			dev_warn(ctrl->device,
+				 "qid %d: authentication setup failed\n", qid);
+			ret = NVME_SC_AUTH_REQUIRED;
+		} else {
+			ret = nvme_auth_wait(ctrl, qid);
+			if (ret)
+				dev_warn(ctrl->device,
+					 "qid %u: authentication failed\n", qid);
+		}
+	}
 	kfree(data);
 	return ret;
 }
@@ -552,6 +586,8 @@ static const match_table_t opt_tokens = {
 	{ NVMF_OPT_TOS,			"tos=%d"		},
 	{ NVMF_OPT_FAIL_FAST_TMO,	"fast_io_fail_tmo=%d"	},
 	{ NVMF_OPT_DISCOVERY,		"discovery"		},
+	{ NVMF_OPT_DHCHAP_SECRET,	"dhchap_secret=%s"	},
+	{ NVMF_OPT_DHCHAP_CTRL_SECRET,	"dhchap_ctrl_secret=%s"	},
 	{ NVMF_OPT_ERR,			NULL			}
 };
 
@@ -833,6 +869,34 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
 		case NVMF_OPT_DISCOVERY:
 			opts->discovery_nqn = true;
 			break;
+		case NVMF_OPT_DHCHAP_SECRET:
+			p = match_strdup(args);
+			if (!p) {
+				ret = -ENOMEM;
+				goto out;
+			}
+			if (strlen(p) < 11 || strncmp(p, "DHHC-1:", 7)) {
+				pr_err("Invalid DH-CHAP secret %s\n", p);
+				ret = -EINVAL;
+				goto out;
+			}
+			kfree(opts->dhchap_secret);
+			opts->dhchap_secret = p;
+			break;
+		case NVMF_OPT_DHCHAP_CTRL_SECRET:
+			p = match_strdup(args);
+			if (!p) {
+				ret = -ENOMEM;
+				goto out;
+			}
+			if (strlen(p) < 11 || strncmp(p, "DHHC-1:", 7)) {
+				pr_err("Invalid DH-CHAP secret %s\n", p);
+				ret = -EINVAL;
+				goto out;
+			}
+			kfree(opts->dhchap_ctrl_secret);
+			opts->dhchap_ctrl_secret = p;
+			break;
 		default:
 			pr_warn("unknown parameter or missing value '%s' in ctrl creation request\n",
 				p);
@@ -951,6 +1015,7 @@ void nvmf_free_options(struct nvmf_ctrl_options *opts)
 	kfree(opts->subsysnqn);
 	kfree(opts->host_traddr);
 	kfree(opts->host_iface);
+	kfree(opts->dhchap_secret);
 	kfree(opts);
 }
 EXPORT_SYMBOL_GPL(nvmf_free_options);
@@ -960,7 +1025,8 @@ EXPORT_SYMBOL_GPL(nvmf_free_options);
 				 NVMF_OPT_KATO | NVMF_OPT_HOSTNQN | \
 				 NVMF_OPT_HOST_ID | NVMF_OPT_DUP_CONNECT |\
 				 NVMF_OPT_DISABLE_SQFLOW | NVMF_OPT_DISCOVERY |\
-				 NVMF_OPT_FAIL_FAST_TMO)
+				 NVMF_OPT_FAIL_FAST_TMO | NVMF_OPT_DHCHAP_SECRET |\
+				 NVMF_OPT_DHCHAP_CTRL_SECRET)
 
 static struct nvme_ctrl *
 nvmf_create_ctrl(struct device *dev, const char *buf)
@@ -1196,7 +1262,14 @@ static void __exit nvmf_exit(void)
 	BUILD_BUG_ON(sizeof(struct nvmf_connect_command) != 64);
 	BUILD_BUG_ON(sizeof(struct nvmf_property_get_command) != 64);
 	BUILD_BUG_ON(sizeof(struct nvmf_property_set_command) != 64);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_send_command) != 64);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_receive_command) != 64);
 	BUILD_BUG_ON(sizeof(struct nvmf_connect_data) != 1024);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_negotiate_data) != 8);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_challenge_data) != 16);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_reply_data) != 16);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_success1_data) != 16);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_success2_data) != 16);
 }
 
 MODULE_LICENSE("GPL v2");
diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h
index 1e3a09cad961..15c142b277ab 100644
--- a/drivers/nvme/host/fabrics.h
+++ b/drivers/nvme/host/fabrics.h
@@ -68,6 +68,8 @@ enum {
 	NVMF_OPT_FAIL_FAST_TMO	= 1 << 20,
 	NVMF_OPT_HOST_IFACE	= 1 << 21,
 	NVMF_OPT_DISCOVERY	= 1 << 22,
+	NVMF_OPT_DHCHAP_SECRET	= 1 << 23,
+	NVMF_OPT_DHCHAP_CTRL_SECRET = 1 << 24,
 };
 
 /**
@@ -97,6 +99,9 @@ enum {
  * @max_reconnects: maximum number of allowed reconnect attempts before removing
  *              the controller, (-1) means reconnect forever, zero means remove
  *              immediately;
+ * @dhchap_secret: DH-HMAC-CHAP secret
+ * @dhchap_ctrl_secret: DH-HMAC-CHAP controller secret for bi-directional
+ *              authentication
  * @disable_sqflow: disable controller sq flow control
  * @hdr_digest: generate/verify header digest (TCP)
  * @data_digest: generate/verify data digest (TCP)
@@ -121,6 +126,8 @@ struct nvmf_ctrl_options {
 	unsigned int		kato;
 	struct nvmf_host	*host;
 	int			max_reconnects;
+	char			*dhchap_secret;
+	char			*dhchap_ctrl_secret;
 	bool			disable_sqflow;
 	bool			hdr_digest;
 	bool			data_digest;
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 1ea908d43e17..f110437a3190 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -323,6 +323,15 @@ struct nvme_ctrl {
 	struct work_struct ana_work;
 #endif
 
+#ifdef CONFIG_NVME_AUTH
+	struct work_struct dhchap_auth_work;
+	struct list_head dhchap_auth_list;
+	struct mutex dhchap_auth_mutex;
+	struct nvme_dhchap_key *host_key;
+	struct nvme_dhchap_key *ctrl_key;
+	u16 transaction;
+#endif
+
 	/* Power saving configuration */
 	u64 ps_max_latency_us;
 	bool apst_enabled;
@@ -928,6 +937,28 @@ static inline bool nvme_ctrl_sgl_supported(struct nvme_ctrl *ctrl)
 	return ctrl->sgls & ((1 << 0) | (1 << 1));
 }
 
+#ifdef CONFIG_NVME_AUTH
+void nvme_auth_init_ctrl(struct nvme_ctrl *ctrl);
+void nvme_auth_stop(struct nvme_ctrl *ctrl);
+int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid);
+int nvme_auth_wait(struct nvme_ctrl *ctrl, int qid);
+void nvme_auth_reset(struct nvme_ctrl *ctrl);
+void nvme_auth_free(struct nvme_ctrl *ctrl);
+int nvme_auth_generate_key(u8 *secret, struct nvme_dhchap_key **ret_key);
+#else
+static inline void nvme_auth_init_ctrl(struct nvme_ctrl *ctrl) {};
+static inline void nvme_auth_stop(struct nvme_ctrl *ctrl) {};
+static inline int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid)
+{
+	return -EPROTONOSUPPORT;
+}
+static inline int nvme_auth_wait(struct nvme_ctrl *ctrl, int qid)
+{
+	return NVME_SC_AUTH_REQUIRED;
+}
+static inline void nvme_auth_free(struct nvme_ctrl *ctrl) {};
+#endif
+
 u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 			 u8 opcode);
 int nvme_execute_passthru_rq(struct request *rq);
diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index d9f19d901313..bd63568c0068 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -1197,6 +1197,7 @@ static void nvme_rdma_error_recovery_work(struct work_struct *work)
 	struct nvme_rdma_ctrl *ctrl = container_of(work,
 			struct nvme_rdma_ctrl, err_work);
 
+	nvme_auth_stop(&ctrl->ctrl);
 	nvme_stop_keep_alive(&ctrl->ctrl);
 	flush_work(&ctrl->ctrl.async_event_work);
 	nvme_rdma_teardown_io_queues(ctrl, false);
diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index ad3a2bf2f1e9..be52902b5f3b 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -2174,6 +2174,7 @@ static void nvme_tcp_error_recovery_work(struct work_struct *work)
 				struct nvme_tcp_ctrl, err_work);
 	struct nvme_ctrl *ctrl = &tcp_ctrl->ctrl;
 
+	nvme_auth_stop(ctrl);
 	nvme_stop_keep_alive(ctrl);
 	flush_work(&ctrl->async_event_work);
 	nvme_tcp_teardown_io_queues(ctrl, false);
diff --git a/drivers/nvme/host/trace.c b/drivers/nvme/host/trace.c
index 2a89c5aa0790..1c36fcedea20 100644
--- a/drivers/nvme/host/trace.c
+++ b/drivers/nvme/host/trace.c
@@ -287,6 +287,34 @@ static const char *nvme_trace_fabrics_property_get(struct trace_seq *p, u8 *spc)
 	return ret;
 }
 
+static const char *nvme_trace_fabrics_auth_send(struct trace_seq *p, u8 *spc)
+{
+	const char *ret = trace_seq_buffer_ptr(p);
+	u8 spsp0 = spc[1];
+	u8 spsp1 = spc[2];
+	u8 secp = spc[3];
+	u32 tl = get_unaligned_le32(spc + 4);
+
+	trace_seq_printf(p, "spsp0=%02x, spsp1=%02x, secp=%02x, tl=%u",
+			 spsp0, spsp1, secp, tl);
+	trace_seq_putc(p, 0);
+	return ret;
+}
+
+static const char *nvme_trace_fabrics_auth_receive(struct trace_seq *p, u8 *spc)
+{
+	const char *ret = trace_seq_buffer_ptr(p);
+	u8 spsp0 = spc[1];
+	u8 spsp1 = spc[2];
+	u8 secp = spc[3];
+	u32 al = get_unaligned_le32(spc + 4);
+
+	trace_seq_printf(p, "spsp0=%02x, spsp1=%02x, secp=%02x, al=%u",
+			 spsp0, spsp1, secp, al);
+	trace_seq_putc(p, 0);
+	return ret;
+}
+
 static const char *nvme_trace_fabrics_common(struct trace_seq *p, u8 *spc)
 {
 	const char *ret = trace_seq_buffer_ptr(p);
@@ -306,6 +334,10 @@ const char *nvme_trace_parse_fabrics_cmd(struct trace_seq *p,
 		return nvme_trace_fabrics_connect(p, spc);
 	case nvme_fabrics_type_property_get:
 		return nvme_trace_fabrics_property_get(p, spc);
+	case nvme_fabrics_type_auth_send:
+		return nvme_trace_fabrics_auth_send(p, spc);
+	case nvme_fabrics_type_auth_receive:
+		return nvme_trace_fabrics_auth_receive(p, spc);
 	default:
 		return nvme_trace_fabrics_common(p, spc);
 	}
-- 
2.29.2

Re: [PATCH 06/11] nvme: Implement In-Band authentication

[Hannes Reinecke]

On 3/24/22 17:53, Chaitanya Kulkarni wrote:
> On 3/23/22 00:12, Hannes Reinecke wrote:
>> Implement NVMe-oF In-Band authentication according to NVMe TPAR 8006.
>> This patch adds two new fabric options 'dhchap_secret' to specify the
>> pre-shared key (in ASCII respresentation according to NVMe 2.0 section
>> 8.13.5.8 'Secret representation') and 'dhchap_ctrl_secret' to specify
>> the pre-shared controller key for bi-directional authentication of both
>> the host and the controller.
>> Re-authentication can be triggered by writing the PSK into the new
>> controller sysfs attribute 'dhchap_secret' or 'dhchap_ctrl_secret'.
>>
>> Signed-off-by: Hannes Reinecke <hare@suse.de>
>> ---
>>    drivers/nvme/host/Kconfig   |   11 +
>>    drivers/nvme/host/Makefile  |    1 +
>>    drivers/nvme/host/auth.c    | 1140 +++++++++++++++++++++++++++++++++++
>>    drivers/nvme/host/auth.h    |   32 +
>>    drivers/nvme/host/core.c    |  141 ++++-
>>    drivers/nvme/host/fabrics.c |   79 ++-
>>    drivers/nvme/host/fabrics.h |    7 +
>>    drivers/nvme/host/nvme.h    |   31 +
>>    drivers/nvme/host/rdma.c    |    1 +
>>    drivers/nvme/host/tcp.c     |    1 +
>>    drivers/nvme/host/trace.c   |   32 +
>>    11 files changed, 1469 insertions(+), 7 deletions(-)
>>    create mode 100644 drivers/nvme/host/auth.c
>>    create mode 100644 drivers/nvme/host/auth.h
>>
>> diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig
>> index d6d056963c06..dd0e91fb0615 100644
>> --- a/drivers/nvme/host/Kconfig
>> +++ b/drivers/nvme/host/Kconfig
>> @@ -91,3 +91,14 @@ config NVME_TCP
>>    	  from https://github.com/linux-nvme/nvme-cli.
>>    
>>    	  If unsure, say N.
>> +
>> +config NVME_AUTH
>> +	bool "NVM Express over Fabrics In-Band Authentication"
>> +	depends on NVME_CORE
>> +	select CRYPTO_HMAC
>> +	select CRYPTO_SHA256
>> +	select CRYPTO_SHA512
>> +	help
>> +	  This provides support for NVMe over Fabrics In-Band Authentication.
>> +
>> +	  If unsure, say N.
>> diff --git a/drivers/nvme/host/Makefile b/drivers/nvme/host/Makefile
>> index 476c5c988496..7755f5e3b281 100644
>> --- a/drivers/nvme/host/Makefile
>> +++ b/drivers/nvme/host/Makefile
>> @@ -15,6 +15,7 @@ nvme-core-$(CONFIG_NVME_MULTIPATH)	+= multipath.o
>>    nvme-core-$(CONFIG_BLK_DEV_ZONED)	+= zns.o
>>    nvme-core-$(CONFIG_FAULT_INJECTION_DEBUG_FS)	+= fault_inject.o
>>    nvme-core-$(CONFIG_NVME_HWMON)		+= hwmon.o
>> +nvme-core-$(CONFIG_NVME_AUTH)		+= auth.o
>>    
>>    nvme-y					+= pci.o
>>    
>> diff --git a/drivers/nvme/host/auth.c b/drivers/nvme/host/auth.c
>> new file mode 100644
>> index 000000000000..4bca4ba1ccea
>> --- /dev/null
>> +++ b/drivers/nvme/host/auth.c
>> @@ -0,0 +1,1140 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * Copyright (c) 2020 Hannes Reinecke, SUSE Linux
>> + */
>> +
>> +#include <linux/crc32.h>
>> +#include <linux/base64.h>
>> +#include <linux/prandom.h>
>> +#include <asm/unaligned.h>
>> +#include <crypto/hash.h>
>> +#include <crypto/dh.h>
>> +#include "nvme.h"
>> +#include "fabrics.h"
>> +#include "auth.h"
>> +
>> +static u32 nvme_dhchap_seqnum;
>> +static DEFINE_MUTEX(nvme_dhchap_mutex);
>> +
>> +struct nvme_dhchap_queue_context {
>> +	struct list_head entry;
>> +	struct work_struct auth_work;
>> +	struct nvme_ctrl *ctrl;
>> +	struct crypto_shash *shash_tfm;
>> +	void *buf;
>> +	size_t buf_size;
>> +	int qid;
>> +	int error;
>> +	u32 s1;
>> +	u32 s2;
>> +	u16 transaction;
>> +	u8 status;
>> +	u8 hash_id;
>> +	size_t hash_len;
>> +	u8 dhgroup_id;
>> +	u8 c1[64];
>> +	u8 c2[64];
>> +	u8 response[64];
>> +	u8 *host_response;
>> +};
>> +
>> +u32 nvme_auth_get_seqnum(void)
>> +{
>> +	u32 seqnum;
>> +
>> +	mutex_lock(&nvme_dhchap_mutex);
>> +	if (!nvme_dhchap_seqnum)
>> +		nvme_dhchap_seqnum = prandom_u32();
>> +	else {
>> +		nvme_dhchap_seqnum++;
>> +		if (!nvme_dhchap_seqnum)
>> +			nvme_dhchap_seqnum++;
>> +	}
>> +	seqnum = nvme_dhchap_seqnum;
>> +	mutex_unlock(&nvme_dhchap_mutex);
>> +	return seqnum;
>> +}
>> +EXPORT_SYMBOL_GPL(nvme_auth_get_seqnum);
>> +
>> +static struct nvme_auth_dhgroup_map {
>> +	u8 id;
>> +	const char name[16];
>> +	const char kpp[16];
>> +} dhgroup_map[] = {
>> +	{ .id = NVME_AUTH_DHGROUP_NULL,
>> +	  .name = "null", .kpp = "null" },
>> +	{ .id = NVME_AUTH_DHGROUP_2048,
>> +	  .name = "ffdhe2048", .kpp = "ffdhe2048(dh)" },
>> +	{ .id = NVME_AUTH_DHGROUP_3072,
>> +	  .name = "ffdhe3072", .kpp = "ffdhe3072(dh)" },
>> +	{ .id = NVME_AUTH_DHGROUP_4096,
>> +	  .name = "ffdhe4096", .kpp = "ffdhe4096(dh)" },
>> +	{ .id = NVME_AUTH_DHGROUP_6144,
>> +	  .name = "ffdhe6144", .kpp = "ffdhe6144(dh)" },
>> +	{ .id = NVME_AUTH_DHGROUP_8192,
>> +	  .name = "ffdhe8192", .kpp = "ffdhe8192(dh)" },
>> +};
>> +
>> +const char *nvme_auth_dhgroup_name(u8 dhgroup_id)
>> +{
>> +	int i;
>> +
>> +	for (i = 0; i < ARRAY_SIZE(dhgroup_map); i++) {
>> +		if (dhgroup_map[i].id == dhgroup_id)
>> +			return dhgroup_map[i].name;
>> +	}
>> +	return NULL;
>> +}
>> +EXPORT_SYMBOL_GPL(nvme_auth_dhgroup_name);
>> +
>> +const char *nvme_auth_dhgroup_kpp(u8 dhgroup_id)
>> +{
>> +	int i;
>> +
>> +	for (i = 0; i < ARRAY_SIZE(dhgroup_map); i++) {
>> +		if (dhgroup_map[i].id == dhgroup_id)
>> +			return dhgroup_map[i].kpp;
>> +	}
>> +	return NULL;
>> +}
>> +EXPORT_SYMBOL_GPL(nvme_auth_dhgroup_kpp);
>> +
>> +u8 nvme_auth_dhgroup_id(const char *dhgroup_name)
>> +{
>> +	int i;
>> +
>> +	for (i = 0; i < ARRAY_SIZE(dhgroup_map); i++) {
>> +		if (!strncmp(dhgroup_map[i].name, dhgroup_name,
>> +			     strlen(dhgroup_map[i].name)))
>> +			return dhgroup_map[i].id;
>> +	}
>> +	return NVME_AUTH_DHGROUP_INVALID;
>> +}
>> +EXPORT_SYMBOL_GPL(nvme_auth_dhgroup_id);
>> +
>> +static struct nvme_dhchap_hash_map {
>> +	int id;
>> +	int len;
>> +	const char hmac[15];
>> +	const char digest[15];
>> +} hash_map[] = {
>> +	{.id = NVME_AUTH_HASH_SHA256, .len = 32,
>> +	 .hmac = "hmac(sha256)", .digest = "sha256" },
>> +	{.id = NVME_AUTH_HASH_SHA384, .len = 48,
>> +	 .hmac = "hmac(sha384)", .digest = "sha384" },
>> +	{.id = NVME_AUTH_HASH_SHA512, .len = 64,
>> +	 .hmac = "hmac(sha512)", .digest = "sha512" },
>> +};
>> +
>> +const char *nvme_auth_hmac_name(u8 hmac_id)
>> +{
>> +	int i;
>> +
>> +	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
>> +		if (hash_map[i].id == hmac_id)
>> +			return hash_map[i].hmac;
>> +	}
>> +	return NULL;
>> +}
>> +EXPORT_SYMBOL_GPL(nvme_auth_hmac_name);
>> +
>> +const char *nvme_auth_digest_name(u8 hmac_id)
>> +{
>> +	int i;
>> +
>> +	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
>> +		if (hash_map[i].id == hmac_id)
>> +			return hash_map[i].digest;
>> +	}
>> +	return NULL;
>> +}
>> +EXPORT_SYMBOL_GPL(nvme_auth_digest_name);
>> +
>> +u8 nvme_auth_hmac_id(const char *hmac_name)
>> +{
>> +	int i;
>> +
>> +	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
>> +		if (!strncmp(hash_map[i].hmac, hmac_name,
>> +			     strlen(hash_map[i].hmac)))
>> +			return hash_map[i].id;
>> +	}
>> +	return NVME_AUTH_HASH_INVALID;
>> +}
>> +EXPORT_SYMBOL_GPL(nvme_auth_hmac_id);
>> +
>> +size_t nvme_auth_hmac_hash_len(u8 hmac_id)
>> +{
>> +	int i;
>> +
>> +	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
>> +		if (hash_map[i].id == hmac_id)
>> +			return hash_map[i].len;
>> +	}
>> +	return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(nvme_auth_hmac_hash_len);
> 
> we can remove loops in the above helpers and the id member in
> the struct with the help of the sparse array, why not :-
> 
Sure, good idea. Will be doing that.

[ .. ]
>> +static int nvme_auth_receive(struct nvme_ctrl *ctrl, int qid,
>> +		void *buf, size_t al)
>> +{
>> +	struct nvme_command cmd = {};
>> +	blk_mq_req_flags_t flags = nvme_auth_flags_from_qid(qid);
>> +	struct request_queue *q = nvme_auth_queue_from_qid(ctrl, qid);
>> +	int ret;
>> +
>> +	cmd.auth_receive.opcode = nvme_fabrics_command;
>> +	cmd.auth_receive.fctype = nvme_fabrics_type_auth_receive;
>> +	cmd.auth_receive.secp = NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER;
>> +	cmd.auth_receive.spsp0 = 0x01;
>> +	cmd.auth_receive.spsp1 = 0x01;
>> +	cmd.auth_receive.al = cpu_to_le32(al);
>> +
>> +	ret = __nvme_submit_sync_cmd(q, &cmd, NULL, buf, al, 0,
>> +				     qid == 0 ? NVME_QID_ANY : qid,
>> +				     0, flags);
>> +	if (ret > 0) {
>> +		dev_warn(ctrl->device,
>> +			 "qid %d auth_recv failed with status %x\n", qid, ret);
>> +		ret = -EIO;
>> +	} else if (ret < 0) {
>> +		dev_err(ctrl->device,
>> +			"qid %d auth_recv failed with error %d\n", qid, ret);
>> +	}
>> +
>> +	return ret;
>> +}
>> +
> 
> Why not use something like this ? It reduces the duplicate code and
> need for macros :-
> 
> static int nvme_auth_send_recv_common(bool send, struct nvme_ctrl *ctrl,
>                                         int qid, void *buf, size_t buflen)
> 
> {
>           strucy nvme_cmd cmd = { };
>           blk_mq_req_flags_t flags;
>           struct request_queue *q;
>           int ret;
> 
>           flags = qid == 0 ? 0 : BLK_MQ_REQ_RESERVED | BLK_MQ_REQ_NOWAIT;
>           q = qid == 0 ? ctrl->fabrics_q : ctrl->connect_q;
> 
>           /* auth send/recv share common offset for the various fields in
> cmd */
>           cmd.auth_send.opcode = nvme_fabrics_command;
>           cmd.auth_send.secp = NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER;
>           cmd.auth_send.spsp0 = 0x01;
>           cmd.auth_send.spsp1 = 0x01;
>           cmd.auth_send.al = cpu_to_le32(buflen);
>           cmd.auth_receive.fctype = send ? nvme_fabrics_type_auth_send :
>                                            nvme_fabrics_type_auth_receive;
> 
>           ret = nvme_submit_sync_cmd(q, &cmd, NULL, buf, buflen, 0,
>                                      qid == 0 ? NVME_QID_ANY : qid,
>                                      0, flags);
>           if (ret > 0)
>                   dev_warn(ctrl->device,
>                           "qid %d fctype 0x%x failed with status %d\n", qid,
>                           cmd.auth_send.fctype, ret);
>           else if (ret < 0)
>                   dev_err(ctrl->device,
>                           "qid %d fctype 0x%x failed with error %d\n", qid,
>                           cmd.auth_send.fctype, ret);
>           return ret;
> }
> 
> 
Naa. If we were going down that route we should add a new structure for 
nvme_auth_common, and use that for the common function.
And then the wrapper can just cast nvme_auth_send and nvme_auth_recv to 
the common structure.
I'll give it a go.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer

Re: [PATCH 06/11] nvme: Implement In-Band authentication

[Chaitanya Kulkarni]

On 3/23/22 00:12, Hannes Reinecke wrote:
> Implement NVMe-oF In-Band authentication according to NVMe TPAR 8006.
> This patch adds two new fabric options 'dhchap_secret' to specify the
> pre-shared key (in ASCII respresentation according to NVMe 2.0 section
> 8.13.5.8 'Secret representation') and 'dhchap_ctrl_secret' to specify
> the pre-shared controller key for bi-directional authentication of both
> the host and the controller.
> Re-authentication can be triggered by writing the PSK into the new
> controller sysfs attribute 'dhchap_secret' or 'dhchap_ctrl_secret'.
> 
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>   drivers/nvme/host/Kconfig   |   11 +
>   drivers/nvme/host/Makefile  |    1 +
>   drivers/nvme/host/auth.c    | 1140 +++++++++++++++++++++++++++++++++++
>   drivers/nvme/host/auth.h    |   32 +
>   drivers/nvme/host/core.c    |  141 ++++-
>   drivers/nvme/host/fabrics.c |   79 ++-
>   drivers/nvme/host/fabrics.h |    7 +
>   drivers/nvme/host/nvme.h    |   31 +
>   drivers/nvme/host/rdma.c    |    1 +
>   drivers/nvme/host/tcp.c     |    1 +
>   drivers/nvme/host/trace.c   |   32 +
>   11 files changed, 1469 insertions(+), 7 deletions(-)
>   create mode 100644 drivers/nvme/host/auth.c
>   create mode 100644 drivers/nvme/host/auth.h
> 
> diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig
> index d6d056963c06..dd0e91fb0615 100644
> --- a/drivers/nvme/host/Kconfig
> +++ b/drivers/nvme/host/Kconfig
> @@ -91,3 +91,14 @@ config NVME_TCP
>   	  from https://github.com/linux-nvme/nvme-cli.
>   
>   	  If unsure, say N.
> +
> +config NVME_AUTH
> +	bool "NVM Express over Fabrics In-Band Authentication"
> +	depends on NVME_CORE
> +	select CRYPTO_HMAC
> +	select CRYPTO_SHA256
> +	select CRYPTO_SHA512
> +	help
> +	  This provides support for NVMe over Fabrics In-Band Authentication.
> +
> +	  If unsure, say N.
> diff --git a/drivers/nvme/host/Makefile b/drivers/nvme/host/Makefile
> index 476c5c988496..7755f5e3b281 100644
> --- a/drivers/nvme/host/Makefile
> +++ b/drivers/nvme/host/Makefile
> @@ -15,6 +15,7 @@ nvme-core-$(CONFIG_NVME_MULTIPATH)	+= multipath.o
>   nvme-core-$(CONFIG_BLK_DEV_ZONED)	+= zns.o
>   nvme-core-$(CONFIG_FAULT_INJECTION_DEBUG_FS)	+= fault_inject.o
>   nvme-core-$(CONFIG_NVME_HWMON)		+= hwmon.o
> +nvme-core-$(CONFIG_NVME_AUTH)		+= auth.o
>   
>   nvme-y					+= pci.o
>   
> diff --git a/drivers/nvme/host/auth.c b/drivers/nvme/host/auth.c
> new file mode 100644
> index 000000000000..4bca4ba1ccea
> --- /dev/null
> +++ b/drivers/nvme/host/auth.c
> @@ -0,0 +1,1140 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Copyright (c) 2020 Hannes Reinecke, SUSE Linux
> + */
> +
> +#include <linux/crc32.h>
> +#include <linux/base64.h>
> +#include <linux/prandom.h>
> +#include <asm/unaligned.h>
> +#include <crypto/hash.h>
> +#include <crypto/dh.h>
> +#include "nvme.h"
> +#include "fabrics.h"
> +#include "auth.h"
> +
> +static u32 nvme_dhchap_seqnum;
> +static DEFINE_MUTEX(nvme_dhchap_mutex);
> +
> +struct nvme_dhchap_queue_context {
> +	struct list_head entry;
> +	struct work_struct auth_work;
> +	struct nvme_ctrl *ctrl;
> +	struct crypto_shash *shash_tfm;
> +	void *buf;
> +	size_t buf_size;
> +	int qid;
> +	int error;
> +	u32 s1;
> +	u32 s2;
> +	u16 transaction;
> +	u8 status;
> +	u8 hash_id;
> +	size_t hash_len;
> +	u8 dhgroup_id;
> +	u8 c1[64];
> +	u8 c2[64];
> +	u8 response[64];
> +	u8 *host_response;
> +};
> +
> +u32 nvme_auth_get_seqnum(void)
> +{
> +	u32 seqnum;
> +
> +	mutex_lock(&nvme_dhchap_mutex);
> +	if (!nvme_dhchap_seqnum)
> +		nvme_dhchap_seqnum = prandom_u32();
> +	else {
> +		nvme_dhchap_seqnum++;
> +		if (!nvme_dhchap_seqnum)
> +			nvme_dhchap_seqnum++;
> +	}
> +	seqnum = nvme_dhchap_seqnum;
> +	mutex_unlock(&nvme_dhchap_mutex);
> +	return seqnum;
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_get_seqnum);
> +
> +static struct nvme_auth_dhgroup_map {
> +	u8 id;
> +	const char name[16];
> +	const char kpp[16];
> +} dhgroup_map[] = {
> +	{ .id = NVME_AUTH_DHGROUP_NULL,
> +	  .name = "null", .kpp = "null" },
> +	{ .id = NVME_AUTH_DHGROUP_2048,
> +	  .name = "ffdhe2048", .kpp = "ffdhe2048(dh)" },
> +	{ .id = NVME_AUTH_DHGROUP_3072,
> +	  .name = "ffdhe3072", .kpp = "ffdhe3072(dh)" },
> +	{ .id = NVME_AUTH_DHGROUP_4096,
> +	  .name = "ffdhe4096", .kpp = "ffdhe4096(dh)" },
> +	{ .id = NVME_AUTH_DHGROUP_6144,
> +	  .name = "ffdhe6144", .kpp = "ffdhe6144(dh)" },
> +	{ .id = NVME_AUTH_DHGROUP_8192,
> +	  .name = "ffdhe8192", .kpp = "ffdhe8192(dh)" },
> +};
> +
> +const char *nvme_auth_dhgroup_name(u8 dhgroup_id)
> +{
> +	int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(dhgroup_map); i++) {
> +		if (dhgroup_map[i].id == dhgroup_id)
> +			return dhgroup_map[i].name;
> +	}
> +	return NULL;
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_dhgroup_name);
> +
> +const char *nvme_auth_dhgroup_kpp(u8 dhgroup_id)
> +{
> +	int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(dhgroup_map); i++) {
> +		if (dhgroup_map[i].id == dhgroup_id)
> +			return dhgroup_map[i].kpp;
> +	}
> +	return NULL;
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_dhgroup_kpp);
> +
> +u8 nvme_auth_dhgroup_id(const char *dhgroup_name)
> +{
> +	int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(dhgroup_map); i++) {
> +		if (!strncmp(dhgroup_map[i].name, dhgroup_name,
> +			     strlen(dhgroup_map[i].name)))
> +			return dhgroup_map[i].id;
> +	}
> +	return NVME_AUTH_DHGROUP_INVALID;
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_dhgroup_id);
> +
> +static struct nvme_dhchap_hash_map {
> +	int id;
> +	int len;
> +	const char hmac[15];
> +	const char digest[15];
> +} hash_map[] = {
> +	{.id = NVME_AUTH_HASH_SHA256, .len = 32,
> +	 .hmac = "hmac(sha256)", .digest = "sha256" },
> +	{.id = NVME_AUTH_HASH_SHA384, .len = 48,
> +	 .hmac = "hmac(sha384)", .digest = "sha384" },
> +	{.id = NVME_AUTH_HASH_SHA512, .len = 64,
> +	 .hmac = "hmac(sha512)", .digest = "sha512" },
> +};
> +
> +const char *nvme_auth_hmac_name(u8 hmac_id)
> +{
> +	int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
> +		if (hash_map[i].id == hmac_id)
> +			return hash_map[i].hmac;
> +	}
> +	return NULL;
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_hmac_name);
> +
> +const char *nvme_auth_digest_name(u8 hmac_id)
> +{
> +	int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
> +		if (hash_map[i].id == hmac_id)
> +			return hash_map[i].digest;
> +	}
> +	return NULL;
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_digest_name);
> +
> +u8 nvme_auth_hmac_id(const char *hmac_name)
> +{
> +	int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
> +		if (!strncmp(hash_map[i].hmac, hmac_name,
> +			     strlen(hash_map[i].hmac)))
> +			return hash_map[i].id;
> +	}
> +	return NVME_AUTH_HASH_INVALID;
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_hmac_id);
> +
> +size_t nvme_auth_hmac_hash_len(u8 hmac_id)
> +{
> +	int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
> +		if (hash_map[i].id == hmac_id)
> +			return hash_map[i].len;
> +	}
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_hmac_hash_len);

we can remove loops in the above helpers and the id member in
the struct with the help of the sparse array, why not :-

/* Defined hash functions for DH-HMAC-CHAP authentication */
enum { 

        NVME_AUTH_HASH_SHA256   = 0x01,
        NVME_AUTH_HASH_SHA384   = 0x02,
        NVME_AUTH_HASH_SHA512   = 0x03,
        NVME_AUTH_HASH_INVALID  = 0xff,
};

static struct nvme_dhchap_hash_map {
	int len;
	const char *hmac;
	const char *digest;
} hash_map[] = {
	[NVME_AUTH_HASH_SHA256] = {	.len = 32,
					.hmac = "hmac(sha256)",
					.digest = "sha256"
				},
	[NVME_AUTH_HASH_SHA384] = {	.len = 48,
					.hmac = "hmac(sha384)",
					.digest = "sha384"
				},
	[NVME_AUTH_HASH_SHA512] = {	.len = 64,
					.hmac = "hmac(sha512)",
					.digest = "sha512"
				},
};

const char *nvme_auth_hmac_name(unsigned int hmac_id)
{
	if ((hmac_id > ARRAY_SIZE(hash_map)) || !hash_map[hmac_id].hmac)
		return NULL;
	return hash_map[hmac_id].hmac;
}

const char *nvme_auth_digest_name(unsigned int hmac_id)
{
	if ((hmac_id > ARRAY_SIZE(hash_map)) || !hash_map[hmac_id].hmac)
		return NULL;
	return hash_map[hmac_id].digest;
}

size_t nvme_auth_hmac_hash_len(unsigned int hmac_id)
{
	if ((hmac_id > ARRAY_SIZE(hash_map)) || !hash_map[hmac_id].hmac)
		return 0;
	return hash_map[hmac_id].len;
}

unsigned int nvme_auth_hmac_id(const char *hmac_name)
{
	int i;

	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
		if (!hash_map[i].hmac)
			continue;
		if (!strncmp(hash_map[i].hmac, hmac_name,
			     strlen(hash_map[i].hmac)))
			return i;
	}
	return NVME_AUTH_HASH_INVALID;
}

See [1] that has a test program execution, same can be applied to
nvme_dhchap_hash_map.

> +
> +struct nvme_dhchap_key *nvme_auth_extract_key(unsigned char *secret,
> +					      u8 key_hash)
> +{
> +	struct nvme_dhchap_key *key;
> +	unsigned char *p;
> +	u32 crc;
> +	int ret, key_len;
> +	size_t allocated_len = strlen(secret);
> +
> +	/* Secret might be affixed with a ':' */
> +	p = strrchr(secret, ':');
> +	if (p)
> +		allocated_len = p - secret;
> +	key = kzalloc(sizeof(*key), GFP_KERNEL);
> +	if (!key)
> +		return ERR_PTR(-ENOMEM);
> +	key->key = kzalloc(allocated_len, GFP_KERNEL);
> +	if (!key->key) {
> +		ret = -ENOMEM;
> +		goto out_free_key;
> +	}
> +
> +	key_len = base64_decode(secret, allocated_len, key->key);
> +	if (key_len < 0) {
> +		pr_debug("base64 key decoding error %d\n",
> +			 key_len);
> +		ret = key_len;
> +		goto out_free_secret;
> +	}
> +
> +	if (key_len != 36 && key_len != 52 &&
> +	    key_len != 68) {
> +		pr_err("Invalid DH-HMAC-CHAP key len %d\n",
> +		       key_len);
> +		ret = -EINVAL;
> +		goto out_free_secret;
> +	}
> +
> +	if (key_hash > 0 &&
> +	    (key_len - 4) != nvme_auth_hmac_hash_len(key_hash)) {
> +		pr_err("Invalid DH-HMAC-CHAP key len %d for %s\n", key_len,
> +		       nvme_auth_hmac_name(key_hash));
> +		ret = -EINVAL;
> +		goto out_free_secret;
> +	}
> +
> +	/* The last four bytes is the CRC in little-endian format */
> +	key_len -= 4;
> +	/*
> +	 * The linux implementation doesn't do pre- and post-increments,
> +	 * so we have to do it manually.
> +	 */
> +	crc = ~crc32(~0, key->key, key_len);
> +
> +	if (get_unaligned_le32(key->key + key_len) != crc) {
> +		pr_err("DH-HMAC-CHAP key crc mismatch (key %08x, crc %08x)\n",
> +		       get_unaligned_le32(key->key + key_len), crc);
> +		ret = -EKEYREJECTED;
> +		goto out_free_secret;
> +	}
> +	key->len = key_len;
> +	key->hash = key_hash;
> +	return key;
> +out_free_secret:
> +	kfree_sensitive(key->key);
> +out_free_key:
> +	kfree(key);
> +	return ERR_PTR(ret);
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_extract_key);
> +
> +void nvme_auth_free_key(struct nvme_dhchap_key *key)
> +{
> +	if (!key)
> +		return;
> +	kfree_sensitive(key->key);
> +	kfree(key);
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_free_key);
> +
> +u8 *nvme_auth_transform_key(struct nvme_dhchap_key *key, char *nqn)
> +{
> +	const char *hmac_name = nvme_auth_hmac_name(key->hash);
> +	struct crypto_shash *key_tfm;
> +	struct shash_desc *shash;
> +	u8 *transformed_key;
> +	int ret;
> +
> +	if (key->hash == 0) {
> +		transformed_key = kmemdup(key->key, key->len, GFP_KERNEL);
> +		return transformed_key ? transformed_key : ERR_PTR(-ENOMEM);
> +	}
> +
> +	if (!key || !key->key) {
> +		pr_warn("No key specified\n");
> +		return ERR_PTR(-ENOKEY);
> +	}
> +	if (!hmac_name) {
> +		pr_warn("Invalid key hash id %d\n", key->hash);
> +		return ERR_PTR(-EINVAL);
> +	}
> +
> +	key_tfm = crypto_alloc_shash(hmac_name, 0, 0);
> +	if (IS_ERR(key_tfm))
> +		return (u8 *)key_tfm;
> +
> +	shash = kmalloc(sizeof(struct shash_desc) +
> +			crypto_shash_descsize(key_tfm),
> +			GFP_KERNEL);
> +	if (!shash) {
> +		ret = -ENOMEM;
> +		goto out_free_key;
> +	}
> +
> +	transformed_key = kzalloc(crypto_shash_digestsize(key_tfm), GFP_KERNEL);
> +	if (!transformed_key) {
> +		ret = -ENOMEM;
> +		goto out_free_shash;
> +	}
> +
> +	shash->tfm = key_tfm;
> +	ret = crypto_shash_setkey(key_tfm, key->key, key->len);
> +	if (ret < 0)
> +		goto out_free_shash;
> +	ret = crypto_shash_init(shash);
> +	if (ret < 0)
> +		goto out_free_shash;
> +	ret = crypto_shash_update(shash, nqn, strlen(nqn));
> +	if (ret < 0)
> +		goto out_free_shash;
> +	ret = crypto_shash_update(shash, "NVMe-over-Fabrics", 17);
> +	if (ret < 0)
> +		goto out_free_shash;
> +	ret = crypto_shash_final(shash, transformed_key);
> +out_free_shash:
> +	kfree(shash);
> +out_free_key:
> +	crypto_free_shash(key_tfm);
> +	if (ret < 0) {
> +		kfree_sensitive(transformed_key);
> +		return ERR_PTR(ret);
> +	}
> +	return transformed_key;
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_transform_key);
> +
> +#define nvme_auth_flags_from_qid(qid) \
> +	(qid == 0) ? 0 : BLK_MQ_REQ_NOWAIT | BLK_MQ_REQ_RESERVED
> +#define nvme_auth_queue_from_qid(ctrl, qid) \
> +	(qid == 0) ? (ctrl)->fabrics_q : (ctrl)->connect_q
> +
> +static int nvme_auth_send(struct nvme_ctrl *ctrl, int qid,
> +		void *data, size_t tl)
> +{
> +	struct nvme_command cmd = {};
> +	blk_mq_req_flags_t flags = nvme_auth_flags_from_qid(qid);
> +	struct request_queue *q = nvme_auth_queue_from_qid(ctrl, qid);
> +	int ret;
> +
> +	cmd.auth_send.opcode = nvme_fabrics_command;
> +	cmd.auth_send.fctype = nvme_fabrics_type_auth_send;
> +	cmd.auth_send.secp = NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER;
> +	cmd.auth_send.spsp0 = 0x01;
> +	cmd.auth_send.spsp1 = 0x01;
> +	cmd.auth_send.tl = cpu_to_le32(tl);
> +
> +	ret = __nvme_submit_sync_cmd(q, &cmd, NULL, data, tl, 0,
> +				     qid == 0 ? NVME_QID_ANY : qid,
> +				     0, flags);
> +	if (ret > 0)
> +		dev_warn(ctrl->device,
> +			"qid %d auth_send failed with status %d\n", qid, ret);
> +	else if (ret < 0)
> +		dev_err(ctrl->device,
> +			"qid %d auth_send failed with error %d\n", qid, ret);
> +	return ret;
> +}
> +
> +static int nvme_auth_receive(struct nvme_ctrl *ctrl, int qid,
> +		void *buf, size_t al)
> +{
> +	struct nvme_command cmd = {};
> +	blk_mq_req_flags_t flags = nvme_auth_flags_from_qid(qid);
> +	struct request_queue *q = nvme_auth_queue_from_qid(ctrl, qid);
> +	int ret;
> +
> +	cmd.auth_receive.opcode = nvme_fabrics_command;
> +	cmd.auth_receive.fctype = nvme_fabrics_type_auth_receive;
> +	cmd.auth_receive.secp = NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER;
> +	cmd.auth_receive.spsp0 = 0x01;
> +	cmd.auth_receive.spsp1 = 0x01;
> +	cmd.auth_receive.al = cpu_to_le32(al);
> +
> +	ret = __nvme_submit_sync_cmd(q, &cmd, NULL, buf, al, 0,
> +				     qid == 0 ? NVME_QID_ANY : qid,
> +				     0, flags);
> +	if (ret > 0) {
> +		dev_warn(ctrl->device,
> +			 "qid %d auth_recv failed with status %x\n", qid, ret);
> +		ret = -EIO;
> +	} else if (ret < 0) {
> +		dev_err(ctrl->device,
> +			"qid %d auth_recv failed with error %d\n", qid, ret);
> +	}
> +
> +	return ret;
> +}
> +

Why not use something like this ? It reduces the duplicate code and
need for macros :-

static int nvme_auth_send_recv_common(bool send, struct nvme_ctrl *ctrl,
                                       int qid, void *buf, size_t buflen)

{
         strucy nvme_cmd cmd = { };
         blk_mq_req_flags_t flags;
         struct request_queue *q;
         int ret;

         flags = qid == 0 ? 0 : BLK_MQ_REQ_RESERVED | BLK_MQ_REQ_NOWAIT;
         q = qid == 0 ? ctrl->fabrics_q : ctrl->connect_q;

         /* auth send/recv share common offset for the various fields in 
cmd */
         cmd.auth_send.opcode = nvme_fabrics_command;
         cmd.auth_send.secp = NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER;
         cmd.auth_send.spsp0 = 0x01;
         cmd.auth_send.spsp1 = 0x01;
         cmd.auth_send.al = cpu_to_le32(buflen);
         cmd.auth_receive.fctype = send ? nvme_fabrics_type_auth_send :
                                          nvme_fabrics_type_auth_receive;

         ret = nvme_submit_sync_cmd(q, &cmd, NULL, buf, buflen, 0,
                                    qid == 0 ? NVME_QID_ANY : qid,
                                    0, flags);
         if (ret > 0)
                 dev_warn(ctrl->device,
                         "qid %d fctype 0x%x failed with status %d\n", qid,
                         cmd.auth_send.fctype, ret);
         else if (ret < 0)
                 dev_err(ctrl->device,
                         "qid %d fctype 0x%x failed with error %d\n", qid,
                         cmd.auth_send.fctype, ret);
         return ret;
}


static int nvme_auth_send(struct nvme_ctrl *c, int qid, void *data, 
size_t tl)
{
         return nvme_auth_send_recv_common(true, c, qid, data, tl);
}

static int nvme_auth_receive(struct nvme_ctrl *ctrl, int qid,
                 void *buf, size_t al)
{
         return nvme_auth_send_recv_common(false, c, qid, data, tl);

}



> +static int nvme_auth_receive_validate(struct nvme_ctrl *ctrl, int qid,
> +		struct nvmf_auth_dhchap_failure_data *data,
> +		u16 transaction, u8 expected_msg)
> +{
> +	dev_dbg(ctrl->device, "%s: qid %d auth_type %d auth_id %x\n",
> +		__func__, qid, data->auth_type, data->auth_id);
> +
> +	if (data->auth_type == NVME_AUTH_COMMON_MESSAGES &&
> +	    data->auth_id == NVME_AUTH_DHCHAP_MESSAGE_FAILURE1) {
> +		return data->rescode_exp;
> +	}
> +	if (data->auth_type != NVME_AUTH_DHCHAP_MESSAGES ||
> +	    data->auth_id != expected_msg) {
> +		dev_warn(ctrl->device,
> +			 "qid %d invalid message %02x/%02x\n",
> +			 qid, data->auth_type, data->auth_id);
> +		return NVME_AUTH_DHCHAP_FAILURE_INCORRECT_MESSAGE;
> +	}
> +	if (le16_to_cpu(data->t_id) != transaction) {
> +		dev_warn(ctrl->device,
> +			 "qid %d invalid transaction ID %d\n",
> +			 qid, le16_to_cpu(data->t_id));
> +		return NVME_AUTH_DHCHAP_FAILURE_INCORRECT_MESSAGE;
> +	}
> +	return 0;
> +}
> +
[...]


[1] # cat  a.c
#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <stdlib.h>
#include <fcntl.h>
#include <unistd.h>
#include <string.h>
#include <sys/uio.h>
#include <stdlib.h>

#define COUNT (10)
#define LEN (1024)

#define ARRAY_SIZE(arr) ((sizeof(arr) / sizeof(arr[0])))

/* Defined hash functions for DH-HMAC-CHAP authentication */
enum { 

        NVME_AUTH_HASH_SHA256   = 0x01,
        NVME_AUTH_HASH_SHA384   = 0x02,
        NVME_AUTH_HASH_SHA512   = 0x03,
        NVME_AUTH_HASH_INVALID  = 0xff,
};

static struct nvme_dhchap_hash_map {
	int len;
	const char *hmac;
	const char *digest;
} hash_map[] = {
	[NVME_AUTH_HASH_SHA256] = {	.len = 32,
					.hmac = "hmac(sha256)",
					.digest = "sha256"
				},
	[NVME_AUTH_HASH_SHA384] = {	.len = 48,
					.hmac = "hmac(sha384)",
					.digest = "sha384"
				},
	[NVME_AUTH_HASH_SHA512] = {	.len = 64,
					.hmac = "hmac(sha512)",
					.digest = "sha512"
				},
};

const char *nvme_auth_hmac_name(unsigned int hmac_id)
{
	if ((hmac_id > ARRAY_SIZE(hash_map)) || !hash_map[hmac_id].hmac)
		return NULL;
	return hash_map[hmac_id].hmac;
}

const char *nvme_auth_digest_name(unsigned int hmac_id)
{
	if ((hmac_id > ARRAY_SIZE(hash_map)) || !hash_map[hmac_id].hmac)
		return NULL;
	return hash_map[hmac_id].digest;
}

size_t nvme_auth_hmac_hash_len(unsigned int hmac_id)
{
	if ((hmac_id > ARRAY_SIZE(hash_map)) || !hash_map[hmac_id].hmac)
		return 0;
	return hash_map[hmac_id].len;
}

unsigned int nvme_auth_hmac_id(const char *hmac_name)
{
	int i;

	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
		if (!hash_map[i].hmac)
			continue;
		if (!strncmp(hash_map[i].hmac, hmac_name,
			     strlen(hash_map[i].hmac)))
			return i;
	}
	return NVME_AUTH_HASH_INVALID;
}

int main(void)
{
	printf("-------------------------------------------------------\n");
	printf("hmac_id = 0x%s name 0x%x\n", "hmac(sha256)",
			nvme_auth_hmac_id("hmac(sha256)"));
	printf("hmac_id = 0x%s name 0x%x\n", "hmac(sha384)",
			nvme_auth_hmac_id("hmac(sha384)"));
	printf("hmac_id = 0x%s name 0x%x\n", "hmac(sha512)",
			nvme_auth_hmac_id("hmac(sha512)"));
	printf("hmac_id = 0x%s name 0x%x\n", "",
			nvme_auth_hmac_id(""));
	printf("hmac_id = 0x%s name 0x%x\n", "asdfadsf",
			nvme_auth_hmac_id("asdfadsf"));

	printf("-------------------------------------------------------\n");
	printf("hmac_id = 0x%x name %s\n", NVME_AUTH_HASH_SHA256,
			nvme_auth_hmac_name(NVME_AUTH_HASH_SHA256));
	printf("hmac_id = 0x%x name %s\n", NVME_AUTH_HASH_SHA384,
			nvme_auth_hmac_name(NVME_AUTH_HASH_SHA384));
	printf("hmac_id = 0x%x name %s\n", NVME_AUTH_HASH_SHA512,
			nvme_auth_hmac_name(NVME_AUTH_HASH_SHA512));
	printf("hmac_id = 0x%x name %s\n", 0x05,
			nvme_auth_hmac_name(0x05));
	printf("hmac_id = 0x%x name %s\n", 0x0ff,
			nvme_auth_hmac_name(0xff));
	printf("hmac_id = 0x%x name %s\n", 0x0ff1,
			nvme_auth_hmac_name(0xff1));

	printf("-------------------------------------------------------\n");
	printf("hmac_id = 0x%x digest %s\n", NVME_AUTH_HASH_SHA256,
			nvme_auth_digest_name(NVME_AUTH_HASH_SHA256));
	printf("hmac_id = 0x%x digest %s\n", NVME_AUTH_HASH_SHA384,
			nvme_auth_digest_name(NVME_AUTH_HASH_SHA384));
	printf("hmac_id = 0x%x digest %s\n", NVME_AUTH_HASH_SHA512,
			nvme_auth_digest_name(NVME_AUTH_HASH_SHA512));
	printf("hmac_id = 0x%x digest %s\n", 0x05,
			nvme_auth_digest_name(0x05));
	printf("hmac_id = 0x%x digest %s\n", 0x0ff,
			nvme_auth_digest_name(0xff));
	printf("hmac_id = 0x%x digest %s\n", 0x0ff1,
			nvme_auth_digest_name(0xff1));

	printf("-------------------------------------------------------\n");
	printf("hmac_id = 0x%x len %d\n", NVME_AUTH_HASH_SHA256,
			nvme_auth_hmac_hash_len(NVME_AUTH_HASH_SHA256));
	printf("hmac_id = 0x%x len %d\n", NVME_AUTH_HASH_SHA384,
			nvme_auth_hmac_hash_len(NVME_AUTH_HASH_SHA384));
	printf("hmac_id = 0x%x len %d\n", NVME_AUTH_HASH_SHA512,
			nvme_auth_hmac_hash_len(NVME_AUTH_HASH_SHA512));
	printf("hmac_id = 0x%x len %d\n", 0x05,
			nvme_auth_hmac_hash_len(0x05));
	printf("hmac_id = 0x%x len %d\n", 0x0ff,
			nvme_auth_hmac_hash_len(0xff));
	printf("hmac_id = 0x%x len %d\n", 0x0ff1,
			nvme_auth_hmac_hash_len(0xff1));

         return 0;
}
nvme (nvme-5.18) # gcc a.c
nvme (nvme-5.18) # ./a.out
-------------------------------------------------------
hmac_id = 0xhmac(sha256) name 0x1
hmac_id = 0xhmac(sha384) name 0x2
hmac_id = 0xhmac(sha512) name 0x3
hmac_id = 0x name 0xff
hmac_id = 0xasdfadsf name 0xff
-------------------------------------------------------
hmac_id = 0x1 name hmac(sha256)
hmac_id = 0x2 name hmac(sha384)
hmac_id = 0x3 name hmac(sha512)
hmac_id = 0x5 name (null)
hmac_id = 0xff name (null)
hmac_id = 0xff1 name (null)
-------------------------------------------------------
hmac_id = 0x1 digest sha256
hmac_id = 0x2 digest sha384
hmac_id = 0x3 digest sha512
hmac_id = 0x5 digest (null)
hmac_id = 0xff digest (null)
hmac_id = 0xff1 digest (null)
-------------------------------------------------------
hmac_id = 0x1 len 32
hmac_id = 0x2 len 48
hmac_id = 0x3 len 64
hmac_id = 0x5 len 0
hmac_id = 0xff len 0
hmac_id = 0xff1 len 0

[PATCH 06/11] nvme: Implement In-Band authentication

[Hannes Reinecke]

Implement NVMe-oF In-Band authentication according to NVMe TPAR 8006.
This patch adds two new fabric options 'dhchap_secret' to specify the
pre-shared key (in ASCII respresentation according to NVMe 2.0 section
8.13.5.8 'Secret representation') and 'dhchap_ctrl_secret' to specify
the pre-shared controller key for bi-directional authentication of both
the host and the controller.
Re-authentication can be triggered by writing the PSK into the new
controller sysfs attribute 'dhchap_secret' or 'dhchap_ctrl_secret'.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/nvme/host/Kconfig   |   11 +
 drivers/nvme/host/Makefile  |    1 +
 drivers/nvme/host/auth.c    | 1140 +++++++++++++++++++++++++++++++++++
 drivers/nvme/host/auth.h    |   32 +
 drivers/nvme/host/core.c    |  141 ++++-
 drivers/nvme/host/fabrics.c |   79 ++-
 drivers/nvme/host/fabrics.h |    7 +
 drivers/nvme/host/nvme.h    |   31 +
 drivers/nvme/host/rdma.c    |    1 +
 drivers/nvme/host/tcp.c     |    1 +
 drivers/nvme/host/trace.c   |   32 +
 11 files changed, 1469 insertions(+), 7 deletions(-)
 create mode 100644 drivers/nvme/host/auth.c
 create mode 100644 drivers/nvme/host/auth.h

diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig
index d6d056963c06..dd0e91fb0615 100644
--- a/drivers/nvme/host/Kconfig
+++ b/drivers/nvme/host/Kconfig
@@ -91,3 +91,14 @@ config NVME_TCP
 	  from https://github.com/linux-nvme/nvme-cli.
 
 	  If unsure, say N.
+
+config NVME_AUTH
+	bool "NVM Express over Fabrics In-Band Authentication"
+	depends on NVME_CORE
+	select CRYPTO_HMAC
+	select CRYPTO_SHA256
+	select CRYPTO_SHA512
+	help
+	  This provides support for NVMe over Fabrics In-Band Authentication.
+
+	  If unsure, say N.
diff --git a/drivers/nvme/host/Makefile b/drivers/nvme/host/Makefile
index 476c5c988496..7755f5e3b281 100644
--- a/drivers/nvme/host/Makefile
+++ b/drivers/nvme/host/Makefile
@@ -15,6 +15,7 @@ nvme-core-$(CONFIG_NVME_MULTIPATH)	+= multipath.o
 nvme-core-$(CONFIG_BLK_DEV_ZONED)	+= zns.o
 nvme-core-$(CONFIG_FAULT_INJECTION_DEBUG_FS)	+= fault_inject.o
 nvme-core-$(CONFIG_NVME_HWMON)		+= hwmon.o
+nvme-core-$(CONFIG_NVME_AUTH)		+= auth.o
 
 nvme-y					+= pci.o
 
diff --git a/drivers/nvme/host/auth.c b/drivers/nvme/host/auth.c
new file mode 100644
index 000000000000..4bca4ba1ccea
--- /dev/null
+++ b/drivers/nvme/host/auth.c
@@ -0,0 +1,1140 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2020 Hannes Reinecke, SUSE Linux
+ */
+
+#include <linux/crc32.h>
+#include <linux/base64.h>
+#include <linux/prandom.h>
+#include <asm/unaligned.h>
+#include <crypto/hash.h>
+#include <crypto/dh.h>
+#include "nvme.h"
+#include "fabrics.h"
+#include "auth.h"
+
+static u32 nvme_dhchap_seqnum;
+static DEFINE_MUTEX(nvme_dhchap_mutex);
+
+struct nvme_dhchap_queue_context {
+	struct list_head entry;
+	struct work_struct auth_work;
+	struct nvme_ctrl *ctrl;
+	struct crypto_shash *shash_tfm;
+	void *buf;
+	size_t buf_size;
+	int qid;
+	int error;
+	u32 s1;
+	u32 s2;
+	u16 transaction;
+	u8 status;
+	u8 hash_id;
+	size_t hash_len;
+	u8 dhgroup_id;
+	u8 c1[64];
+	u8 c2[64];
+	u8 response[64];
+	u8 *host_response;
+};
+
+u32 nvme_auth_get_seqnum(void)
+{
+	u32 seqnum;
+
+	mutex_lock(&nvme_dhchap_mutex);
+	if (!nvme_dhchap_seqnum)
+		nvme_dhchap_seqnum = prandom_u32();
+	else {
+		nvme_dhchap_seqnum++;
+		if (!nvme_dhchap_seqnum)
+			nvme_dhchap_seqnum++;
+	}
+	seqnum = nvme_dhchap_seqnum;
+	mutex_unlock(&nvme_dhchap_mutex);
+	return seqnum;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_get_seqnum);
+
+static struct nvme_auth_dhgroup_map {
+	u8 id;
+	const char name[16];
+	const char kpp[16];
+} dhgroup_map[] = {
+	{ .id = NVME_AUTH_DHGROUP_NULL,
+	  .name = "null", .kpp = "null" },
+	{ .id = NVME_AUTH_DHGROUP_2048,
+	  .name = "ffdhe2048", .kpp = "ffdhe2048(dh)" },
+	{ .id = NVME_AUTH_DHGROUP_3072,
+	  .name = "ffdhe3072", .kpp = "ffdhe3072(dh)" },
+	{ .id = NVME_AUTH_DHGROUP_4096,
+	  .name = "ffdhe4096", .kpp = "ffdhe4096(dh)" },
+	{ .id = NVME_AUTH_DHGROUP_6144,
+	  .name = "ffdhe6144", .kpp = "ffdhe6144(dh)" },
+	{ .id = NVME_AUTH_DHGROUP_8192,
+	  .name = "ffdhe8192", .kpp = "ffdhe8192(dh)" },
+};
+
+const char *nvme_auth_dhgroup_name(u8 dhgroup_id)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(dhgroup_map); i++) {
+		if (dhgroup_map[i].id == dhgroup_id)
+			return dhgroup_map[i].name;
+	}
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_dhgroup_name);
+
+const char *nvme_auth_dhgroup_kpp(u8 dhgroup_id)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(dhgroup_map); i++) {
+		if (dhgroup_map[i].id == dhgroup_id)
+			return dhgroup_map[i].kpp;
+	}
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_dhgroup_kpp);
+
+u8 nvme_auth_dhgroup_id(const char *dhgroup_name)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(dhgroup_map); i++) {
+		if (!strncmp(dhgroup_map[i].name, dhgroup_name,
+			     strlen(dhgroup_map[i].name)))
+			return dhgroup_map[i].id;
+	}
+	return NVME_AUTH_DHGROUP_INVALID;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_dhgroup_id);
+
+static struct nvme_dhchap_hash_map {
+	int id;
+	int len;
+	const char hmac[15];
+	const char digest[15];
+} hash_map[] = {
+	{.id = NVME_AUTH_HASH_SHA256, .len = 32,
+	 .hmac = "hmac(sha256)", .digest = "sha256" },
+	{.id = NVME_AUTH_HASH_SHA384, .len = 48,
+	 .hmac = "hmac(sha384)", .digest = "sha384" },
+	{.id = NVME_AUTH_HASH_SHA512, .len = 64,
+	 .hmac = "hmac(sha512)", .digest = "sha512" },
+};
+
+const char *nvme_auth_hmac_name(u8 hmac_id)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
+		if (hash_map[i].id == hmac_id)
+			return hash_map[i].hmac;
+	}
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_hmac_name);
+
+const char *nvme_auth_digest_name(u8 hmac_id)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
+		if (hash_map[i].id == hmac_id)
+			return hash_map[i].digest;
+	}
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_digest_name);
+
+u8 nvme_auth_hmac_id(const char *hmac_name)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
+		if (!strncmp(hash_map[i].hmac, hmac_name,
+			     strlen(hash_map[i].hmac)))
+			return hash_map[i].id;
+	}
+	return NVME_AUTH_HASH_INVALID;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_hmac_id);
+
+size_t nvme_auth_hmac_hash_len(u8 hmac_id)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
+		if (hash_map[i].id == hmac_id)
+			return hash_map[i].len;
+	}
+	return 0;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_hmac_hash_len);
+
+struct nvme_dhchap_key *nvme_auth_extract_key(unsigned char *secret,
+					      u8 key_hash)
+{
+	struct nvme_dhchap_key *key;
+	unsigned char *p;
+	u32 crc;
+	int ret, key_len;
+	size_t allocated_len = strlen(secret);
+
+	/* Secret might be affixed with a ':' */
+	p = strrchr(secret, ':');
+	if (p)
+		allocated_len = p - secret;
+	key = kzalloc(sizeof(*key), GFP_KERNEL);
+	if (!key)
+		return ERR_PTR(-ENOMEM);
+	key->key = kzalloc(allocated_len, GFP_KERNEL);
+	if (!key->key) {
+		ret = -ENOMEM;
+		goto out_free_key;
+	}
+
+	key_len = base64_decode(secret, allocated_len, key->key);
+	if (key_len < 0) {
+		pr_debug("base64 key decoding error %d\n",
+			 key_len);
+		ret = key_len;
+		goto out_free_secret;
+	}
+
+	if (key_len != 36 && key_len != 52 &&
+	    key_len != 68) {
+		pr_err("Invalid DH-HMAC-CHAP key len %d\n",
+		       key_len);
+		ret = -EINVAL;
+		goto out_free_secret;
+	}
+
+	if (key_hash > 0 &&
+	    (key_len - 4) != nvme_auth_hmac_hash_len(key_hash)) {
+		pr_err("Invalid DH-HMAC-CHAP key len %d for %s\n", key_len,
+		       nvme_auth_hmac_name(key_hash));
+		ret = -EINVAL;
+		goto out_free_secret;
+	}
+
+	/* The last four bytes is the CRC in little-endian format */
+	key_len -= 4;
+	/*
+	 * The linux implementation doesn't do pre- and post-increments,
+	 * so we have to do it manually.
+	 */
+	crc = ~crc32(~0, key->key, key_len);
+
+	if (get_unaligned_le32(key->key + key_len) != crc) {
+		pr_err("DH-HMAC-CHAP key crc mismatch (key %08x, crc %08x)\n",
+		       get_unaligned_le32(key->key + key_len), crc);
+		ret = -EKEYREJECTED;
+		goto out_free_secret;
+	}
+	key->len = key_len;
+	key->hash = key_hash;
+	return key;
+out_free_secret:
+	kfree_sensitive(key->key);
+out_free_key:
+	kfree(key);
+	return ERR_PTR(ret);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_extract_key);
+
+void nvme_auth_free_key(struct nvme_dhchap_key *key)
+{
+	if (!key)
+		return;
+	kfree_sensitive(key->key);
+	kfree(key);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_free_key);
+
+u8 *nvme_auth_transform_key(struct nvme_dhchap_key *key, char *nqn)
+{
+	const char *hmac_name = nvme_auth_hmac_name(key->hash);
+	struct crypto_shash *key_tfm;
+	struct shash_desc *shash;
+	u8 *transformed_key;
+	int ret;
+
+	if (key->hash == 0) {
+		transformed_key = kmemdup(key->key, key->len, GFP_KERNEL);
+		return transformed_key ? transformed_key : ERR_PTR(-ENOMEM);
+	}
+
+	if (!key || !key->key) {
+		pr_warn("No key specified\n");
+		return ERR_PTR(-ENOKEY);
+	}
+	if (!hmac_name) {
+		pr_warn("Invalid key hash id %d\n", key->hash);
+		return ERR_PTR(-EINVAL);
+	}
+
+	key_tfm = crypto_alloc_shash(hmac_name, 0, 0);
+	if (IS_ERR(key_tfm))
+		return (u8 *)key_tfm;
+
+	shash = kmalloc(sizeof(struct shash_desc) +
+			crypto_shash_descsize(key_tfm),
+			GFP_KERNEL);
+	if (!shash) {
+		ret = -ENOMEM;
+		goto out_free_key;
+	}
+
+	transformed_key = kzalloc(crypto_shash_digestsize(key_tfm), GFP_KERNEL);
+	if (!transformed_key) {
+		ret = -ENOMEM;
+		goto out_free_shash;
+	}
+
+	shash->tfm = key_tfm;
+	ret = crypto_shash_setkey(key_tfm, key->key, key->len);
+	if (ret < 0)
+		goto out_free_shash;
+	ret = crypto_shash_init(shash);
+	if (ret < 0)
+		goto out_free_shash;
+	ret = crypto_shash_update(shash, nqn, strlen(nqn));
+	if (ret < 0)
+		goto out_free_shash;
+	ret = crypto_shash_update(shash, "NVMe-over-Fabrics", 17);
+	if (ret < 0)
+		goto out_free_shash;
+	ret = crypto_shash_final(shash, transformed_key);
+out_free_shash:
+	kfree(shash);
+out_free_key:
+	crypto_free_shash(key_tfm);
+	if (ret < 0) {
+		kfree_sensitive(transformed_key);
+		return ERR_PTR(ret);
+	}
+	return transformed_key;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_transform_key);
+
+#define nvme_auth_flags_from_qid(qid) \
+	(qid == 0) ? 0 : BLK_MQ_REQ_NOWAIT | BLK_MQ_REQ_RESERVED
+#define nvme_auth_queue_from_qid(ctrl, qid) \
+	(qid == 0) ? (ctrl)->fabrics_q : (ctrl)->connect_q
+
+static int nvme_auth_send(struct nvme_ctrl *ctrl, int qid,
+		void *data, size_t tl)
+{
+	struct nvme_command cmd = {};
+	blk_mq_req_flags_t flags = nvme_auth_flags_from_qid(qid);
+	struct request_queue *q = nvme_auth_queue_from_qid(ctrl, qid);
+	int ret;
+
+	cmd.auth_send.opcode = nvme_fabrics_command;
+	cmd.auth_send.fctype = nvme_fabrics_type_auth_send;
+	cmd.auth_send.secp = NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER;
+	cmd.auth_send.spsp0 = 0x01;
+	cmd.auth_send.spsp1 = 0x01;
+	cmd.auth_send.tl = cpu_to_le32(tl);
+
+	ret = __nvme_submit_sync_cmd(q, &cmd, NULL, data, tl, 0,
+				     qid == 0 ? NVME_QID_ANY : qid,
+				     0, flags);
+	if (ret > 0)
+		dev_warn(ctrl->device,
+			"qid %d auth_send failed with status %d\n", qid, ret);
+	else if (ret < 0)
+		dev_err(ctrl->device,
+			"qid %d auth_send failed with error %d\n", qid, ret);
+	return ret;
+}
+
+static int nvme_auth_receive(struct nvme_ctrl *ctrl, int qid,
+		void *buf, size_t al)
+{
+	struct nvme_command cmd = {};
+	blk_mq_req_flags_t flags = nvme_auth_flags_from_qid(qid);
+	struct request_queue *q = nvme_auth_queue_from_qid(ctrl, qid);
+	int ret;
+
+	cmd.auth_receive.opcode = nvme_fabrics_command;
+	cmd.auth_receive.fctype = nvme_fabrics_type_auth_receive;
+	cmd.auth_receive.secp = NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER;
+	cmd.auth_receive.spsp0 = 0x01;
+	cmd.auth_receive.spsp1 = 0x01;
+	cmd.auth_receive.al = cpu_to_le32(al);
+
+	ret = __nvme_submit_sync_cmd(q, &cmd, NULL, buf, al, 0,
+				     qid == 0 ? NVME_QID_ANY : qid,
+				     0, flags);
+	if (ret > 0) {
+		dev_warn(ctrl->device,
+			 "qid %d auth_recv failed with status %x\n", qid, ret);
+		ret = -EIO;
+	} else if (ret < 0) {
+		dev_err(ctrl->device,
+			"qid %d auth_recv failed with error %d\n", qid, ret);
+	}
+
+	return ret;
+}
+
+static int nvme_auth_receive_validate(struct nvme_ctrl *ctrl, int qid,
+		struct nvmf_auth_dhchap_failure_data *data,
+		u16 transaction, u8 expected_msg)
+{
+	dev_dbg(ctrl->device, "%s: qid %d auth_type %d auth_id %x\n",
+		__func__, qid, data->auth_type, data->auth_id);
+
+	if (data->auth_type == NVME_AUTH_COMMON_MESSAGES &&
+	    data->auth_id == NVME_AUTH_DHCHAP_MESSAGE_FAILURE1) {
+		return data->rescode_exp;
+	}
+	if (data->auth_type != NVME_AUTH_DHCHAP_MESSAGES ||
+	    data->auth_id != expected_msg) {
+		dev_warn(ctrl->device,
+			 "qid %d invalid message %02x/%02x\n",
+			 qid, data->auth_type, data->auth_id);
+		return NVME_AUTH_DHCHAP_FAILURE_INCORRECT_MESSAGE;
+	}
+	if (le16_to_cpu(data->t_id) != transaction) {
+		dev_warn(ctrl->device,
+			 "qid %d invalid transaction ID %d\n",
+			 qid, le16_to_cpu(data->t_id));
+		return NVME_AUTH_DHCHAP_FAILURE_INCORRECT_MESSAGE;
+	}
+	return 0;
+}
+
+static int nvme_auth_set_dhchap_negotiate_data(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_negotiate_data *data = chap->buf;
+	size_t size = sizeof(*data) + sizeof(union nvmf_auth_protocol);
+
+	if (chap->buf_size < size) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return -EINVAL;
+	}
+	memset((u8 *)chap->buf, 0, size);
+	data->auth_type = NVME_AUTH_COMMON_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE;
+	data->t_id = cpu_to_le16(chap->transaction);
+	data->sc_c = 0; /* No secure channel concatenation */
+	data->napd = 1;
+	data->auth_protocol[0].dhchap.authid = NVME_AUTH_DHCHAP_AUTH_ID;
+	data->auth_protocol[0].dhchap.halen = 3;
+	data->auth_protocol[0].dhchap.dhlen = 6;
+	data->auth_protocol[0].dhchap.idlist[0] = NVME_AUTH_HASH_SHA256;
+	data->auth_protocol[0].dhchap.idlist[1] = NVME_AUTH_HASH_SHA384;
+	data->auth_protocol[0].dhchap.idlist[2] = NVME_AUTH_HASH_SHA512;
+	data->auth_protocol[0].dhchap.idlist[30] = NVME_AUTH_DHGROUP_NULL;
+	data->auth_protocol[0].dhchap.idlist[31] = NVME_AUTH_DHGROUP_2048;
+	data->auth_protocol[0].dhchap.idlist[32] = NVME_AUTH_DHGROUP_3072;
+	data->auth_protocol[0].dhchap.idlist[33] = NVME_AUTH_DHGROUP_4096;
+	data->auth_protocol[0].dhchap.idlist[34] = NVME_AUTH_DHGROUP_6144;
+	data->auth_protocol[0].dhchap.idlist[35] = NVME_AUTH_DHGROUP_8192;
+
+	return size;
+}
+
+static int nvme_auth_process_dhchap_challenge(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_challenge_data *data = chap->buf;
+	u16 dhvlen = le16_to_cpu(data->dhvlen);
+	size_t size = sizeof(*data) + data->hl + dhvlen;
+	const char *hmac_name, *kpp_name;
+
+	if (chap->buf_size < size) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return NVME_SC_INVALID_FIELD;
+	}
+
+	hmac_name = nvme_auth_hmac_name(data->hashid);
+	if (!hmac_name) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid HASH ID %d\n",
+			 chap->qid, data->hashid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
+		return NVME_SC_INVALID_FIELD;
+	}
+
+	if (chap->hash_id == data->hashid && chap->shash_tfm &&
+	    !strcmp(crypto_shash_alg_name(chap->shash_tfm), hmac_name) &&
+	    crypto_shash_digestsize(chap->shash_tfm) == data->hl) {
+		dev_dbg(ctrl->device,
+			"qid %d: reuse existing hash %s\n",
+			chap->qid, hmac_name);
+		goto select_kpp;
+	}
+
+	/* Reset if hash cannot be reused */
+	if (chap->shash_tfm) {
+		crypto_free_shash(chap->shash_tfm);
+		chap->hash_id = 0;
+		chap->hash_len = 0;
+	}
+	chap->shash_tfm = crypto_alloc_shash(hmac_name, 0,
+					     CRYPTO_ALG_ALLOCATES_MEMORY);
+	if (IS_ERR(chap->shash_tfm)) {
+		dev_warn(ctrl->device,
+			 "qid %d: failed to allocate hash %s, error %ld\n",
+			 chap->qid, hmac_name, PTR_ERR(chap->shash_tfm));
+		chap->shash_tfm = NULL;
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_FAILED;
+		return NVME_SC_AUTH_REQUIRED;
+	}
+
+	if (crypto_shash_digestsize(chap->shash_tfm) != data->hl) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid hash length %d\n",
+			 chap->qid, data->hl);
+		crypto_free_shash(chap->shash_tfm);
+		chap->shash_tfm = NULL;
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
+		return NVME_SC_AUTH_REQUIRED;
+	}
+
+	/* Reset host response if the hash had been changed */
+	if (chap->hash_id != data->hashid) {
+		kfree(chap->host_response);
+		chap->host_response = NULL;
+	}
+
+	chap->hash_id = data->hashid;
+	chap->hash_len = data->hl;
+	dev_dbg(ctrl->device, "qid %d: selected hash %s\n",
+		chap->qid, hmac_name);
+
+select_kpp:
+	kpp_name = nvme_auth_dhgroup_kpp(data->dhgid);
+	if (!kpp_name) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid DH group id %d\n",
+			 chap->qid, data->dhgid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
+		return NVME_SC_AUTH_REQUIRED;
+	}
+
+	if (data->dhgid != NVME_AUTH_DHGROUP_NULL) {
+		dev_warn(ctrl->device,
+			 "qid %d: unsupported DH group %s\n",
+			 chap->qid, kpp_name);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
+		return NVME_SC_AUTH_REQUIRED;
+	} else if (dhvlen != 0) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid DH value for NULL DH\n",
+			 chap->qid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return NVME_SC_INVALID_FIELD;
+	}
+	chap->dhgroup_id = data->dhgid;
+
+	chap->s1 = le32_to_cpu(data->seqnum);
+	memcpy(chap->c1, data->cval, chap->hash_len);
+
+	return 0;
+}
+
+static int nvme_auth_set_dhchap_reply_data(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_reply_data *data = chap->buf;
+	size_t size = sizeof(*data);
+
+	size += 2 * chap->hash_len;
+
+	if (chap->buf_size < size) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return -EINVAL;
+	}
+
+	memset(chap->buf, 0, size);
+	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_REPLY;
+	data->t_id = cpu_to_le16(chap->transaction);
+	data->hl = chap->hash_len;
+	data->dhvlen = 0;
+	memcpy(data->rval, chap->response, chap->hash_len);
+	if (ctrl->opts->dhchap_ctrl_secret) {
+		get_random_bytes(chap->c2, chap->hash_len);
+		data->cvalid = 1;
+		chap->s2 = nvme_auth_get_seqnum();
+		memcpy(data->rval + chap->hash_len, chap->c2,
+		       chap->hash_len);
+		dev_dbg(ctrl->device, "%s: qid %d ctrl challenge %*ph\n",
+			__func__, chap->qid, (int)chap->hash_len, chap->c2);
+	} else {
+		memset(chap->c2, 0, chap->hash_len);
+		chap->s2 = 0;
+	}
+	data->seqnum = cpu_to_le32(chap->s2);
+	return size;
+}
+
+static int nvme_auth_process_dhchap_success1(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_success1_data *data = chap->buf;
+	size_t size = sizeof(*data);
+
+	if (ctrl->opts->dhchap_ctrl_secret)
+		size += chap->hash_len;
+
+	if (chap->buf_size < size) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+		return NVME_SC_INVALID_FIELD;
+	}
+
+	if (data->hl != chap->hash_len) {
+		dev_warn(ctrl->device,
+			 "qid %d: invalid hash length %u\n",
+			 chap->qid, data->hl);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
+		return NVME_SC_INVALID_FIELD;
+	}
+
+	/* Just print out information for the admin queue */
+	if (chap->qid == 0)
+		dev_info(ctrl->device,
+			 "qid 0: authenticated with hash %s dhgroup %s\n",
+			 nvme_auth_hmac_name(chap->hash_id),
+			 nvme_auth_dhgroup_name(chap->dhgroup_id));
+
+	if (!data->rvalid)
+		return 0;
+
+	/* Validate controller response */
+	if (memcmp(chap->response, data->rval, data->hl)) {
+		dev_dbg(ctrl->device, "%s: qid %d ctrl response %*ph\n",
+			__func__, chap->qid, (int)chap->hash_len, data->rval);
+		dev_dbg(ctrl->device, "%s: qid %d host response %*ph\n",
+			__func__, chap->qid, (int)chap->hash_len,
+			chap->response);
+		dev_warn(ctrl->device,
+			 "qid %d: controller authentication failed\n",
+			 chap->qid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_FAILED;
+		return NVME_SC_AUTH_REQUIRED;
+	}
+
+	/* Just print out information for the admin queue */
+	if (chap->qid == 0)
+		dev_info(ctrl->device,
+			 "qid 0: controller authenticated\n");
+	return 0;
+}
+
+static int nvme_auth_set_dhchap_success2_data(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_success2_data *data = chap->buf;
+	size_t size = sizeof(*data);
+
+	memset(chap->buf, 0, size);
+	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_SUCCESS2;
+	data->t_id = cpu_to_le16(chap->transaction);
+
+	return size;
+}
+
+static int nvme_auth_set_dhchap_failure2_data(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	struct nvmf_auth_dhchap_failure_data *data = chap->buf;
+	size_t size = sizeof(*data);
+
+	memset(chap->buf, 0, size);
+	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_FAILURE2;
+	data->t_id = cpu_to_le16(chap->transaction);
+	data->rescode = NVME_AUTH_DHCHAP_FAILURE_REASON_FAILED;
+	data->rescode_exp = chap->status;
+
+	return size;
+}
+
+static int nvme_auth_dhchap_setup_host_response(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
+	u8 buf[4], *challenge = chap->c1;
+	int ret;
+
+	dev_dbg(ctrl->device, "%s: qid %d host response seq %u transaction %d\n",
+		__func__, chap->qid, chap->s1, chap->transaction);
+
+	if (!chap->host_response) {
+		chap->host_response = nvme_auth_transform_key(ctrl->host_key,
+						ctrl->opts->host->nqn);
+		if (IS_ERR(chap->host_response)) {
+			ret = PTR_ERR(chap->host_response);
+			chap->host_response = NULL;
+			return ret;
+		}
+	} else {
+		dev_dbg(ctrl->device, "%s: qid %d re-using host response\n",
+			__func__, chap->qid);
+	}
+
+	ret = crypto_shash_setkey(chap->shash_tfm,
+			chap->host_response, ctrl->host_key->len);
+	if (ret) {
+		dev_warn(ctrl->device, "qid %d: failed to set key, error %d\n",
+			 chap->qid, ret);
+		goto out;
+	}
+
+	shash->tfm = chap->shash_tfm;
+	ret = crypto_shash_init(shash);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, challenge, chap->hash_len);
+	if (ret)
+		goto out;
+	put_unaligned_le32(chap->s1, buf);
+	ret = crypto_shash_update(shash, buf, 4);
+	if (ret)
+		goto out;
+	put_unaligned_le16(chap->transaction, buf);
+	ret = crypto_shash_update(shash, buf, 2);
+	if (ret)
+		goto out;
+	memset(buf, 0, sizeof(buf));
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, "HostHost", 8);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
+				  strlen(ctrl->opts->host->nqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
+			    strlen(ctrl->opts->subsysnqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_final(shash, chap->response);
+out:
+	if (challenge != chap->c1)
+		kfree(challenge);
+	return ret;
+}
+
+static int nvme_auth_dhchap_setup_ctrl_response(struct nvme_ctrl *ctrl,
+		struct nvme_dhchap_queue_context *chap)
+{
+	SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
+	u8 *ctrl_response;
+	u8 buf[4], *challenge = chap->c2;
+	int ret;
+
+	ctrl_response = nvme_auth_transform_key(ctrl->ctrl_key,
+				ctrl->opts->subsysnqn);
+	if (IS_ERR(ctrl_response)) {
+		ret = PTR_ERR(ctrl_response);
+		return ret;
+	}
+	ret = crypto_shash_setkey(chap->shash_tfm,
+			ctrl_response, ctrl->ctrl_key->len);
+	if (ret) {
+		dev_warn(ctrl->device, "qid %d: failed to set key, error %d\n",
+			 chap->qid, ret);
+		goto out;
+	}
+
+	dev_dbg(ctrl->device, "%s: qid %d ctrl response seq %u transaction %d\n",
+		__func__, chap->qid, chap->s2, chap->transaction);
+	dev_dbg(ctrl->device, "%s: qid %d challenge %*ph\n",
+		__func__, chap->qid, (int)chap->hash_len, challenge);
+	dev_dbg(ctrl->device, "%s: qid %d subsysnqn %s\n",
+		__func__, chap->qid, ctrl->opts->subsysnqn);
+	dev_dbg(ctrl->device, "%s: qid %d hostnqn %s\n",
+		__func__, chap->qid, ctrl->opts->host->nqn);
+	shash->tfm = chap->shash_tfm;
+	ret = crypto_shash_init(shash);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, challenge, chap->hash_len);
+	if (ret)
+		goto out;
+	put_unaligned_le32(chap->s2, buf);
+	ret = crypto_shash_update(shash, buf, 4);
+	if (ret)
+		goto out;
+	put_unaligned_le16(chap->transaction, buf);
+	ret = crypto_shash_update(shash, buf, 2);
+	if (ret)
+		goto out;
+	memset(buf, 0, 4);
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, "Controller", 10);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
+				  strlen(ctrl->opts->subsysnqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
+				  strlen(ctrl->opts->host->nqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_final(shash, chap->response);
+out:
+	if (challenge != chap->c2)
+		kfree(challenge);
+	return ret;
+}
+
+int nvme_auth_generate_key(u8 *secret, struct nvme_dhchap_key **ret_key)
+{
+	struct nvme_dhchap_key *key;
+	u8 key_hash;
+
+	if (!secret) {
+		*ret_key = NULL;
+		return 0;
+	}
+
+	if (sscanf(secret, "DHHC-1:%hhd:%*s:", &key_hash) != 1)
+		return -EINVAL;
+
+	/* Pass in the secret without the 'DHHC-1:XX:' prefix */
+	key = nvme_auth_extract_key(secret + 10, key_hash);
+	if (IS_ERR(key)) {
+		return PTR_ERR(key);
+	}
+
+	*ret_key = key;
+	return 0;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_generate_key);
+
+static void __nvme_auth_reset(struct nvme_dhchap_queue_context *chap)
+{
+	chap->status = 0;
+	chap->error = 0;
+	chap->s1 = 0;
+	chap->s2 = 0;
+	chap->transaction = 0;
+	memset(chap->c1, 0, sizeof(chap->c1));
+	memset(chap->c2, 0, sizeof(chap->c2));
+}
+
+static void __nvme_auth_free(struct nvme_dhchap_queue_context *chap)
+{
+	__nvme_auth_reset(chap);
+	if (chap->shash_tfm)
+		crypto_free_shash(chap->shash_tfm);
+	kfree_sensitive(chap->host_response);
+	kfree(chap->buf);
+	kfree(chap);
+}
+
+static void __nvme_auth_work(struct work_struct *work)
+{
+	struct nvme_dhchap_queue_context *chap =
+		container_of(work, struct nvme_dhchap_queue_context, auth_work);
+	struct nvme_ctrl *ctrl = chap->ctrl;
+	size_t tl;
+	int ret = 0;
+
+	chap->transaction = ctrl->transaction++;
+
+	/* DH-HMAC-CHAP Step 1: send negotiate */
+	dev_dbg(ctrl->device, "%s: qid %d send negotiate\n",
+		__func__, chap->qid);
+	ret = nvme_auth_set_dhchap_negotiate_data(ctrl, chap);
+	if (ret < 0) {
+		chap->error = ret;
+		return;
+	}
+	tl = ret;
+	ret = nvme_auth_send(ctrl, chap->qid, chap->buf, tl);
+	if (ret) {
+		chap->error = ret;
+		return;
+	}
+
+	/* DH-HMAC-CHAP Step 2: receive challenge */
+	dev_dbg(ctrl->device, "%s: qid %d receive challenge\n",
+		__func__, chap->qid);
+
+	memset(chap->buf, 0, chap->buf_size);
+	ret = nvme_auth_receive(ctrl, chap->qid, chap->buf, chap->buf_size);
+	if (ret) {
+		dev_warn(ctrl->device,
+			 "qid %d failed to receive challenge, %s %d\n",
+			 chap->qid, ret < 0 ? "error" : "nvme status", ret);
+		chap->error = ret;
+		return;
+	}
+	ret = nvme_auth_receive_validate(ctrl, chap->qid, chap->buf, chap->transaction,
+					 NVME_AUTH_DHCHAP_MESSAGE_CHALLENGE);
+	if (ret) {
+		chap->status = ret;
+		chap->error = NVME_SC_AUTH_REQUIRED;
+		return;
+	}
+
+	ret = nvme_auth_process_dhchap_challenge(ctrl, chap);
+	if (ret) {
+		/* Invalid challenge parameters */
+		goto fail2;
+	}
+
+	dev_dbg(ctrl->device, "%s: qid %d host response\n",
+		__func__, chap->qid);
+	ret = nvme_auth_dhchap_setup_host_response(ctrl, chap);
+	if (ret)
+		goto fail2;
+
+	/* DH-HMAC-CHAP Step 3: send reply */
+	dev_dbg(ctrl->device, "%s: qid %d send reply\n",
+		__func__, chap->qid);
+	ret = nvme_auth_set_dhchap_reply_data(ctrl, chap);
+	if (ret < 0)
+		goto fail2;
+
+	tl = ret;
+	ret = nvme_auth_send(ctrl, chap->qid, chap->buf, tl);
+	if (ret)
+		goto fail2;
+
+	/* DH-HMAC-CHAP Step 4: receive success1 */
+	dev_dbg(ctrl->device, "%s: qid %d receive success1\n",
+		__func__, chap->qid);
+
+	memset(chap->buf, 0, chap->buf_size);
+	ret = nvme_auth_receive(ctrl, chap->qid, chap->buf, chap->buf_size);
+	if (ret) {
+		dev_warn(ctrl->device,
+			 "qid %d failed to receive success1, %s %d\n",
+			 chap->qid, ret < 0 ? "error" : "nvme status", ret);
+		chap->error = ret;
+		return;
+	}
+	ret = nvme_auth_receive_validate(ctrl, chap->qid,
+					 chap->buf, chap->transaction,
+					 NVME_AUTH_DHCHAP_MESSAGE_SUCCESS1);
+	if (ret) {
+		chap->status = ret;
+		chap->error = NVME_SC_AUTH_REQUIRED;
+		return;
+	}
+
+	if (ctrl->opts->dhchap_ctrl_secret) {
+		dev_dbg(ctrl->device,
+			"%s: qid %d controller response\n",
+			__func__, chap->qid);
+		ret = nvme_auth_dhchap_setup_ctrl_response(ctrl, chap);
+		if (ret)
+			goto fail2;
+	}
+
+	ret = nvme_auth_process_dhchap_success1(ctrl, chap);
+	if (ret) {
+		/* Controller authentication failed */
+		goto fail2;
+	}
+
+	/* DH-HMAC-CHAP Step 5: send success2 */
+	dev_dbg(ctrl->device, "%s: qid %d send success2\n",
+		__func__, chap->qid);
+	tl = nvme_auth_set_dhchap_success2_data(ctrl, chap);
+	ret = nvme_auth_send(ctrl, chap->qid, chap->buf, tl);
+	if (!ret) {
+		chap->error = 0;
+		return;
+	}
+
+fail2:
+	dev_dbg(ctrl->device, "%s: qid %d send failure2, status %x\n",
+		__func__, chap->qid, chap->status);
+	tl = nvme_auth_set_dhchap_failure2_data(ctrl, chap);
+	ret = nvme_auth_send(ctrl, chap->qid, chap->buf, tl);
+	if (!ret)
+		ret = -EPROTO;
+	chap->error = ret;
+}
+
+int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid)
+{
+	struct nvme_dhchap_queue_context *chap;
+
+	if (!ctrl->host_key) {
+		dev_warn(ctrl->device, "qid %d: no key\n", qid);
+		return -ENOKEY;
+	}
+
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	/* Check if the context is already queued */
+	list_for_each_entry(chap, &ctrl->dhchap_auth_list, entry) {
+		WARN_ON(!chap->buf);
+		if (chap->qid == qid) {
+			dev_dbg(ctrl->device, "qid %d: re-using context\n", qid);
+			mutex_unlock(&ctrl->dhchap_auth_mutex);
+			flush_work(&chap->auth_work);
+			__nvme_auth_reset(chap);
+			queue_work(nvme_wq, &chap->auth_work);
+			return 0;
+		}
+	}
+	chap = kzalloc(sizeof(*chap), GFP_KERNEL);
+	if (!chap) {
+		mutex_unlock(&ctrl->dhchap_auth_mutex);
+		return -ENOMEM;
+	}
+	chap->qid = (qid == NVME_QID_ANY) ? 0 : qid;
+	chap->ctrl = ctrl;
+
+	/*
+	 * Allocate a large enough buffer for the entire negotiation:
+	 * 4k should be enough to ffdhe8192.
+	 */
+	chap->buf_size = 4096;
+	chap->buf = kzalloc(chap->buf_size, GFP_KERNEL);
+	if (!chap->buf) {
+		mutex_unlock(&ctrl->dhchap_auth_mutex);
+		kfree(chap);
+		return -ENOMEM;
+	}
+
+	INIT_WORK(&chap->auth_work, __nvme_auth_work);
+	list_add(&chap->entry, &ctrl->dhchap_auth_list);
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+	queue_work(nvme_wq, &chap->auth_work);
+	return 0;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_negotiate);
+
+int nvme_auth_wait(struct nvme_ctrl *ctrl, int qid)
+{
+	struct nvme_dhchap_queue_context *chap;
+	int ret;
+
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	list_for_each_entry(chap, &ctrl->dhchap_auth_list, entry) {
+		if (chap->qid != qid)
+			continue;
+		mutex_unlock(&ctrl->dhchap_auth_mutex);
+		flush_work(&chap->auth_work);
+		ret = chap->error;
+		return ret;
+	}
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+	return -ENXIO;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_wait);
+
+void nvme_auth_reset(struct nvme_ctrl *ctrl)
+{
+	struct nvme_dhchap_queue_context *chap;
+
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	list_for_each_entry(chap, &ctrl->dhchap_auth_list, entry) {
+		mutex_unlock(&ctrl->dhchap_auth_mutex);
+		flush_work(&chap->auth_work);
+		__nvme_auth_reset(chap);
+	}
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_reset);
+
+static void nvme_dhchap_auth_work(struct work_struct *work)
+{
+	struct nvme_ctrl *ctrl =
+		container_of(work, struct nvme_ctrl, dhchap_auth_work);
+	int ret, q;
+
+	/* Authenticate admin queue first */
+	ret = nvme_auth_negotiate(ctrl, 0);
+	if (ret) {
+		dev_warn(ctrl->device,
+			 "qid 0: error %d setting up authentication\n", ret);
+		return;
+	}
+	ret = nvme_auth_wait(ctrl, 0);
+	if (ret) {
+		dev_warn(ctrl->device,
+			 "qid 0: authentication failed\n");
+		return;
+	}
+
+	for (q = 1; q < ctrl->queue_count; q++) {
+		ret = nvme_auth_negotiate(ctrl, q);
+		if (ret) {
+			dev_warn(ctrl->device,
+				 "qid %d: error %d setting up authentication\n",
+				 q, ret);
+			break;
+		}
+	}
+
+	/*
+	 * Failure is a soft-state; credentials remain valid until
+	 * the controller terminates the connection.
+	 */
+}
+
+void nvme_auth_init_ctrl(struct nvme_ctrl *ctrl)
+{
+	INIT_LIST_HEAD(&ctrl->dhchap_auth_list);
+	INIT_WORK(&ctrl->dhchap_auth_work, nvme_dhchap_auth_work);
+	mutex_init(&ctrl->dhchap_auth_mutex);
+	if (!ctrl->opts)
+		return;
+	nvme_auth_generate_key(ctrl->opts->dhchap_secret, &ctrl->host_key);
+	nvme_auth_generate_key(ctrl->opts->dhchap_ctrl_secret, &ctrl->ctrl_key);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_init_ctrl);
+
+void nvme_auth_stop(struct nvme_ctrl *ctrl)
+{
+	struct nvme_dhchap_queue_context *chap = NULL, *tmp;
+
+	cancel_work_sync(&ctrl->dhchap_auth_work);
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	list_for_each_entry_safe(chap, tmp, &ctrl->dhchap_auth_list, entry)
+		cancel_work_sync(&chap->auth_work);
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+}
+EXPORT_SYMBOL_GPL(nvme_auth_stop);
+
+void nvme_auth_free(struct nvme_ctrl *ctrl)
+{
+	struct nvme_dhchap_queue_context *chap = NULL, *tmp;
+
+	mutex_lock(&ctrl->dhchap_auth_mutex);
+	list_for_each_entry_safe(chap, tmp, &ctrl->dhchap_auth_list, entry) {
+		list_del_init(&chap->entry);
+		flush_work(&chap->auth_work);
+		__nvme_auth_free(chap);
+	}
+	mutex_unlock(&ctrl->dhchap_auth_mutex);
+	if (ctrl->host_key) {
+		nvme_auth_free_key(ctrl->host_key);
+		ctrl->host_key = NULL;
+	}
+	if (ctrl->ctrl_key) {
+		nvme_auth_free_key(ctrl->ctrl_key);
+		ctrl->ctrl_key = NULL;
+	}
+}
+EXPORT_SYMBOL_GPL(nvme_auth_free);
diff --git a/drivers/nvme/host/auth.h b/drivers/nvme/host/auth.h
new file mode 100644
index 000000000000..73d6ec63a5c8
--- /dev/null
+++ b/drivers/nvme/host/auth.h
@@ -0,0 +1,32 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2021 Hannes Reinecke, SUSE Software Solutions
+ */
+
+#ifndef _NVME_AUTH_H
+#define _NVME_AUTH_H
+
+#include <crypto/kpp.h>
+
+struct nvme_dhchap_key {
+	u8 *key;
+	size_t len;
+	u8 hash;
+};
+
+u32 nvme_auth_get_seqnum(void);
+const char *nvme_auth_dhgroup_name(u8 dhgroup_id);
+const char *nvme_auth_dhgroup_kpp(u8 dhgroup_id);
+u8 nvme_auth_dhgroup_id(const char *dhgroup_name);
+
+const char *nvme_auth_hmac_name(u8 hmac_id);
+const char *nvme_auth_digest_name(u8 hmac_id);
+size_t nvme_auth_hmac_hash_len(u8 hmac_id);
+u8 nvme_auth_hmac_id(const char *hmac_name);
+
+struct nvme_dhchap_key *nvme_auth_extract_key(unsigned char *secret,
+					      u8 key_hash);
+void nvme_auth_free_key(struct nvme_dhchap_key *key);
+u8 *nvme_auth_transform_key(struct nvme_dhchap_key *key, char *nqn);
+
+#endif /* _NVME_AUTH_H */
diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index cd6eac8e3dd6..dc10ba7cfc6c 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -24,6 +24,7 @@
 
 #include "nvme.h"
 #include "fabrics.h"
+#include "auth.h"
 
 #define CREATE_TRACE_POINTS
 #include "trace.h"
@@ -334,6 +335,7 @@ enum nvme_disposition {
 	COMPLETE,
 	RETRY,
 	FAILOVER,
+	AUTHENTICATE,
 };
 
 static inline enum nvme_disposition nvme_decide_disposition(struct request *req)
@@ -341,6 +343,9 @@ static inline enum nvme_disposition nvme_decide_disposition(struct request *req)
 	if (likely(nvme_req(req)->status == 0))
 		return COMPLETE;
 
+	if ((nvme_req(req)->status & 0x7ff) == NVME_SC_AUTH_REQUIRED)
+		return AUTHENTICATE;
+
 	if (blk_noretry_request(req) ||
 	    (nvme_req(req)->status & NVME_SC_DNR) ||
 	    nvme_req(req)->retries >= nvme_max_retries)
@@ -379,11 +384,13 @@ static inline void nvme_end_req(struct request *req)
 
 void nvme_complete_rq(struct request *req)
 {
+	struct nvme_ctrl *ctrl = nvme_req(req)->ctrl;
+
 	trace_nvme_complete_rq(req);
 	nvme_cleanup_cmd(req);
 
-	if (nvme_req(req)->ctrl->kas)
-		nvme_req(req)->ctrl->comp_seen = true;
+	if (ctrl->kas)
+		ctrl->comp_seen = true;
 
 	switch (nvme_decide_disposition(req)) {
 	case COMPLETE:
@@ -395,6 +402,14 @@ void nvme_complete_rq(struct request *req)
 	case FAILOVER:
 		nvme_failover_req(req);
 		return;
+	case AUTHENTICATE:
+#ifdef CONFIG_NVME_AUTH
+		queue_work(nvme_wq, &ctrl->dhchap_auth_work);
+		nvme_retry_req(req);
+#else
+		nvme_end_req(req);
+#endif
+		return;
 	}
 }
 EXPORT_SYMBOL_GPL(nvme_complete_rq);
@@ -706,7 +721,9 @@ bool __nvme_check_ready(struct nvme_ctrl *ctrl, struct request *rq,
 		switch (ctrl->state) {
 		case NVME_CTRL_CONNECTING:
 			if (blk_rq_is_passthrough(rq) && nvme_is_fabrics(req->cmd) &&
-			    req->cmd->fabrics.fctype == nvme_fabrics_type_connect)
+			    (req->cmd->fabrics.fctype == nvme_fabrics_type_connect ||
+			     req->cmd->fabrics.fctype == nvme_fabrics_type_auth_send ||
+			     req->cmd->fabrics.fctype == nvme_fabrics_type_auth_receive))
 				return true;
 			break;
 		default:
@@ -3563,6 +3580,108 @@ static ssize_t dctype_show(struct device *dev,
 }
 static DEVICE_ATTR_RO(dctype);
 
+#ifdef CONFIG_NVME_AUTH
+static ssize_t nvme_ctrl_dhchap_secret_show(struct device *dev,
+		struct device_attribute *attr, char *buf)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	struct nvmf_ctrl_options *opts = ctrl->opts;
+
+	if (!opts->dhchap_secret)
+		return sysfs_emit(buf, "none\n");
+	return sysfs_emit(buf, "%s\n", opts->dhchap_secret);
+}
+
+static ssize_t nvme_ctrl_dhchap_secret_store(struct device *dev,
+		struct device_attribute *attr, const char *buf, size_t count)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	struct nvmf_ctrl_options *opts = ctrl->opts;
+	char *dhchap_secret;
+
+	if (!ctrl->opts->dhchap_secret)
+		return -EINVAL;
+	if (count < 7)
+		return -EINVAL;
+	if (memcmp(buf, "DHHC-1:", 7))
+		return -EINVAL;
+
+	dhchap_secret = kzalloc(count + 1, GFP_KERNEL);
+	if (!dhchap_secret)
+		return -ENOMEM;
+	memcpy(dhchap_secret, buf, count);
+	nvme_auth_stop(ctrl);
+	if (strcmp(dhchap_secret, opts->dhchap_secret)) {
+		int ret;
+
+		ret = nvme_auth_generate_key(dhchap_secret, &ctrl->host_key);
+		if (ret)
+			return ret;
+		kfree(opts->dhchap_secret);
+		opts->dhchap_secret = dhchap_secret;
+		/* Key has changed; re-authentication with new key */
+		nvme_auth_reset(ctrl);
+	}
+	/* Start re-authentication */
+	dev_info(ctrl->device, "re-authenticating controller\n");
+	queue_work(nvme_wq, &ctrl->dhchap_auth_work);
+
+	return count;
+}
+DEVICE_ATTR(dhchap_secret, S_IRUGO | S_IWUSR,
+	nvme_ctrl_dhchap_secret_show, nvme_ctrl_dhchap_secret_store);
+
+static ssize_t nvme_ctrl_dhchap_ctrl_secret_show(struct device *dev,
+		struct device_attribute *attr, char *buf)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	struct nvmf_ctrl_options *opts = ctrl->opts;
+
+	if (!opts->dhchap_ctrl_secret)
+		return sysfs_emit(buf, "none\n");
+	return sysfs_emit(buf, "%s\n", opts->dhchap_ctrl_secret);
+}
+
+static ssize_t nvme_ctrl_dhchap_ctrl_secret_store(struct device *dev,
+		struct device_attribute *attr, const char *buf, size_t count)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	struct nvmf_ctrl_options *opts = ctrl->opts;
+	char *dhchap_secret;
+
+	if (!ctrl->opts->dhchap_ctrl_secret)
+		return -EINVAL;
+	if (count < 7)
+		return -EINVAL;
+	if (memcmp(buf, "DHHC-1:", 7))
+		return -EINVAL;
+
+	dhchap_secret = kzalloc(count + 1, GFP_KERNEL);
+	if (!dhchap_secret)
+		return -ENOMEM;
+	memcpy(dhchap_secret, buf, count);
+	nvme_auth_stop(ctrl);
+	if (strcmp(dhchap_secret, opts->dhchap_ctrl_secret)) {
+		int ret;
+
+		ret = nvme_auth_generate_key(dhchap_secret, &ctrl->ctrl_key);
+		if (ret)
+			return ret;
+		kfree(opts->dhchap_ctrl_secret);
+		opts->dhchap_ctrl_secret = dhchap_secret;
+		/* Key has changed; re-authentication with new key */
+		nvme_auth_reset(ctrl);
+	}
+	/* Start re-authentication */
+	dev_info(ctrl->device, "re-authenticating controller\n");
+	queue_work(nvme_wq, &ctrl->dhchap_auth_work);
+
+	return count;
+}
+DEVICE_ATTR(dhchap_ctrl_secret, S_IRUGO | S_IWUSR,
+	nvme_ctrl_dhchap_ctrl_secret_show, nvme_ctrl_dhchap_ctrl_secret_store);
+#endif
+
 static struct attribute *nvme_dev_attrs[] = {
 	&dev_attr_reset_controller.attr,
 	&dev_attr_rescan_controller.attr,
@@ -3586,6 +3705,10 @@ static struct attribute *nvme_dev_attrs[] = {
 	&dev_attr_kato.attr,
 	&dev_attr_cntrltype.attr,
 	&dev_attr_dctype.attr,
+#ifdef CONFIG_NVME_AUTH
+	&dev_attr_dhchap_secret.attr,
+	&dev_attr_dhchap_ctrl_secret.attr,
+#endif
 	NULL
 };
 
@@ -3609,6 +3732,10 @@ static umode_t nvme_dev_attrs_are_visible(struct kobject *kobj,
 		return 0;
 	if (a == &dev_attr_fast_io_fail_tmo.attr && !ctrl->opts)
 		return 0;
+#ifdef CONFIG_NVME_AUTH
+	if (a == &dev_attr_dhchap_secret.attr && !ctrl->opts)
+		return 0;
+#endif
 
 	return a->mode;
 }
@@ -4450,8 +4577,10 @@ static void nvme_handle_aen_notice(struct nvme_ctrl *ctrl, u32 result)
 		 * recovery actions from interfering with the controller's
 		 * firmware activation.
 		 */
-		if (nvme_change_ctrl_state(ctrl, NVME_CTRL_RESETTING))
+		if (nvme_change_ctrl_state(ctrl, NVME_CTRL_RESETTING)) {
+			nvme_auth_stop(ctrl);
 			queue_work(nvme_wq, &ctrl->fw_act_work);
+		}
 		break;
 #ifdef CONFIG_NVME_MULTIPATH
 	case NVME_AER_NOTICE_ANA:
@@ -4498,6 +4627,7 @@ EXPORT_SYMBOL_GPL(nvme_complete_async_event);
 void nvme_stop_ctrl(struct nvme_ctrl *ctrl)
 {
 	nvme_mpath_stop(ctrl);
+	nvme_auth_stop(ctrl);
 	nvme_stop_keep_alive(ctrl);
 	nvme_stop_failfast_work(ctrl);
 	flush_work(&ctrl->async_event_work);
@@ -4554,6 +4684,8 @@ static void nvme_free_ctrl(struct device *dev)
 
 	nvme_free_cels(ctrl);
 	nvme_mpath_uninit(ctrl);
+	nvme_auth_stop(ctrl);
+	nvme_auth_free(ctrl);
 	__free_page(ctrl->discard_page);
 
 	if (subsys) {
@@ -4644,6 +4776,7 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev,
 
 	nvme_fault_inject_init(&ctrl->fault_inject, dev_name(ctrl->device));
 	nvme_mpath_init_ctrl(ctrl);
+	nvme_auth_init_ctrl(ctrl);
 
 	return 0;
 out_free_name:
diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index 953e3076aab1..f5786a60931c 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -369,6 +369,7 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)
 	union nvme_result res;
 	struct nvmf_connect_data *data;
 	int ret;
+	u32 result;
 
 	cmd.connect.opcode = nvme_fabrics_command;
 	cmd.connect.fctype = nvme_fabrics_type_connect;
@@ -401,8 +402,25 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)
 		goto out_free_data;
 	}
 
-	ctrl->cntlid = le16_to_cpu(res.u16);
-
+	result = le32_to_cpu(res.u32);
+	ctrl->cntlid = result & 0xFFFF;
+	if ((result >> 16) & 2) {
+		/* Authentication required */
+		ret = nvme_auth_negotiate(ctrl, 0);
+		if (ret) {
+			dev_warn(ctrl->device,
+				 "qid 0: authentication setup failed\n");
+			ret = NVME_SC_AUTH_REQUIRED;
+			goto out_free_data;
+		}
+		ret = nvme_auth_wait(ctrl, 0);
+		if (ret)
+			dev_warn(ctrl->device,
+				 "qid 0: authentication failed\n");
+		else
+			dev_info(ctrl->device,
+				 "qid 0: authenticated\n");
+	}
 out_free_data:
 	kfree(data);
 	return ret;
@@ -435,6 +453,7 @@ int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, u16 qid)
 	struct nvmf_connect_data *data;
 	union nvme_result res;
 	int ret;
+	u32 result;
 
 	cmd.connect.opcode = nvme_fabrics_command;
 	cmd.connect.fctype = nvme_fabrics_type_connect;
@@ -460,6 +479,21 @@ int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, u16 qid)
 		nvmf_log_connect_error(ctrl, ret, le32_to_cpu(res.u32),
 				       &cmd, data);
 	}
+	result = le32_to_cpu(res.u32);
+	if ((result >> 16) & 2) {
+		/* Authentication required */
+		ret = nvme_auth_negotiate(ctrl, qid);
+		if (ret) {
+			dev_warn(ctrl->device,
+				 "qid %d: authentication setup failed\n", qid);
+			ret = NVME_SC_AUTH_REQUIRED;
+		} else {
+			ret = nvme_auth_wait(ctrl, qid);
+			if (ret)
+				dev_warn(ctrl->device,
+					 "qid %u: authentication failed\n", qid);
+		}
+	}
 	kfree(data);
 	return ret;
 }
@@ -552,6 +586,8 @@ static const match_table_t opt_tokens = {
 	{ NVMF_OPT_TOS,			"tos=%d"		},
 	{ NVMF_OPT_FAIL_FAST_TMO,	"fast_io_fail_tmo=%d"	},
 	{ NVMF_OPT_DISCOVERY,		"discovery"		},
+	{ NVMF_OPT_DHCHAP_SECRET,	"dhchap_secret=%s"	},
+	{ NVMF_OPT_DHCHAP_CTRL_SECRET,	"dhchap_ctrl_secret=%s"	},
 	{ NVMF_OPT_ERR,			NULL			}
 };
 
@@ -833,6 +869,34 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
 		case NVMF_OPT_DISCOVERY:
 			opts->discovery_nqn = true;
 			break;
+		case NVMF_OPT_DHCHAP_SECRET:
+			p = match_strdup(args);
+			if (!p) {
+				ret = -ENOMEM;
+				goto out;
+			}
+			if (strlen(p) < 11 || strncmp(p, "DHHC-1:", 7)) {
+				pr_err("Invalid DH-CHAP secret %s\n", p);
+				ret = -EINVAL;
+				goto out;
+			}
+			kfree(opts->dhchap_secret);
+			opts->dhchap_secret = p;
+			break;
+		case NVMF_OPT_DHCHAP_CTRL_SECRET:
+			p = match_strdup(args);
+			if (!p) {
+				ret = -ENOMEM;
+				goto out;
+			}
+			if (strlen(p) < 11 || strncmp(p, "DHHC-1:", 7)) {
+				pr_err("Invalid DH-CHAP secret %s\n", p);
+				ret = -EINVAL;
+				goto out;
+			}
+			kfree(opts->dhchap_ctrl_secret);
+			opts->dhchap_ctrl_secret = p;
+			break;
 		default:
 			pr_warn("unknown parameter or missing value '%s' in ctrl creation request\n",
 				p);
@@ -951,6 +1015,7 @@ void nvmf_free_options(struct nvmf_ctrl_options *opts)
 	kfree(opts->subsysnqn);
 	kfree(opts->host_traddr);
 	kfree(opts->host_iface);
+	kfree(opts->dhchap_secret);
 	kfree(opts);
 }
 EXPORT_SYMBOL_GPL(nvmf_free_options);
@@ -960,7 +1025,8 @@ EXPORT_SYMBOL_GPL(nvmf_free_options);
 				 NVMF_OPT_KATO | NVMF_OPT_HOSTNQN | \
 				 NVMF_OPT_HOST_ID | NVMF_OPT_DUP_CONNECT |\
 				 NVMF_OPT_DISABLE_SQFLOW | NVMF_OPT_DISCOVERY |\
-				 NVMF_OPT_FAIL_FAST_TMO)
+				 NVMF_OPT_FAIL_FAST_TMO | NVMF_OPT_DHCHAP_SECRET |\
+				 NVMF_OPT_DHCHAP_CTRL_SECRET)
 
 static struct nvme_ctrl *
 nvmf_create_ctrl(struct device *dev, const char *buf)
@@ -1196,7 +1262,14 @@ static void __exit nvmf_exit(void)
 	BUILD_BUG_ON(sizeof(struct nvmf_connect_command) != 64);
 	BUILD_BUG_ON(sizeof(struct nvmf_property_get_command) != 64);
 	BUILD_BUG_ON(sizeof(struct nvmf_property_set_command) != 64);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_send_command) != 64);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_receive_command) != 64);
 	BUILD_BUG_ON(sizeof(struct nvmf_connect_data) != 1024);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_negotiate_data) != 8);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_challenge_data) != 16);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_reply_data) != 16);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_success1_data) != 16);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_success2_data) != 16);
 }
 
 MODULE_LICENSE("GPL v2");
diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h
index 1e3a09cad961..15c142b277ab 100644
--- a/drivers/nvme/host/fabrics.h
+++ b/drivers/nvme/host/fabrics.h
@@ -68,6 +68,8 @@ enum {
 	NVMF_OPT_FAIL_FAST_TMO	= 1 << 20,
 	NVMF_OPT_HOST_IFACE	= 1 << 21,
 	NVMF_OPT_DISCOVERY	= 1 << 22,
+	NVMF_OPT_DHCHAP_SECRET	= 1 << 23,
+	NVMF_OPT_DHCHAP_CTRL_SECRET = 1 << 24,
 };
 
 /**
@@ -97,6 +99,9 @@ enum {
  * @max_reconnects: maximum number of allowed reconnect attempts before removing
  *              the controller, (-1) means reconnect forever, zero means remove
  *              immediately;
+ * @dhchap_secret: DH-HMAC-CHAP secret
+ * @dhchap_ctrl_secret: DH-HMAC-CHAP controller secret for bi-directional
+ *              authentication
  * @disable_sqflow: disable controller sq flow control
  * @hdr_digest: generate/verify header digest (TCP)
  * @data_digest: generate/verify data digest (TCP)
@@ -121,6 +126,8 @@ struct nvmf_ctrl_options {
 	unsigned int		kato;
 	struct nvmf_host	*host;
 	int			max_reconnects;
+	char			*dhchap_secret;
+	char			*dhchap_ctrl_secret;
 	bool			disable_sqflow;
 	bool			hdr_digest;
 	bool			data_digest;
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 1ea908d43e17..f110437a3190 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -323,6 +323,15 @@ struct nvme_ctrl {
 	struct work_struct ana_work;
 #endif
 
+#ifdef CONFIG_NVME_AUTH
+	struct work_struct dhchap_auth_work;
+	struct list_head dhchap_auth_list;
+	struct mutex dhchap_auth_mutex;
+	struct nvme_dhchap_key *host_key;
+	struct nvme_dhchap_key *ctrl_key;
+	u16 transaction;
+#endif
+
 	/* Power saving configuration */
 	u64 ps_max_latency_us;
 	bool apst_enabled;
@@ -928,6 +937,28 @@ static inline bool nvme_ctrl_sgl_supported(struct nvme_ctrl *ctrl)
 	return ctrl->sgls & ((1 << 0) | (1 << 1));
 }
 
+#ifdef CONFIG_NVME_AUTH
+void nvme_auth_init_ctrl(struct nvme_ctrl *ctrl);
+void nvme_auth_stop(struct nvme_ctrl *ctrl);
+int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid);
+int nvme_auth_wait(struct nvme_ctrl *ctrl, int qid);
+void nvme_auth_reset(struct nvme_ctrl *ctrl);
+void nvme_auth_free(struct nvme_ctrl *ctrl);
+int nvme_auth_generate_key(u8 *secret, struct nvme_dhchap_key **ret_key);
+#else
+static inline void nvme_auth_init_ctrl(struct nvme_ctrl *ctrl) {};
+static inline void nvme_auth_stop(struct nvme_ctrl *ctrl) {};
+static inline int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid)
+{
+	return -EPROTONOSUPPORT;
+}
+static inline int nvme_auth_wait(struct nvme_ctrl *ctrl, int qid)
+{
+	return NVME_SC_AUTH_REQUIRED;
+}
+static inline void nvme_auth_free(struct nvme_ctrl *ctrl) {};
+#endif
+
 u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 			 u8 opcode);
 int nvme_execute_passthru_rq(struct request *rq);
diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index d9f19d901313..bd63568c0068 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -1197,6 +1197,7 @@ static void nvme_rdma_error_recovery_work(struct work_struct *work)
 	struct nvme_rdma_ctrl *ctrl = container_of(work,
 			struct nvme_rdma_ctrl, err_work);
 
+	nvme_auth_stop(&ctrl->ctrl);
 	nvme_stop_keep_alive(&ctrl->ctrl);
 	flush_work(&ctrl->ctrl.async_event_work);
 	nvme_rdma_teardown_io_queues(ctrl, false);
diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index ad3a2bf2f1e9..be52902b5f3b 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -2174,6 +2174,7 @@ static void nvme_tcp_error_recovery_work(struct work_struct *work)
 				struct nvme_tcp_ctrl, err_work);
 	struct nvme_ctrl *ctrl = &tcp_ctrl->ctrl;
 
+	nvme_auth_stop(ctrl);
 	nvme_stop_keep_alive(ctrl);
 	flush_work(&ctrl->async_event_work);
 	nvme_tcp_teardown_io_queues(ctrl, false);
diff --git a/drivers/nvme/host/trace.c b/drivers/nvme/host/trace.c
index 2a89c5aa0790..1c36fcedea20 100644
--- a/drivers/nvme/host/trace.c
+++ b/drivers/nvme/host/trace.c
@@ -287,6 +287,34 @@ static const char *nvme_trace_fabrics_property_get(struct trace_seq *p, u8 *spc)
 	return ret;
 }
 
+static const char *nvme_trace_fabrics_auth_send(struct trace_seq *p, u8 *spc)
+{
+	const char *ret = trace_seq_buffer_ptr(p);
+	u8 spsp0 = spc[1];
+	u8 spsp1 = spc[2];
+	u8 secp = spc[3];
+	u32 tl = get_unaligned_le32(spc + 4);
+
+	trace_seq_printf(p, "spsp0=%02x, spsp1=%02x, secp=%02x, tl=%u",
+			 spsp0, spsp1, secp, tl);
+	trace_seq_putc(p, 0);
+	return ret;
+}
+
+static const char *nvme_trace_fabrics_auth_receive(struct trace_seq *p, u8 *spc)
+{
+	const char *ret = trace_seq_buffer_ptr(p);
+	u8 spsp0 = spc[1];
+	u8 spsp1 = spc[2];
+	u8 secp = spc[3];
+	u32 al = get_unaligned_le32(spc + 4);
+
+	trace_seq_printf(p, "spsp0=%02x, spsp1=%02x, secp=%02x, al=%u",
+			 spsp0, spsp1, secp, al);
+	trace_seq_putc(p, 0);
+	return ret;
+}
+
 static const char *nvme_trace_fabrics_common(struct trace_seq *p, u8 *spc)
 {
 	const char *ret = trace_seq_buffer_ptr(p);
@@ -306,6 +334,10 @@ const char *nvme_trace_parse_fabrics_cmd(struct trace_seq *p,
 		return nvme_trace_fabrics_connect(p, spc);
 	case nvme_fabrics_type_property_get:
 		return nvme_trace_fabrics_property_get(p, spc);
+	case nvme_fabrics_type_auth_send:
+		return nvme_trace_fabrics_auth_send(p, spc);
+	case nvme_fabrics_type_auth_receive:
+		return nvme_trace_fabrics_auth_receive(p, spc);
 	default:
 		return nvme_trace_fabrics_common(p, spc);
 	}
-- 
2.29.2

Re: [PATCH 06/11] nvme: Implement In-Band authentication

[Vladislav Bolkhovitin]

On 7/21/21 9:08 AM, Hannes Reinecke wrote:
> On 7/20/21 10:27 PM, Vladislav Bolkhovitin wrote:
>>
>> On 7/16/21 2:04 PM, Hannes Reinecke wrote:
>>
>> [...]
>>
>>> +struct nvmet_dhchap_hash_map {
>>> +	int id;
>>> +	int hash_len;
>>> +	const char hmac[15];
>>> +	const char digest[15];
>>> +} hash_map[] = {
>>> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA256,
>>> +	 .hash_len = 32,
>>> +	 .hmac = "hmac(sha256)", .digest = "sha256" },
>>> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA384,
>>> +	 .hash_len = 48,
>>> +	 .hmac = "hmac(sha384)", .digest = "sha384" },
>>> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA512,
>>> +	 .hash_len = 64,
>>> +	 .hmac = "hmac(sha512)", .digest = "sha512" },
>>> +};
>>
>> "hmac()" is always here, so why not to just auto-generate hmac(sha512)
>> from sha512?
>>
> 
> ... all part of the learning curve ...
> If that's true then of course I can auto-generate the hmac name.

As far as I understand, this naming convention is as stable as any other
interface in the kernel.

Vlad

Re: [PATCH 06/11] nvme: Implement In-Band authentication

[Vladislav Bolkhovitin]

On 7/21/21 9:08 AM, Hannes Reinecke wrote:
> On 7/20/21 10:27 PM, Vladislav Bolkhovitin wrote:
>>
>> On 7/16/21 2:04 PM, Hannes Reinecke wrote:
>>
>> [...]
>>
>>> +struct nvmet_dhchap_hash_map {
>>> +	int id;
>>> +	int hash_len;
>>> +	const char hmac[15];
>>> +	const char digest[15];
>>> +} hash_map[] = {
>>> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA256,
>>> +	 .hash_len = 32,
>>> +	 .hmac = "hmac(sha256)", .digest = "sha256" },
>>> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA384,
>>> +	 .hash_len = 48,
>>> +	 .hmac = "hmac(sha384)", .digest = "sha384" },
>>> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA512,
>>> +	 .hash_len = 64,
>>> +	 .hmac = "hmac(sha512)", .digest = "sha512" },
>>> +};
>>
>> "hmac()" is always here, so why not to just auto-generate hmac(sha512)
>> from sha512?
>>
> 
> ... all part of the learning curve ...
> If that's true then of course I can auto-generate the hmac name.

As far as I understand, this naming convention is as stable as any other
interface in the kernel.

Vlad

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 06/11] nvme: Implement In-Band authentication

[Hannes Reinecke]

On 7/20/21 10:28 PM, Vladislav Bolkhovitin wrote:
> 
> On 7/18/21 3:21 PM, Hannes Reinecke wrote:
>> On 7/17/21 9:22 AM, Sagi Grimberg wrote:
>>>> Implement NVMe-oF In-Band authentication. This patch adds two new
>>>> fabric options 'dhchap_key' to specify the PSK
>>>
>>> pre-shared-key.
>>>
>>> Also, we need a sysfs knob to rotate the key that will trigger
>>> re-authentication or even a simple controller(s-plural) reset, so this
>>> should go beyond just the connection string.
>>>
>>
>> Yeah, re-authentication currently is not implemented. I first wanted to
>> get this patchset out such that we can settle on the userspace interface
>> (both from host and target).
>> I'll have to think on how we should handle authentication; one of the
>> really interesting cases would be when one malicious admin will _just_
>> send a 'negotiate' command to the controller. As per spec the controller
>> will be waiting for an 'authentication receive' command to send a
>> 'challenge' payload back to the host. But that will never come, so as it
>> stands currently the controller is required to abort the connection.
>> Not very nice.
> 
> Yes, in this case after some reasonable timeout (I would suggest 10-15
> seconds) the controller expected to abort connection and clean up all
> allocated resources.
> 
> To handle DoS possibility to make too many such "orphan" negotiations,
> hence consume all controller memory, some additional handling is needed.
> For simplicity as a first step I would suggest to have a global limit on
> number of currently being authenticated connections.
> 
> [...]
> 
>>>> +    chap->key = nvme_auth_extract_secret(ctrl->opts->dhchap_secret,
>>>> +                         &chap->key_len);
>>>> +    if (IS_ERR(chap->key)) {
>>>> +        ret = PTR_ERR(chap->key);
>>>> +        chap->key = NULL;
>>>> +        return ret;
>>>> +    }
>>>> +
>>>> +    if (key_hash == 0)
>>>> +        return 0;
>>>> +
>>>> +    hmac_name = nvme_auth_hmac_name(key_hash);
>>>> +    if (!hmac_name) {
>>>> +        pr_debug("Invalid key hash id %d\n", key_hash);
>>>> +        return -EKEYREJECTED;
>>>> +    }
>>>
>>> Why does the user influence the hmac used? isn't that is driven
>>> by the susbsystem?
>>>
>>> I don't think that the user should choose in this level.
>>>
>>
>> That is another weirdness of the spec.
>> The _secret_ will be hashed with a specific function, and that function
>> is stated in the transport representation.
>> (Cf section "DH-HMAC-CHAP Security Requirements").
>> This is _not_ the hash function used by the authentication itself, which
>> will be selected by the protocol.
>> So it's not the user here, but rather the transport specification of the
>> key which selects the hash algorithm.
> 
> Yes, good catch. It looks as a minor errata material to specify that
> hash function here is implementation specific.
> 
> I would suggest to just hardcode SHA512 here. Users don't have to be
> confused by this.
> 
Sure, can do. My reasoning was that the target absolutely has to support
the hash functions specified in the PSK, so that will be a safe bet to
choose for the hash function in the protocol itself.
(Any other hash function _might_ not be preset on the target.)
But if the PSK does not specify a hash the target need to pick one; and
for that of course we can use SHA512.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		           Kernel Storage Architect
hare@suse.de			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), GF: Felix Imendörffer

Re: [PATCH 06/11] nvme: Implement In-Band authentication

[Hannes Reinecke]

On 7/20/21 10:28 PM, Vladislav Bolkhovitin wrote:
> 
> On 7/18/21 3:21 PM, Hannes Reinecke wrote:
>> On 7/17/21 9:22 AM, Sagi Grimberg wrote:
>>>> Implement NVMe-oF In-Band authentication. This patch adds two new
>>>> fabric options 'dhchap_key' to specify the PSK
>>>
>>> pre-shared-key.
>>>
>>> Also, we need a sysfs knob to rotate the key that will trigger
>>> re-authentication or even a simple controller(s-plural) reset, so this
>>> should go beyond just the connection string.
>>>
>>
>> Yeah, re-authentication currently is not implemented. I first wanted to
>> get this patchset out such that we can settle on the userspace interface
>> (both from host and target).
>> I'll have to think on how we should handle authentication; one of the
>> really interesting cases would be when one malicious admin will _just_
>> send a 'negotiate' command to the controller. As per spec the controller
>> will be waiting for an 'authentication receive' command to send a
>> 'challenge' payload back to the host. But that will never come, so as it
>> stands currently the controller is required to abort the connection.
>> Not very nice.
> 
> Yes, in this case after some reasonable timeout (I would suggest 10-15
> seconds) the controller expected to abort connection and clean up all
> allocated resources.
> 
> To handle DoS possibility to make too many such "orphan" negotiations,
> hence consume all controller memory, some additional handling is needed.
> For simplicity as a first step I would suggest to have a global limit on
> number of currently being authenticated connections.
> 
> [...]
> 
>>>> +    chap->key = nvme_auth_extract_secret(ctrl->opts->dhchap_secret,
>>>> +                         &chap->key_len);
>>>> +    if (IS_ERR(chap->key)) {
>>>> +        ret = PTR_ERR(chap->key);
>>>> +        chap->key = NULL;
>>>> +        return ret;
>>>> +    }
>>>> +
>>>> +    if (key_hash == 0)
>>>> +        return 0;
>>>> +
>>>> +    hmac_name = nvme_auth_hmac_name(key_hash);
>>>> +    if (!hmac_name) {
>>>> +        pr_debug("Invalid key hash id %d\n", key_hash);
>>>> +        return -EKEYREJECTED;
>>>> +    }
>>>
>>> Why does the user influence the hmac used? isn't that is driven
>>> by the susbsystem?
>>>
>>> I don't think that the user should choose in this level.
>>>
>>
>> That is another weirdness of the spec.
>> The _secret_ will be hashed with a specific function, and that function
>> is stated in the transport representation.
>> (Cf section "DH-HMAC-CHAP Security Requirements").
>> This is _not_ the hash function used by the authentication itself, which
>> will be selected by the protocol.
>> So it's not the user here, but rather the transport specification of the
>> key which selects the hash algorithm.
> 
> Yes, good catch. It looks as a minor errata material to specify that
> hash function here is implementation specific.
> 
> I would suggest to just hardcode SHA512 here. Users don't have to be
> confused by this.
> 
Sure, can do. My reasoning was that the target absolutely has to support
the hash functions specified in the PSK, so that will be a safe bet to
choose for the hash function in the protocol itself.
(Any other hash function _might_ not be preset on the target.)
But if the PSK does not specify a hash the target need to pick one; and
for that of course we can use SHA512.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		           Kernel Storage Architect
hare@suse.de			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), GF: Felix Imendörffer

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 06/11] nvme: Implement In-Band authentication

[Hannes Reinecke]

On 7/20/21 10:27 PM, Vladislav Bolkhovitin wrote:
> 
> On 7/16/21 2:04 PM, Hannes Reinecke wrote:
> 
> [...]
> 
>> +struct nvmet_dhchap_hash_map {
>> +	int id;
>> +	int hash_len;
>> +	const char hmac[15];
>> +	const char digest[15];
>> +} hash_map[] = {
>> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA256,
>> +	 .hash_len = 32,
>> +	 .hmac = "hmac(sha256)", .digest = "sha256" },
>> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA384,
>> +	 .hash_len = 48,
>> +	 .hmac = "hmac(sha384)", .digest = "sha384" },
>> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA512,
>> +	 .hash_len = 64,
>> +	 .hmac = "hmac(sha512)", .digest = "sha512" },
>> +};
> 
> "hmac()" is always here, so why not to just auto-generate hmac(sha512)
> from sha512?
> 

... all part of the learning curve ...
If that's true then of course I can auto-generate the hmac name.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		           Kernel Storage Architect
hare@suse.de			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), GF: Felix Imendörffer

Re: [PATCH 06/11] nvme: Implement In-Band authentication

[Hannes Reinecke]

On 7/20/21 10:27 PM, Vladislav Bolkhovitin wrote:
> 
> On 7/16/21 2:04 PM, Hannes Reinecke wrote:
> 
> [...]
> 
>> +struct nvmet_dhchap_hash_map {
>> +	int id;
>> +	int hash_len;
>> +	const char hmac[15];
>> +	const char digest[15];
>> +} hash_map[] = {
>> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA256,
>> +	 .hash_len = 32,
>> +	 .hmac = "hmac(sha256)", .digest = "sha256" },
>> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA384,
>> +	 .hash_len = 48,
>> +	 .hmac = "hmac(sha384)", .digest = "sha384" },
>> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA512,
>> +	 .hash_len = 64,
>> +	 .hmac = "hmac(sha512)", .digest = "sha512" },
>> +};
> 
> "hmac()" is always here, so why not to just auto-generate hmac(sha512)
> from sha512?
> 

... all part of the learning curve ...
If that's true then of course I can auto-generate the hmac name.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		           Kernel Storage Architect
hare@suse.de			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), GF: Felix Imendörffer

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 06/11] nvme: Implement In-Band authentication

[Vladislav Bolkhovitin]

On 7/18/21 3:21 PM, Hannes Reinecke wrote:
> On 7/17/21 9:22 AM, Sagi Grimberg wrote:
>>> Implement NVMe-oF In-Band authentication. This patch adds two new
>>> fabric options 'dhchap_key' to specify the PSK
>>
>> pre-shared-key.
>>
>> Also, we need a sysfs knob to rotate the key that will trigger
>> re-authentication or even a simple controller(s-plural) reset, so this
>> should go beyond just the connection string.
>>
> 
> Yeah, re-authentication currently is not implemented. I first wanted to
> get this patchset out such that we can settle on the userspace interface
> (both from host and target).
> I'll have to think on how we should handle authentication; one of the
> really interesting cases would be when one malicious admin will _just_
> send a 'negotiate' command to the controller. As per spec the controller
> will be waiting for an 'authentication receive' command to send a
> 'challenge' payload back to the host. But that will never come, so as it
> stands currently the controller is required to abort the connection.
> Not very nice.

Yes, in this case after some reasonable timeout (I would suggest 10-15
seconds) the controller expected to abort connection and clean up all
allocated resources.

To handle DoS possibility to make too many such "orphan" negotiations,
hence consume all controller memory, some additional handling is needed.
For simplicity as a first step I would suggest to have a global limit on
number of currently being authenticated connections.

[...]

>>> +    chap->key = nvme_auth_extract_secret(ctrl->opts->dhchap_secret,
>>> +                         &chap->key_len);
>>> +    if (IS_ERR(chap->key)) {
>>> +        ret = PTR_ERR(chap->key);
>>> +        chap->key = NULL;
>>> +        return ret;
>>> +    }
>>> +
>>> +    if (key_hash == 0)
>>> +        return 0;
>>> +
>>> +    hmac_name = nvme_auth_hmac_name(key_hash);
>>> +    if (!hmac_name) {
>>> +        pr_debug("Invalid key hash id %d\n", key_hash);
>>> +        return -EKEYREJECTED;
>>> +    }
>>
>> Why does the user influence the hmac used? isn't that is driven
>> by the susbsystem?
>>
>> I don't think that the user should choose in this level.
>>
> 
> That is another weirdness of the spec.
> The _secret_ will be hashed with a specific function, and that function
> is stated in the transport representation.
> (Cf section "DH-HMAC-CHAP Security Requirements").
> This is _not_ the hash function used by the authentication itself, which
> will be selected by the protocol.
> So it's not the user here, but rather the transport specification of the
> key which selects the hash algorithm.

Yes, good catch. It looks as a minor errata material to specify that
hash function here is implementation specific.

I would suggest to just hardcode SHA512 here. Users don't have to be
confused by this.

Vlad

Re: [PATCH 06/11] nvme: Implement In-Band authentication

[Vladislav Bolkhovitin]

On 7/18/21 3:21 PM, Hannes Reinecke wrote:
> On 7/17/21 9:22 AM, Sagi Grimberg wrote:
>>> Implement NVMe-oF In-Band authentication. This patch adds two new
>>> fabric options 'dhchap_key' to specify the PSK
>>
>> pre-shared-key.
>>
>> Also, we need a sysfs knob to rotate the key that will trigger
>> re-authentication or even a simple controller(s-plural) reset, so this
>> should go beyond just the connection string.
>>
> 
> Yeah, re-authentication currently is not implemented. I first wanted to
> get this patchset out such that we can settle on the userspace interface
> (both from host and target).
> I'll have to think on how we should handle authentication; one of the
> really interesting cases would be when one malicious admin will _just_
> send a 'negotiate' command to the controller. As per spec the controller
> will be waiting for an 'authentication receive' command to send a
> 'challenge' payload back to the host. But that will never come, so as it
> stands currently the controller is required to abort the connection.
> Not very nice.

Yes, in this case after some reasonable timeout (I would suggest 10-15
seconds) the controller expected to abort connection and clean up all
allocated resources.

To handle DoS possibility to make too many such "orphan" negotiations,
hence consume all controller memory, some additional handling is needed.
For simplicity as a first step I would suggest to have a global limit on
number of currently being authenticated connections.

[...]

>>> +    chap->key = nvme_auth_extract_secret(ctrl->opts->dhchap_secret,
>>> +                         &chap->key_len);
>>> +    if (IS_ERR(chap->key)) {
>>> +        ret = PTR_ERR(chap->key);
>>> +        chap->key = NULL;
>>> +        return ret;
>>> +    }
>>> +
>>> +    if (key_hash == 0)
>>> +        return 0;
>>> +
>>> +    hmac_name = nvme_auth_hmac_name(key_hash);
>>> +    if (!hmac_name) {
>>> +        pr_debug("Invalid key hash id %d\n", key_hash);
>>> +        return -EKEYREJECTED;
>>> +    }
>>
>> Why does the user influence the hmac used? isn't that is driven
>> by the susbsystem?
>>
>> I don't think that the user should choose in this level.
>>
> 
> That is another weirdness of the spec.
> The _secret_ will be hashed with a specific function, and that function
> is stated in the transport representation.
> (Cf section "DH-HMAC-CHAP Security Requirements").
> This is _not_ the hash function used by the authentication itself, which
> will be selected by the protocol.
> So it's not the user here, but rather the transport specification of the
> key which selects the hash algorithm.

Yes, good catch. It looks as a minor errata material to specify that
hash function here is implementation specific.

I would suggest to just hardcode SHA512 here. Users don't have to be
confused by this.

Vlad

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 06/11] nvme: Implement In-Band authentication

[Vladislav Bolkhovitin]

On 7/16/21 2:04 PM, Hannes Reinecke wrote:

[...]

> +struct nvmet_dhchap_hash_map {
> +	int id;
> +	int hash_len;
> +	const char hmac[15];
> +	const char digest[15];
> +} hash_map[] = {
> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA256,
> +	 .hash_len = 32,
> +	 .hmac = "hmac(sha256)", .digest = "sha256" },
> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA384,
> +	 .hash_len = 48,
> +	 .hmac = "hmac(sha384)", .digest = "sha384" },
> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA512,
> +	 .hash_len = 64,
> +	 .hmac = "hmac(sha512)", .digest = "sha512" },
> +};

"hmac()" is always here, so why not to just auto-generate hmac(sha512)
from sha512?

Vlad

Re: [PATCH 06/11] nvme: Implement In-Band authentication

[Vladislav Bolkhovitin]

On 7/16/21 2:04 PM, Hannes Reinecke wrote:

[...]

> +struct nvmet_dhchap_hash_map {
> +	int id;
> +	int hash_len;
> +	const char hmac[15];
> +	const char digest[15];
> +} hash_map[] = {
> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA256,
> +	 .hash_len = 32,
> +	 .hmac = "hmac(sha256)", .digest = "sha256" },
> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA384,
> +	 .hash_len = 48,
> +	 .hmac = "hmac(sha384)", .digest = "sha384" },
> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA512,
> +	 .hash_len = 64,
> +	 .hmac = "hmac(sha512)", .digest = "sha512" },
> +};

"hmac()" is always here, so why not to just auto-generate hmac(sha512)
from sha512?

Vlad

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 06/11] nvme: Implement In-Band authentication

[Sagi Grimberg]

>>> Implement NVMe-oF In-Band authentication. This patch adds two new
>>> fabric options 'dhchap_key' to specify the PSK
>>
>> pre-shared-key.
>>
>> Also, we need a sysfs knob to rotate the key that will trigger
>> re-authentication or even a simple controller(s-plural) reset, so this
>> should go beyond just the connection string.
>>
> 
> Yeah, re-authentication currently is not implemented. I first wanted to 
> get this patchset out such that we can settle on the userspace interface 
> (both from host and target).

I think this is mandatory from the start (at the very least from the
host side) because credentials get rotated.

The flow imo needs to be that the target allows to update the
credentials even when the subsystem is exposed and connected, but
keep all existing hosts connected such that only new connections will
need the new credentials. (an incremental step, which would be optional
is to allow it for a configurable grace period and then disconnect
existing hosts, which would force new connections with updated
credentials).

 From the host side, we need to expose a controller (or more
appropriately a subsystem) sysfs file to override the existing key. That
can initially just trigger a controller reset, and incrementally can do
a graceful re-authentication).

> I'll have to think on how we should handle authentication; one of the 
> really interesting cases would be when one malicious admin will _just_ 
> send a 'negotiate' command to the controller. As per spec the controller 
> will be waiting for an 'authentication receive' command to send a 
> 'challenge' payload back to the host. But that will never come, so as it 
> stands currently the controller is required to abort the connection.
> Not very nice.

Well, at the moment we can keep it open and discuss this in the TWG.

>> P.S. can you add also the nvme-cli code in the next go?
>>
> Oh, sure. It's already sitting around in my local repo (surprise, 
> surprise); will be ending it out next time.

Great, thanks.

>>> and 'dhchap_authenticate'
>>> to request bi-directional authentication of both the host and the 
>>> controller.
>>
>> bidirectional? not uni-directional?
>>
> 
> Yeah, that's a bit of a misnomer. When a PSK is specified, the 
> controller will start the authentication protocol such that the 
> _controller_ can validate the host. If the host wants to authenticate 
> the controller is needs to set this flag.
> Hence bi-directional authentication.
> But I'm the first to admit that this is poor wording for the flag.

It is misleading. But I'll need to go look again because I didn't
see the host authenticating the controller..

>>> Signed-off-by: Hannes Reinecke <hare@suse.de>
>>> ---
>>>   drivers/nvme/host/Kconfig   |  11 +
>>>   drivers/nvme/host/Makefile  |   1 +
>>>   drivers/nvme/host/auth.c    | 813 ++++++++++++++++++++++++++++++++++++
>>>   drivers/nvme/host/auth.h    |  23 +
>>>   drivers/nvme/host/core.c    |  77 +++-
>>>   drivers/nvme/host/fabrics.c |  65 ++-
>>>   drivers/nvme/host/fabrics.h |   8 +
>>>   drivers/nvme/host/nvme.h    |  15 +
>>>   drivers/nvme/host/trace.c   |  32 ++
>>>   9 files changed, 1041 insertions(+), 4 deletions(-)
>>>   create mode 100644 drivers/nvme/host/auth.c
>>>   create mode 100644 drivers/nvme/host/auth.h
>>>
>>> diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig
>>> index c3f3d77f1aac..853c546305e9 100644
>>> --- a/drivers/nvme/host/Kconfig
>>> +++ b/drivers/nvme/host/Kconfig
>>> @@ -85,3 +85,14 @@ config NVME_TCP
>>>         from https://github.com/linux-nvme/nvme-cli.
>>>         If unsure, say N.
>>> +
>>> +config NVME_AUTH
>>> +    bool "NVM Express over Fabrics In-Band Authentication"
>>> +    depends on NVME_TCP
>>> +    select CRYPTO_SHA256
>>> +    select CRYPTO_SHA512
>>> +    help
>>> +      This provides support for NVMe over Fabrics In-Band 
>>> Authentication
>>> +      for the NVMe over TCP transport.
>>
>> In this form, nothing is specific to nvme-tcp here afaict.
>>
> 
> Indeed. I guess we can leave out the nvme-tcp reference here.
> 
>>> +
>>> +      If unsure, say N.
>>> diff --git a/drivers/nvme/host/Makefile b/drivers/nvme/host/Makefile
>>> index cbc509784b2e..03748a55a12b 100644
>>> --- a/drivers/nvme/host/Makefile
>>> +++ b/drivers/nvme/host/Makefile
>>> @@ -20,6 +20,7 @@ nvme-core-$(CONFIG_NVME_HWMON)        += hwmon.o
>>>   nvme-y                    += pci.o
>>>   nvme-fabrics-y                += fabrics.o
>>> +nvme-fabrics-$(CONFIG_NVME_AUTH)    += auth.o
>>>   nvme-rdma-y                += rdma.o
>>> diff --git a/drivers/nvme/host/auth.c b/drivers/nvme/host/auth.c
>>> new file mode 100644
>>> index 000000000000..448a3adebea6
>>> --- /dev/null
>>> +++ b/drivers/nvme/host/auth.c
>>> @@ -0,0 +1,813 @@
>>> +// SPDX-License-Identifier: GPL-2.0
>>> +/*
>>> + * Copyright (c) 2020 Hannes Reinecke, SUSE Linux
>>> + */
>>> +
>>> +#include <linux/crc32.h>
>>> +#include <linux/base64.h>
>>> +#include <asm/unaligned.h>
>>> +#include <crypto/hash.h>
>>> +#include <crypto/kpp.h>
>>> +#include "nvme.h"
>>> +#include "fabrics.h"
>>> +#include "auth.h"
>>> +
>>> +static u32 nvme_dhchap_seqnum;
>>> +
>>> +struct nvme_dhchap_context {
>>
>> Maybe nvme_dhchap_queue_context ?
>>
>> I'm thinking that we should perhaps split
>> it to host-wide, subsys-wide and queue specific
>> auth contexts?
>>
>> Let's see...
>>
> 
> Interestingly enough, that's what I did for the target side.
> For the host side I found it easier that way, as then we'll have a 
> single authentication context which can be deleted after authentication 
> finished, and be sure that we removed _all_ information.
> Security and all that.
> Splitting it off would require to remove information on three different 
> places, and observing life-time rules for them.
> So more of a chance to mess things up :-)

I understand what you are saying, but this way it will be challanging
to cross check stuff. It is also where things sort of belong...

>>> +    struct crypto_shash *shash_tfm;
>>> +    unsigned char *key;
>>> +    size_t key_len;
>>> +    int qid;
>>> +    u32 s1;
>>> +    u32 s2;
>>> +    u16 transaction;
>>> +    u8 status;
>>> +    u8 hash_id;
>>> +    u8 hash_len;
>>> +    u8 c1[64];
>>> +    u8 c2[64];
>>> +    u8 response[64];
>>> +    u8 *ctrl_key;
>>> +    int ctrl_key_len;
>>> +    u8 *host_key;
>>> +    int host_key_len;
>>> +    u8 *sess_key;
>>> +    int sess_key_len;
>>> +};
>>> +
>>> +struct nvmet_dhchap_hash_map {
>>
>> nvmet?
>>
> 
> Yeah; originally I coded that for the target side, and only later moved 
> it into the host side to have it usable for both.
> Will be fixing it up.
> 
>>> +    int id;
>>> +    int hash_len;
>>> +    const char hmac[15];
>>> +    const char digest[15];
>>> +} hash_map[] = {
>>> +    {.id = NVME_AUTH_DHCHAP_HASH_SHA256,
>>> +     .hash_len = 32,
>>> +     .hmac = "hmac(sha256)", .digest = "sha256" },
>>> +    {.id = NVME_AUTH_DHCHAP_HASH_SHA384,
>>> +     .hash_len = 48,
>>> +     .hmac = "hmac(sha384)", .digest = "sha384" },
>>> +    {.id = NVME_AUTH_DHCHAP_HASH_SHA512,
>>> +     .hash_len = 64,
>>> +     .hmac = "hmac(sha512)", .digest = "sha512" },
>>> +};
>>> +
>>> +const char *nvme_auth_hmac_name(int hmac_id)
>>
>> Should these arrays be static?
>>
> 
> Definitely.
> 
>>> +{
>>> +    int i;
>>> +
>>> +    for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
>>> +        if (hash_map[i].id == hmac_id)
>>> +            return hash_map[i].hmac;
>>> +    }
>>> +    return NULL;
>>> +}
>>> +EXPORT_SYMBOL_GPL(nvme_auth_hmac_name);
>>> +
>>> +const char *nvme_auth_digest_name(int hmac_id)
>>> +{
>>> +    int i;
>>> +
>>> +    for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
>>> +        if (hash_map[i].id == hmac_id)
>>> +            return hash_map[i].digest;
>>> +    }
>>> +    return NULL;
>>> +}
>>> +EXPORT_SYMBOL_GPL(nvme_auth_digest_name);
>>> +
>>> +int nvme_auth_hmac_len(int hmac_id)
>>> +{
>>> +    int i;
>>> +
>>> +    for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
>>> +        if (hash_map[i].id == hmac_id)
>>> +            return hash_map[i].hash_len;
>>> +    }
>>> +    return -1;
>>> +}
>>> +EXPORT_SYMBOL_GPL(nvme_auth_hmac_len);
>>> +
>>> +int nvme_auth_hmac_id(const char *hmac_name)
>>> +{
>>> +    int i;
>>> +
>>> +    for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
>>> +        if (!strncmp(hash_map[i].hmac, hmac_name,
>>> +                 strlen(hash_map[i].hmac)))
>>> +            return hash_map[i].id;
>>> +    }
>>> +    return -1;
>>> +}
>>> +EXPORT_SYMBOL_GPL(nvme_auth_hmac_id);
>>> +
>>> +unsigned char *nvme_auth_extract_secret(unsigned char *dhchap_secret,
>>> +                    size_t *dhchap_key_len)
>>> +{
>>> +    unsigned char *dhchap_key;
>>> +    u32 crc;
>>> +    int key_len;
>>> +    size_t allocated_len;
>>> +
>>> +    allocated_len = strlen(dhchap_secret) - 10;
>>
>> the 10 feels like a magic here, should at least note this is the
>> "DHHC-1:..." prefix.
>>
> 
> It _is_ magic. And it might even be better to just pass in the string 
> _without_ the DHHC-1: prefix.

I don't think that the user should pass in a key in that form at all,
its none of its concern the compliant NVMe representation is.

> 
>>> +    dhchap_key = kzalloc(allocated_len, GFP_KERNEL);
>>> +    if (!dhchap_key)
>>> +        return ERR_PTR(-ENOMEM);
>>> +
>>> +    key_len = base64_decode(dhchap_secret + 10,
>>> +                allocated_len, dhchap_key);
>>> +    if (key_len != 36 && key_len != 52 &&
>>> +        key_len != 68) {
>>> +        pr_debug("Invalid DH-HMAC-CHAP key len %d\n",
>>> +             key_len);
>>> +        kfree(dhchap_key);
>>> +        return ERR_PTR(-EINVAL);
>>> +    }
>>> +    pr_debug("DH-HMAC-CHAP Key: %*ph\n",
>>> +         (int)key_len, dhchap_key);
>>
>> One can argue if even printing this is problematic..
>>
> 
> Debugging scaffolding. You wouldn't believe how many things can go wrong...
> 
> And yes, that should be removed.

Cool.

>>> +
>>> +    /* The last four bytes is the CRC in little-endian format */
>>> +    key_len -= 4;
>>> +    /*
>>> +     * The linux implementation doesn't do pre- and post-increments,
>>> +     * so we have to do it manually.
>>> +     */
>>> +    crc = ~crc32(~0, dhchap_key, key_len);
>>> +
>>> +    if (get_unaligned_le32(dhchap_key + key_len) != crc) {
>>> +        pr_debug("DH-HMAC-CHAP crc mismatch (key %08x, crc %08x)\n",
>>> +               get_unaligned_le32(dhchap_key + key_len), crc);
>>> +        kfree(dhchap_key);
>>> +        return ERR_PTR(-EKEYREJECTED);
>>> +    }
>>> +    *dhchap_key_len = key_len;
>>> +    return dhchap_key;
>>> +}
>>> +EXPORT_SYMBOL_GPL(nvme_auth_extract_secret);
>>> +
>>> +static int nvme_auth_send(struct nvme_ctrl *ctrl, int qid,
>>> +              void *data, size_t tl)
>>> +{
>>> +    struct nvme_command cmd = {};
>>> +    blk_mq_req_flags_t flags = qid == NVME_QID_ANY ?
>>> +        0 : BLK_MQ_REQ_NOWAIT | BLK_MQ_REQ_RESERVED;
>>> +    struct request_queue *q = qid == NVME_QID_ANY ?
>>> +        ctrl->fabrics_q : ctrl->connect_q;
>>> +    int ret;
>>> +
>>> +    cmd.auth_send.opcode = nvme_fabrics_command;
>>> +    cmd.auth_send.fctype = nvme_fabrics_type_auth_send;
>>> +    cmd.auth_send.secp = NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER;
>>> +    cmd.auth_send.spsp0 = 0x01;
>>> +    cmd.auth_send.spsp1 = 0x01;
>>> +    cmd.auth_send.tl = tl;
>>> +
>>> +    ret = __nvme_submit_sync_cmd(q, &cmd, NULL, data, tl, 0, qid,
>>> +                     0, flags);
>>> +    if (ret)
>>> +        dev_dbg(ctrl->device,
>>> +            "%s: qid %d error %d\n", __func__, qid, ret);
>>
>> Maybe a little more informative print rather than __func__ ?
>>
> 
> Yes, can do.
> 
>>> +    return ret;
>>> +}
>>> +
>>> +static int nvme_auth_receive(struct nvme_ctrl *ctrl, int qid,
>>> +                 void *buf, size_t al,
>>> +                 u16 transaction, u8 expected_msg )
>>> +{
>>> +    struct nvme_command cmd = {};
>>> +    struct nvmf_auth_dhchap_failure_data *data = buf;
>>> +    blk_mq_req_flags_t flags = qid == NVME_QID_ANY ?
>>> +        0 : BLK_MQ_REQ_NOWAIT | BLK_MQ_REQ_RESERVED;
>>> +    struct request_queue *q = qid == NVME_QID_ANY ?
>>> +        ctrl->fabrics_q : ctrl->connect_q;
>>> +    int ret;
>>> +
>>> +    cmd.auth_receive.opcode = nvme_fabrics_command;
>>> +    cmd.auth_receive.fctype = nvme_fabrics_type_auth_receive;
>>> +    cmd.auth_receive.secp = NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER;
>>> +    cmd.auth_receive.spsp0 = 0x01;
>>> +    cmd.auth_receive.spsp1 = 0x01;
>>> +    cmd.auth_receive.al = al;
>>> +
>>> +    ret = __nvme_submit_sync_cmd(q, &cmd, NULL, buf, al, 0, qid,
>>> +                     0, flags);
>>> +    if (ret > 0) {
>>> +        dev_dbg(ctrl->device, "%s: qid %d nvme status %x\n",
>>> +            __func__, qid, ret);
>>> +        ret = -EIO;
>>> +    }
>>> +    if (ret < 0) {
>>> +        dev_dbg(ctrl->device, "%s: qid %d error %d\n",
>>> +            __func__, qid, ret);
>>> +        return ret;
>>> +    }
>>> +    dev_dbg(ctrl->device, "%s: qid %d auth_type %d auth_id %x\n",
>>> +        __func__, qid, data->auth_type, data->auth_id);
>>> +    if (data->auth_type == NVME_AUTH_COMMON_MESSAGES &&
>>> +        data->auth_id == NVME_AUTH_DHCHAP_MESSAGE_FAILURE1) {
>>> +        return data->reason_code_explanation;
>>> +    }
>>> +    if (data->auth_type != NVME_AUTH_DHCHAP_MESSAGES ||
>>> +        data->auth_id != expected_msg) {
>>> +        dev_warn(ctrl->device,
>>> +             "qid %d invalid message %02x/%02x\n",
>>> +             qid, data->auth_type, data->auth_id);
>>> +        return NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
>>> +    }
>>> +    if (le16_to_cpu(data->t_id) != transaction) {
>>> +        dev_warn(ctrl->device,
>>> +             "qid %d invalid transaction ID %d\n",
>>> +             qid, le16_to_cpu(data->t_id));
>>> +        return NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
>>> +    }
>>> +
>>> +    return 0;
>>> +}
>>> +
>>> +static int nvme_auth_dhchap_negotiate(struct nvme_ctrl *ctrl,
>>> +                      struct nvme_dhchap_context *chap,
>>> +                      void *buf, size_t buf_size)
>>
>> Maybe nvme_auth_set_dhchap_negotiate_data ?
>>
> 
> These are the individual steps in the state machine later on, so I 
> wanted to keep the names identical.
> But I'm open to suggestions.
> 
>>> +{
>>> +    struct nvmf_auth_dhchap_negotiate_data *data = buf;
>>> +    size_t size = sizeof(*data) + sizeof(union nvmf_auth_protocol);
>>> +
>>> +    if (buf_size < size)
>>> +        return -EINVAL;
>>> +
>>> +    memset((u8 *)buf, 0, size);
>>> +    data->auth_type = NVME_AUTH_COMMON_MESSAGES;
>>> +    data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE;
>>> +    data->t_id = cpu_to_le16(chap->transaction);
>>> +    data->sc_c = 0; /* No secure channel concatenation */
>>> +    data->napd = 1;
>>> +    data->auth_protocol[0].dhchap.authid = NVME_AUTH_DHCHAP_AUTH_ID;
>>> +    data->auth_protocol[0].dhchap.halen = 3;
>>> +    data->auth_protocol[0].dhchap.dhlen = 1;
>>> +    data->auth_protocol[0].dhchap.idlist[0] = 
>>> NVME_AUTH_DHCHAP_HASH_SHA256;
>>> +    data->auth_protocol[0].dhchap.idlist[1] = 
>>> NVME_AUTH_DHCHAP_HASH_SHA384;
>>> +    data->auth_protocol[0].dhchap.idlist[2] = 
>>> NVME_AUTH_DHCHAP_HASH_SHA512;
>>> +    data->auth_protocol[0].dhchap.idlist[3] = 
>>> NVME_AUTH_DHCHAP_DHGROUP_NULL;
>> You should comment that this routine expects buf to have enough
>> room for both negotiate and auth_proto structures.
>>
> Hmm. I do a check for the overall size at the start, so I'm not sure 
> what this will buy us.
> And actually, anyone wanting to make sense of the implementation would 
> need to look at the spec anyway.

Unrelated to the spec, just makes the code more readable. There is
an assumption on the buffer size passed, so it would be nice to
document it in the code (so it is less likely to break in the future
in the presence of assumptions).

>>> +
>>> +    return size;
>>> +}
>>> +
>>> +static int nvme_auth_dhchap_challenge(struct nvme_ctrl *ctrl,
>>> +                      struct nvme_dhchap_context *chap,
>>> +                      void *buf, size_t buf_size)
>>
>> Maybe nvme_auth_process_dhchap_challange ?
>>
> 
> See above. I'd rather have consistent names for the state machine.
> But I can change them to 'nvme_process_chchap_<statename>'
> 
>>> +{
>>> +    struct nvmf_auth_dhchap_challenge_data *data = buf;
>>> +    size_t size = sizeof(*data) + data->hl + data->dhvlen;
>>> +    const char *gid_name;
>>> +
>>> +    if (buf_size < size) {
>>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
>>> +        return -ENOMSG;
>>> +    }
>>> +
>>> +    if (data->hashid != NVME_AUTH_DHCHAP_HASH_SHA256 &&
>>> +        data->hashid != NVME_AUTH_DHCHAP_HASH_SHA384 &&
>>> +        data->hashid != NVME_AUTH_DHCHAP_HASH_SHA512) {
>>> +        dev_warn(ctrl->device,
>>> +             "qid %d: DH-HMAC-CHAP: invalid HASH ID %d\n",
>>> +             chap->qid, data->hashid);
>>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
>>> +        return -EPROTO;
>>> +    }
>>> +    switch (data->dhgid) {
>>> +    case NVME_AUTH_DHCHAP_DHGROUP_NULL:
>>> +        gid_name = "null";
>>> +        break;
>>> +    default:
>>> +        gid_name = NULL;
>>> +        break;
>>> +    }
>>> +    if (!gid_name) {
>>> +        dev_warn(ctrl->device,
>>> +             "qid %d: DH-HMAC-CHAP: invalid DH group id %d\n",
>>> +             chap->qid, data->dhgid);
>>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
>>> +        return -EPROTO;
>>> +    }
>>
>> Maybe some spaces between condition blocks?
>>
> 
> Ok.
> 
>>> +    if (data->dhgid != NVME_AUTH_DHCHAP_DHGROUP_NULL) {
>>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
>>> +        return -EPROTO;
>>> +    }
>>> +    if (data->dhgid == NVME_AUTH_DHCHAP_DHGROUP_NULL && data->dhvlen 
>>> != 0) {
>>> +        dev_warn(ctrl->device,
>>> +             "qid %d: DH-HMAC-CHAP: invalid DH value for NULL DH\n",
>>> +            chap->qid);
>>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
>>> +        return -EPROTO;
>>> +    }
>>> +    dev_dbg(ctrl->device, "%s: qid %d requested hash id %d\n",
>>> +        __func__, chap->qid, data->hashid);
>>> +    if (nvme_auth_hmac_len(data->hashid) != data->hl) {
>>> +        dev_warn(ctrl->device,
>>> +             "qid %d: DH-HMAC-CHAP: invalid hash length\n",
>>> +            chap->qid);
>>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
>>> +        return -EPROTO;
>>> +    }
>>> +    chap->hash_id = data->hashid;
>>> +    chap->hash_len = data->hl;
>>> +    chap->s1 = le32_to_cpu(data->seqnum);
>>> +    memcpy(chap->c1, data->cval, chap->hash_len);
>>> +
>>> +    return 0;
>>> +}
>>> +
>>> +static int nvme_auth_dhchap_reply(struct nvme_ctrl *ctrl,
>>> +                  struct nvme_dhchap_context *chap,
>>> +                  void *buf, size_t buf_size)
>>
>> nvme_auth_set_dhchap_reply
>>
> 
> Ah. Now I see what you're getting at.
> Okay, will be changing it.
> 
>>> +{
>>> +    struct nvmf_auth_dhchap_reply_data *data = buf;
>>> +    size_t size = sizeof(*data);
>>> +
>>> +    size += 2 * chap->hash_len;
>>> +    if (ctrl->opts->dhchap_auth) {
>>
>> The ctrl opts is not clear to me. what is dhchap_auth
>> mean?
>>
> As stated above, this is for bi-directional authentication.
> And yes, it is poor wording.
> 
> 'dhchap_bidirectional' ?

dhchap_auth_ctrl maybe.

>> Also shouldn't these params be lifted to the subsys?
>>
> 
> I kinda like to have it all encapsulated in a common per-queue 
> structure; on the host side this one isn't even attached to anything, so 
> any new authentication attempt will allocate a new one, with no chance 
> of accidentally re-using existing values.
> I thought this to be a rather nice property for a state-machine.

I understand, but different controllers for a single subsystem
should not behave differently. Meaning for one you authenticate
and the other you don't (or with different keys).

> 
>>> +        get_random_bytes(chap->c2, chap->hash_len);
>>> +        chap->s2 = nvme_dhchap_seqnum++;
>>> +    } else
>>> +        memset(chap->c2, 0, chap->hash_len);
>>> +
>>> +    if (chap->host_key_len)
>>> +        size += chap->host_key_len;
>>> +
>>> +    if (buf_size < size)
>>> +        return -EINVAL;
>>> +
>>> +    memset(buf, 0, size);
>>> +    data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
>>> +    data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_REPLY;
>>> +    data->t_id = cpu_to_le16(chap->transaction);
>>> +    data->hl = chap->hash_len;
>>> +    data->dhvlen = chap->host_key_len;
>>> +    data->seqnum = cpu_to_le32(chap->s2);
>>> +    memcpy(data->rval, chap->response, chap->hash_len);
>>> +    if (ctrl->opts->dhchap_auth) {
>>> +        dev_dbg(ctrl->device, "%s: qid %d ctrl challenge %*ph\n",
>>> +            __func__, chap->qid,
>>> +            chap->hash_len, chap->c2);
>>> +        data->cvalid = 1;
>>> +        memcpy(data->rval + chap->hash_len, chap->c2,
>>> +               chap->hash_len);
>>> +    }
>>> +    if (chap->host_key_len)
>>> +        memcpy(data->rval + 2 * chap->hash_len, chap->host_key,
>>> +               chap->host_key_len);
>>> +
>>> +    return size;
>>> +}
>>> +
>>> +static int nvme_auth_dhchap_success1(struct nvme_ctrl *ctrl,
>>> +                     struct nvme_dhchap_context *chap,
>>> +                     void *buf, size_t buf_size)
>>
>> nvme_auth_process_dhchap_success1
>>
> 
> OK.
> 
>>> +{
>>> +    struct nvmf_auth_dhchap_success1_data *data = buf;
>>> +    size_t size = sizeof(*data);
>>> +
>>> +    if (ctrl->opts->dhchap_auth)
>>> +        size += chap->hash_len;
>>> +
>>> +
>>> +    if (buf_size < size) {
>>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
>>> +        return -ENOMSG;
>>> +    }
>>> +
>>> +    if (data->hl != chap->hash_len) {
>>> +        dev_warn(ctrl->device,
>>> +             "qid %d: DH-HMAC-CHAP: invalid hash length %d\n",
>>> +             chap->qid, data->hl);
>>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
>>> +        return -EPROTO;
>>> +    }
>>> +
>>> +    if (!data->rvalid)
>>> +        return 0;
>>> +
>>> +    /* Validate controller response */
>>> +    if (memcmp(chap->response, data->rval, data->hl)) {
>>> +        dev_dbg(ctrl->device, "%s: qid %d ctrl response %*ph\n",
>>> +            __func__, chap->qid, chap->hash_len, data->rval);
>>> +        dev_dbg(ctrl->device, "%s: qid %d host response %*ph\n",
>>> +            __func__, chap->qid, chap->hash_len, chap->response);
>>> +        dev_warn(ctrl->device,
>>> +             "qid %d: DH-HMAC-CHAP: controller authentication 
>>> failed\n",
>>> +             chap->qid);
>>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
>>> +        return -EPROTO;
>>> +    }
>>> +    dev_info(ctrl->device,
>>> +         "qid %d: DH-HMAC-CHAP: controller authenticated\n",
>>> +        chap->qid);
>>> +    return 0;
>>> +}
>>> +
>>> +static int nvme_auth_dhchap_success2(struct nvme_ctrl *ctrl,
>>> +                     struct nvme_dhchap_context *chap,
>>> +                     void *buf, size_t buf_size)
>>
>> same
>>
>>> +{
>>> +    struct nvmf_auth_dhchap_success2_data *data = buf;
>>> +    size_t size = sizeof(*data);
>>> +
>>> +    memset(buf, 0, size);
>>> +    data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
>>> +    data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_SUCCESS2;
>>> +    data->t_id = cpu_to_le16(chap->transaction);
>>> +
>>> +    return size;
>>> +}
>>> +
>>> +static int nvme_auth_dhchap_failure2(struct nvme_ctrl *ctrl,
>>> +                     struct nvme_dhchap_context *chap,
>>> +                     void *buf, size_t buf_size)
>>
>> same
>>
>>> +{
>>> +    struct nvmf_auth_dhchap_failure_data *data = buf;
>>> +    size_t size = sizeof(*data);
>>> +
>>> +    memset(buf, 0, size);
>>> +    data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
>>> +    data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_FAILURE2;
>>> +    data->t_id = cpu_to_le16(chap->transaction);
>>> +    data->reason_code = 1;
>>> +    data->reason_code_explanation = chap->status;
>>> +
>>> +    return size;
>>> +}
>>> +
>>> +int nvme_auth_select_hash(struct nvme_ctrl *ctrl,
>>> +              struct nvme_dhchap_context *chap)
>>
>> Maybe _select_hf (hash function)? not a must, just sticks
>> to the spec language.
>>
> 
> Hmm. Will be checking.
> 
>>> +{
>>> +    char *hash_name;
>>> +    int ret;
>>> +
>>> +    switch (chap->hash_id) {
>>> +    case NVME_AUTH_DHCHAP_HASH_SHA256:
>>> +        hash_name = "hmac(sha256)";
>>> +        break;
>>> +    case NVME_AUTH_DHCHAP_HASH_SHA384:
>>> +        hash_name = "hmac(sha384)";
>>> +        break;
>>> +    case NVME_AUTH_DHCHAP_HASH_SHA512:
>>> +        hash_name = "hmac(sha512)";
>>> +        break;
>>> +    default:
>>> +        hash_name = NULL;
>>> +        break;
>>> +    }
>>> +    if (!hash_name) {
>>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
>>> +        return -EPROTO;
>>> +    }
>>> +    chap->shash_tfm = crypto_alloc_shash(hash_name, 0,
>>> +                         CRYPTO_ALG_ALLOCATES_MEMORY);
>>> +    if (IS_ERR(chap->shash_tfm)) {
>>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
>>> +        chap->shash_tfm = NULL;
>>> +        return -EPROTO;
>>> +    }
>>> +    if (!chap->key) {
>>> +        dev_warn(ctrl->device, "qid %d: cannot select hash, no key\n",
>>> +             chap->qid);
>>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
>>> +        crypto_free_shash(chap->shash_tfm);
>>
>> Wouldn't it better to check this before allocating the tfm?
>>
> 
> Indeed. Will be changing it.
> 
>>> +        chap->shash_tfm = NULL;
>>> +        return -EINVAL;
>>> +    }
>>> +    ret = crypto_shash_setkey(chap->shash_tfm, chap->key, 
>>> chap->key_len);
>>> +    if (ret) {
>>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
>>> +        crypto_free_shash(chap->shash_tfm);
>>> +        chap->shash_tfm = NULL;
>>> +        return ret;
>>> +    }
>>> +    dev_info(ctrl->device, "qid %d: DH-HMAC_CHAP: selected hash %s\n",
>>> +         chap->qid, hash_name);
>>> +    return 0;
>>> +}
>>> +
>>> +static int nvme_auth_dhchap_host_response(struct nvme_ctrl *ctrl,
>>> +                      struct nvme_dhchap_context *chap)
>>> +{
>>> +    SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
>>> +    u8 buf[4], *challenge = chap->c1;
>>> +    int ret;
>>> +
>>> +    dev_dbg(ctrl->device, "%s: qid %d host response seq %d 
>>> transaction %d\n",
>>> +        __func__, chap->qid, chap->s1, chap->transaction);
>>> +    shash->tfm = chap->shash_tfm;
>>> +    ret = crypto_shash_init(shash);
>>> +    if (ret)
>>> +        goto out;
>>> +    ret = crypto_shash_update(shash, challenge, chap->hash_len);
>>> +    if (ret)
>>> +        goto out;
>>> +    put_unaligned_le32(chap->s1, buf);
>>> +    ret = crypto_shash_update(shash, buf, 4);
>>> +    if (ret)
>>> +        goto out;
>>> +    put_unaligned_le16(chap->transaction, buf);
>>> +    ret = crypto_shash_update(shash, buf, 2);
>>> +    if (ret)
>>> +        goto out;
>>> +    memset(buf, 0, sizeof(buf));
>>> +    ret = crypto_shash_update(shash, buf, 1);
>>> +    if (ret)
>>> +        goto out;
>>> +    ret = crypto_shash_update(shash, "HostHost", 8);
>>
>> HostHost ? Can you refer me to the specific section
>> that talks about this?
>>
> 
> NVMe 2.0 section DH-HMAC-CHAP_Reply Message, paragraph Response Value.
> HostHost.
> 
>> Would be good to have a comment on the format fed to the
>> shash.
>>
> 
> Yes, will be doing so.
> 
>>> +    if (ret)
>>> +        goto out;
>>> +    ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
>>> +                  strlen(ctrl->opts->host->nqn));
>>> +    if (ret)
>>> +        goto out;
>>> +    ret = crypto_shash_update(shash, buf, 1);
>>> +    if (ret)
>>> +        goto out;
>>> +    ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
>>> +                strlen(ctrl->opts->subsysnqn));
>>> +    if (ret)
>>> +        goto out;
>>> +    ret = crypto_shash_final(shash, chap->response);
>>> +out:
>>> +    return ret;
>>> +}
>>> +
>>> +static int nvme_auth_dhchap_ctrl_response(struct nvme_ctrl *ctrl,
>>> +                      struct nvme_dhchap_context *chap)
>>> +{
>>> +    SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
>>> +    u8 buf[4], *challenge = chap->c2;
>>> +    int ret;
>>> +
>>> +    dev_dbg(ctrl->device, "%s: qid %d host response seq %d 
>>> transaction %d\n",
>>> +        __func__, chap->qid, chap->s2, chap->transaction);
>>> +    dev_dbg(ctrl->device, "%s: qid %d challenge %*ph\n",
>>> +        __func__, chap->qid, chap->hash_len, challenge);
>>> +    dev_dbg(ctrl->device, "%s: qid %d subsysnqn %s\n",
>>> +        __func__, chap->qid, ctrl->opts->subsysnqn);
>>> +    dev_dbg(ctrl->device, "%s: qid %d hostnqn %s\n",
>>> +        __func__, chap->qid, ctrl->opts->host->nqn);
>>> +    shash->tfm = chap->shash_tfm;
>>> +    ret = crypto_shash_init(shash);
>>> +    if (ret)
>>> +        goto out;
>>> +    ret = crypto_shash_update(shash, challenge, chap->hash_len);
>>> +    if (ret)
>>> +        goto out;
>>> +    put_unaligned_le32(chap->s2, buf);
>>> +    ret = crypto_shash_update(shash, buf, 4);
>>> +    if (ret)
>>> +        goto out;
>>> +    put_unaligned_le16(chap->transaction, buf);
>>> +    ret = crypto_shash_update(shash, buf, 2);
>>> +    if (ret)
>>> +        goto out;
>>> +    memset(buf, 0, 4);
>>> +    ret = crypto_shash_update(shash, buf, 1);
>>> +    if (ret)
>>> +        goto out;
>>> +    ret = crypto_shash_update(shash, "Controller", 10);
>>> +    if (ret)
>>> +        goto out;
>>> +    ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
>>> +                  strlen(ctrl->opts->subsysnqn));
>>> +    if (ret)
>>> +        goto out;
>>> +    ret = crypto_shash_update(shash, buf, 1);
>>> +    if (ret)
>>> +        goto out;
>>> +    ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
>>> +                  strlen(ctrl->opts->host->nqn));
>>> +    if (ret)
>>> +        goto out;
>>> +    ret = crypto_shash_final(shash, chap->response);
>>> +out:
>>> +    return ret;
>>> +}
>>> +
>>> +int nvme_auth_generate_key(struct nvme_ctrl *ctrl,
>>> +               struct nvme_dhchap_context *chap)
>>> +{
>>> +    int ret;
>>> +    u8 key_hash;
>>> +    const char *hmac_name;
>>> +    struct crypto_shash *key_tfm;
>>> +
>>> +    if (sscanf(ctrl->opts->dhchap_secret, "DHHC-1:%hhd:%*s:",
>>> +           &key_hash) != 1)
>>> +        return -EINVAL;
>>
>> I'd expect that the user will pass in a secret key (as binary)
>> the the driver will build the spec compliant formatted string no?
>>  > Am I not reading this correctly?
>>
> 
> I'm under the impression that this is the format into which the 
> User/Admin will get hold of the secret key.
> Spec says:
> 
> '... all NVMe over Fabrics entities shall support the following ASCII
> representation of secrets ...'
> 
> And as the userspace interface is the only way how the user/admin 
> _could_ interact with the NVMe over Fabrics entities in Linux I guess 
> we'll need to be able to parse it.

Right... But who is responsible for crcing and encoding it? nvme-cli?

> We sure could allow a binary secret, too, but then what would be the 
> point in converting it into the secret representation?
> The protocol revolves around the binary secret, not the transport 
> representation.

I am not sure I understand who is responsible for represnting the key
this way in Linux?

>>> +
>>> +    chap->key = nvme_auth_extract_secret(ctrl->opts->dhchap_secret,
>>> +                         &chap->key_len);
>>> +    if (IS_ERR(chap->key)) {
>>> +        ret = PTR_ERR(chap->key);
>>> +        chap->key = NULL;
>>> +        return ret;
>>> +    }
>>> +
>>> +    if (key_hash == 0)
>>> +        return 0;
>>> +
>>> +    hmac_name = nvme_auth_hmac_name(key_hash);
>>> +    if (!hmac_name) {
>>> +        pr_debug("Invalid key hash id %d\n", key_hash);
>>> +        return -EKEYREJECTED;
>>> +    }
>>
>> Why does the user influence the hmac used? isn't that is driven
>> by the susbsystem?
>>
>> I don't think that the user should choose in this level.
>>
> 
> That is another weirdness of the spec.
> The _secret_ will be hashed with a specific function, and that function 
> is stated in the transport representation.
> (Cf section "DH-HMAC-CHAP Security Requirements").
> This is _not_ the hash function used by the authentication itself, which 
> will be selected by the protocol.

Yes, I see it now, and it is indeed confusing.

> So it's not the user here, but rather the transport specification of the 
> key which selects the hash algorithm.

What do you mean by the transport specification?

>>> +
>>> +    key_tfm = crypto_alloc_shash(hmac_name, 0, 0);
>>> +    if (IS_ERR(key_tfm)) {
>>> +        kfree(chap->key);
>>> +        chap->key = NULL;
>>> +        ret = PTR_ERR(key_tfm);
>>
>> You set ret and later return 0? I think that the success
>> path in the else clause is hard to read and error prone...
>>
> 
> Do I? Will need to fix it up.
> 
>>> +    } else {
>>> +        SHASH_DESC_ON_STACK(shash, key_tfm);
>>> +
>>> +        shash->tfm = key_tfm;
>>> +        ret = crypto_shash_setkey(key_tfm, chap->key,
>>> +                      chap->key_len);
>>> +        if (ret < 0) {
>>> +            crypto_free_shash(key_tfm);
>>> +            kfree(chap->key);
>>> +            chap->key = NULL;
>>> +            return ret;
>>> +        }
>>> +        crypto_shash_init(shash);
>>> +        crypto_shash_update(shash, ctrl->opts->host->nqn,
>>> +                    strlen(ctrl->opts->host->nqn));
>>> +        crypto_shash_update(shash, "NVMe-over-Fabrics", 17);
>>> +        crypto_shash_final(shash, chap->key);
>>> +        crypto_free_shash(key_tfm);
>>
>> Shouldn't these be done when preparing the dh-hmac-chap reply?
>>
> 
> By setting the hash here I avoid having to pass the required hash 
> function for the secret transformation.
> I could be doing the entire secret transformation thingie when preparing 
> the reply; reason why I did it here is that _having_ a secret is the 
> precondition to everything else, so I wanted to check upfront for that.
> But I'll check what would happen if I move it.

Now that I understand that this is not the authentication transformation
its ok I guess. Please add a comment in the code so its clearer.

>>> +    }
>>> +    return 0;
>>> +}
>>> +
>>> +void nvme_auth_free(struct nvme_dhchap_context *chap)
>>> +{
>>> +    if (chap->shash_tfm)
>>> +        crypto_free_shash(chap->shash_tfm);
>>> +    if (chap->key)
>>> +        kfree(chap->key);
>>> +    if (chap->ctrl_key)
>>> +        kfree(chap->ctrl_key);
>>> +    if (chap->host_key)
>>> +        kfree(chap->host_key);
>>> +    if (chap->sess_key)
>>> +        kfree(chap->sess_key);
>>
>> No need to check null for kfree...
>>
> 
> Will be fixing it up.
> 
>>> +    kfree(chap);
>>> +}
>>> +
>>> +int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid)
>>> +{
>>> +    struct nvme_dhchap_context *chap;
>>> +    void *buf;
>>> +    size_t buf_size, tl;
>>> +    int ret = 0;
>>> +
>>> +    chap = kzalloc(sizeof(*chap), GFP_KERNEL);
>>> +    if (!chap)
>>> +        return -ENOMEM;
>>> +    chap->qid = qid;
>>> +    chap->transaction = ctrl->transaction++;
>>> +
>>> +    ret = nvme_auth_generate_key(ctrl, chap);
>>> +    if (ret) {
>>> +        dev_dbg(ctrl->device, "%s: failed to generate key, error %d\n",
>>> +            __func__, ret);
>>> +        nvme_auth_free(chap);
>>> +        return ret;
>>> +    }
>>> +
>>> +    /*
>>> +     * Allocate a large enough buffer for the entire negotiation:
>>> +     * 4k should be enough to ffdhe8192.
>>> +     */
>>> +    buf_size = 4096;
>>> +    buf = kzalloc(buf_size, GFP_KERNEL);
>>> +    if (!buf) {
>>> +        ret = -ENOMEM;
>>> +        goto out;
>>> +    }
>>> +
>>> +    /* DH-HMAC-CHAP Step 1: send negotiate */
>>
>> I'd consider breaking these into sub-routines.
>>
> 
> Which ones? The preparation step?

I'm thinking:
1. nvme_auth_initiate_negotiation
    - nvme_auth_set_dhchap_negotiate_data
    - nvme_auth_send
2. nvme_auth_do_challange
    - nvme_auth_receive
    - nvme_auth_process_dhchap_challange
    - nvme_auth_select_hash
    - nvme_auth_dhchap_host_response
    - nvme_auth_set_dhchap_reply
    - nvme_auth_send
    - nvme_auth_receive
    - nvme_auth_process_dhchap_success1
3. if (ctrl->opts->dhchap_auth_ctrl)
    - nvme_auth_dhchap_authenticate_ctrl
      (e.g. nvme_auth_dhchap_ctrl_response)
4. nvme_auth_acknowledge_transaction
    - nvme_auth_set_dhchap_success2
    - nvme_auth_send

if steps 1,2,3 failed, goto target will have a func:
5. nvme_auth_fail_transaction
    - nvme_auth_set_dhchap_failure2
    - nvme_auth_send

> Sure, can do.
> 
>>> +    dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP negotiate\n",
>>> +        __func__, qid);
>>> +    ret = nvme_auth_dhchap_negotiate(ctrl, chap, buf, buf_size);
>>> +    if (ret < 0)
>>> +        goto out;
>>> +    tl = ret;
>>> +    ret = nvme_auth_send(ctrl, qid, buf, tl);
>>> +    if (ret)
>>> +        goto out;
>>> +
>>> +    memset(buf, 0, buf_size);
>>> +    ret = nvme_auth_receive(ctrl, qid, buf, buf_size, 
>>> chap->transaction,
>>> +                NVME_AUTH_DHCHAP_MESSAGE_CHALLENGE);
>>> +    if (ret < 0) {
>>> +        dev_dbg(ctrl->device,
>>> +            "%s: qid %d DH-HMAC-CHAP failed to receive challenge\n",
>>> +            __func__, qid);
>>> +        goto out;
>>> +    }
>>> +    if (ret > 0) {
>>> +        chap->status = ret;
>>> +        goto fail1;
>>> +    }
>>> +
>>> +    /* DH-HMAC-CHAP Step 2: receive challenge */
>>> +    dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP challenge\n",
>>> +        __func__, qid);
>>> +
>>> +    ret = nvme_auth_dhchap_challenge(ctrl, chap, buf, buf_size);
>>> +    if (ret) {
>>> +        /* Invalid parameters for negotiate */
>>> +        goto fail2;
>>> +    }
>>> +
>>> +    dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP select hash\n",
>>> +        __func__, qid);
>>> +    ret = nvme_auth_select_hash(ctrl, chap);
>>> +    if (ret)
>>> +        goto fail2;
>>> +
>>> +    dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP host response\n",
>>> +        __func__, qid);
>>> +    ret = nvme_auth_dhchap_host_response(ctrl, chap);
>>> +    if (ret)
>>> +        goto fail2;
>>> +
>>> +    /* DH-HMAC-CHAP Step 3: send reply */
>>> +    dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP reply\n",
>>> +        __func__, qid);
>>> +    ret = nvme_auth_dhchap_reply(ctrl, chap, buf, buf_size);
>>> +    if (ret < 0)
>>> +        goto fail2;
>>> +
>>> +    tl = ret;
>>> +    ret = nvme_auth_send(ctrl, qid, buf, tl);
>>> +    if (ret)
>>> +        goto fail2;
>>> +
>>> +    memset(buf, 0, buf_size);
>>> +    ret = nvme_auth_receive(ctrl, qid, buf, buf_size, 
>>> chap->transaction,
>>> +                NVME_AUTH_DHCHAP_MESSAGE_SUCCESS1);
>>> +    if (ret < 0) {
>>> +        dev_dbg(ctrl->device,
>>> +            "%s: qid %d DH-HMAC-CHAP failed to receive success1\n",
>>> +            __func__, qid);
>>> +        goto out;
>>> +    }
>>> +    if (ret > 0) {
>>> +        chap->status = ret;
>>> +        goto fail1;
>>> +    }
>>> +
>>> +    if (ctrl->opts->dhchap_auth) {
>>> +        dev_dbg(ctrl->device,
>>> +            "%s: qid %d DH-HMAC-CHAP controller response\n",
>>> +            __func__, qid);
>>> +        ret = nvme_auth_dhchap_ctrl_response(ctrl, chap);
>>> +        if (ret)
>>> +            goto fail2;
>>> +    }
>>> +
>>> +    /* DH-HMAC-CHAP Step 4: receive success1 */
>>> +    dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP success1\n",
>>> +        __func__, qid);
>>> +    ret = nvme_auth_dhchap_success1(ctrl, chap, buf, buf_size);
>>> +    if (ret < 0) {
>>> +        /* Controller authentication failed */
>>> +        goto fail2;
>>> +    }
>>> +    tl = ret;
>>> +    /* DH-HMAC-CHAP Step 5: send success2 */
>>> +    dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP success2\n",
>>> +        __func__, qid);
>>> +    tl = nvme_auth_dhchap_success2(ctrl, chap, buf, buf_size);
>>> +    ret = nvme_auth_send(ctrl, qid, buf, tl);
>>> +    if (!ret)
>>> +        goto out;
>>> +
>>> +fail1:
>>> +    dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP failure1, status 
>>> %x\n",
>>> +        __func__, qid, chap->status);
>>> +    goto out;
>>> +
>>> +fail2:
>>> +    dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP failure2, status 
>>> %x\n",
>>> +        __func__, qid, chap->status);
>>> +    tl = nvme_auth_dhchap_failure2(ctrl, chap, buf, buf_size);
>>> +    ret = nvme_auth_send(ctrl, qid, buf, tl);
>>> +
>>> +out:
>>> +    if (!ret && chap->status)
>>> +        ret = -EPROTO;
>>> +    if (!ret) {
>>> +        ctrl->dhchap_hash = chap->hash_id;
>>> +    }
>>> +    kfree(buf);
>>> +    nvme_auth_free(chap);
>>> +    return ret;
>>> +}
>>> diff --git a/drivers/nvme/host/auth.h b/drivers/nvme/host/auth.h
>>> new file mode 100644
>>> index 000000000000..4950b1cb9470
>>> --- /dev/null
>>> +++ b/drivers/nvme/host/auth.h
>>> @@ -0,0 +1,23 @@
>>> +/* SPDX-License-Identifier: GPL-2.0 */
>>> +/*
>>> + * Copyright (c) 2021 Hannes Reinecke, SUSE Software Solutions
>>> + */
>>> +
>>> +#ifndef _NVME_AUTH_H
>>> +#define _NVME_AUTH_H
>>> +
>>> +const char *nvme_auth_dhgroup_name(int dhgroup_id);
>>> +int nvme_auth_dhgroup_pubkey_size(int dhgroup_id);
>>> +int nvme_auth_dhgroup_privkey_size(int dhgroup_id);
>>> +const char *nvme_auth_dhgroup_kpp(int dhgroup_id);
>>> +int nvme_auth_dhgroup_id(const char *dhgroup_name);
>>> +
>>> +const char *nvme_auth_hmac_name(int hmac_id);
>>> +const char *nvme_auth_digest_name(int hmac_id);
>>> +int nvme_auth_hmac_id(const char *hmac_name);
>>> +int nvme_auth_hmac_len(int hmac_len);
>>> +
>>> +unsigned char *nvme_auth_extract_secret(unsigned char *dhchap_secret,
>>> +                    size_t *dhchap_key_len);
>>> +
>>> +#endif /* _NVME_AUTH_H */
>>> diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
>>> index 11779be42186..7ce9b666dc09 100644
>>> --- a/drivers/nvme/host/core.c
>>> +++ b/drivers/nvme/host/core.c
>>> @@ -708,7 +708,9 @@ bool __nvme_check_ready(struct nvme_ctrl *ctrl, 
>>> struct request *rq,
>>>           switch (ctrl->state) {
>>>           case NVME_CTRL_CONNECTING:
>>>               if (blk_rq_is_passthrough(rq) && 
>>> nvme_is_fabrics(req->cmd) &&
>>> -                req->cmd->fabrics.fctype == nvme_fabrics_type_connect)
>>> +                (req->cmd->fabrics.fctype == 
>>> nvme_fabrics_type_connect ||
>>> +                 req->cmd->fabrics.fctype == 
>>> nvme_fabrics_type_auth_send ||
>>> +                 req->cmd->fabrics.fctype == 
>>> nvme_fabrics_type_auth_receive))
>>>                   return true;
>>>               break;
>>>           default:
>>> @@ -3426,6 +3428,66 @@ static ssize_t 
>>> nvme_ctrl_fast_io_fail_tmo_store(struct device *dev,
>>>   static DEVICE_ATTR(fast_io_fail_tmo, S_IRUGO | S_IWUSR,
>>>       nvme_ctrl_fast_io_fail_tmo_show, 
>>> nvme_ctrl_fast_io_fail_tmo_store);
>>> +#ifdef CONFIG_NVME_AUTH
>>> +struct nvmet_dhchap_hash_map {
>>> +    int id;
>>> +    const char name[15];
>>> +} hash_map[] = {
>>> +    {.id = NVME_AUTH_DHCHAP_HASH_SHA256,
>>> +     .name = "hmac(sha256)", },
>>> +    {.id = NVME_AUTH_DHCHAP_HASH_SHA384,
>>> +     .name = "hmac(sha384)", },
>>> +    {.id = NVME_AUTH_DHCHAP_HASH_SHA512,
>>> +     .name = "hmac(sha512)", },
>>> +};
>>> +
>>> +static ssize_t dhchap_hash_show(struct device *dev,
>>> +    struct device_attribute *attr, char *buf)
>>> +{
>>> +    struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
>>> +    int i;
>>> +
>>> +    for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
>>> +        if (hash_map[i].id == ctrl->dhchap_hash)
>>> +            return sprintf(buf, "%s\n", hash_map[i].name);
>>> +    }
>>> +    return sprintf(buf, "none\n");
>>> +}
>>> +DEVICE_ATTR_RO(dhchap_hash);
>>> +
>>> +struct nvmet_dhchap_group_map {
>>> +    int id;
>>> +    const char name[15];
>>> +} dhgroup_map[] = {
>>> +    {.id = NVME_AUTH_DHCHAP_DHGROUP_NULL,
>>> +     .name = "NULL", },
>>> +    {.id = NVME_AUTH_DHCHAP_DHGROUP_2048,
>>> +     .name = "ffdhe2048", },
>>> +    {.id = NVME_AUTH_DHCHAP_DHGROUP_3072,
>>> +     .name = "ffdhe3072", },
>>> +    {.id = NVME_AUTH_DHCHAP_DHGROUP_4096,
>>> +     .name = "ffdhe4096", },
>>> +    {.id = NVME_AUTH_DHCHAP_DHGROUP_6144,
>>> +     .name = "ffdhe6144", },
>>> +    {.id = NVME_AUTH_DHCHAP_DHGROUP_8192,
>>> +     .name = "ffdhe8192", },
>>> +};
>>> +
>>> +static ssize_t dhchap_dhgroup_show(struct device *dev,
>>> +    struct device_attribute *attr, char *buf)
>>> +{
>>> +    struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
>>> +    int i;
>>> +
>>> +    for (i = 0; i < ARRAY_SIZE(dhgroup_map); i++) {
>>> +        if (hash_map[i].id == ctrl->dhchap_dhgroup)
>>> +            return sprintf(buf, "%s\n", dhgroup_map[i].name);
>>> +    }
>>> +    return sprintf(buf, "none\n");
>>> +}
>>> +DEVICE_ATTR_RO(dhchap_dhgroup);
>>> +#endif
>>> +
>>>   static struct attribute *nvme_dev_attrs[] = {
>>>       &dev_attr_reset_controller.attr,
>>>       &dev_attr_rescan_controller.attr,
>>> @@ -3447,6 +3509,10 @@ static struct attribute *nvme_dev_attrs[] = {
>>>       &dev_attr_reconnect_delay.attr,
>>>       &dev_attr_fast_io_fail_tmo.attr,
>>>       &dev_attr_kato.attr,
>>> +#ifdef CONFIG_NVME_AUTH
>>> +    &dev_attr_dhchap_hash.attr,
>>> +    &dev_attr_dhchap_dhgroup.attr,
>>> +#endif
>>>       NULL
>>>   };
>>> @@ -3470,6 +3536,10 @@ static umode_t 
>>> nvme_dev_attrs_are_visible(struct kobject *kobj,
>>>           return 0;
>>>       if (a == &dev_attr_fast_io_fail_tmo.attr && !ctrl->opts)
>>>           return 0;
>>> +#ifdef CONFIG_NVME_AUTH
>>> +    if (a == &dev_attr_dhchap_hash.attr && !ctrl->opts)
>>> +        return 0;
>>> +#endif
>>>       return a->mode;
>>>   }
>>> @@ -4581,6 +4651,11 @@ static inline void _nvme_check_size(void)
>>>       BUILD_BUG_ON(sizeof(struct nvme_smart_log) != 512);
>>>       BUILD_BUG_ON(sizeof(struct nvme_dbbuf) != 64);
>>>       BUILD_BUG_ON(sizeof(struct nvme_directive_cmd) != 64);
>>> +    BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_negotiate_data) != 8);
>>> +    BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_challenge_data) != 16);
>>> +    BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_reply_data) != 16);
>>> +    BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_success1_data) != 16);
>>> +    BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_success2_data) != 16);
>>>   }
>>> diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
>>> index a5469fd9d4c3..6404ab9b604b 100644
>>> --- a/drivers/nvme/host/fabrics.c
>>> +++ b/drivers/nvme/host/fabrics.c
>>> @@ -366,6 +366,7 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)
>>>       union nvme_result res;
>>>       struct nvmf_connect_data *data;
>>>       int ret;
>>> +    u32 result;
>>>       cmd.connect.opcode = nvme_fabrics_command;
>>>       cmd.connect.fctype = nvme_fabrics_type_connect;
>>> @@ -398,8 +399,18 @@ int nvmf_connect_admin_queue(struct nvme_ctrl 
>>> *ctrl)
>>>           goto out_free_data;
>>>       }
>>> -    ctrl->cntlid = le16_to_cpu(res.u16);
>>> -
>>> +    result = le32_to_cpu(res.u32);
>>> +    ctrl->cntlid = result & 0xFFFF;
>>> +    if ((result >> 16) & 2) {
>>> +        /* Authentication required */
>>> +        ret = nvme_auth_negotiate(ctrl, NVME_QID_ANY);
>>> +        if (ret)
>>> +            dev_warn(ctrl->device,
>>> +                 "qid 0: authentication failed\n");
>>> +        else
>>> +            dev_info(ctrl->device,
>>> +                 "qid 0: authenticated\n");
>>
>> info is too chatty.
>>
> 
> Hmm. I know I need to work on logging...
> 
>>> +    }
>>>   out_free_data:
>>>       kfree(data);
>>>       return ret;
>>> @@ -432,6 +443,7 @@ int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, 
>>> u16 qid)
>>>       struct nvmf_connect_data *data;
>>>       union nvme_result res;
>>>       int ret;
>>> +    u32 result;
>>>       cmd.connect.opcode = nvme_fabrics_command;
>>>       cmd.connect.fctype = nvme_fabrics_type_connect;
>>> @@ -457,6 +469,17 @@ int nvmf_connect_io_queue(struct nvme_ctrl 
>>> *ctrl, u16 qid)
>>>           nvmf_log_connect_error(ctrl, ret, le32_to_cpu(res.u32),
>>>                          &cmd, data);
>>>       }
>>> +    result = le32_to_cpu(res.u32);
>>> +    if ((result >> 16) & 2) {
>>> +        /* Authentication required */
>>> +        ret = nvme_auth_negotiate(ctrl, qid);
>>> +        if (ret)
>>> +            dev_warn(ctrl->device,
>>> +                 "qid %u: authentication failed\n", qid);
>>> +        else
>>> +            dev_info(ctrl->device,
>>> +                 "qid %u: authenticated\n", qid);
>>> +    }
>>>       kfree(data);
>>>       return ret;
>>>   }
>>> @@ -548,6 +571,9 @@ static const match_table_t opt_tokens = {
>>>       { NVMF_OPT_NR_POLL_QUEUES,    "nr_poll_queues=%d"    },
>>>       { NVMF_OPT_TOS,            "tos=%d"        },
>>>       { NVMF_OPT_FAIL_FAST_TMO,    "fast_io_fail_tmo=%d"    },
>>> +    { NVMF_OPT_DHCHAP_SECRET,    "dhchap_secret=%s"    },
>>> +    { NVMF_OPT_DHCHAP_AUTH,        "authenticate"        },
>>> +    { NVMF_OPT_DHCHAP_GROUP,    "dhchap_group=%s"    },
>>
>> Isn't the group driven by the subsystem? also why is there a
>> "authenticate" boolean? what is it good for?
>>
> Ah. Right. Of course, the 'group' is pointless here.
> And the 'authenticate' bool is the abovementioned bidirectional 
> authentication.
> I _do_ need to give it another name.
> 
>>>       { NVMF_OPT_ERR,            NULL            }
>>>   };
>>> @@ -824,6 +850,35 @@ static int nvmf_parse_options(struct 
>>> nvmf_ctrl_options *opts,
>>>               }
>>>               opts->tos = token;
>>>               break;
>>> +        case NVMF_OPT_DHCHAP_SECRET:
>>> +            p = match_strdup(args);
>>> +            if (!p) {
>>> +                ret = -ENOMEM;
>>> +                goto out;
>>> +            }
>>> +            if (strncmp(p, "DHHC-1:00:", 10)) {
>>> +                pr_err("Invalid DH-CHAP secret %s\n", p);
>>> +                ret = -EINVAL;
>>> +                goto out;
>>> +            }
>>> +            kfree(opts->dhchap_secret);
>>> +            opts->dhchap_secret = p;
>>> +            break;
>>> +        case NVMF_OPT_DHCHAP_AUTH:
>>> +            opts->dhchap_auth = true;
>>> +            break;
>>> +        case NVMF_OPT_DHCHAP_GROUP:
>>> +            if (match_int(args, &token)) {
>>> +                ret = -EINVAL;
>>> +                goto out;
>>> +            }
>>> +            if (token <= 0) {
>>> +                pr_err("Invalid dhchap_group %d\n", token);
>>> +                ret = -EINVAL;
>>> +                goto out;
>>> +            }
>>> +            opts->dhchap_group = token;
>>> +            break;
>>>           default:
>>>               pr_warn("unknown parameter or missing value '%s' in 
>>> ctrl creation request\n",
>>>                   p);
>>> @@ -942,6 +997,7 @@ void nvmf_free_options(struct nvmf_ctrl_options 
>>> *opts)
>>>       kfree(opts->subsysnqn);
>>>       kfree(opts->host_traddr);
>>>       kfree(opts->host_iface);
>>> +    kfree(opts->dhchap_secret);
>>>       kfree(opts);
>>>   }
>>>   EXPORT_SYMBOL_GPL(nvmf_free_options);
>>> @@ -951,7 +1007,10 @@ EXPORT_SYMBOL_GPL(nvmf_free_options);
>>>                    NVMF_OPT_KATO | NVMF_OPT_HOSTNQN | \
>>>                    NVMF_OPT_HOST_ID | NVMF_OPT_DUP_CONNECT |\
>>>                    NVMF_OPT_DISABLE_SQFLOW |\
>>> -                 NVMF_OPT_FAIL_FAST_TMO)
>>> +                 NVMF_OPT_CTRL_LOSS_TMO |\
>>> +                 NVMF_OPT_FAIL_FAST_TMO |\
>>> +                 NVMF_OPT_DHCHAP_SECRET |\
>>> +                 NVMF_OPT_DHCHAP_AUTH | NVMF_OPT_DHCHAP_GROUP)
>>>   static struct nvme_ctrl *
>>>   nvmf_create_ctrl(struct device *dev, const char *buf)
>>> diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h
>>> index a146cb903869..535bc544f0f6 100644
>>> --- a/drivers/nvme/host/fabrics.h
>>> +++ b/drivers/nvme/host/fabrics.h
>>> @@ -67,6 +67,9 @@ enum {
>>>       NVMF_OPT_TOS        = 1 << 19,
>>>       NVMF_OPT_FAIL_FAST_TMO    = 1 << 20,
>>>       NVMF_OPT_HOST_IFACE    = 1 << 21,
>>> +    NVMF_OPT_DHCHAP_SECRET    = 1 << 22,
>>> +    NVMF_OPT_DHCHAP_AUTH    = 1 << 23,
>>> +    NVMF_OPT_DHCHAP_GROUP    = 1 << 24,
>>>   };
>>>   /**
>>> @@ -96,6 +99,8 @@ enum {
>>>    * @max_reconnects: maximum number of allowed reconnect attempts 
>>> before removing
>>>    *              the controller, (-1) means reconnect forever, zero 
>>> means remove
>>>    *              immediately;
>>> + * @dhchap_secret: DH-HMAC-CHAP secret
>>> + * @dhchap_auth: DH-HMAC-CHAP authenticate controller
>>>    * @disable_sqflow: disable controller sq flow control
>>>    * @hdr_digest: generate/verify header digest (TCP)
>>>    * @data_digest: generate/verify data digest (TCP)
>>> @@ -120,6 +125,9 @@ struct nvmf_ctrl_options {
>>>       unsigned int        kato;
>>>       struct nvmf_host    *host;
>>>       int            max_reconnects;
>>> +    char            *dhchap_secret;
>>> +    int            dhchap_group;
>>> +    bool            dhchap_auth;
>>>       bool            disable_sqflow;
>>>       bool            hdr_digest;
>>>       bool            data_digest;
>>> diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
>>> index 18ef8dd03a90..bcd5b8276c26 100644
>>> --- a/drivers/nvme/host/nvme.h
>>> +++ b/drivers/nvme/host/nvme.h
>>> @@ -328,6 +328,12 @@ struct nvme_ctrl {
>>>       struct work_struct ana_work;
>>>   #endif
>>> +#ifdef CONFIG_NVME_AUTH
>>> +    u16 transaction;
>>> +    u8 dhchap_hash;
>>> +    u8 dhchap_dhgroup;
>>
>> Do multiple controllers in the same subsystem have different
>> params? no, so I think these should be lifted to subsys.
>>
> 
> It doesn't actually say in the spec; it always refers to the params as 
> being set by the controller.
> So it could be either; maybe we should ask for clafication at the fmds 
> call.

We should, but I'd be surprised that different controllers in the same
subsystem can authenticate difrerently...

Re: [PATCH 06/11] nvme: Implement In-Band authentication

[Sagi Grimberg]

>>> Implement NVMe-oF In-Band authentication. This patch adds two new
>>> fabric options 'dhchap_key' to specify the PSK
>>
>> pre-shared-key.
>>
>> Also, we need a sysfs knob to rotate the key that will trigger
>> re-authentication or even a simple controller(s-plural) reset, so this
>> should go beyond just the connection string.
>>
> 
> Yeah, re-authentication currently is not implemented. I first wanted to 
> get this patchset out such that we can settle on the userspace interface 
> (both from host and target).

I think this is mandatory from the start (at the very least from the
host side) because credentials get rotated.

The flow imo needs to be that the target allows to update the
credentials even when the subsystem is exposed and connected, but
keep all existing hosts connected such that only new connections will
need the new credentials. (an incremental step, which would be optional
is to allow it for a configurable grace period and then disconnect
existing hosts, which would force new connections with updated
credentials).

 From the host side, we need to expose a controller (or more
appropriately a subsystem) sysfs file to override the existing key. That
can initially just trigger a controller reset, and incrementally can do
a graceful re-authentication).

> I'll have to think on how we should handle authentication; one of the 
> really interesting cases would be when one malicious admin will _just_ 
> send a 'negotiate' command to the controller. As per spec the controller 
> will be waiting for an 'authentication receive' command to send a 
> 'challenge' payload back to the host. But that will never come, so as it 
> stands currently the controller is required to abort the connection.
> Not very nice.

Well, at the moment we can keep it open and discuss this in the TWG.

>> P.S. can you add also the nvme-cli code in the next go?
>>
> Oh, sure. It's already sitting around in my local repo (surprise, 
> surprise); will be ending it out next time.

Great, thanks.

>>> and 'dhchap_authenticate'
>>> to request bi-directional authentication of both the host and the 
>>> controller.
>>
>> bidirectional? not uni-directional?
>>
> 
> Yeah, that's a bit of a misnomer. When a PSK is specified, the 
> controller will start the authentication protocol such that the 
> _controller_ can validate the host. If the host wants to authenticate 
> the controller is needs to set this flag.
> Hence bi-directional authentication.
> But I'm the first to admit that this is poor wording for the flag.

It is misleading. But I'll need to go look again because I didn't
see the host authenticating the controller..

>>> Signed-off-by: Hannes Reinecke <hare@suse.de>
>>> ---
>>>   drivers/nvme/host/Kconfig   |  11 +
>>>   drivers/nvme/host/Makefile  |   1 +
>>>   drivers/nvme/host/auth.c    | 813 ++++++++++++++++++++++++++++++++++++
>>>   drivers/nvme/host/auth.h    |  23 +
>>>   drivers/nvme/host/core.c    |  77 +++-
>>>   drivers/nvme/host/fabrics.c |  65 ++-
>>>   drivers/nvme/host/fabrics.h |   8 +
>>>   drivers/nvme/host/nvme.h    |  15 +
>>>   drivers/nvme/host/trace.c   |  32 ++
>>>   9 files changed, 1041 insertions(+), 4 deletions(-)
>>>   create mode 100644 drivers/nvme/host/auth.c
>>>   create mode 100644 drivers/nvme/host/auth.h
>>>
>>> diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig
>>> index c3f3d77f1aac..853c546305e9 100644
>>> --- a/drivers/nvme/host/Kconfig
>>> +++ b/drivers/nvme/host/Kconfig
>>> @@ -85,3 +85,14 @@ config NVME_TCP
>>>         from https://github.com/linux-nvme/nvme-cli.
>>>         If unsure, say N.
>>> +
>>> +config NVME_AUTH
>>> +    bool "NVM Express over Fabrics In-Band Authentication"
>>> +    depends on NVME_TCP
>>> +    select CRYPTO_SHA256
>>> +    select CRYPTO_SHA512
>>> +    help
>>> +      This provides support for NVMe over Fabrics In-Band 
>>> Authentication
>>> +      for the NVMe over TCP transport.
>>
>> In this form, nothing is specific to nvme-tcp here afaict.
>>
> 
> Indeed. I guess we can leave out the nvme-tcp reference here.
> 
>>> +
>>> +      If unsure, say N.
>>> diff --git a/drivers/nvme/host/Makefile b/drivers/nvme/host/Makefile
>>> index cbc509784b2e..03748a55a12b 100644
>>> --- a/drivers/nvme/host/Makefile
>>> +++ b/drivers/nvme/host/Makefile
>>> @@ -20,6 +20,7 @@ nvme-core-$(CONFIG_NVME_HWMON)        += hwmon.o
>>>   nvme-y                    += pci.o
>>>   nvme-fabrics-y                += fabrics.o
>>> +nvme-fabrics-$(CONFIG_NVME_AUTH)    += auth.o
>>>   nvme-rdma-y                += rdma.o
>>> diff --git a/drivers/nvme/host/auth.c b/drivers/nvme/host/auth.c
>>> new file mode 100644
>>> index 000000000000..448a3adebea6
>>> --- /dev/null
>>> +++ b/drivers/nvme/host/auth.c
>>> @@ -0,0 +1,813 @@
>>> +// SPDX-License-Identifier: GPL-2.0
>>> +/*
>>> + * Copyright (c) 2020 Hannes Reinecke, SUSE Linux
>>> + */
>>> +
>>> +#include <linux/crc32.h>
>>> +#include <linux/base64.h>
>>> +#include <asm/unaligned.h>
>>> +#include <crypto/hash.h>
>>> +#include <crypto/kpp.h>
>>> +#include "nvme.h"
>>> +#include "fabrics.h"
>>> +#include "auth.h"
>>> +
>>> +static u32 nvme_dhchap_seqnum;
>>> +
>>> +struct nvme_dhchap_context {
>>
>> Maybe nvme_dhchap_queue_context ?
>>
>> I'm thinking that we should perhaps split
>> it to host-wide, subsys-wide and queue specific
>> auth contexts?
>>
>> Let's see...
>>
> 
> Interestingly enough, that's what I did for the target side.
> For the host side I found it easier that way, as then we'll have a 
> single authentication context which can be deleted after authentication 
> finished, and be sure that we removed _all_ information.
> Security and all that.
> Splitting it off would require to remove information on three different 
> places, and observing life-time rules for them.
> So more of a chance to mess things up :-)

I understand what you are saying, but this way it will be challanging
to cross check stuff. It is also where things sort of belong...

>>> +    struct crypto_shash *shash_tfm;
>>> +    unsigned char *key;
>>> +    size_t key_len;
>>> +    int qid;
>>> +    u32 s1;
>>> +    u32 s2;
>>> +    u16 transaction;
>>> +    u8 status;
>>> +    u8 hash_id;
>>> +    u8 hash_len;
>>> +    u8 c1[64];
>>> +    u8 c2[64];
>>> +    u8 response[64];
>>> +    u8 *ctrl_key;
>>> +    int ctrl_key_len;
>>> +    u8 *host_key;
>>> +    int host_key_len;
>>> +    u8 *sess_key;
>>> +    int sess_key_len;
>>> +};
>>> +
>>> +struct nvmet_dhchap_hash_map {
>>
>> nvmet?
>>
> 
> Yeah; originally I coded that for the target side, and only later moved 
> it into the host side to have it usable for both.
> Will be fixing it up.
> 
>>> +    int id;
>>> +    int hash_len;
>>> +    const char hmac[15];
>>> +    const char digest[15];
>>> +} hash_map[] = {
>>> +    {.id = NVME_AUTH_DHCHAP_HASH_SHA256,
>>> +     .hash_len = 32,
>>> +     .hmac = "hmac(sha256)", .digest = "sha256" },
>>> +    {.id = NVME_AUTH_DHCHAP_HASH_SHA384,
>>> +     .hash_len = 48,
>>> +     .hmac = "hmac(sha384)", .digest = "sha384" },
>>> +    {.id = NVME_AUTH_DHCHAP_HASH_SHA512,
>>> +     .hash_len = 64,
>>> +     .hmac = "hmac(sha512)", .digest = "sha512" },
>>> +};
>>> +
>>> +const char *nvme_auth_hmac_name(int hmac_id)
>>
>> Should these arrays be static?
>>
> 
> Definitely.
> 
>>> +{
>>> +    int i;
>>> +
>>> +    for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
>>> +        if (hash_map[i].id == hmac_id)
>>> +            return hash_map[i].hmac;
>>> +    }
>>> +    return NULL;
>>> +}
>>> +EXPORT_SYMBOL_GPL(nvme_auth_hmac_name);
>>> +
>>> +const char *nvme_auth_digest_name(int hmac_id)
>>> +{
>>> +    int i;
>>> +
>>> +    for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
>>> +        if (hash_map[i].id == hmac_id)
>>> +            return hash_map[i].digest;
>>> +    }
>>> +    return NULL;
>>> +}
>>> +EXPORT_SYMBOL_GPL(nvme_auth_digest_name);
>>> +
>>> +int nvme_auth_hmac_len(int hmac_id)
>>> +{
>>> +    int i;
>>> +
>>> +    for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
>>> +        if (hash_map[i].id == hmac_id)
>>> +            return hash_map[i].hash_len;
>>> +    }
>>> +    return -1;
>>> +}
>>> +EXPORT_SYMBOL_GPL(nvme_auth_hmac_len);
>>> +
>>> +int nvme_auth_hmac_id(const char *hmac_name)
>>> +{
>>> +    int i;
>>> +
>>> +    for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
>>> +        if (!strncmp(hash_map[i].hmac, hmac_name,
>>> +                 strlen(hash_map[i].hmac)))
>>> +            return hash_map[i].id;
>>> +    }
>>> +    return -1;
>>> +}
>>> +EXPORT_SYMBOL_GPL(nvme_auth_hmac_id);
>>> +
>>> +unsigned char *nvme_auth_extract_secret(unsigned char *dhchap_secret,
>>> +                    size_t *dhchap_key_len)
>>> +{
>>> +    unsigned char *dhchap_key;
>>> +    u32 crc;
>>> +    int key_len;
>>> +    size_t allocated_len;
>>> +
>>> +    allocated_len = strlen(dhchap_secret) - 10;
>>
>> the 10 feels like a magic here, should at least note this is the
>> "DHHC-1:..." prefix.
>>
> 
> It _is_ magic. And it might even be better to just pass in the string 
> _without_ the DHHC-1: prefix.

I don't think that the user should pass in a key in that form at all,
its none of its concern the compliant NVMe representation is.

> 
>>> +    dhchap_key = kzalloc(allocated_len, GFP_KERNEL);
>>> +    if (!dhchap_key)
>>> +        return ERR_PTR(-ENOMEM);
>>> +
>>> +    key_len = base64_decode(dhchap_secret + 10,
>>> +                allocated_len, dhchap_key);
>>> +    if (key_len != 36 && key_len != 52 &&
>>> +        key_len != 68) {
>>> +        pr_debug("Invalid DH-HMAC-CHAP key len %d\n",
>>> +             key_len);
>>> +        kfree(dhchap_key);
>>> +        return ERR_PTR(-EINVAL);
>>> +    }
>>> +    pr_debug("DH-HMAC-CHAP Key: %*ph\n",
>>> +         (int)key_len, dhchap_key);
>>
>> One can argue if even printing this is problematic..
>>
> 
> Debugging scaffolding. You wouldn't believe how many things can go wrong...
> 
> And yes, that should be removed.

Cool.

>>> +
>>> +    /* The last four bytes is the CRC in little-endian format */
>>> +    key_len -= 4;
>>> +    /*
>>> +     * The linux implementation doesn't do pre- and post-increments,
>>> +     * so we have to do it manually.
>>> +     */
>>> +    crc = ~crc32(~0, dhchap_key, key_len);
>>> +
>>> +    if (get_unaligned_le32(dhchap_key + key_len) != crc) {
>>> +        pr_debug("DH-HMAC-CHAP crc mismatch (key %08x, crc %08x)\n",
>>> +               get_unaligned_le32(dhchap_key + key_len), crc);
>>> +        kfree(dhchap_key);
>>> +        return ERR_PTR(-EKEYREJECTED);
>>> +    }
>>> +    *dhchap_key_len = key_len;
>>> +    return dhchap_key;
>>> +}
>>> +EXPORT_SYMBOL_GPL(nvme_auth_extract_secret);
>>> +
>>> +static int nvme_auth_send(struct nvme_ctrl *ctrl, int qid,
>>> +              void *data, size_t tl)
>>> +{
>>> +    struct nvme_command cmd = {};
>>> +    blk_mq_req_flags_t flags = qid == NVME_QID_ANY ?
>>> +        0 : BLK_MQ_REQ_NOWAIT | BLK_MQ_REQ_RESERVED;
>>> +    struct request_queue *q = qid == NVME_QID_ANY ?
>>> +        ctrl->fabrics_q : ctrl->connect_q;
>>> +    int ret;
>>> +
>>> +    cmd.auth_send.opcode = nvme_fabrics_command;
>>> +    cmd.auth_send.fctype = nvme_fabrics_type_auth_send;
>>> +    cmd.auth_send.secp = NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER;
>>> +    cmd.auth_send.spsp0 = 0x01;
>>> +    cmd.auth_send.spsp1 = 0x01;
>>> +    cmd.auth_send.tl = tl;
>>> +
>>> +    ret = __nvme_submit_sync_cmd(q, &cmd, NULL, data, tl, 0, qid,
>>> +                     0, flags);
>>> +    if (ret)
>>> +        dev_dbg(ctrl->device,
>>> +            "%s: qid %d error %d\n", __func__, qid, ret);
>>
>> Maybe a little more informative print rather than __func__ ?
>>
> 
> Yes, can do.
> 
>>> +    return ret;
>>> +}
>>> +
>>> +static int nvme_auth_receive(struct nvme_ctrl *ctrl, int qid,
>>> +                 void *buf, size_t al,
>>> +                 u16 transaction, u8 expected_msg )
>>> +{
>>> +    struct nvme_command cmd = {};
>>> +    struct nvmf_auth_dhchap_failure_data *data = buf;
>>> +    blk_mq_req_flags_t flags = qid == NVME_QID_ANY ?
>>> +        0 : BLK_MQ_REQ_NOWAIT | BLK_MQ_REQ_RESERVED;
>>> +    struct request_queue *q = qid == NVME_QID_ANY ?
>>> +        ctrl->fabrics_q : ctrl->connect_q;
>>> +    int ret;
>>> +
>>> +    cmd.auth_receive.opcode = nvme_fabrics_command;
>>> +    cmd.auth_receive.fctype = nvme_fabrics_type_auth_receive;
>>> +    cmd.auth_receive.secp = NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER;
>>> +    cmd.auth_receive.spsp0 = 0x01;
>>> +    cmd.auth_receive.spsp1 = 0x01;
>>> +    cmd.auth_receive.al = al;
>>> +
>>> +    ret = __nvme_submit_sync_cmd(q, &cmd, NULL, buf, al, 0, qid,
>>> +                     0, flags);
>>> +    if (ret > 0) {
>>> +        dev_dbg(ctrl->device, "%s: qid %d nvme status %x\n",
>>> +            __func__, qid, ret);
>>> +        ret = -EIO;
>>> +    }
>>> +    if (ret < 0) {
>>> +        dev_dbg(ctrl->device, "%s: qid %d error %d\n",
>>> +            __func__, qid, ret);
>>> +        return ret;
>>> +    }
>>> +    dev_dbg(ctrl->device, "%s: qid %d auth_type %d auth_id %x\n",
>>> +        __func__, qid, data->auth_type, data->auth_id);
>>> +    if (data->auth_type == NVME_AUTH_COMMON_MESSAGES &&
>>> +        data->auth_id == NVME_AUTH_DHCHAP_MESSAGE_FAILURE1) {
>>> +        return data->reason_code_explanation;
>>> +    }
>>> +    if (data->auth_type != NVME_AUTH_DHCHAP_MESSAGES ||
>>> +        data->auth_id != expected_msg) {
>>> +        dev_warn(ctrl->device,
>>> +             "qid %d invalid message %02x/%02x\n",
>>> +             qid, data->auth_type, data->auth_id);
>>> +        return NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
>>> +    }
>>> +    if (le16_to_cpu(data->t_id) != transaction) {
>>> +        dev_warn(ctrl->device,
>>> +             "qid %d invalid transaction ID %d\n",
>>> +             qid, le16_to_cpu(data->t_id));
>>> +        return NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
>>> +    }
>>> +
>>> +    return 0;
>>> +}
>>> +
>>> +static int nvme_auth_dhchap_negotiate(struct nvme_ctrl *ctrl,
>>> +                      struct nvme_dhchap_context *chap,
>>> +                      void *buf, size_t buf_size)
>>
>> Maybe nvme_auth_set_dhchap_negotiate_data ?
>>
> 
> These are the individual steps in the state machine later on, so I 
> wanted to keep the names identical.
> But I'm open to suggestions.
> 
>>> +{
>>> +    struct nvmf_auth_dhchap_negotiate_data *data = buf;
>>> +    size_t size = sizeof(*data) + sizeof(union nvmf_auth_protocol);
>>> +
>>> +    if (buf_size < size)
>>> +        return -EINVAL;
>>> +
>>> +    memset((u8 *)buf, 0, size);
>>> +    data->auth_type = NVME_AUTH_COMMON_MESSAGES;
>>> +    data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE;
>>> +    data->t_id = cpu_to_le16(chap->transaction);
>>> +    data->sc_c = 0; /* No secure channel concatenation */
>>> +    data->napd = 1;
>>> +    data->auth_protocol[0].dhchap.authid = NVME_AUTH_DHCHAP_AUTH_ID;
>>> +    data->auth_protocol[0].dhchap.halen = 3;
>>> +    data->auth_protocol[0].dhchap.dhlen = 1;
>>> +    data->auth_protocol[0].dhchap.idlist[0] = 
>>> NVME_AUTH_DHCHAP_HASH_SHA256;
>>> +    data->auth_protocol[0].dhchap.idlist[1] = 
>>> NVME_AUTH_DHCHAP_HASH_SHA384;
>>> +    data->auth_protocol[0].dhchap.idlist[2] = 
>>> NVME_AUTH_DHCHAP_HASH_SHA512;
>>> +    data->auth_protocol[0].dhchap.idlist[3] = 
>>> NVME_AUTH_DHCHAP_DHGROUP_NULL;
>> You should comment that this routine expects buf to have enough
>> room for both negotiate and auth_proto structures.
>>
> Hmm. I do a check for the overall size at the start, so I'm not sure 
> what this will buy us.
> And actually, anyone wanting to make sense of the implementation would 
> need to look at the spec anyway.

Unrelated to the spec, just makes the code more readable. There is
an assumption on the buffer size passed, so it would be nice to
document it in the code (so it is less likely to break in the future
in the presence of assumptions).

>>> +
>>> +    return size;
>>> +}
>>> +
>>> +static int nvme_auth_dhchap_challenge(struct nvme_ctrl *ctrl,
>>> +                      struct nvme_dhchap_context *chap,
>>> +                      void *buf, size_t buf_size)
>>
>> Maybe nvme_auth_process_dhchap_challange ?
>>
> 
> See above. I'd rather have consistent names for the state machine.
> But I can change them to 'nvme_process_chchap_<statename>'
> 
>>> +{
>>> +    struct nvmf_auth_dhchap_challenge_data *data = buf;
>>> +    size_t size = sizeof(*data) + data->hl + data->dhvlen;
>>> +    const char *gid_name;
>>> +
>>> +    if (buf_size < size) {
>>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
>>> +        return -ENOMSG;
>>> +    }
>>> +
>>> +    if (data->hashid != NVME_AUTH_DHCHAP_HASH_SHA256 &&
>>> +        data->hashid != NVME_AUTH_DHCHAP_HASH_SHA384 &&
>>> +        data->hashid != NVME_AUTH_DHCHAP_HASH_SHA512) {
>>> +        dev_warn(ctrl->device,
>>> +             "qid %d: DH-HMAC-CHAP: invalid HASH ID %d\n",
>>> +             chap->qid, data->hashid);
>>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
>>> +        return -EPROTO;
>>> +    }
>>> +    switch (data->dhgid) {
>>> +    case NVME_AUTH_DHCHAP_DHGROUP_NULL:
>>> +        gid_name = "null";
>>> +        break;
>>> +    default:
>>> +        gid_name = NULL;
>>> +        break;
>>> +    }
>>> +    if (!gid_name) {
>>> +        dev_warn(ctrl->device,
>>> +             "qid %d: DH-HMAC-CHAP: invalid DH group id %d\n",
>>> +             chap->qid, data->dhgid);
>>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
>>> +        return -EPROTO;
>>> +    }
>>
>> Maybe some spaces between condition blocks?
>>
> 
> Ok.
> 
>>> +    if (data->dhgid != NVME_AUTH_DHCHAP_DHGROUP_NULL) {
>>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
>>> +        return -EPROTO;
>>> +    }
>>> +    if (data->dhgid == NVME_AUTH_DHCHAP_DHGROUP_NULL && data->dhvlen 
>>> != 0) {
>>> +        dev_warn(ctrl->device,
>>> +             "qid %d: DH-HMAC-CHAP: invalid DH value for NULL DH\n",
>>> +            chap->qid);
>>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
>>> +        return -EPROTO;
>>> +    }
>>> +    dev_dbg(ctrl->device, "%s: qid %d requested hash id %d\n",
>>> +        __func__, chap->qid, data->hashid);
>>> +    if (nvme_auth_hmac_len(data->hashid) != data->hl) {
>>> +        dev_warn(ctrl->device,
>>> +             "qid %d: DH-HMAC-CHAP: invalid hash length\n",
>>> +            chap->qid);
>>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
>>> +        return -EPROTO;
>>> +    }
>>> +    chap->hash_id = data->hashid;
>>> +    chap->hash_len = data->hl;
>>> +    chap->s1 = le32_to_cpu(data->seqnum);
>>> +    memcpy(chap->c1, data->cval, chap->hash_len);
>>> +
>>> +    return 0;
>>> +}
>>> +
>>> +static int nvme_auth_dhchap_reply(struct nvme_ctrl *ctrl,
>>> +                  struct nvme_dhchap_context *chap,
>>> +                  void *buf, size_t buf_size)
>>
>> nvme_auth_set_dhchap_reply
>>
> 
> Ah. Now I see what you're getting at.
> Okay, will be changing it.
> 
>>> +{
>>> +    struct nvmf_auth_dhchap_reply_data *data = buf;
>>> +    size_t size = sizeof(*data);
>>> +
>>> +    size += 2 * chap->hash_len;
>>> +    if (ctrl->opts->dhchap_auth) {
>>
>> The ctrl opts is not clear to me. what is dhchap_auth
>> mean?
>>
> As stated above, this is for bi-directional authentication.
> And yes, it is poor wording.
> 
> 'dhchap_bidirectional' ?

dhchap_auth_ctrl maybe.

>> Also shouldn't these params be lifted to the subsys?
>>
> 
> I kinda like to have it all encapsulated in a common per-queue 
> structure; on the host side this one isn't even attached to anything, so 
> any new authentication attempt will allocate a new one, with no chance 
> of accidentally re-using existing values.
> I thought this to be a rather nice property for a state-machine.

I understand, but different controllers for a single subsystem
should not behave differently. Meaning for one you authenticate
and the other you don't (or with different keys).

> 
>>> +        get_random_bytes(chap->c2, chap->hash_len);
>>> +        chap->s2 = nvme_dhchap_seqnum++;
>>> +    } else
>>> +        memset(chap->c2, 0, chap->hash_len);
>>> +
>>> +    if (chap->host_key_len)
>>> +        size += chap->host_key_len;
>>> +
>>> +    if (buf_size < size)
>>> +        return -EINVAL;
>>> +
>>> +    memset(buf, 0, size);
>>> +    data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
>>> +    data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_REPLY;
>>> +    data->t_id = cpu_to_le16(chap->transaction);
>>> +    data->hl = chap->hash_len;
>>> +    data->dhvlen = chap->host_key_len;
>>> +    data->seqnum = cpu_to_le32(chap->s2);
>>> +    memcpy(data->rval, chap->response, chap->hash_len);
>>> +    if (ctrl->opts->dhchap_auth) {
>>> +        dev_dbg(ctrl->device, "%s: qid %d ctrl challenge %*ph\n",
>>> +            __func__, chap->qid,
>>> +            chap->hash_len, chap->c2);
>>> +        data->cvalid = 1;
>>> +        memcpy(data->rval + chap->hash_len, chap->c2,
>>> +               chap->hash_len);
>>> +    }
>>> +    if (chap->host_key_len)
>>> +        memcpy(data->rval + 2 * chap->hash_len, chap->host_key,
>>> +               chap->host_key_len);
>>> +
>>> +    return size;
>>> +}
>>> +
>>> +static int nvme_auth_dhchap_success1(struct nvme_ctrl *ctrl,
>>> +                     struct nvme_dhchap_context *chap,
>>> +                     void *buf, size_t buf_size)
>>
>> nvme_auth_process_dhchap_success1
>>
> 
> OK.
> 
>>> +{
>>> +    struct nvmf_auth_dhchap_success1_data *data = buf;
>>> +    size_t size = sizeof(*data);
>>> +
>>> +    if (ctrl->opts->dhchap_auth)
>>> +        size += chap->hash_len;
>>> +
>>> +
>>> +    if (buf_size < size) {
>>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
>>> +        return -ENOMSG;
>>> +    }
>>> +
>>> +    if (data->hl != chap->hash_len) {
>>> +        dev_warn(ctrl->device,
>>> +             "qid %d: DH-HMAC-CHAP: invalid hash length %d\n",
>>> +             chap->qid, data->hl);
>>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
>>> +        return -EPROTO;
>>> +    }
>>> +
>>> +    if (!data->rvalid)
>>> +        return 0;
>>> +
>>> +    /* Validate controller response */
>>> +    if (memcmp(chap->response, data->rval, data->hl)) {
>>> +        dev_dbg(ctrl->device, "%s: qid %d ctrl response %*ph\n",
>>> +            __func__, chap->qid, chap->hash_len, data->rval);
>>> +        dev_dbg(ctrl->device, "%s: qid %d host response %*ph\n",
>>> +            __func__, chap->qid, chap->hash_len, chap->response);
>>> +        dev_warn(ctrl->device,
>>> +             "qid %d: DH-HMAC-CHAP: controller authentication 
>>> failed\n",
>>> +             chap->qid);
>>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
>>> +        return -EPROTO;
>>> +    }
>>> +    dev_info(ctrl->device,
>>> +         "qid %d: DH-HMAC-CHAP: controller authenticated\n",
>>> +        chap->qid);
>>> +    return 0;
>>> +}
>>> +
>>> +static int nvme_auth_dhchap_success2(struct nvme_ctrl *ctrl,
>>> +                     struct nvme_dhchap_context *chap,
>>> +                     void *buf, size_t buf_size)
>>
>> same
>>
>>> +{
>>> +    struct nvmf_auth_dhchap_success2_data *data = buf;
>>> +    size_t size = sizeof(*data);
>>> +
>>> +    memset(buf, 0, size);
>>> +    data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
>>> +    data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_SUCCESS2;
>>> +    data->t_id = cpu_to_le16(chap->transaction);
>>> +
>>> +    return size;
>>> +}
>>> +
>>> +static int nvme_auth_dhchap_failure2(struct nvme_ctrl *ctrl,
>>> +                     struct nvme_dhchap_context *chap,
>>> +                     void *buf, size_t buf_size)
>>
>> same
>>
>>> +{
>>> +    struct nvmf_auth_dhchap_failure_data *data = buf;
>>> +    size_t size = sizeof(*data);
>>> +
>>> +    memset(buf, 0, size);
>>> +    data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
>>> +    data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_FAILURE2;
>>> +    data->t_id = cpu_to_le16(chap->transaction);
>>> +    data->reason_code = 1;
>>> +    data->reason_code_explanation = chap->status;
>>> +
>>> +    return size;
>>> +}
>>> +
>>> +int nvme_auth_select_hash(struct nvme_ctrl *ctrl,
>>> +              struct nvme_dhchap_context *chap)
>>
>> Maybe _select_hf (hash function)? not a must, just sticks
>> to the spec language.
>>
> 
> Hmm. Will be checking.
> 
>>> +{
>>> +    char *hash_name;
>>> +    int ret;
>>> +
>>> +    switch (chap->hash_id) {
>>> +    case NVME_AUTH_DHCHAP_HASH_SHA256:
>>> +        hash_name = "hmac(sha256)";
>>> +        break;
>>> +    case NVME_AUTH_DHCHAP_HASH_SHA384:
>>> +        hash_name = "hmac(sha384)";
>>> +        break;
>>> +    case NVME_AUTH_DHCHAP_HASH_SHA512:
>>> +        hash_name = "hmac(sha512)";
>>> +        break;
>>> +    default:
>>> +        hash_name = NULL;
>>> +        break;
>>> +    }
>>> +    if (!hash_name) {
>>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
>>> +        return -EPROTO;
>>> +    }
>>> +    chap->shash_tfm = crypto_alloc_shash(hash_name, 0,
>>> +                         CRYPTO_ALG_ALLOCATES_MEMORY);
>>> +    if (IS_ERR(chap->shash_tfm)) {
>>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
>>> +        chap->shash_tfm = NULL;
>>> +        return -EPROTO;
>>> +    }
>>> +    if (!chap->key) {
>>> +        dev_warn(ctrl->device, "qid %d: cannot select hash, no key\n",
>>> +             chap->qid);
>>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
>>> +        crypto_free_shash(chap->shash_tfm);
>>
>> Wouldn't it better to check this before allocating the tfm?
>>
> 
> Indeed. Will be changing it.
> 
>>> +        chap->shash_tfm = NULL;
>>> +        return -EINVAL;
>>> +    }
>>> +    ret = crypto_shash_setkey(chap->shash_tfm, chap->key, 
>>> chap->key_len);
>>> +    if (ret) {
>>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
>>> +        crypto_free_shash(chap->shash_tfm);
>>> +        chap->shash_tfm = NULL;
>>> +        return ret;
>>> +    }
>>> +    dev_info(ctrl->device, "qid %d: DH-HMAC_CHAP: selected hash %s\n",
>>> +         chap->qid, hash_name);
>>> +    return 0;
>>> +}
>>> +
>>> +static int nvme_auth_dhchap_host_response(struct nvme_ctrl *ctrl,
>>> +                      struct nvme_dhchap_context *chap)
>>> +{
>>> +    SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
>>> +    u8 buf[4], *challenge = chap->c1;
>>> +    int ret;
>>> +
>>> +    dev_dbg(ctrl->device, "%s: qid %d host response seq %d 
>>> transaction %d\n",
>>> +        __func__, chap->qid, chap->s1, chap->transaction);
>>> +    shash->tfm = chap->shash_tfm;
>>> +    ret = crypto_shash_init(shash);
>>> +    if (ret)
>>> +        goto out;
>>> +    ret = crypto_shash_update(shash, challenge, chap->hash_len);
>>> +    if (ret)
>>> +        goto out;
>>> +    put_unaligned_le32(chap->s1, buf);
>>> +    ret = crypto_shash_update(shash, buf, 4);
>>> +    if (ret)
>>> +        goto out;
>>> +    put_unaligned_le16(chap->transaction, buf);
>>> +    ret = crypto_shash_update(shash, buf, 2);
>>> +    if (ret)
>>> +        goto out;
>>> +    memset(buf, 0, sizeof(buf));
>>> +    ret = crypto_shash_update(shash, buf, 1);
>>> +    if (ret)
>>> +        goto out;
>>> +    ret = crypto_shash_update(shash, "HostHost", 8);
>>
>> HostHost ? Can you refer me to the specific section
>> that talks about this?
>>
> 
> NVMe 2.0 section DH-HMAC-CHAP_Reply Message, paragraph Response Value.
> HostHost.
> 
>> Would be good to have a comment on the format fed to the
>> shash.
>>
> 
> Yes, will be doing so.
> 
>>> +    if (ret)
>>> +        goto out;
>>> +    ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
>>> +                  strlen(ctrl->opts->host->nqn));
>>> +    if (ret)
>>> +        goto out;
>>> +    ret = crypto_shash_update(shash, buf, 1);
>>> +    if (ret)
>>> +        goto out;
>>> +    ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
>>> +                strlen(ctrl->opts->subsysnqn));
>>> +    if (ret)
>>> +        goto out;
>>> +    ret = crypto_shash_final(shash, chap->response);
>>> +out:
>>> +    return ret;
>>> +}
>>> +
>>> +static int nvme_auth_dhchap_ctrl_response(struct nvme_ctrl *ctrl,
>>> +                      struct nvme_dhchap_context *chap)
>>> +{
>>> +    SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
>>> +    u8 buf[4], *challenge = chap->c2;
>>> +    int ret;
>>> +
>>> +    dev_dbg(ctrl->device, "%s: qid %d host response seq %d 
>>> transaction %d\n",
>>> +        __func__, chap->qid, chap->s2, chap->transaction);
>>> +    dev_dbg(ctrl->device, "%s: qid %d challenge %*ph\n",
>>> +        __func__, chap->qid, chap->hash_len, challenge);
>>> +    dev_dbg(ctrl->device, "%s: qid %d subsysnqn %s\n",
>>> +        __func__, chap->qid, ctrl->opts->subsysnqn);
>>> +    dev_dbg(ctrl->device, "%s: qid %d hostnqn %s\n",
>>> +        __func__, chap->qid, ctrl->opts->host->nqn);
>>> +    shash->tfm = chap->shash_tfm;
>>> +    ret = crypto_shash_init(shash);
>>> +    if (ret)
>>> +        goto out;
>>> +    ret = crypto_shash_update(shash, challenge, chap->hash_len);
>>> +    if (ret)
>>> +        goto out;
>>> +    put_unaligned_le32(chap->s2, buf);
>>> +    ret = crypto_shash_update(shash, buf, 4);
>>> +    if (ret)
>>> +        goto out;
>>> +    put_unaligned_le16(chap->transaction, buf);
>>> +    ret = crypto_shash_update(shash, buf, 2);
>>> +    if (ret)
>>> +        goto out;
>>> +    memset(buf, 0, 4);
>>> +    ret = crypto_shash_update(shash, buf, 1);
>>> +    if (ret)
>>> +        goto out;
>>> +    ret = crypto_shash_update(shash, "Controller", 10);
>>> +    if (ret)
>>> +        goto out;
>>> +    ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
>>> +                  strlen(ctrl->opts->subsysnqn));
>>> +    if (ret)
>>> +        goto out;
>>> +    ret = crypto_shash_update(shash, buf, 1);
>>> +    if (ret)
>>> +        goto out;
>>> +    ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
>>> +                  strlen(ctrl->opts->host->nqn));
>>> +    if (ret)
>>> +        goto out;
>>> +    ret = crypto_shash_final(shash, chap->response);
>>> +out:
>>> +    return ret;
>>> +}
>>> +
>>> +int nvme_auth_generate_key(struct nvme_ctrl *ctrl,
>>> +               struct nvme_dhchap_context *chap)
>>> +{
>>> +    int ret;
>>> +    u8 key_hash;
>>> +    const char *hmac_name;
>>> +    struct crypto_shash *key_tfm;
>>> +
>>> +    if (sscanf(ctrl->opts->dhchap_secret, "DHHC-1:%hhd:%*s:",
>>> +           &key_hash) != 1)
>>> +        return -EINVAL;
>>
>> I'd expect that the user will pass in a secret key (as binary)
>> the the driver will build the spec compliant formatted string no?
>>  > Am I not reading this correctly?
>>
> 
> I'm under the impression that this is the format into which the 
> User/Admin will get hold of the secret key.
> Spec says:
> 
> '... all NVMe over Fabrics entities shall support the following ASCII
> representation of secrets ...'
> 
> And as the userspace interface is the only way how the user/admin 
> _could_ interact with the NVMe over Fabrics entities in Linux I guess 
> we'll need to be able to parse it.

Right... But who is responsible for crcing and encoding it? nvme-cli?

> We sure could allow a binary secret, too, but then what would be the 
> point in converting it into the secret representation?
> The protocol revolves around the binary secret, not the transport 
> representation.

I am not sure I understand who is responsible for represnting the key
this way in Linux?

>>> +
>>> +    chap->key = nvme_auth_extract_secret(ctrl->opts->dhchap_secret,
>>> +                         &chap->key_len);
>>> +    if (IS_ERR(chap->key)) {
>>> +        ret = PTR_ERR(chap->key);
>>> +        chap->key = NULL;
>>> +        return ret;
>>> +    }
>>> +
>>> +    if (key_hash == 0)
>>> +        return 0;
>>> +
>>> +    hmac_name = nvme_auth_hmac_name(key_hash);
>>> +    if (!hmac_name) {
>>> +        pr_debug("Invalid key hash id %d\n", key_hash);
>>> +        return -EKEYREJECTED;
>>> +    }
>>
>> Why does the user influence the hmac used? isn't that is driven
>> by the susbsystem?
>>
>> I don't think that the user should choose in this level.
>>
> 
> That is another weirdness of the spec.
> The _secret_ will be hashed with a specific function, and that function 
> is stated in the transport representation.
> (Cf section "DH-HMAC-CHAP Security Requirements").
> This is _not_ the hash function used by the authentication itself, which 
> will be selected by the protocol.

Yes, I see it now, and it is indeed confusing.

> So it's not the user here, but rather the transport specification of the 
> key which selects the hash algorithm.

What do you mean by the transport specification?

>>> +
>>> +    key_tfm = crypto_alloc_shash(hmac_name, 0, 0);
>>> +    if (IS_ERR(key_tfm)) {
>>> +        kfree(chap->key);
>>> +        chap->key = NULL;
>>> +        ret = PTR_ERR(key_tfm);
>>
>> You set ret and later return 0? I think that the success
>> path in the else clause is hard to read and error prone...
>>
> 
> Do I? Will need to fix it up.
> 
>>> +    } else {
>>> +        SHASH_DESC_ON_STACK(shash, key_tfm);
>>> +
>>> +        shash->tfm = key_tfm;
>>> +        ret = crypto_shash_setkey(key_tfm, chap->key,
>>> +                      chap->key_len);
>>> +        if (ret < 0) {
>>> +            crypto_free_shash(key_tfm);
>>> +            kfree(chap->key);
>>> +            chap->key = NULL;
>>> +            return ret;
>>> +        }
>>> +        crypto_shash_init(shash);
>>> +        crypto_shash_update(shash, ctrl->opts->host->nqn,
>>> +                    strlen(ctrl->opts->host->nqn));
>>> +        crypto_shash_update(shash, "NVMe-over-Fabrics", 17);
>>> +        crypto_shash_final(shash, chap->key);
>>> +        crypto_free_shash(key_tfm);
>>
>> Shouldn't these be done when preparing the dh-hmac-chap reply?
>>
> 
> By setting the hash here I avoid having to pass the required hash 
> function for the secret transformation.
> I could be doing the entire secret transformation thingie when preparing 
> the reply; reason why I did it here is that _having_ a secret is the 
> precondition to everything else, so I wanted to check upfront for that.
> But I'll check what would happen if I move it.

Now that I understand that this is not the authentication transformation
its ok I guess. Please add a comment in the code so its clearer.

>>> +    }
>>> +    return 0;
>>> +}
>>> +
>>> +void nvme_auth_free(struct nvme_dhchap_context *chap)
>>> +{
>>> +    if (chap->shash_tfm)
>>> +        crypto_free_shash(chap->shash_tfm);
>>> +    if (chap->key)
>>> +        kfree(chap->key);
>>> +    if (chap->ctrl_key)
>>> +        kfree(chap->ctrl_key);
>>> +    if (chap->host_key)
>>> +        kfree(chap->host_key);
>>> +    if (chap->sess_key)
>>> +        kfree(chap->sess_key);
>>
>> No need to check null for kfree...
>>
> 
> Will be fixing it up.
> 
>>> +    kfree(chap);
>>> +}
>>> +
>>> +int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid)
>>> +{
>>> +    struct nvme_dhchap_context *chap;
>>> +    void *buf;
>>> +    size_t buf_size, tl;
>>> +    int ret = 0;
>>> +
>>> +    chap = kzalloc(sizeof(*chap), GFP_KERNEL);
>>> +    if (!chap)
>>> +        return -ENOMEM;
>>> +    chap->qid = qid;
>>> +    chap->transaction = ctrl->transaction++;
>>> +
>>> +    ret = nvme_auth_generate_key(ctrl, chap);
>>> +    if (ret) {
>>> +        dev_dbg(ctrl->device, "%s: failed to generate key, error %d\n",
>>> +            __func__, ret);
>>> +        nvme_auth_free(chap);
>>> +        return ret;
>>> +    }
>>> +
>>> +    /*
>>> +     * Allocate a large enough buffer for the entire negotiation:
>>> +     * 4k should be enough to ffdhe8192.
>>> +     */
>>> +    buf_size = 4096;
>>> +    buf = kzalloc(buf_size, GFP_KERNEL);
>>> +    if (!buf) {
>>> +        ret = -ENOMEM;
>>> +        goto out;
>>> +    }
>>> +
>>> +    /* DH-HMAC-CHAP Step 1: send negotiate */
>>
>> I'd consider breaking these into sub-routines.
>>
> 
> Which ones? The preparation step?

I'm thinking:
1. nvme_auth_initiate_negotiation
    - nvme_auth_set_dhchap_negotiate_data
    - nvme_auth_send
2. nvme_auth_do_challange
    - nvme_auth_receive
    - nvme_auth_process_dhchap_challange
    - nvme_auth_select_hash
    - nvme_auth_dhchap_host_response
    - nvme_auth_set_dhchap_reply
    - nvme_auth_send
    - nvme_auth_receive
    - nvme_auth_process_dhchap_success1
3. if (ctrl->opts->dhchap_auth_ctrl)
    - nvme_auth_dhchap_authenticate_ctrl
      (e.g. nvme_auth_dhchap_ctrl_response)
4. nvme_auth_acknowledge_transaction
    - nvme_auth_set_dhchap_success2
    - nvme_auth_send

if steps 1,2,3 failed, goto target will have a func:
5. nvme_auth_fail_transaction
    - nvme_auth_set_dhchap_failure2
    - nvme_auth_send

> Sure, can do.
> 
>>> +    dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP negotiate\n",
>>> +        __func__, qid);
>>> +    ret = nvme_auth_dhchap_negotiate(ctrl, chap, buf, buf_size);
>>> +    if (ret < 0)
>>> +        goto out;
>>> +    tl = ret;
>>> +    ret = nvme_auth_send(ctrl, qid, buf, tl);
>>> +    if (ret)
>>> +        goto out;
>>> +
>>> +    memset(buf, 0, buf_size);
>>> +    ret = nvme_auth_receive(ctrl, qid, buf, buf_size, 
>>> chap->transaction,
>>> +                NVME_AUTH_DHCHAP_MESSAGE_CHALLENGE);
>>> +    if (ret < 0) {
>>> +        dev_dbg(ctrl->device,
>>> +            "%s: qid %d DH-HMAC-CHAP failed to receive challenge\n",
>>> +            __func__, qid);
>>> +        goto out;
>>> +    }
>>> +    if (ret > 0) {
>>> +        chap->status = ret;
>>> +        goto fail1;
>>> +    }
>>> +
>>> +    /* DH-HMAC-CHAP Step 2: receive challenge */
>>> +    dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP challenge\n",
>>> +        __func__, qid);
>>> +
>>> +    ret = nvme_auth_dhchap_challenge(ctrl, chap, buf, buf_size);
>>> +    if (ret) {
>>> +        /* Invalid parameters for negotiate */
>>> +        goto fail2;
>>> +    }
>>> +
>>> +    dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP select hash\n",
>>> +        __func__, qid);
>>> +    ret = nvme_auth_select_hash(ctrl, chap);
>>> +    if (ret)
>>> +        goto fail2;
>>> +
>>> +    dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP host response\n",
>>> +        __func__, qid);
>>> +    ret = nvme_auth_dhchap_host_response(ctrl, chap);
>>> +    if (ret)
>>> +        goto fail2;
>>> +
>>> +    /* DH-HMAC-CHAP Step 3: send reply */
>>> +    dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP reply\n",
>>> +        __func__, qid);
>>> +    ret = nvme_auth_dhchap_reply(ctrl, chap, buf, buf_size);
>>> +    if (ret < 0)
>>> +        goto fail2;
>>> +
>>> +    tl = ret;
>>> +    ret = nvme_auth_send(ctrl, qid, buf, tl);
>>> +    if (ret)
>>> +        goto fail2;
>>> +
>>> +    memset(buf, 0, buf_size);
>>> +    ret = nvme_auth_receive(ctrl, qid, buf, buf_size, 
>>> chap->transaction,
>>> +                NVME_AUTH_DHCHAP_MESSAGE_SUCCESS1);
>>> +    if (ret < 0) {
>>> +        dev_dbg(ctrl->device,
>>> +            "%s: qid %d DH-HMAC-CHAP failed to receive success1\n",
>>> +            __func__, qid);
>>> +        goto out;
>>> +    }
>>> +    if (ret > 0) {
>>> +        chap->status = ret;
>>> +        goto fail1;
>>> +    }
>>> +
>>> +    if (ctrl->opts->dhchap_auth) {
>>> +        dev_dbg(ctrl->device,
>>> +            "%s: qid %d DH-HMAC-CHAP controller response\n",
>>> +            __func__, qid);
>>> +        ret = nvme_auth_dhchap_ctrl_response(ctrl, chap);
>>> +        if (ret)
>>> +            goto fail2;
>>> +    }
>>> +
>>> +    /* DH-HMAC-CHAP Step 4: receive success1 */
>>> +    dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP success1\n",
>>> +        __func__, qid);
>>> +    ret = nvme_auth_dhchap_success1(ctrl, chap, buf, buf_size);
>>> +    if (ret < 0) {
>>> +        /* Controller authentication failed */
>>> +        goto fail2;
>>> +    }
>>> +    tl = ret;
>>> +    /* DH-HMAC-CHAP Step 5: send success2 */
>>> +    dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP success2\n",
>>> +        __func__, qid);
>>> +    tl = nvme_auth_dhchap_success2(ctrl, chap, buf, buf_size);
>>> +    ret = nvme_auth_send(ctrl, qid, buf, tl);
>>> +    if (!ret)
>>> +        goto out;
>>> +
>>> +fail1:
>>> +    dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP failure1, status 
>>> %x\n",
>>> +        __func__, qid, chap->status);
>>> +    goto out;
>>> +
>>> +fail2:
>>> +    dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP failure2, status 
>>> %x\n",
>>> +        __func__, qid, chap->status);
>>> +    tl = nvme_auth_dhchap_failure2(ctrl, chap, buf, buf_size);
>>> +    ret = nvme_auth_send(ctrl, qid, buf, tl);
>>> +
>>> +out:
>>> +    if (!ret && chap->status)
>>> +        ret = -EPROTO;
>>> +    if (!ret) {
>>> +        ctrl->dhchap_hash = chap->hash_id;
>>> +    }
>>> +    kfree(buf);
>>> +    nvme_auth_free(chap);
>>> +    return ret;
>>> +}
>>> diff --git a/drivers/nvme/host/auth.h b/drivers/nvme/host/auth.h
>>> new file mode 100644
>>> index 000000000000..4950b1cb9470
>>> --- /dev/null
>>> +++ b/drivers/nvme/host/auth.h
>>> @@ -0,0 +1,23 @@
>>> +/* SPDX-License-Identifier: GPL-2.0 */
>>> +/*
>>> + * Copyright (c) 2021 Hannes Reinecke, SUSE Software Solutions
>>> + */
>>> +
>>> +#ifndef _NVME_AUTH_H
>>> +#define _NVME_AUTH_H
>>> +
>>> +const char *nvme_auth_dhgroup_name(int dhgroup_id);
>>> +int nvme_auth_dhgroup_pubkey_size(int dhgroup_id);
>>> +int nvme_auth_dhgroup_privkey_size(int dhgroup_id);
>>> +const char *nvme_auth_dhgroup_kpp(int dhgroup_id);
>>> +int nvme_auth_dhgroup_id(const char *dhgroup_name);
>>> +
>>> +const char *nvme_auth_hmac_name(int hmac_id);
>>> +const char *nvme_auth_digest_name(int hmac_id);
>>> +int nvme_auth_hmac_id(const char *hmac_name);
>>> +int nvme_auth_hmac_len(int hmac_len);
>>> +
>>> +unsigned char *nvme_auth_extract_secret(unsigned char *dhchap_secret,
>>> +                    size_t *dhchap_key_len);
>>> +
>>> +#endif /* _NVME_AUTH_H */
>>> diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
>>> index 11779be42186..7ce9b666dc09 100644
>>> --- a/drivers/nvme/host/core.c
>>> +++ b/drivers/nvme/host/core.c
>>> @@ -708,7 +708,9 @@ bool __nvme_check_ready(struct nvme_ctrl *ctrl, 
>>> struct request *rq,
>>>           switch (ctrl->state) {
>>>           case NVME_CTRL_CONNECTING:
>>>               if (blk_rq_is_passthrough(rq) && 
>>> nvme_is_fabrics(req->cmd) &&
>>> -                req->cmd->fabrics.fctype == nvme_fabrics_type_connect)
>>> +                (req->cmd->fabrics.fctype == 
>>> nvme_fabrics_type_connect ||
>>> +                 req->cmd->fabrics.fctype == 
>>> nvme_fabrics_type_auth_send ||
>>> +                 req->cmd->fabrics.fctype == 
>>> nvme_fabrics_type_auth_receive))
>>>                   return true;
>>>               break;
>>>           default:
>>> @@ -3426,6 +3428,66 @@ static ssize_t 
>>> nvme_ctrl_fast_io_fail_tmo_store(struct device *dev,
>>>   static DEVICE_ATTR(fast_io_fail_tmo, S_IRUGO | S_IWUSR,
>>>       nvme_ctrl_fast_io_fail_tmo_show, 
>>> nvme_ctrl_fast_io_fail_tmo_store);
>>> +#ifdef CONFIG_NVME_AUTH
>>> +struct nvmet_dhchap_hash_map {
>>> +    int id;
>>> +    const char name[15];
>>> +} hash_map[] = {
>>> +    {.id = NVME_AUTH_DHCHAP_HASH_SHA256,
>>> +     .name = "hmac(sha256)", },
>>> +    {.id = NVME_AUTH_DHCHAP_HASH_SHA384,
>>> +     .name = "hmac(sha384)", },
>>> +    {.id = NVME_AUTH_DHCHAP_HASH_SHA512,
>>> +     .name = "hmac(sha512)", },
>>> +};
>>> +
>>> +static ssize_t dhchap_hash_show(struct device *dev,
>>> +    struct device_attribute *attr, char *buf)
>>> +{
>>> +    struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
>>> +    int i;
>>> +
>>> +    for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
>>> +        if (hash_map[i].id == ctrl->dhchap_hash)
>>> +            return sprintf(buf, "%s\n", hash_map[i].name);
>>> +    }
>>> +    return sprintf(buf, "none\n");
>>> +}
>>> +DEVICE_ATTR_RO(dhchap_hash);
>>> +
>>> +struct nvmet_dhchap_group_map {
>>> +    int id;
>>> +    const char name[15];
>>> +} dhgroup_map[] = {
>>> +    {.id = NVME_AUTH_DHCHAP_DHGROUP_NULL,
>>> +     .name = "NULL", },
>>> +    {.id = NVME_AUTH_DHCHAP_DHGROUP_2048,
>>> +     .name = "ffdhe2048", },
>>> +    {.id = NVME_AUTH_DHCHAP_DHGROUP_3072,
>>> +     .name = "ffdhe3072", },
>>> +    {.id = NVME_AUTH_DHCHAP_DHGROUP_4096,
>>> +     .name = "ffdhe4096", },
>>> +    {.id = NVME_AUTH_DHCHAP_DHGROUP_6144,
>>> +     .name = "ffdhe6144", },
>>> +    {.id = NVME_AUTH_DHCHAP_DHGROUP_8192,
>>> +     .name = "ffdhe8192", },
>>> +};
>>> +
>>> +static ssize_t dhchap_dhgroup_show(struct device *dev,
>>> +    struct device_attribute *attr, char *buf)
>>> +{
>>> +    struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
>>> +    int i;
>>> +
>>> +    for (i = 0; i < ARRAY_SIZE(dhgroup_map); i++) {
>>> +        if (hash_map[i].id == ctrl->dhchap_dhgroup)
>>> +            return sprintf(buf, "%s\n", dhgroup_map[i].name);
>>> +    }
>>> +    return sprintf(buf, "none\n");
>>> +}
>>> +DEVICE_ATTR_RO(dhchap_dhgroup);
>>> +#endif
>>> +
>>>   static struct attribute *nvme_dev_attrs[] = {
>>>       &dev_attr_reset_controller.attr,
>>>       &dev_attr_rescan_controller.attr,
>>> @@ -3447,6 +3509,10 @@ static struct attribute *nvme_dev_attrs[] = {
>>>       &dev_attr_reconnect_delay.attr,
>>>       &dev_attr_fast_io_fail_tmo.attr,
>>>       &dev_attr_kato.attr,
>>> +#ifdef CONFIG_NVME_AUTH
>>> +    &dev_attr_dhchap_hash.attr,
>>> +    &dev_attr_dhchap_dhgroup.attr,
>>> +#endif
>>>       NULL
>>>   };
>>> @@ -3470,6 +3536,10 @@ static umode_t 
>>> nvme_dev_attrs_are_visible(struct kobject *kobj,
>>>           return 0;
>>>       if (a == &dev_attr_fast_io_fail_tmo.attr && !ctrl->opts)
>>>           return 0;
>>> +#ifdef CONFIG_NVME_AUTH
>>> +    if (a == &dev_attr_dhchap_hash.attr && !ctrl->opts)
>>> +        return 0;
>>> +#endif
>>>       return a->mode;
>>>   }
>>> @@ -4581,6 +4651,11 @@ static inline void _nvme_check_size(void)
>>>       BUILD_BUG_ON(sizeof(struct nvme_smart_log) != 512);
>>>       BUILD_BUG_ON(sizeof(struct nvme_dbbuf) != 64);
>>>       BUILD_BUG_ON(sizeof(struct nvme_directive_cmd) != 64);
>>> +    BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_negotiate_data) != 8);
>>> +    BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_challenge_data) != 16);
>>> +    BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_reply_data) != 16);
>>> +    BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_success1_data) != 16);
>>> +    BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_success2_data) != 16);
>>>   }
>>> diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
>>> index a5469fd9d4c3..6404ab9b604b 100644
>>> --- a/drivers/nvme/host/fabrics.c
>>> +++ b/drivers/nvme/host/fabrics.c
>>> @@ -366,6 +366,7 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)
>>>       union nvme_result res;
>>>       struct nvmf_connect_data *data;
>>>       int ret;
>>> +    u32 result;
>>>       cmd.connect.opcode = nvme_fabrics_command;
>>>       cmd.connect.fctype = nvme_fabrics_type_connect;
>>> @@ -398,8 +399,18 @@ int nvmf_connect_admin_queue(struct nvme_ctrl 
>>> *ctrl)
>>>           goto out_free_data;
>>>       }
>>> -    ctrl->cntlid = le16_to_cpu(res.u16);
>>> -
>>> +    result = le32_to_cpu(res.u32);
>>> +    ctrl->cntlid = result & 0xFFFF;
>>> +    if ((result >> 16) & 2) {
>>> +        /* Authentication required */
>>> +        ret = nvme_auth_negotiate(ctrl, NVME_QID_ANY);
>>> +        if (ret)
>>> +            dev_warn(ctrl->device,
>>> +                 "qid 0: authentication failed\n");
>>> +        else
>>> +            dev_info(ctrl->device,
>>> +                 "qid 0: authenticated\n");
>>
>> info is too chatty.
>>
> 
> Hmm. I know I need to work on logging...
> 
>>> +    }
>>>   out_free_data:
>>>       kfree(data);
>>>       return ret;
>>> @@ -432,6 +443,7 @@ int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, 
>>> u16 qid)
>>>       struct nvmf_connect_data *data;
>>>       union nvme_result res;
>>>       int ret;
>>> +    u32 result;
>>>       cmd.connect.opcode = nvme_fabrics_command;
>>>       cmd.connect.fctype = nvme_fabrics_type_connect;
>>> @@ -457,6 +469,17 @@ int nvmf_connect_io_queue(struct nvme_ctrl 
>>> *ctrl, u16 qid)
>>>           nvmf_log_connect_error(ctrl, ret, le32_to_cpu(res.u32),
>>>                          &cmd, data);
>>>       }
>>> +    result = le32_to_cpu(res.u32);
>>> +    if ((result >> 16) & 2) {
>>> +        /* Authentication required */
>>> +        ret = nvme_auth_negotiate(ctrl, qid);
>>> +        if (ret)
>>> +            dev_warn(ctrl->device,
>>> +                 "qid %u: authentication failed\n", qid);
>>> +        else
>>> +            dev_info(ctrl->device,
>>> +                 "qid %u: authenticated\n", qid);
>>> +    }
>>>       kfree(data);
>>>       return ret;
>>>   }
>>> @@ -548,6 +571,9 @@ static const match_table_t opt_tokens = {
>>>       { NVMF_OPT_NR_POLL_QUEUES,    "nr_poll_queues=%d"    },
>>>       { NVMF_OPT_TOS,            "tos=%d"        },
>>>       { NVMF_OPT_FAIL_FAST_TMO,    "fast_io_fail_tmo=%d"    },
>>> +    { NVMF_OPT_DHCHAP_SECRET,    "dhchap_secret=%s"    },
>>> +    { NVMF_OPT_DHCHAP_AUTH,        "authenticate"        },
>>> +    { NVMF_OPT_DHCHAP_GROUP,    "dhchap_group=%s"    },
>>
>> Isn't the group driven by the subsystem? also why is there a
>> "authenticate" boolean? what is it good for?
>>
> Ah. Right. Of course, the 'group' is pointless here.
> And the 'authenticate' bool is the abovementioned bidirectional 
> authentication.
> I _do_ need to give it another name.
> 
>>>       { NVMF_OPT_ERR,            NULL            }
>>>   };
>>> @@ -824,6 +850,35 @@ static int nvmf_parse_options(struct 
>>> nvmf_ctrl_options *opts,
>>>               }
>>>               opts->tos = token;
>>>               break;
>>> +        case NVMF_OPT_DHCHAP_SECRET:
>>> +            p = match_strdup(args);
>>> +            if (!p) {
>>> +                ret = -ENOMEM;
>>> +                goto out;
>>> +            }
>>> +            if (strncmp(p, "DHHC-1:00:", 10)) {
>>> +                pr_err("Invalid DH-CHAP secret %s\n", p);
>>> +                ret = -EINVAL;
>>> +                goto out;
>>> +            }
>>> +            kfree(opts->dhchap_secret);
>>> +            opts->dhchap_secret = p;
>>> +            break;
>>> +        case NVMF_OPT_DHCHAP_AUTH:
>>> +            opts->dhchap_auth = true;
>>> +            break;
>>> +        case NVMF_OPT_DHCHAP_GROUP:
>>> +            if (match_int(args, &token)) {
>>> +                ret = -EINVAL;
>>> +                goto out;
>>> +            }
>>> +            if (token <= 0) {
>>> +                pr_err("Invalid dhchap_group %d\n", token);
>>> +                ret = -EINVAL;
>>> +                goto out;
>>> +            }
>>> +            opts->dhchap_group = token;
>>> +            break;
>>>           default:
>>>               pr_warn("unknown parameter or missing value '%s' in 
>>> ctrl creation request\n",
>>>                   p);
>>> @@ -942,6 +997,7 @@ void nvmf_free_options(struct nvmf_ctrl_options 
>>> *opts)
>>>       kfree(opts->subsysnqn);
>>>       kfree(opts->host_traddr);
>>>       kfree(opts->host_iface);
>>> +    kfree(opts->dhchap_secret);
>>>       kfree(opts);
>>>   }
>>>   EXPORT_SYMBOL_GPL(nvmf_free_options);
>>> @@ -951,7 +1007,10 @@ EXPORT_SYMBOL_GPL(nvmf_free_options);
>>>                    NVMF_OPT_KATO | NVMF_OPT_HOSTNQN | \
>>>                    NVMF_OPT_HOST_ID | NVMF_OPT_DUP_CONNECT |\
>>>                    NVMF_OPT_DISABLE_SQFLOW |\
>>> -                 NVMF_OPT_FAIL_FAST_TMO)
>>> +                 NVMF_OPT_CTRL_LOSS_TMO |\
>>> +                 NVMF_OPT_FAIL_FAST_TMO |\
>>> +                 NVMF_OPT_DHCHAP_SECRET |\
>>> +                 NVMF_OPT_DHCHAP_AUTH | NVMF_OPT_DHCHAP_GROUP)
>>>   static struct nvme_ctrl *
>>>   nvmf_create_ctrl(struct device *dev, const char *buf)
>>> diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h
>>> index a146cb903869..535bc544f0f6 100644
>>> --- a/drivers/nvme/host/fabrics.h
>>> +++ b/drivers/nvme/host/fabrics.h
>>> @@ -67,6 +67,9 @@ enum {
>>>       NVMF_OPT_TOS        = 1 << 19,
>>>       NVMF_OPT_FAIL_FAST_TMO    = 1 << 20,
>>>       NVMF_OPT_HOST_IFACE    = 1 << 21,
>>> +    NVMF_OPT_DHCHAP_SECRET    = 1 << 22,
>>> +    NVMF_OPT_DHCHAP_AUTH    = 1 << 23,
>>> +    NVMF_OPT_DHCHAP_GROUP    = 1 << 24,
>>>   };
>>>   /**
>>> @@ -96,6 +99,8 @@ enum {
>>>    * @max_reconnects: maximum number of allowed reconnect attempts 
>>> before removing
>>>    *              the controller, (-1) means reconnect forever, zero 
>>> means remove
>>>    *              immediately;
>>> + * @dhchap_secret: DH-HMAC-CHAP secret
>>> + * @dhchap_auth: DH-HMAC-CHAP authenticate controller
>>>    * @disable_sqflow: disable controller sq flow control
>>>    * @hdr_digest: generate/verify header digest (TCP)
>>>    * @data_digest: generate/verify data digest (TCP)
>>> @@ -120,6 +125,9 @@ struct nvmf_ctrl_options {
>>>       unsigned int        kato;
>>>       struct nvmf_host    *host;
>>>       int            max_reconnects;
>>> +    char            *dhchap_secret;
>>> +    int            dhchap_group;
>>> +    bool            dhchap_auth;
>>>       bool            disable_sqflow;
>>>       bool            hdr_digest;
>>>       bool            data_digest;
>>> diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
>>> index 18ef8dd03a90..bcd5b8276c26 100644
>>> --- a/drivers/nvme/host/nvme.h
>>> +++ b/drivers/nvme/host/nvme.h
>>> @@ -328,6 +328,12 @@ struct nvme_ctrl {
>>>       struct work_struct ana_work;
>>>   #endif
>>> +#ifdef CONFIG_NVME_AUTH
>>> +    u16 transaction;
>>> +    u8 dhchap_hash;
>>> +    u8 dhchap_dhgroup;
>>
>> Do multiple controllers in the same subsystem have different
>> params? no, so I think these should be lifted to subsys.
>>
> 
> It doesn't actually say in the spec; it always refers to the params as 
> being set by the controller.
> So it could be either; maybe we should ask for clafication at the fmds 
> call.

We should, but I'd be surprised that different controllers in the same
subsystem can authenticate difrerently...

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 06/11] nvme: Implement In-Band authentication

[Stephan Müller]

Am Sonntag, 18. Juli 2021, 14:43:43 CEST schrieb Hannes Reinecke:

Hi Hannes,

> >> +	size += 2 * chap->hash_len;
> >> +	if (ctrl->opts->dhchap_auth) {
> >> +		get_random_bytes(chap->c2, chap->hash_len);
> > 
> > Why are you using CRYPTO_RNG_DEFAULT when you are using get_random_bytes
> > here?
> Errm ... do I?
> Seems that my crypto ignorance is showing here; 'get_random_bytes()' is
> the usual function we're using for drivers; if there is another way for
> crypto please enlighten me.

Apologies, I was looking at CONFIG_RNG where you set CRYPTO_RNG_DEFAULT. 
Please ignore my comment here.

Ciao
Stephan

Re: [PATCH 06/11] nvme: Implement In-Band authentication

[Stephan Müller]

Am Sonntag, 18. Juli 2021, 14:43:43 CEST schrieb Hannes Reinecke:

Hi Hannes,

> >> +	size += 2 * chap->hash_len;
> >> +	if (ctrl->opts->dhchap_auth) {
> >> +		get_random_bytes(chap->c2, chap->hash_len);
> > 
> > Why are you using CRYPTO_RNG_DEFAULT when you are using get_random_bytes
> > here?
> Errm ... do I?
> Seems that my crypto ignorance is showing here; 'get_random_bytes()' is
> the usual function we're using for drivers; if there is another way for
> crypto please enlighten me.

Apologies, I was looking at CONFIG_RNG where you set CRYPTO_RNG_DEFAULT. 
Please ignore my comment here.

Ciao
Stephan



_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 06/11] nvme: Implement In-Band authentication

[Hannes Reinecke]

On 7/17/21 6:49 PM, Stephan Müller wrote:
> Am Freitag, 16. Juli 2021, 13:04:23 CEST schrieb Hannes Reinecke:
> 
> Hi Hannes,
> 
>> Implement NVMe-oF In-Band authentication. This patch adds two new
>> fabric options 'dhchap_key' to specify the PSK and 'dhchap_authenticate'
>> to request bi-directional authentication of both the host and the
>> controller.
>>
>> Signed-off-by: Hannes Reinecke <hare@suse.de>
>> ---
>>   drivers/nvme/host/Kconfig   |  11 +
>>   drivers/nvme/host/Makefile  |   1 +
>>   drivers/nvme/host/auth.c    | 813 ++++++++++++++++++++++++++++++++++++
>>   drivers/nvme/host/auth.h    |  23 +
>>   drivers/nvme/host/core.c    |  77 +++-
>>   drivers/nvme/host/fabrics.c |  65 ++-
>>   drivers/nvme/host/fabrics.h |   8 +
>>   drivers/nvme/host/nvme.h    |  15 +
>>   drivers/nvme/host/trace.c   |  32 ++
>>   9 files changed, 1041 insertions(+), 4 deletions(-)
>>   create mode 100644 drivers/nvme/host/auth.c
>>   create mode 100644 drivers/nvme/host/auth.h
>>
>> diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig
>> index c3f3d77f1aac..853c546305e9 100644
>> --- a/drivers/nvme/host/Kconfig
>> +++ b/drivers/nvme/host/Kconfig
>> @@ -85,3 +85,14 @@ config NVME_TCP
>>   	  from https://github.com/linux-nvme/nvme-cli.
>>
>>   	  If unsure, say N.
>> +
>> +config NVME_AUTH
>> +	bool "NVM Express over Fabrics In-Band Authentication"
>> +	depends on NVME_TCP
>> +	select CRYPTO_SHA256
>> +	select CRYPTO_SHA512
> 
> What about adding CRYPTO_HMAC here?
> 

Yes, you are correct. Will be fixing it.

>> +	help
>> +	  This provides support for NVMe over Fabrics In-Band Authentication
>> +	  for the NVMe over TCP transport.
>> +
>> +	  If unsure, say N.
>> diff --git a/drivers/nvme/host/Makefile b/drivers/nvme/host/Makefile
>> index cbc509784b2e..03748a55a12b 100644
>> --- a/drivers/nvme/host/Makefile
>> +++ b/drivers/nvme/host/Makefile
>> @@ -20,6 +20,7 @@ nvme-core-$(CONFIG_NVME_HWMON)		+= hwmon.o
>>   nvme-y					+= pci.o
>>
>>   nvme-fabrics-y				+= fabrics.o
>> +nvme-fabrics-$(CONFIG_NVME_AUTH)	+= auth.o
>>
>>   nvme-rdma-y				+= rdma.o
>>
>> diff --git a/drivers/nvme/host/auth.c b/drivers/nvme/host/auth.c
>> new file mode 100644
>> index 000000000000..448a3adebea6
>> --- /dev/null
>> +++ b/drivers/nvme/host/auth.c
>> @@ -0,0 +1,813 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * Copyright (c) 2020 Hannes Reinecke, SUSE Linux
>> + */
>> +
>> +#include <linux/crc32.h>
>> +#include <linux/base64.h>
>> +#include <asm/unaligned.h>
>> +#include <crypto/hash.h>
>> +#include <crypto/kpp.h>
>> +#include "nvme.h"
>> +#include "fabrics.h"
>> +#include "auth.h"
>> +
>> +static u32 nvme_dhchap_seqnum;
>> +
>> +struct nvme_dhchap_context {
>> +	struct crypto_shash *shash_tfm;
>> +	unsigned char *key;
>> +	size_t key_len;
>> +	int qid;
>> +	u32 s1;
>> +	u32 s2;
>> +	u16 transaction;
>> +	u8 status;
>> +	u8 hash_id;
>> +	u8 hash_len;
>> +	u8 c1[64];
>> +	u8 c2[64];
>> +	u8 response[64];
>> +	u8 *ctrl_key;
>> +	int ctrl_key_len;
>> +	u8 *host_key;
>> +	int host_key_len;
>> +	u8 *sess_key;
>> +	int sess_key_len;
>> +};
>> +
>> +struct nvmet_dhchap_hash_map {
>> +	int id;
>> +	int hash_len;
>> +	const char hmac[15];
>> +	const char digest[15];
>> +} hash_map[] = {
>> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA256,
>> +	 .hash_len = 32,
>> +	 .hmac = "hmac(sha256)", .digest = "sha256" },
>> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA384,
>> +	 .hash_len = 48,
>> +	 .hmac = "hmac(sha384)", .digest = "sha384" },
>> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA512,
>> +	 .hash_len = 64,
>> +	 .hmac = "hmac(sha512)", .digest = "sha512" },
>> +};
>> +
>> +const char *nvme_auth_hmac_name(int hmac_id)
>> +{
>> +	int i;
>> +
>> +	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
>> +		if (hash_map[i].id == hmac_id)
>> +			return hash_map[i].hmac;
>> +	}
>> +	return NULL;
>> +}
>> +EXPORT_SYMBOL_GPL(nvme_auth_hmac_name);
>> +
>> +const char *nvme_auth_digest_name(int hmac_id)
>> +{
>> +	int i;
>> +
>> +	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
>> +		if (hash_map[i].id == hmac_id)
>> +			return hash_map[i].digest;
>> +	}
>> +	return NULL;
>> +}
>> +EXPORT_SYMBOL_GPL(nvme_auth_digest_name);
>> +
>> +int nvme_auth_hmac_len(int hmac_id)
>> +{
>> +	int i;
>> +
>> +	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
>> +		if (hash_map[i].id == hmac_id)
>> +			return hash_map[i].hash_len;
>> +	}
>> +	return -1;
>> +}
>> +EXPORT_SYMBOL_GPL(nvme_auth_hmac_len);
>> +
>> +int nvme_auth_hmac_id(const char *hmac_name)
>> +{
>> +	int i;
>> +
>> +	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
>> +		if (!strncmp(hash_map[i].hmac, hmac_name,
>> +			     strlen(hash_map[i].hmac)))
>> +			return hash_map[i].id;
>> +	}
>> +	return -1;
>> +}
>> +EXPORT_SYMBOL_GPL(nvme_auth_hmac_id);
>> +
>> +unsigned char *nvme_auth_extract_secret(unsigned char *dhchap_secret,
>> +					size_t *dhchap_key_len)
>> +{
>> +	unsigned char *dhchap_key;
>> +	u32 crc;
>> +	int key_len;
>> +	size_t allocated_len;
>> +
>> +	allocated_len = strlen(dhchap_secret) - 10;
> 
> Are you sure that the string is always at least 10 bytes long? If so, can you
> please add a comment to it?
> 
> Also, is it guaranteed that we have an ASCII string? Note, a secret sounds to
> be like a binary string which may contain \0 as an appropriate value.
> 

The string will always be in the transport encoding as specified in the 
NVMe Base specification v2.0. Any other string will be rejected by the 
ioctl interface.

>> +	dhchap_key = kzalloc(allocated_len, GFP_KERNEL);
> 
> What about aligning it to CRYPTO_MINALIGN_ATTR to save a memcpy in
> shash_final?
> 

Wasn't aware that I need to do that. Will be fixing it up.

>> +	if (!dhchap_key)
>> +		return ERR_PTR(-ENOMEM);
>> +
>> +	key_len = base64_decode(dhchap_secret + 10,
>> +				allocated_len, dhchap_key);
>> +	if (key_len != 36 && key_len != 52 &&
>> +	    key_len != 68) {
>> +		pr_debug("Invalid DH-HMAC-CHAP key len %d\n",
>> +			 key_len);
>> +		kfree(dhchap_key);
>> +		return ERR_PTR(-EINVAL);
>> +	}
>> +	pr_debug("DH-HMAC-CHAP Key: %*ph\n",
>> +		 (int)key_len, dhchap_key);
>> +
>> +	/* The last four bytes is the CRC in little-endian format */
>> +	key_len -= 4;
>> +	/*
>> +	 * The linux implementation doesn't do pre- and post-increments,
>> +	 * so we have to do it manually.
>> +	 */
>> +	crc = ~crc32(~0, dhchap_key, key_len);
>> +
>> +	if (get_unaligned_le32(dhchap_key + key_len) != crc) {
>> +		pr_debug("DH-HMAC-CHAP crc mismatch (key %08x, crc %08x)\n",
>> +		       get_unaligned_le32(dhchap_key + key_len), crc);
>> +		kfree(dhchap_key);
>> +		return ERR_PTR(-EKEYREJECTED);
>> +	}
>> +	*dhchap_key_len = key_len;
>> +	return dhchap_key;
>> +}
>> +EXPORT_SYMBOL_GPL(nvme_auth_extract_secret);
>> +
>> +static int nvme_auth_send(struct nvme_ctrl *ctrl, int qid,
>> +			  void *data, size_t tl)
>> +{
>> +	struct nvme_command cmd = {};
>> +	blk_mq_req_flags_t flags = qid == NVME_QID_ANY ?
>> +		0 : BLK_MQ_REQ_NOWAIT | BLK_MQ_REQ_RESERVED;
>> +	struct request_queue *q = qid == NVME_QID_ANY ?
>> +		ctrl->fabrics_q : ctrl->connect_q;
>> +	int ret;
>> +
>> +	cmd.auth_send.opcode = nvme_fabrics_command;
>> +	cmd.auth_send.fctype = nvme_fabrics_type_auth_send;
>> +	cmd.auth_send.secp = NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER;
>> +	cmd.auth_send.spsp0 = 0x01;
>> +	cmd.auth_send.spsp1 = 0x01;
>> +	cmd.auth_send.tl = tl;
>> +
>> +	ret = __nvme_submit_sync_cmd(q, &cmd, NULL, data, tl, 0, qid,
>> +				     0, flags);
>> +	if (ret)
>> +		dev_dbg(ctrl->device,
>> +			"%s: qid %d error %d\n", __func__, qid, ret);
>> +	return ret;
>> +}
>> +
>> +static int nvme_auth_receive(struct nvme_ctrl *ctrl, int qid,
>> +			     void *buf, size_t al,
>> +			     u16 transaction, u8 expected_msg )
>> +{
>> +	struct nvme_command cmd = {};
>> +	struct nvmf_auth_dhchap_failure_data *data = buf;
>> +	blk_mq_req_flags_t flags = qid == NVME_QID_ANY ?
>> +		0 : BLK_MQ_REQ_NOWAIT | BLK_MQ_REQ_RESERVED;
>> +	struct request_queue *q = qid == NVME_QID_ANY ?
>> +		ctrl->fabrics_q : ctrl->connect_q;
>> +	int ret;
>> +
>> +	cmd.auth_receive.opcode = nvme_fabrics_command;
>> +	cmd.auth_receive.fctype = nvme_fabrics_type_auth_receive;
>> +	cmd.auth_receive.secp = NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER;
>> +	cmd.auth_receive.spsp0 = 0x01;
>> +	cmd.auth_receive.spsp1 = 0x01;
>> +	cmd.auth_receive.al = al;
>> +
>> +	ret = __nvme_submit_sync_cmd(q, &cmd, NULL, buf, al, 0, qid,
>> +				     0, flags);
>> +	if (ret > 0) {
>> +		dev_dbg(ctrl->device, "%s: qid %d nvme status %x\n",
>> +			__func__, qid, ret);
>> +		ret = -EIO;
>> +	}
>> +	if (ret < 0) {
>> +		dev_dbg(ctrl->device, "%s: qid %d error %d\n",
>> +			__func__, qid, ret);
>> +		return ret;
>> +	}
>> +	dev_dbg(ctrl->device, "%s: qid %d auth_type %d auth_id %x\n",
>> +		__func__, qid, data->auth_type, data->auth_id);
>> +	if (data->auth_type == NVME_AUTH_COMMON_MESSAGES &&
>> +	    data->auth_id == NVME_AUTH_DHCHAP_MESSAGE_FAILURE1) {
>> +		return data->reason_code_explanation;
>> +	}
>> +	if (data->auth_type != NVME_AUTH_DHCHAP_MESSAGES ||
>> +	    data->auth_id != expected_msg) {
>> +		dev_warn(ctrl->device,
>> +			 "qid %d invalid message %02x/%02x\n",
>> +			 qid, data->auth_type, data->auth_id);
>> +		return NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
>> +	}
>> +	if (le16_to_cpu(data->t_id) != transaction) {
>> +		dev_warn(ctrl->device,
>> +			 "qid %d invalid transaction ID %d\n",
>> +			 qid, le16_to_cpu(data->t_id));
>> +		return NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +static int nvme_auth_dhchap_negotiate(struct nvme_ctrl *ctrl,
>> +				      struct nvme_dhchap_context *chap,
>> +				      void *buf, size_t buf_size)
>> +{
>> +	struct nvmf_auth_dhchap_negotiate_data *data = buf;
>> +	size_t size = sizeof(*data) + sizeof(union nvmf_auth_protocol);
>> +
>> +	if (buf_size < size)
>> +		return -EINVAL;
>> +
>> +	memset((u8 *)buf, 0, size);
>> +	data->auth_type = NVME_AUTH_COMMON_MESSAGES;
>> +	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE;
>> +	data->t_id = cpu_to_le16(chap->transaction);
>> +	data->sc_c = 0; /* No secure channel concatenation */
>> +	data->napd = 1;
>> +	data->auth_protocol[0].dhchap.authid = NVME_AUTH_DHCHAP_AUTH_ID;
>> +	data->auth_protocol[0].dhchap.halen = 3;
>> +	data->auth_protocol[0].dhchap.dhlen = 1;
>> +	data->auth_protocol[0].dhchap.idlist[0] = NVME_AUTH_DHCHAP_HASH_SHA256;
>> +	data->auth_protocol[0].dhchap.idlist[1] = NVME_AUTH_DHCHAP_HASH_SHA384;
>> +	data->auth_protocol[0].dhchap.idlist[2] = NVME_AUTH_DHCHAP_HASH_SHA512;
>> +	data->auth_protocol[0].dhchap.idlist[3] = NVME_AUTH_DHCHAP_DHGROUP_NULL;
>> +
>> +	return size;
>> +}
>> +
>> +static int nvme_auth_dhchap_challenge(struct nvme_ctrl *ctrl,
>> +				      struct nvme_dhchap_context *chap,
>> +				      void *buf, size_t buf_size)
>> +{
>> +	struct nvmf_auth_dhchap_challenge_data *data = buf;
>> +	size_t size = sizeof(*data) + data->hl + data->dhvlen;
>> +	const char *gid_name;
>> +
>> +	if (buf_size < size) {
>> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
>> +		return -ENOMSG;
>> +	}
>> +
>> +	if (data->hashid != NVME_AUTH_DHCHAP_HASH_SHA256 &&
>> +	    data->hashid != NVME_AUTH_DHCHAP_HASH_SHA384 &&
>> +	    data->hashid != NVME_AUTH_DHCHAP_HASH_SHA512) {
>> +		dev_warn(ctrl->device,
>> +			 "qid %d: DH-HMAC-CHAP: invalid HASH ID %d\n",
>> +			 chap->qid, data->hashid);
>> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
>> +		return -EPROTO;
>> +	}
>> +	switch (data->dhgid) {
>> +	case NVME_AUTH_DHCHAP_DHGROUP_NULL:
>> +		gid_name = "null";
>> +		break;
>> +	default:
>> +		gid_name = NULL;
>> +		break;
>> +	}
>> +	if (!gid_name) {
>> +		dev_warn(ctrl->device,
>> +			 "qid %d: DH-HMAC-CHAP: invalid DH group id %d\n",
>> +			 chap->qid, data->dhgid);
>> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
>> +		return -EPROTO;
>> +	}
>> +	if (data->dhgid != NVME_AUTH_DHCHAP_DHGROUP_NULL) {
>> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
>> +		return -EPROTO;
>> +	}
>> +	if (data->dhgid == NVME_AUTH_DHCHAP_DHGROUP_NULL && data->dhvlen != 0) {
>> +		dev_warn(ctrl->device,
>> +			 "qid %d: DH-HMAC-CHAP: invalid DH value for NULL DH\n",
>> +			chap->qid);
>> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
>> +		return -EPROTO;
>> +	}
>> +	dev_dbg(ctrl->device, "%s: qid %d requested hash id %d\n",
>> +		__func__, chap->qid, data->hashid);
>> +	if (nvme_auth_hmac_len(data->hashid) != data->hl) {
>> +		dev_warn(ctrl->device,
>> +			 "qid %d: DH-HMAC-CHAP: invalid hash length\n",
>> +			chap->qid);
>> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
>> +		return -EPROTO;
>> +	}
>> +	chap->hash_id = data->hashid;
>> +	chap->hash_len = data->hl;
>> +	chap->s1 = le32_to_cpu(data->seqnum);
>> +	memcpy(chap->c1, data->cval, chap->hash_len);
>> +
>> +	return 0;
>> +}
>> +
>> +static int nvme_auth_dhchap_reply(struct nvme_ctrl *ctrl,
>> +				  struct nvme_dhchap_context *chap,
>> +				  void *buf, size_t buf_size)
>> +{
>> +	struct nvmf_auth_dhchap_reply_data *data = buf;
>> +	size_t size = sizeof(*data);
>> +
>> +	size += 2 * chap->hash_len;
>> +	if (ctrl->opts->dhchap_auth) {
>> +		get_random_bytes(chap->c2, chap->hash_len);
> 
> Why are you using CRYPTO_RNG_DEFAULT when you are using get_random_bytes here?
> 

Errm ... do I?
Seems that my crypto ignorance is showing here; 'get_random_bytes()' is 
the usual function we're using for drivers; if there is another way for 
crypto please enlighten me.

>> +		chap->s2 = nvme_dhchap_seqnum++;
>> +	} else
>> +		memset(chap->c2, 0, chap->hash_len);
>> +
>> +	if (chap->host_key_len)
>> +		size += chap->host_key_len;
>> +
>> +	if (buf_size < size)
>> +		return -EINVAL;
>> +
>> +	memset(buf, 0, size);
>> +	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
>> +	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_REPLY;
>> +	data->t_id = cpu_to_le16(chap->transaction);
>> +	data->hl = chap->hash_len;
>> +	data->dhvlen = chap->host_key_len;
>> +	data->seqnum = cpu_to_le32(chap->s2);
>> +	memcpy(data->rval, chap->response, chap->hash_len);
>> +	if (ctrl->opts->dhchap_auth) {
>> +		dev_dbg(ctrl->device, "%s: qid %d ctrl challenge %*ph\n",
>> +			__func__, chap->qid,
>> +			chap->hash_len, chap->c2);
>> +		data->cvalid = 1;
>> +		memcpy(data->rval + chap->hash_len, chap->c2,
>> +		       chap->hash_len);
>> +	}
>> +	if (chap->host_key_len)
>> +		memcpy(data->rval + 2 * chap->hash_len, chap->host_key,
>> +		       chap->host_key_len);
>> +
>> +	return size;
>> +}
>> +
>> +static int nvme_auth_dhchap_success1(struct nvme_ctrl *ctrl,
>> +				     struct nvme_dhchap_context *chap,
>> +				     void *buf, size_t buf_size)
>> +{
>> +	struct nvmf_auth_dhchap_success1_data *data = buf;
>> +	size_t size = sizeof(*data);
>> +
>> +	if (ctrl->opts->dhchap_auth)
>> +		size += chap->hash_len;
>> +
>> +
>> +	if (buf_size < size) {
>> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
>> +		return -ENOMSG;
>> +	}
>> +
>> +	if (data->hl != chap->hash_len) {
>> +		dev_warn(ctrl->device,
>> +			 "qid %d: DH-HMAC-CHAP: invalid hash length %d\n",
>> +			 chap->qid, data->hl);
>> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
>> +		return -EPROTO;
>> +	}
>> +
>> +	if (!data->rvalid)
>> +		return 0;
>> +
>> +	/* Validate controller response */
>> +	if (memcmp(chap->response, data->rval, data->hl)) {
>> +		dev_dbg(ctrl->device, "%s: qid %d ctrl response %*ph\n",
>> +			__func__, chap->qid, chap->hash_len, data->rval);
>> +		dev_dbg(ctrl->device, "%s: qid %d host response %*ph\n",
>> +			__func__, chap->qid, chap->hash_len, chap->response);
>> +		dev_warn(ctrl->device,
>> +			 "qid %d: DH-HMAC-CHAP: controller authentication failed\n",
>> +			 chap->qid);
>> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
>> +		return -EPROTO;
>> +	}
>> +	dev_info(ctrl->device,
>> +		 "qid %d: DH-HMAC-CHAP: controller authenticated\n",
>> +		chap->qid);
>> +	return 0;
>> +}
>> +
>> +static int nvme_auth_dhchap_success2(struct nvme_ctrl *ctrl,
>> +				     struct nvme_dhchap_context *chap,
>> +				     void *buf, size_t buf_size)
>> +{
>> +	struct nvmf_auth_dhchap_success2_data *data = buf;
>> +	size_t size = sizeof(*data);
>> +
>> +	memset(buf, 0, size);
>> +	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
>> +	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_SUCCESS2;
>> +	data->t_id = cpu_to_le16(chap->transaction);
>> +
>> +	return size;
>> +}
>> +
>> +static int nvme_auth_dhchap_failure2(struct nvme_ctrl *ctrl,
>> +				     struct nvme_dhchap_context *chap,
>> +				     void *buf, size_t buf_size)
>> +{
>> +	struct nvmf_auth_dhchap_failure_data *data = buf;
>> +	size_t size = sizeof(*data);
>> +
>> +	memset(buf, 0, size);
>> +	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
>> +	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_FAILURE2;
>> +	data->t_id = cpu_to_le16(chap->transaction);
>> +	data->reason_code = 1;
>> +	data->reason_code_explanation = chap->status;
>> +
>> +	return size;
>> +}
>> +
>> +int nvme_auth_select_hash(struct nvme_ctrl *ctrl,
>> +			  struct nvme_dhchap_context *chap)
>> +{
>> +	char *hash_name;
>> +	int ret;
>> +
>> +	switch (chap->hash_id) {
>> +	case NVME_AUTH_DHCHAP_HASH_SHA256:
>> +		hash_name = "hmac(sha256)";
>> +		break;
>> +	case NVME_AUTH_DHCHAP_HASH_SHA384:
>> +		hash_name = "hmac(sha384)";
>> +		break;
>> +	case NVME_AUTH_DHCHAP_HASH_SHA512:
>> +		hash_name = "hmac(sha512)";
>> +		break;
>> +	default:
>> +		hash_name = NULL;
>> +		break;
>> +	}
>> +	if (!hash_name) {
>> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
>> +		return -EPROTO;
>> +	}
>> +	chap->shash_tfm = crypto_alloc_shash(hash_name, 0,
>> +					     CRYPTO_ALG_ALLOCATES_MEMORY);
>> +	if (IS_ERR(chap->shash_tfm)) {
>> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
>> +		chap->shash_tfm = NULL;
>> +		return -EPROTO;
>> +	}
>> +	if (!chap->key) {
>> +		dev_warn(ctrl->device, "qid %d: cannot select hash, no key\n",
>> +			 chap->qid);
>> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
>> +		crypto_free_shash(chap->shash_tfm);
>> +		chap->shash_tfm = NULL;
>> +		return -EINVAL;
>> +	}
>> +	ret = crypto_shash_setkey(chap->shash_tfm, chap->key, chap->key_len);
>> +	if (ret) {
>> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
>> +		crypto_free_shash(chap->shash_tfm);
>> +		chap->shash_tfm = NULL;
>> +		return ret;
>> +	}
>> +	dev_info(ctrl->device, "qid %d: DH-HMAC_CHAP: selected hash %s\n",
>> +		 chap->qid, hash_name);
>> +	return 0;
>> +}
>> +
>> +static int nvme_auth_dhchap_host_response(struct nvme_ctrl *ctrl,
>> +					  struct nvme_dhchap_context *chap)
>> +{
>> +	SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
>> +	u8 buf[4], *challenge = chap->c1;
>> +	int ret;
>> +
>> +	dev_dbg(ctrl->device, "%s: qid %d host response seq %d transaction
> %d\n",
>> +		__func__, chap->qid, chap->s1, chap->transaction);
>> +	shash->tfm = chap->shash_tfm;
>> +	ret = crypto_shash_init(shash);
>> +	if (ret)
>> +		goto out;
>> +	ret = crypto_shash_update(shash, challenge, chap->hash_len);
>> +	if (ret)
>> +		goto out;
>> +	put_unaligned_le32(chap->s1, buf);
>> +	ret = crypto_shash_update(shash, buf, 4);
>> +	if (ret)
>> +		goto out;
>> +	put_unaligned_le16(chap->transaction, buf);
>> +	ret = crypto_shash_update(shash, buf, 2);
>> +	if (ret)
>> +		goto out;
>> +	memset(buf, 0, sizeof(buf));
>> +	ret = crypto_shash_update(shash, buf, 1);
>> +	if (ret)
>> +		goto out;
>> +	ret = crypto_shash_update(shash, "HostHost", 8);
>> +	if (ret)
>> +		goto out;
>> +	ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
>> +				  strlen(ctrl->opts->host->nqn));
>> +	if (ret)
>> +		goto out;
>> +	ret = crypto_shash_update(shash, buf, 1);
>> +	if (ret)
>> +		goto out;
>> +	ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
>> +			    strlen(ctrl->opts->subsysnqn));
>> +	if (ret)
>> +		goto out;
>> +	ret = crypto_shash_final(shash, chap->response);
>> +out:
>> +	return ret;
>> +}
>> +
>> +static int nvme_auth_dhchap_ctrl_response(struct nvme_ctrl *ctrl,
>> +					  struct nvme_dhchap_context *chap)
>> +{
>> +	SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
>> +	u8 buf[4], *challenge = chap->c2;
>> +	int ret;
>> +
>> +	dev_dbg(ctrl->device, "%s: qid %d host response seq %d transaction
> %d\n",
>> +		__func__, chap->qid, chap->s2, chap->transaction);
>> +	dev_dbg(ctrl->device, "%s: qid %d challenge %*ph\n",
>> +		__func__, chap->qid, chap->hash_len, challenge);
>> +	dev_dbg(ctrl->device, "%s: qid %d subsysnqn %s\n",
>> +		__func__, chap->qid, ctrl->opts->subsysnqn);
>> +	dev_dbg(ctrl->device, "%s: qid %d hostnqn %s\n",
>> +		__func__, chap->qid, ctrl->opts->host->nqn);
>> +	shash->tfm = chap->shash_tfm;
>> +	ret = crypto_shash_init(shash);
>> +	if (ret)
>> +		goto out;
>> +	ret = crypto_shash_update(shash, challenge, chap->hash_len);
>> +	if (ret)
>> +		goto out;
>> +	put_unaligned_le32(chap->s2, buf);
>> +	ret = crypto_shash_update(shash, buf, 4);
>> +	if (ret)
>> +		goto out;
>> +	put_unaligned_le16(chap->transaction, buf);
>> +	ret = crypto_shash_update(shash, buf, 2);
>> +	if (ret)
>> +		goto out;
>> +	memset(buf, 0, 4);
>> +	ret = crypto_shash_update(shash, buf, 1);
>> +	if (ret)
>> +		goto out;
>> +	ret = crypto_shash_update(shash, "Controller", 10);
>> +	if (ret)
>> +		goto out;
>> +	ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
>> +				  strlen(ctrl->opts->subsysnqn));
>> +	if (ret)
>> +		goto out;
>> +	ret = crypto_shash_update(shash, buf, 1);
>> +	if (ret)
>> +		goto out;
>> +	ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
>> +				  strlen(ctrl->opts->host->nqn));
>> +	if (ret)
>> +		goto out;
>> +	ret = crypto_shash_final(shash, chap->response);
>> +out:
>> +	return ret;
>> +}
>> +
>> +int nvme_auth_generate_key(struct nvme_ctrl *ctrl,
>> +			   struct nvme_dhchap_context *chap)
>> +{
>> +	int ret;
>> +	u8 key_hash;
>> +	const char *hmac_name;
>> +	struct crypto_shash *key_tfm;
>> +
>> +	if (sscanf(ctrl->opts->dhchap_secret, "DHHC-1:%hhd:%*s:",
>> +		   &key_hash) != 1)
>> +		return -EINVAL;
>> +
>> +	chap->key = nvme_auth_extract_secret(ctrl->opts->dhchap_secret,
>> +					     &chap->key_len);
>> +	if (IS_ERR(chap->key)) {
>> +		ret = PTR_ERR(chap->key);
>> +		chap->key = NULL;
>> +		return ret;
>> +	}
>> +
>> +	if (key_hash == 0)
>> +		return 0;
>> +
>> +	hmac_name = nvme_auth_hmac_name(key_hash);
>> +	if (!hmac_name) {
>> +		pr_debug("Invalid key hash id %d\n", key_hash);
>> +		return -EKEYREJECTED;
>> +	}
>> +
>> +	key_tfm = crypto_alloc_shash(hmac_name, 0, 0);
>> +	if (IS_ERR(key_tfm)) {
>> +		kfree(chap->key);
>> +		chap->key = NULL;
>> +		ret = PTR_ERR(key_tfm);
>> +	} else {
>> +		SHASH_DESC_ON_STACK(shash, key_tfm);
>> +
>> +		shash->tfm = key_tfm;
>> +		ret = crypto_shash_setkey(key_tfm, chap->key,
>> +					  chap->key_len);
>> +		if (ret < 0) {
>> +			crypto_free_shash(key_tfm);
>> +			kfree(chap->key);
>> +			chap->key = NULL;
>> +			return ret;
>> +		}
>> +		crypto_shash_init(shash);
>> +		crypto_shash_update(shash, ctrl->opts->host->nqn,
>> +				    strlen(ctrl->opts->host->nqn));
>> +		crypto_shash_update(shash, "NVMe-over-Fabrics", 17);
>> +		crypto_shash_final(shash, chap->key);
>> +		crypto_free_shash(key_tfm);
>> +	}
>> +	return 0;
>> +}
>> +
>> +void nvme_auth_free(struct nvme_dhchap_context *chap)
>> +{
>> +	if (chap->shash_tfm)
>> +		crypto_free_shash(chap->shash_tfm);
>> +	if (chap->key)
>> +		kfree(chap->key);
>> +	if (chap->ctrl_key)
>> +		kfree(chap->ctrl_key);
>> +	if (chap->host_key)
>> +		kfree(chap->host_key);
>> +	if (chap->sess_key)
>> +		kfree(chap->sess_key);
>> +	kfree(chap);
> 
> kfree_sensitive in all cases as all buffers have sensitive data?
> 

Yes, will be fixing it up.

>> +}
>> +
>> +int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid)
>> +{
>> +	struct nvme_dhchap_context *chap;
>> +	void *buf;
>> +	size_t buf_size, tl;
>> +	int ret = 0;
>> +
>> +	chap = kzalloc(sizeof(*chap), GFP_KERNEL);
> 
> Suggestion: make sure that chap->response is aligned to CRYPTO_MINALIGN_ATTR -
> then you would save a memcpy in crypto_shash_final
> 

Ok.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer

Re: [PATCH 06/11] nvme: Implement In-Band authentication

[Hannes Reinecke]

On 7/17/21 6:49 PM, Stephan Müller wrote:
> Am Freitag, 16. Juli 2021, 13:04:23 CEST schrieb Hannes Reinecke:
> 
> Hi Hannes,
> 
>> Implement NVMe-oF In-Band authentication. This patch adds two new
>> fabric options 'dhchap_key' to specify the PSK and 'dhchap_authenticate'
>> to request bi-directional authentication of both the host and the
>> controller.
>>
>> Signed-off-by: Hannes Reinecke <hare@suse.de>
>> ---
>>   drivers/nvme/host/Kconfig   |  11 +
>>   drivers/nvme/host/Makefile  |   1 +
>>   drivers/nvme/host/auth.c    | 813 ++++++++++++++++++++++++++++++++++++
>>   drivers/nvme/host/auth.h    |  23 +
>>   drivers/nvme/host/core.c    |  77 +++-
>>   drivers/nvme/host/fabrics.c |  65 ++-
>>   drivers/nvme/host/fabrics.h |   8 +
>>   drivers/nvme/host/nvme.h    |  15 +
>>   drivers/nvme/host/trace.c   |  32 ++
>>   9 files changed, 1041 insertions(+), 4 deletions(-)
>>   create mode 100644 drivers/nvme/host/auth.c
>>   create mode 100644 drivers/nvme/host/auth.h
>>
>> diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig
>> index c3f3d77f1aac..853c546305e9 100644
>> --- a/drivers/nvme/host/Kconfig
>> +++ b/drivers/nvme/host/Kconfig
>> @@ -85,3 +85,14 @@ config NVME_TCP
>>   	  from https://github.com/linux-nvme/nvme-cli.
>>
>>   	  If unsure, say N.
>> +
>> +config NVME_AUTH
>> +	bool "NVM Express over Fabrics In-Band Authentication"
>> +	depends on NVME_TCP
>> +	select CRYPTO_SHA256
>> +	select CRYPTO_SHA512
> 
> What about adding CRYPTO_HMAC here?
> 

Yes, you are correct. Will be fixing it.

>> +	help
>> +	  This provides support for NVMe over Fabrics In-Band Authentication
>> +	  for the NVMe over TCP transport.
>> +
>> +	  If unsure, say N.
>> diff --git a/drivers/nvme/host/Makefile b/drivers/nvme/host/Makefile
>> index cbc509784b2e..03748a55a12b 100644
>> --- a/drivers/nvme/host/Makefile
>> +++ b/drivers/nvme/host/Makefile
>> @@ -20,6 +20,7 @@ nvme-core-$(CONFIG_NVME_HWMON)		+= hwmon.o
>>   nvme-y					+= pci.o
>>
>>   nvme-fabrics-y				+= fabrics.o
>> +nvme-fabrics-$(CONFIG_NVME_AUTH)	+= auth.o
>>
>>   nvme-rdma-y				+= rdma.o
>>
>> diff --git a/drivers/nvme/host/auth.c b/drivers/nvme/host/auth.c
>> new file mode 100644
>> index 000000000000..448a3adebea6
>> --- /dev/null
>> +++ b/drivers/nvme/host/auth.c
>> @@ -0,0 +1,813 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * Copyright (c) 2020 Hannes Reinecke, SUSE Linux
>> + */
>> +
>> +#include <linux/crc32.h>
>> +#include <linux/base64.h>
>> +#include <asm/unaligned.h>
>> +#include <crypto/hash.h>
>> +#include <crypto/kpp.h>
>> +#include "nvme.h"
>> +#include "fabrics.h"
>> +#include "auth.h"
>> +
>> +static u32 nvme_dhchap_seqnum;
>> +
>> +struct nvme_dhchap_context {
>> +	struct crypto_shash *shash_tfm;
>> +	unsigned char *key;
>> +	size_t key_len;
>> +	int qid;
>> +	u32 s1;
>> +	u32 s2;
>> +	u16 transaction;
>> +	u8 status;
>> +	u8 hash_id;
>> +	u8 hash_len;
>> +	u8 c1[64];
>> +	u8 c2[64];
>> +	u8 response[64];
>> +	u8 *ctrl_key;
>> +	int ctrl_key_len;
>> +	u8 *host_key;
>> +	int host_key_len;
>> +	u8 *sess_key;
>> +	int sess_key_len;
>> +};
>> +
>> +struct nvmet_dhchap_hash_map {
>> +	int id;
>> +	int hash_len;
>> +	const char hmac[15];
>> +	const char digest[15];
>> +} hash_map[] = {
>> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA256,
>> +	 .hash_len = 32,
>> +	 .hmac = "hmac(sha256)", .digest = "sha256" },
>> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA384,
>> +	 .hash_len = 48,
>> +	 .hmac = "hmac(sha384)", .digest = "sha384" },
>> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA512,
>> +	 .hash_len = 64,
>> +	 .hmac = "hmac(sha512)", .digest = "sha512" },
>> +};
>> +
>> +const char *nvme_auth_hmac_name(int hmac_id)
>> +{
>> +	int i;
>> +
>> +	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
>> +		if (hash_map[i].id == hmac_id)
>> +			return hash_map[i].hmac;
>> +	}
>> +	return NULL;
>> +}
>> +EXPORT_SYMBOL_GPL(nvme_auth_hmac_name);
>> +
>> +const char *nvme_auth_digest_name(int hmac_id)
>> +{
>> +	int i;
>> +
>> +	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
>> +		if (hash_map[i].id == hmac_id)
>> +			return hash_map[i].digest;
>> +	}
>> +	return NULL;
>> +}
>> +EXPORT_SYMBOL_GPL(nvme_auth_digest_name);
>> +
>> +int nvme_auth_hmac_len(int hmac_id)
>> +{
>> +	int i;
>> +
>> +	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
>> +		if (hash_map[i].id == hmac_id)
>> +			return hash_map[i].hash_len;
>> +	}
>> +	return -1;
>> +}
>> +EXPORT_SYMBOL_GPL(nvme_auth_hmac_len);
>> +
>> +int nvme_auth_hmac_id(const char *hmac_name)
>> +{
>> +	int i;
>> +
>> +	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
>> +		if (!strncmp(hash_map[i].hmac, hmac_name,
>> +			     strlen(hash_map[i].hmac)))
>> +			return hash_map[i].id;
>> +	}
>> +	return -1;
>> +}
>> +EXPORT_SYMBOL_GPL(nvme_auth_hmac_id);
>> +
>> +unsigned char *nvme_auth_extract_secret(unsigned char *dhchap_secret,
>> +					size_t *dhchap_key_len)
>> +{
>> +	unsigned char *dhchap_key;
>> +	u32 crc;
>> +	int key_len;
>> +	size_t allocated_len;
>> +
>> +	allocated_len = strlen(dhchap_secret) - 10;
> 
> Are you sure that the string is always at least 10 bytes long? If so, can you
> please add a comment to it?
> 
> Also, is it guaranteed that we have an ASCII string? Note, a secret sounds to
> be like a binary string which may contain \0 as an appropriate value.
> 

The string will always be in the transport encoding as specified in the 
NVMe Base specification v2.0. Any other string will be rejected by the 
ioctl interface.

>> +	dhchap_key = kzalloc(allocated_len, GFP_KERNEL);
> 
> What about aligning it to CRYPTO_MINALIGN_ATTR to save a memcpy in
> shash_final?
> 

Wasn't aware that I need to do that. Will be fixing it up.

>> +	if (!dhchap_key)
>> +		return ERR_PTR(-ENOMEM);
>> +
>> +	key_len = base64_decode(dhchap_secret + 10,
>> +				allocated_len, dhchap_key);
>> +	if (key_len != 36 && key_len != 52 &&
>> +	    key_len != 68) {
>> +		pr_debug("Invalid DH-HMAC-CHAP key len %d\n",
>> +			 key_len);
>> +		kfree(dhchap_key);
>> +		return ERR_PTR(-EINVAL);
>> +	}
>> +	pr_debug("DH-HMAC-CHAP Key: %*ph\n",
>> +		 (int)key_len, dhchap_key);
>> +
>> +	/* The last four bytes is the CRC in little-endian format */
>> +	key_len -= 4;
>> +	/*
>> +	 * The linux implementation doesn't do pre- and post-increments,
>> +	 * so we have to do it manually.
>> +	 */
>> +	crc = ~crc32(~0, dhchap_key, key_len);
>> +
>> +	if (get_unaligned_le32(dhchap_key + key_len) != crc) {
>> +		pr_debug("DH-HMAC-CHAP crc mismatch (key %08x, crc %08x)\n",
>> +		       get_unaligned_le32(dhchap_key + key_len), crc);
>> +		kfree(dhchap_key);
>> +		return ERR_PTR(-EKEYREJECTED);
>> +	}
>> +	*dhchap_key_len = key_len;
>> +	return dhchap_key;
>> +}
>> +EXPORT_SYMBOL_GPL(nvme_auth_extract_secret);
>> +
>> +static int nvme_auth_send(struct nvme_ctrl *ctrl, int qid,
>> +			  void *data, size_t tl)
>> +{
>> +	struct nvme_command cmd = {};
>> +	blk_mq_req_flags_t flags = qid == NVME_QID_ANY ?
>> +		0 : BLK_MQ_REQ_NOWAIT | BLK_MQ_REQ_RESERVED;
>> +	struct request_queue *q = qid == NVME_QID_ANY ?
>> +		ctrl->fabrics_q : ctrl->connect_q;
>> +	int ret;
>> +
>> +	cmd.auth_send.opcode = nvme_fabrics_command;
>> +	cmd.auth_send.fctype = nvme_fabrics_type_auth_send;
>> +	cmd.auth_send.secp = NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER;
>> +	cmd.auth_send.spsp0 = 0x01;
>> +	cmd.auth_send.spsp1 = 0x01;
>> +	cmd.auth_send.tl = tl;
>> +
>> +	ret = __nvme_submit_sync_cmd(q, &cmd, NULL, data, tl, 0, qid,
>> +				     0, flags);
>> +	if (ret)
>> +		dev_dbg(ctrl->device,
>> +			"%s: qid %d error %d\n", __func__, qid, ret);
>> +	return ret;
>> +}
>> +
>> +static int nvme_auth_receive(struct nvme_ctrl *ctrl, int qid,
>> +			     void *buf, size_t al,
>> +			     u16 transaction, u8 expected_msg )
>> +{
>> +	struct nvme_command cmd = {};
>> +	struct nvmf_auth_dhchap_failure_data *data = buf;
>> +	blk_mq_req_flags_t flags = qid == NVME_QID_ANY ?
>> +		0 : BLK_MQ_REQ_NOWAIT | BLK_MQ_REQ_RESERVED;
>> +	struct request_queue *q = qid == NVME_QID_ANY ?
>> +		ctrl->fabrics_q : ctrl->connect_q;
>> +	int ret;
>> +
>> +	cmd.auth_receive.opcode = nvme_fabrics_command;
>> +	cmd.auth_receive.fctype = nvme_fabrics_type_auth_receive;
>> +	cmd.auth_receive.secp = NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER;
>> +	cmd.auth_receive.spsp0 = 0x01;
>> +	cmd.auth_receive.spsp1 = 0x01;
>> +	cmd.auth_receive.al = al;
>> +
>> +	ret = __nvme_submit_sync_cmd(q, &cmd, NULL, buf, al, 0, qid,
>> +				     0, flags);
>> +	if (ret > 0) {
>> +		dev_dbg(ctrl->device, "%s: qid %d nvme status %x\n",
>> +			__func__, qid, ret);
>> +		ret = -EIO;
>> +	}
>> +	if (ret < 0) {
>> +		dev_dbg(ctrl->device, "%s: qid %d error %d\n",
>> +			__func__, qid, ret);
>> +		return ret;
>> +	}
>> +	dev_dbg(ctrl->device, "%s: qid %d auth_type %d auth_id %x\n",
>> +		__func__, qid, data->auth_type, data->auth_id);
>> +	if (data->auth_type == NVME_AUTH_COMMON_MESSAGES &&
>> +	    data->auth_id == NVME_AUTH_DHCHAP_MESSAGE_FAILURE1) {
>> +		return data->reason_code_explanation;
>> +	}
>> +	if (data->auth_type != NVME_AUTH_DHCHAP_MESSAGES ||
>> +	    data->auth_id != expected_msg) {
>> +		dev_warn(ctrl->device,
>> +			 "qid %d invalid message %02x/%02x\n",
>> +			 qid, data->auth_type, data->auth_id);
>> +		return NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
>> +	}
>> +	if (le16_to_cpu(data->t_id) != transaction) {
>> +		dev_warn(ctrl->device,
>> +			 "qid %d invalid transaction ID %d\n",
>> +			 qid, le16_to_cpu(data->t_id));
>> +		return NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +static int nvme_auth_dhchap_negotiate(struct nvme_ctrl *ctrl,
>> +				      struct nvme_dhchap_context *chap,
>> +				      void *buf, size_t buf_size)
>> +{
>> +	struct nvmf_auth_dhchap_negotiate_data *data = buf;
>> +	size_t size = sizeof(*data) + sizeof(union nvmf_auth_protocol);
>> +
>> +	if (buf_size < size)
>> +		return -EINVAL;
>> +
>> +	memset((u8 *)buf, 0, size);
>> +	data->auth_type = NVME_AUTH_COMMON_MESSAGES;
>> +	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE;
>> +	data->t_id = cpu_to_le16(chap->transaction);
>> +	data->sc_c = 0; /* No secure channel concatenation */
>> +	data->napd = 1;
>> +	data->auth_protocol[0].dhchap.authid = NVME_AUTH_DHCHAP_AUTH_ID;
>> +	data->auth_protocol[0].dhchap.halen = 3;
>> +	data->auth_protocol[0].dhchap.dhlen = 1;
>> +	data->auth_protocol[0].dhchap.idlist[0] = NVME_AUTH_DHCHAP_HASH_SHA256;
>> +	data->auth_protocol[0].dhchap.idlist[1] = NVME_AUTH_DHCHAP_HASH_SHA384;
>> +	data->auth_protocol[0].dhchap.idlist[2] = NVME_AUTH_DHCHAP_HASH_SHA512;
>> +	data->auth_protocol[0].dhchap.idlist[3] = NVME_AUTH_DHCHAP_DHGROUP_NULL;
>> +
>> +	return size;
>> +}
>> +
>> +static int nvme_auth_dhchap_challenge(struct nvme_ctrl *ctrl,
>> +				      struct nvme_dhchap_context *chap,
>> +				      void *buf, size_t buf_size)
>> +{
>> +	struct nvmf_auth_dhchap_challenge_data *data = buf;
>> +	size_t size = sizeof(*data) + data->hl + data->dhvlen;
>> +	const char *gid_name;
>> +
>> +	if (buf_size < size) {
>> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
>> +		return -ENOMSG;
>> +	}
>> +
>> +	if (data->hashid != NVME_AUTH_DHCHAP_HASH_SHA256 &&
>> +	    data->hashid != NVME_AUTH_DHCHAP_HASH_SHA384 &&
>> +	    data->hashid != NVME_AUTH_DHCHAP_HASH_SHA512) {
>> +		dev_warn(ctrl->device,
>> +			 "qid %d: DH-HMAC-CHAP: invalid HASH ID %d\n",
>> +			 chap->qid, data->hashid);
>> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
>> +		return -EPROTO;
>> +	}
>> +	switch (data->dhgid) {
>> +	case NVME_AUTH_DHCHAP_DHGROUP_NULL:
>> +		gid_name = "null";
>> +		break;
>> +	default:
>> +		gid_name = NULL;
>> +		break;
>> +	}
>> +	if (!gid_name) {
>> +		dev_warn(ctrl->device,
>> +			 "qid %d: DH-HMAC-CHAP: invalid DH group id %d\n",
>> +			 chap->qid, data->dhgid);
>> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
>> +		return -EPROTO;
>> +	}
>> +	if (data->dhgid != NVME_AUTH_DHCHAP_DHGROUP_NULL) {
>> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
>> +		return -EPROTO;
>> +	}
>> +	if (data->dhgid == NVME_AUTH_DHCHAP_DHGROUP_NULL && data->dhvlen != 0) {
>> +		dev_warn(ctrl->device,
>> +			 "qid %d: DH-HMAC-CHAP: invalid DH value for NULL DH\n",
>> +			chap->qid);
>> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
>> +		return -EPROTO;
>> +	}
>> +	dev_dbg(ctrl->device, "%s: qid %d requested hash id %d\n",
>> +		__func__, chap->qid, data->hashid);
>> +	if (nvme_auth_hmac_len(data->hashid) != data->hl) {
>> +		dev_warn(ctrl->device,
>> +			 "qid %d: DH-HMAC-CHAP: invalid hash length\n",
>> +			chap->qid);
>> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
>> +		return -EPROTO;
>> +	}
>> +	chap->hash_id = data->hashid;
>> +	chap->hash_len = data->hl;
>> +	chap->s1 = le32_to_cpu(data->seqnum);
>> +	memcpy(chap->c1, data->cval, chap->hash_len);
>> +
>> +	return 0;
>> +}
>> +
>> +static int nvme_auth_dhchap_reply(struct nvme_ctrl *ctrl,
>> +				  struct nvme_dhchap_context *chap,
>> +				  void *buf, size_t buf_size)
>> +{
>> +	struct nvmf_auth_dhchap_reply_data *data = buf;
>> +	size_t size = sizeof(*data);
>> +
>> +	size += 2 * chap->hash_len;
>> +	if (ctrl->opts->dhchap_auth) {
>> +		get_random_bytes(chap->c2, chap->hash_len);
> 
> Why are you using CRYPTO_RNG_DEFAULT when you are using get_random_bytes here?
> 

Errm ... do I?
Seems that my crypto ignorance is showing here; 'get_random_bytes()' is 
the usual function we're using for drivers; if there is another way for 
crypto please enlighten me.

>> +		chap->s2 = nvme_dhchap_seqnum++;
>> +	} else
>> +		memset(chap->c2, 0, chap->hash_len);
>> +
>> +	if (chap->host_key_len)
>> +		size += chap->host_key_len;
>> +
>> +	if (buf_size < size)
>> +		return -EINVAL;
>> +
>> +	memset(buf, 0, size);
>> +	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
>> +	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_REPLY;
>> +	data->t_id = cpu_to_le16(chap->transaction);
>> +	data->hl = chap->hash_len;
>> +	data->dhvlen = chap->host_key_len;
>> +	data->seqnum = cpu_to_le32(chap->s2);
>> +	memcpy(data->rval, chap->response, chap->hash_len);
>> +	if (ctrl->opts->dhchap_auth) {
>> +		dev_dbg(ctrl->device, "%s: qid %d ctrl challenge %*ph\n",
>> +			__func__, chap->qid,
>> +			chap->hash_len, chap->c2);
>> +		data->cvalid = 1;
>> +		memcpy(data->rval + chap->hash_len, chap->c2,
>> +		       chap->hash_len);
>> +	}
>> +	if (chap->host_key_len)
>> +		memcpy(data->rval + 2 * chap->hash_len, chap->host_key,
>> +		       chap->host_key_len);
>> +
>> +	return size;
>> +}
>> +
>> +static int nvme_auth_dhchap_success1(struct nvme_ctrl *ctrl,
>> +				     struct nvme_dhchap_context *chap,
>> +				     void *buf, size_t buf_size)
>> +{
>> +	struct nvmf_auth_dhchap_success1_data *data = buf;
>> +	size_t size = sizeof(*data);
>> +
>> +	if (ctrl->opts->dhchap_auth)
>> +		size += chap->hash_len;
>> +
>> +
>> +	if (buf_size < size) {
>> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
>> +		return -ENOMSG;
>> +	}
>> +
>> +	if (data->hl != chap->hash_len) {
>> +		dev_warn(ctrl->device,
>> +			 "qid %d: DH-HMAC-CHAP: invalid hash length %d\n",
>> +			 chap->qid, data->hl);
>> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
>> +		return -EPROTO;
>> +	}
>> +
>> +	if (!data->rvalid)
>> +		return 0;
>> +
>> +	/* Validate controller response */
>> +	if (memcmp(chap->response, data->rval, data->hl)) {
>> +		dev_dbg(ctrl->device, "%s: qid %d ctrl response %*ph\n",
>> +			__func__, chap->qid, chap->hash_len, data->rval);
>> +		dev_dbg(ctrl->device, "%s: qid %d host response %*ph\n",
>> +			__func__, chap->qid, chap->hash_len, chap->response);
>> +		dev_warn(ctrl->device,
>> +			 "qid %d: DH-HMAC-CHAP: controller authentication failed\n",
>> +			 chap->qid);
>> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
>> +		return -EPROTO;
>> +	}
>> +	dev_info(ctrl->device,
>> +		 "qid %d: DH-HMAC-CHAP: controller authenticated\n",
>> +		chap->qid);
>> +	return 0;
>> +}
>> +
>> +static int nvme_auth_dhchap_success2(struct nvme_ctrl *ctrl,
>> +				     struct nvme_dhchap_context *chap,
>> +				     void *buf, size_t buf_size)
>> +{
>> +	struct nvmf_auth_dhchap_success2_data *data = buf;
>> +	size_t size = sizeof(*data);
>> +
>> +	memset(buf, 0, size);
>> +	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
>> +	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_SUCCESS2;
>> +	data->t_id = cpu_to_le16(chap->transaction);
>> +
>> +	return size;
>> +}
>> +
>> +static int nvme_auth_dhchap_failure2(struct nvme_ctrl *ctrl,
>> +				     struct nvme_dhchap_context *chap,
>> +				     void *buf, size_t buf_size)
>> +{
>> +	struct nvmf_auth_dhchap_failure_data *data = buf;
>> +	size_t size = sizeof(*data);
>> +
>> +	memset(buf, 0, size);
>> +	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
>> +	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_FAILURE2;
>> +	data->t_id = cpu_to_le16(chap->transaction);
>> +	data->reason_code = 1;
>> +	data->reason_code_explanation = chap->status;
>> +
>> +	return size;
>> +}
>> +
>> +int nvme_auth_select_hash(struct nvme_ctrl *ctrl,
>> +			  struct nvme_dhchap_context *chap)
>> +{
>> +	char *hash_name;
>> +	int ret;
>> +
>> +	switch (chap->hash_id) {
>> +	case NVME_AUTH_DHCHAP_HASH_SHA256:
>> +		hash_name = "hmac(sha256)";
>> +		break;
>> +	case NVME_AUTH_DHCHAP_HASH_SHA384:
>> +		hash_name = "hmac(sha384)";
>> +		break;
>> +	case NVME_AUTH_DHCHAP_HASH_SHA512:
>> +		hash_name = "hmac(sha512)";
>> +		break;
>> +	default:
>> +		hash_name = NULL;
>> +		break;
>> +	}
>> +	if (!hash_name) {
>> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
>> +		return -EPROTO;
>> +	}
>> +	chap->shash_tfm = crypto_alloc_shash(hash_name, 0,
>> +					     CRYPTO_ALG_ALLOCATES_MEMORY);
>> +	if (IS_ERR(chap->shash_tfm)) {
>> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
>> +		chap->shash_tfm = NULL;
>> +		return -EPROTO;
>> +	}
>> +	if (!chap->key) {
>> +		dev_warn(ctrl->device, "qid %d: cannot select hash, no key\n",
>> +			 chap->qid);
>> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
>> +		crypto_free_shash(chap->shash_tfm);
>> +		chap->shash_tfm = NULL;
>> +		return -EINVAL;
>> +	}
>> +	ret = crypto_shash_setkey(chap->shash_tfm, chap->key, chap->key_len);
>> +	if (ret) {
>> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
>> +		crypto_free_shash(chap->shash_tfm);
>> +		chap->shash_tfm = NULL;
>> +		return ret;
>> +	}
>> +	dev_info(ctrl->device, "qid %d: DH-HMAC_CHAP: selected hash %s\n",
>> +		 chap->qid, hash_name);
>> +	return 0;
>> +}
>> +
>> +static int nvme_auth_dhchap_host_response(struct nvme_ctrl *ctrl,
>> +					  struct nvme_dhchap_context *chap)
>> +{
>> +	SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
>> +	u8 buf[4], *challenge = chap->c1;
>> +	int ret;
>> +
>> +	dev_dbg(ctrl->device, "%s: qid %d host response seq %d transaction
> %d\n",
>> +		__func__, chap->qid, chap->s1, chap->transaction);
>> +	shash->tfm = chap->shash_tfm;
>> +	ret = crypto_shash_init(shash);
>> +	if (ret)
>> +		goto out;
>> +	ret = crypto_shash_update(shash, challenge, chap->hash_len);
>> +	if (ret)
>> +		goto out;
>> +	put_unaligned_le32(chap->s1, buf);
>> +	ret = crypto_shash_update(shash, buf, 4);
>> +	if (ret)
>> +		goto out;
>> +	put_unaligned_le16(chap->transaction, buf);
>> +	ret = crypto_shash_update(shash, buf, 2);
>> +	if (ret)
>> +		goto out;
>> +	memset(buf, 0, sizeof(buf));
>> +	ret = crypto_shash_update(shash, buf, 1);
>> +	if (ret)
>> +		goto out;
>> +	ret = crypto_shash_update(shash, "HostHost", 8);
>> +	if (ret)
>> +		goto out;
>> +	ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
>> +				  strlen(ctrl->opts->host->nqn));
>> +	if (ret)
>> +		goto out;
>> +	ret = crypto_shash_update(shash, buf, 1);
>> +	if (ret)
>> +		goto out;
>> +	ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
>> +			    strlen(ctrl->opts->subsysnqn));
>> +	if (ret)
>> +		goto out;
>> +	ret = crypto_shash_final(shash, chap->response);
>> +out:
>> +	return ret;
>> +}
>> +
>> +static int nvme_auth_dhchap_ctrl_response(struct nvme_ctrl *ctrl,
>> +					  struct nvme_dhchap_context *chap)
>> +{
>> +	SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
>> +	u8 buf[4], *challenge = chap->c2;
>> +	int ret;
>> +
>> +	dev_dbg(ctrl->device, "%s: qid %d host response seq %d transaction
> %d\n",
>> +		__func__, chap->qid, chap->s2, chap->transaction);
>> +	dev_dbg(ctrl->device, "%s: qid %d challenge %*ph\n",
>> +		__func__, chap->qid, chap->hash_len, challenge);
>> +	dev_dbg(ctrl->device, "%s: qid %d subsysnqn %s\n",
>> +		__func__, chap->qid, ctrl->opts->subsysnqn);
>> +	dev_dbg(ctrl->device, "%s: qid %d hostnqn %s\n",
>> +		__func__, chap->qid, ctrl->opts->host->nqn);
>> +	shash->tfm = chap->shash_tfm;
>> +	ret = crypto_shash_init(shash);
>> +	if (ret)
>> +		goto out;
>> +	ret = crypto_shash_update(shash, challenge, chap->hash_len);
>> +	if (ret)
>> +		goto out;
>> +	put_unaligned_le32(chap->s2, buf);
>> +	ret = crypto_shash_update(shash, buf, 4);
>> +	if (ret)
>> +		goto out;
>> +	put_unaligned_le16(chap->transaction, buf);
>> +	ret = crypto_shash_update(shash, buf, 2);
>> +	if (ret)
>> +		goto out;
>> +	memset(buf, 0, 4);
>> +	ret = crypto_shash_update(shash, buf, 1);
>> +	if (ret)
>> +		goto out;
>> +	ret = crypto_shash_update(shash, "Controller", 10);
>> +	if (ret)
>> +		goto out;
>> +	ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
>> +				  strlen(ctrl->opts->subsysnqn));
>> +	if (ret)
>> +		goto out;
>> +	ret = crypto_shash_update(shash, buf, 1);
>> +	if (ret)
>> +		goto out;
>> +	ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
>> +				  strlen(ctrl->opts->host->nqn));
>> +	if (ret)
>> +		goto out;
>> +	ret = crypto_shash_final(shash, chap->response);
>> +out:
>> +	return ret;
>> +}
>> +
>> +int nvme_auth_generate_key(struct nvme_ctrl *ctrl,
>> +			   struct nvme_dhchap_context *chap)
>> +{
>> +	int ret;
>> +	u8 key_hash;
>> +	const char *hmac_name;
>> +	struct crypto_shash *key_tfm;
>> +
>> +	if (sscanf(ctrl->opts->dhchap_secret, "DHHC-1:%hhd:%*s:",
>> +		   &key_hash) != 1)
>> +		return -EINVAL;
>> +
>> +	chap->key = nvme_auth_extract_secret(ctrl->opts->dhchap_secret,
>> +					     &chap->key_len);
>> +	if (IS_ERR(chap->key)) {
>> +		ret = PTR_ERR(chap->key);
>> +		chap->key = NULL;
>> +		return ret;
>> +	}
>> +
>> +	if (key_hash == 0)
>> +		return 0;
>> +
>> +	hmac_name = nvme_auth_hmac_name(key_hash);
>> +	if (!hmac_name) {
>> +		pr_debug("Invalid key hash id %d\n", key_hash);
>> +		return -EKEYREJECTED;
>> +	}
>> +
>> +	key_tfm = crypto_alloc_shash(hmac_name, 0, 0);
>> +	if (IS_ERR(key_tfm)) {
>> +		kfree(chap->key);
>> +		chap->key = NULL;
>> +		ret = PTR_ERR(key_tfm);
>> +	} else {
>> +		SHASH_DESC_ON_STACK(shash, key_tfm);
>> +
>> +		shash->tfm = key_tfm;
>> +		ret = crypto_shash_setkey(key_tfm, chap->key,
>> +					  chap->key_len);
>> +		if (ret < 0) {
>> +			crypto_free_shash(key_tfm);
>> +			kfree(chap->key);
>> +			chap->key = NULL;
>> +			return ret;
>> +		}
>> +		crypto_shash_init(shash);
>> +		crypto_shash_update(shash, ctrl->opts->host->nqn,
>> +				    strlen(ctrl->opts->host->nqn));
>> +		crypto_shash_update(shash, "NVMe-over-Fabrics", 17);
>> +		crypto_shash_final(shash, chap->key);
>> +		crypto_free_shash(key_tfm);
>> +	}
>> +	return 0;
>> +}
>> +
>> +void nvme_auth_free(struct nvme_dhchap_context *chap)
>> +{
>> +	if (chap->shash_tfm)
>> +		crypto_free_shash(chap->shash_tfm);
>> +	if (chap->key)
>> +		kfree(chap->key);
>> +	if (chap->ctrl_key)
>> +		kfree(chap->ctrl_key);
>> +	if (chap->host_key)
>> +		kfree(chap->host_key);
>> +	if (chap->sess_key)
>> +		kfree(chap->sess_key);
>> +	kfree(chap);
> 
> kfree_sensitive in all cases as all buffers have sensitive data?
> 

Yes, will be fixing it up.

>> +}
>> +
>> +int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid)
>> +{
>> +	struct nvme_dhchap_context *chap;
>> +	void *buf;
>> +	size_t buf_size, tl;
>> +	int ret = 0;
>> +
>> +	chap = kzalloc(sizeof(*chap), GFP_KERNEL);
> 
> Suggestion: make sure that chap->response is aligned to CRYPTO_MINALIGN_ATTR -
> then you would save a memcpy in crypto_shash_final
> 

Ok.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 06/11] nvme: Implement In-Band authentication

[Hannes Reinecke]

On 7/17/21 9:22 AM, Sagi Grimberg wrote:
>> Implement NVMe-oF In-Band authentication. This patch adds two new
>> fabric options 'dhchap_key' to specify the PSK
> 
> pre-shared-key.
> 
> Also, we need a sysfs knob to rotate the key that will trigger
> re-authentication or even a simple controller(s-plural) reset, so this
> should go beyond just the connection string.
> 

Yeah, re-authentication currently is not implemented. I first wanted to 
get this patchset out such that we can settle on the userspace interface 
(both from host and target).
I'll have to think on how we should handle authentication; one of the 
really interesting cases would be when one malicious admin will _just_ 
send a 'negotiate' command to the controller. As per spec the controller 
will be waiting for an 'authentication receive' command to send a 
'challenge' payload back to the host. But that will never come, so as it 
stands currently the controller is required to abort the connection.
Not very nice.

> P.S. can you add also the nvme-cli code in the next go?
> 
Oh, sure. It's already sitting around in my local repo (surprise, 
surprise); will be ending it out next time.

>> and 'dhchap_authenticate'
>> to request bi-directional authentication of both the host and the 
>> controller.
> 
> bidirectional? not uni-directional?
> 

Yeah, that's a bit of a misnomer. When a PSK is specified, the 
controller will start the authentication protocol such that the 
_controller_ can validate the host. If the host wants to authenticate 
the controller is needs to set this flag.
Hence bi-directional authentication.
But I'm the first to admit that this is poor wording for the flag.

>>
>> Signed-off-by: Hannes Reinecke <hare@suse.de>
>> ---
>>   drivers/nvme/host/Kconfig   |  11 +
>>   drivers/nvme/host/Makefile  |   1 +
>>   drivers/nvme/host/auth.c    | 813 ++++++++++++++++++++++++++++++++++++
>>   drivers/nvme/host/auth.h    |  23 +
>>   drivers/nvme/host/core.c    |  77 +++-
>>   drivers/nvme/host/fabrics.c |  65 ++-
>>   drivers/nvme/host/fabrics.h |   8 +
>>   drivers/nvme/host/nvme.h    |  15 +
>>   drivers/nvme/host/trace.c   |  32 ++
>>   9 files changed, 1041 insertions(+), 4 deletions(-)
>>   create mode 100644 drivers/nvme/host/auth.c
>>   create mode 100644 drivers/nvme/host/auth.h
>>
>> diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig
>> index c3f3d77f1aac..853c546305e9 100644
>> --- a/drivers/nvme/host/Kconfig
>> +++ b/drivers/nvme/host/Kconfig
>> @@ -85,3 +85,14 @@ config NVME_TCP
>>         from https://github.com/linux-nvme/nvme-cli.
>>         If unsure, say N.
>> +
>> +config NVME_AUTH
>> +    bool "NVM Express over Fabrics In-Band Authentication"
>> +    depends on NVME_TCP
>> +    select CRYPTO_SHA256
>> +    select CRYPTO_SHA512
>> +    help
>> +      This provides support for NVMe over Fabrics In-Band Authentication
>> +      for the NVMe over TCP transport.
> 
> In this form, nothing is specific to nvme-tcp here afaict.
> 

Indeed. I guess we can leave out the nvme-tcp reference here.

>> +
>> +      If unsure, say N.
>> diff --git a/drivers/nvme/host/Makefile b/drivers/nvme/host/Makefile
>> index cbc509784b2e..03748a55a12b 100644
>> --- a/drivers/nvme/host/Makefile
>> +++ b/drivers/nvme/host/Makefile
>> @@ -20,6 +20,7 @@ nvme-core-$(CONFIG_NVME_HWMON)        += hwmon.o
>>   nvme-y                    += pci.o
>>   nvme-fabrics-y                += fabrics.o
>> +nvme-fabrics-$(CONFIG_NVME_AUTH)    += auth.o
>>   nvme-rdma-y                += rdma.o
>> diff --git a/drivers/nvme/host/auth.c b/drivers/nvme/host/auth.c
>> new file mode 100644
>> index 000000000000..448a3adebea6
>> --- /dev/null
>> +++ b/drivers/nvme/host/auth.c
>> @@ -0,0 +1,813 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * Copyright (c) 2020 Hannes Reinecke, SUSE Linux
>> + */
>> +
>> +#include <linux/crc32.h>
>> +#include <linux/base64.h>
>> +#include <asm/unaligned.h>
>> +#include <crypto/hash.h>
>> +#include <crypto/kpp.h>
>> +#include "nvme.h"
>> +#include "fabrics.h"
>> +#include "auth.h"
>> +
>> +static u32 nvme_dhchap_seqnum;
>> +
>> +struct nvme_dhchap_context {
> 
> Maybe nvme_dhchap_queue_context ?
> 
> I'm thinking that we should perhaps split
> it to host-wide, subsys-wide and queue specific
> auth contexts?
> 
> Let's see...
> 

Interestingly enough, that's what I did for the target side.
For the host side I found it easier that way, as then we'll have a 
single authentication context which can be deleted after authentication 
finished, and be sure that we removed _all_ information.
Security and all that.
Splitting it off would require to remove information on three different 
places, and observing life-time rules for them.
So more of a chance to mess things up :-)

>> +    struct crypto_shash *shash_tfm;
>> +    unsigned char *key;
>> +    size_t key_len;
>> +    int qid;
>> +    u32 s1;
>> +    u32 s2;
>> +    u16 transaction;
>> +    u8 status;
>> +    u8 hash_id;
>> +    u8 hash_len;
>> +    u8 c1[64];
>> +    u8 c2[64];
>> +    u8 response[64];
>> +    u8 *ctrl_key;
>> +    int ctrl_key_len;
>> +    u8 *host_key;
>> +    int host_key_len;
>> +    u8 *sess_key;
>> +    int sess_key_len;
>> +};
>> +
>> +struct nvmet_dhchap_hash_map {
> 
> nvmet?
> 

Yeah; originally I coded that for the target side, and only later moved 
it into the host side to have it usable for both.
Will be fixing it up.

>> +    int id;
>> +    int hash_len;
>> +    const char hmac[15];
>> +    const char digest[15];
>> +} hash_map[] = {
>> +    {.id = NVME_AUTH_DHCHAP_HASH_SHA256,
>> +     .hash_len = 32,
>> +     .hmac = "hmac(sha256)", .digest = "sha256" },
>> +    {.id = NVME_AUTH_DHCHAP_HASH_SHA384,
>> +     .hash_len = 48,
>> +     .hmac = "hmac(sha384)", .digest = "sha384" },
>> +    {.id = NVME_AUTH_DHCHAP_HASH_SHA512,
>> +     .hash_len = 64,
>> +     .hmac = "hmac(sha512)", .digest = "sha512" },
>> +};
>> +
>> +const char *nvme_auth_hmac_name(int hmac_id)
> 
> Should these arrays be static?
> 

Definitely.

>> +{
>> +    int i;
>> +
>> +    for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
>> +        if (hash_map[i].id == hmac_id)
>> +            return hash_map[i].hmac;
>> +    }
>> +    return NULL;
>> +}
>> +EXPORT_SYMBOL_GPL(nvme_auth_hmac_name);
>> +
>> +const char *nvme_auth_digest_name(int hmac_id)
>> +{
>> +    int i;
>> +
>> +    for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
>> +        if (hash_map[i].id == hmac_id)
>> +            return hash_map[i].digest;
>> +    }
>> +    return NULL;
>> +}
>> +EXPORT_SYMBOL_GPL(nvme_auth_digest_name);
>> +
>> +int nvme_auth_hmac_len(int hmac_id)
>> +{
>> +    int i;
>> +
>> +    for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
>> +        if (hash_map[i].id == hmac_id)
>> +            return hash_map[i].hash_len;
>> +    }
>> +    return -1;
>> +}
>> +EXPORT_SYMBOL_GPL(nvme_auth_hmac_len);
>> +
>> +int nvme_auth_hmac_id(const char *hmac_name)
>> +{
>> +    int i;
>> +
>> +    for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
>> +        if (!strncmp(hash_map[i].hmac, hmac_name,
>> +                 strlen(hash_map[i].hmac)))
>> +            return hash_map[i].id;
>> +    }
>> +    return -1;
>> +}
>> +EXPORT_SYMBOL_GPL(nvme_auth_hmac_id);
>> +
>> +unsigned char *nvme_auth_extract_secret(unsigned char *dhchap_secret,
>> +                    size_t *dhchap_key_len)
>> +{
>> +    unsigned char *dhchap_key;
>> +    u32 crc;
>> +    int key_len;
>> +    size_t allocated_len;
>> +
>> +    allocated_len = strlen(dhchap_secret) - 10;
> 
> the 10 feels like a magic here, should at least note this is the
> "DHHC-1:..." prefix.
> 

It _is_ magic. And it might even be better to just pass in the string 
_without_ the DHHC-1: prefix.

>> +    dhchap_key = kzalloc(allocated_len, GFP_KERNEL);
>> +    if (!dhchap_key)
>> +        return ERR_PTR(-ENOMEM);
>> +
>> +    key_len = base64_decode(dhchap_secret + 10,
>> +                allocated_len, dhchap_key);
>> +    if (key_len != 36 && key_len != 52 &&
>> +        key_len != 68) {
>> +        pr_debug("Invalid DH-HMAC-CHAP key len %d\n",
>> +             key_len);
>> +        kfree(dhchap_key);
>> +        return ERR_PTR(-EINVAL);
>> +    }
>> +    pr_debug("DH-HMAC-CHAP Key: %*ph\n",
>> +         (int)key_len, dhchap_key);
> 
> One can argue if even printing this is problematic..
> 

Debugging scaffolding. You wouldn't believe how many things can go wrong...

And yes, that should be removed.

>> +
>> +    /* The last four bytes is the CRC in little-endian format */
>> +    key_len -= 4;
>> +    /*
>> +     * The linux implementation doesn't do pre- and post-increments,
>> +     * so we have to do it manually.
>> +     */
>> +    crc = ~crc32(~0, dhchap_key, key_len);
>> +
>> +    if (get_unaligned_le32(dhchap_key + key_len) != crc) {
>> +        pr_debug("DH-HMAC-CHAP crc mismatch (key %08x, crc %08x)\n",
>> +               get_unaligned_le32(dhchap_key + key_len), crc);
>> +        kfree(dhchap_key);
>> +        return ERR_PTR(-EKEYREJECTED);
>> +    }
>> +    *dhchap_key_len = key_len;
>> +    return dhchap_key;
>> +}
>> +EXPORT_SYMBOL_GPL(nvme_auth_extract_secret);
>> +
>> +static int nvme_auth_send(struct nvme_ctrl *ctrl, int qid,
>> +              void *data, size_t tl)
>> +{
>> +    struct nvme_command cmd = {};
>> +    blk_mq_req_flags_t flags = qid == NVME_QID_ANY ?
>> +        0 : BLK_MQ_REQ_NOWAIT | BLK_MQ_REQ_RESERVED;
>> +    struct request_queue *q = qid == NVME_QID_ANY ?
>> +        ctrl->fabrics_q : ctrl->connect_q;
>> +    int ret;
>> +
>> +    cmd.auth_send.opcode = nvme_fabrics_command;
>> +    cmd.auth_send.fctype = nvme_fabrics_type_auth_send;
>> +    cmd.auth_send.secp = NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER;
>> +    cmd.auth_send.spsp0 = 0x01;
>> +    cmd.auth_send.spsp1 = 0x01;
>> +    cmd.auth_send.tl = tl;
>> +
>> +    ret = __nvme_submit_sync_cmd(q, &cmd, NULL, data, tl, 0, qid,
>> +                     0, flags);
>> +    if (ret)
>> +        dev_dbg(ctrl->device,
>> +            "%s: qid %d error %d\n", __func__, qid, ret);
> 
> Maybe a little more informative print rather than __func__ ?
> 

Yes, can do.

>> +    return ret;
>> +}
>> +
>> +static int nvme_auth_receive(struct nvme_ctrl *ctrl, int qid,
>> +                 void *buf, size_t al,
>> +                 u16 transaction, u8 expected_msg )
>> +{
>> +    struct nvme_command cmd = {};
>> +    struct nvmf_auth_dhchap_failure_data *data = buf;
>> +    blk_mq_req_flags_t flags = qid == NVME_QID_ANY ?
>> +        0 : BLK_MQ_REQ_NOWAIT | BLK_MQ_REQ_RESERVED;
>> +    struct request_queue *q = qid == NVME_QID_ANY ?
>> +        ctrl->fabrics_q : ctrl->connect_q;
>> +    int ret;
>> +
>> +    cmd.auth_receive.opcode = nvme_fabrics_command;
>> +    cmd.auth_receive.fctype = nvme_fabrics_type_auth_receive;
>> +    cmd.auth_receive.secp = NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER;
>> +    cmd.auth_receive.spsp0 = 0x01;
>> +    cmd.auth_receive.spsp1 = 0x01;
>> +    cmd.auth_receive.al = al;
>> +
>> +    ret = __nvme_submit_sync_cmd(q, &cmd, NULL, buf, al, 0, qid,
>> +                     0, flags);
>> +    if (ret > 0) {
>> +        dev_dbg(ctrl->device, "%s: qid %d nvme status %x\n",
>> +            __func__, qid, ret);
>> +        ret = -EIO;
>> +    }
>> +    if (ret < 0) {
>> +        dev_dbg(ctrl->device, "%s: qid %d error %d\n",
>> +            __func__, qid, ret);
>> +        return ret;
>> +    }
>> +    dev_dbg(ctrl->device, "%s: qid %d auth_type %d auth_id %x\n",
>> +        __func__, qid, data->auth_type, data->auth_id);
>> +    if (data->auth_type == NVME_AUTH_COMMON_MESSAGES &&
>> +        data->auth_id == NVME_AUTH_DHCHAP_MESSAGE_FAILURE1) {
>> +        return data->reason_code_explanation;
>> +    }
>> +    if (data->auth_type != NVME_AUTH_DHCHAP_MESSAGES ||
>> +        data->auth_id != expected_msg) {
>> +        dev_warn(ctrl->device,
>> +             "qid %d invalid message %02x/%02x\n",
>> +             qid, data->auth_type, data->auth_id);
>> +        return NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
>> +    }
>> +    if (le16_to_cpu(data->t_id) != transaction) {
>> +        dev_warn(ctrl->device,
>> +             "qid %d invalid transaction ID %d\n",
>> +             qid, le16_to_cpu(data->t_id));
>> +        return NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
>> +    }
>> +
>> +    return 0;
>> +}
>> +
>> +static int nvme_auth_dhchap_negotiate(struct nvme_ctrl *ctrl,
>> +                      struct nvme_dhchap_context *chap,
>> +                      void *buf, size_t buf_size)
> 
> Maybe nvme_auth_set_dhchap_negotiate_data ?
> 

These are the individual steps in the state machine later on, so I 
wanted to keep the names identical.
But I'm open to suggestions.

>> +{
>> +    struct nvmf_auth_dhchap_negotiate_data *data = buf;
>> +    size_t size = sizeof(*data) + sizeof(union nvmf_auth_protocol);
>> +
>> +    if (buf_size < size)
>> +        return -EINVAL;
>> +
>> +    memset((u8 *)buf, 0, size);
>> +    data->auth_type = NVME_AUTH_COMMON_MESSAGES;
>> +    data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE;
>> +    data->t_id = cpu_to_le16(chap->transaction);
>> +    data->sc_c = 0; /* No secure channel concatenation */
>> +    data->napd = 1;
>> +    data->auth_protocol[0].dhchap.authid = NVME_AUTH_DHCHAP_AUTH_ID;
>> +    data->auth_protocol[0].dhchap.halen = 3;
>> +    data->auth_protocol[0].dhchap.dhlen = 1;
>> +    data->auth_protocol[0].dhchap.idlist[0] = 
>> NVME_AUTH_DHCHAP_HASH_SHA256;
>> +    data->auth_protocol[0].dhchap.idlist[1] = 
>> NVME_AUTH_DHCHAP_HASH_SHA384;
>> +    data->auth_protocol[0].dhchap.idlist[2] = 
>> NVME_AUTH_DHCHAP_HASH_SHA512;
>> +    data->auth_protocol[0].dhchap.idlist[3] = 
>> NVME_AUTH_DHCHAP_DHGROUP_NULL;
> You should comment that this routine expects buf to have enough
> room for both negotiate and auth_proto structures.
> 
Hmm. I do a check for the overall size at the start, so I'm not sure 
what this will buy us.
And actually, anyone wanting to make sense of the implementation would 
need to look at the spec anyway.

>> +
>> +    return size;
>> +}
>> +
>> +static int nvme_auth_dhchap_challenge(struct nvme_ctrl *ctrl,
>> +                      struct nvme_dhchap_context *chap,
>> +                      void *buf, size_t buf_size)
> 
> Maybe nvme_auth_process_dhchap_challange ?
> 

See above. I'd rather have consistent names for the state machine.
But I can change them to 'nvme_process_chchap_<statename>'

>> +{
>> +    struct nvmf_auth_dhchap_challenge_data *data = buf;
>> +    size_t size = sizeof(*data) + data->hl + data->dhvlen;
>> +    const char *gid_name;
>> +
>> +    if (buf_size < size) {
>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
>> +        return -ENOMSG;
>> +    }
>> +
>> +    if (data->hashid != NVME_AUTH_DHCHAP_HASH_SHA256 &&
>> +        data->hashid != NVME_AUTH_DHCHAP_HASH_SHA384 &&
>> +        data->hashid != NVME_AUTH_DHCHAP_HASH_SHA512) {
>> +        dev_warn(ctrl->device,
>> +             "qid %d: DH-HMAC-CHAP: invalid HASH ID %d\n",
>> +             chap->qid, data->hashid);
>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
>> +        return -EPROTO;
>> +    }
>> +    switch (data->dhgid) {
>> +    case NVME_AUTH_DHCHAP_DHGROUP_NULL:
>> +        gid_name = "null";
>> +        break;
>> +    default:
>> +        gid_name = NULL;
>> +        break;
>> +    }
>> +    if (!gid_name) {
>> +        dev_warn(ctrl->device,
>> +             "qid %d: DH-HMAC-CHAP: invalid DH group id %d\n",
>> +             chap->qid, data->dhgid);
>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
>> +        return -EPROTO;
>> +    }
> 
> Maybe some spaces between condition blocks?
> 

Ok.

>> +    if (data->dhgid != NVME_AUTH_DHCHAP_DHGROUP_NULL) {
>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
>> +        return -EPROTO;
>> +    }
>> +    if (data->dhgid == NVME_AUTH_DHCHAP_DHGROUP_NULL && data->dhvlen 
>> != 0) {
>> +        dev_warn(ctrl->device,
>> +             "qid %d: DH-HMAC-CHAP: invalid DH value for NULL DH\n",
>> +            chap->qid);
>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
>> +        return -EPROTO;
>> +    }
>> +    dev_dbg(ctrl->device, "%s: qid %d requested hash id %d\n",
>> +        __func__, chap->qid, data->hashid);
>> +    if (nvme_auth_hmac_len(data->hashid) != data->hl) {
>> +        dev_warn(ctrl->device,
>> +             "qid %d: DH-HMAC-CHAP: invalid hash length\n",
>> +            chap->qid);
>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
>> +        return -EPROTO;
>> +    }
>> +    chap->hash_id = data->hashid;
>> +    chap->hash_len = data->hl;
>> +    chap->s1 = le32_to_cpu(data->seqnum);
>> +    memcpy(chap->c1, data->cval, chap->hash_len);
>> +
>> +    return 0;
>> +}
>> +
>> +static int nvme_auth_dhchap_reply(struct nvme_ctrl *ctrl,
>> +                  struct nvme_dhchap_context *chap,
>> +                  void *buf, size_t buf_size)
> 
> nvme_auth_set_dhchap_reply
> 

Ah. Now I see what you're getting at.
Okay, will be changing it.

>> +{
>> +    struct nvmf_auth_dhchap_reply_data *data = buf;
>> +    size_t size = sizeof(*data);
>> +
>> +    size += 2 * chap->hash_len;
>> +    if (ctrl->opts->dhchap_auth) {
> 
> The ctrl opts is not clear to me. what is dhchap_auth
> mean?
> 
As stated above, this is for bi-directional authentication.
And yes, it is poor wording.

'dhchap_bidirectional' ?

> Also shouldn't these params be lifted to the subsys?
> 

I kinda like to have it all encapsulated in a common per-queue 
structure; on the host side this one isn't even attached to anything, so 
any new authentication attempt will allocate a new one, with no chance 
of accidentally re-using existing values.
I thought this to be a rather nice property for a state-machine.

>> +        get_random_bytes(chap->c2, chap->hash_len);
>> +        chap->s2 = nvme_dhchap_seqnum++;
>> +    } else
>> +        memset(chap->c2, 0, chap->hash_len);
>> +
>> +    if (chap->host_key_len)
>> +        size += chap->host_key_len;
>> +
>> +    if (buf_size < size)
>> +        return -EINVAL;
>> +
>> +    memset(buf, 0, size);
>> +    data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
>> +    data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_REPLY;
>> +    data->t_id = cpu_to_le16(chap->transaction);
>> +    data->hl = chap->hash_len;
>> +    data->dhvlen = chap->host_key_len;
>> +    data->seqnum = cpu_to_le32(chap->s2);
>> +    memcpy(data->rval, chap->response, chap->hash_len);
>> +    if (ctrl->opts->dhchap_auth) {
>> +        dev_dbg(ctrl->device, "%s: qid %d ctrl challenge %*ph\n",
>> +            __func__, chap->qid,
>> +            chap->hash_len, chap->c2);
>> +        data->cvalid = 1;
>> +        memcpy(data->rval + chap->hash_len, chap->c2,
>> +               chap->hash_len);
>> +    }
>> +    if (chap->host_key_len)
>> +        memcpy(data->rval + 2 * chap->hash_len, chap->host_key,
>> +               chap->host_key_len);
>> +
>> +    return size;
>> +}
>> +
>> +static int nvme_auth_dhchap_success1(struct nvme_ctrl *ctrl,
>> +                     struct nvme_dhchap_context *chap,
>> +                     void *buf, size_t buf_size)
> 
> nvme_auth_process_dhchap_success1
> 

OK.

>> +{
>> +    struct nvmf_auth_dhchap_success1_data *data = buf;
>> +    size_t size = sizeof(*data);
>> +
>> +    if (ctrl->opts->dhchap_auth)
>> +        size += chap->hash_len;
>> +
>> +
>> +    if (buf_size < size) {
>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
>> +        return -ENOMSG;
>> +    }
>> +
>> +    if (data->hl != chap->hash_len) {
>> +        dev_warn(ctrl->device,
>> +             "qid %d: DH-HMAC-CHAP: invalid hash length %d\n",
>> +             chap->qid, data->hl);
>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
>> +        return -EPROTO;
>> +    }
>> +
>> +    if (!data->rvalid)
>> +        return 0;
>> +
>> +    /* Validate controller response */
>> +    if (memcmp(chap->response, data->rval, data->hl)) {
>> +        dev_dbg(ctrl->device, "%s: qid %d ctrl response %*ph\n",
>> +            __func__, chap->qid, chap->hash_len, data->rval);
>> +        dev_dbg(ctrl->device, "%s: qid %d host response %*ph\n",
>> +            __func__, chap->qid, chap->hash_len, chap->response);
>> +        dev_warn(ctrl->device,
>> +             "qid %d: DH-HMAC-CHAP: controller authentication failed\n",
>> +             chap->qid);
>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
>> +        return -EPROTO;
>> +    }
>> +    dev_info(ctrl->device,
>> +         "qid %d: DH-HMAC-CHAP: controller authenticated\n",
>> +        chap->qid);
>> +    return 0;
>> +}
>> +
>> +static int nvme_auth_dhchap_success2(struct nvme_ctrl *ctrl,
>> +                     struct nvme_dhchap_context *chap,
>> +                     void *buf, size_t buf_size)
> 
> same
> 
>> +{
>> +    struct nvmf_auth_dhchap_success2_data *data = buf;
>> +    size_t size = sizeof(*data);
>> +
>> +    memset(buf, 0, size);
>> +    data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
>> +    data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_SUCCESS2;
>> +    data->t_id = cpu_to_le16(chap->transaction);
>> +
>> +    return size;
>> +}
>> +
>> +static int nvme_auth_dhchap_failure2(struct nvme_ctrl *ctrl,
>> +                     struct nvme_dhchap_context *chap,
>> +                     void *buf, size_t buf_size)
> 
> same
> 
>> +{
>> +    struct nvmf_auth_dhchap_failure_data *data = buf;
>> +    size_t size = sizeof(*data);
>> +
>> +    memset(buf, 0, size);
>> +    data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
>> +    data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_FAILURE2;
>> +    data->t_id = cpu_to_le16(chap->transaction);
>> +    data->reason_code = 1;
>> +    data->reason_code_explanation = chap->status;
>> +
>> +    return size;
>> +}
>> +
>> +int nvme_auth_select_hash(struct nvme_ctrl *ctrl,
>> +              struct nvme_dhchap_context *chap)
> 
> Maybe _select_hf (hash function)? not a must, just sticks
> to the spec language.
> 

Hmm. Will be checking.

>> +{
>> +    char *hash_name;
>> +    int ret;
>> +
>> +    switch (chap->hash_id) {
>> +    case NVME_AUTH_DHCHAP_HASH_SHA256:
>> +        hash_name = "hmac(sha256)";
>> +        break;
>> +    case NVME_AUTH_DHCHAP_HASH_SHA384:
>> +        hash_name = "hmac(sha384)";
>> +        break;
>> +    case NVME_AUTH_DHCHAP_HASH_SHA512:
>> +        hash_name = "hmac(sha512)";
>> +        break;
>> +    default:
>> +        hash_name = NULL;
>> +        break;
>> +    }
>> +    if (!hash_name) {
>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
>> +        return -EPROTO;
>> +    }
>> +    chap->shash_tfm = crypto_alloc_shash(hash_name, 0,
>> +                         CRYPTO_ALG_ALLOCATES_MEMORY);
>> +    if (IS_ERR(chap->shash_tfm)) {
>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
>> +        chap->shash_tfm = NULL;
>> +        return -EPROTO;
>> +    }
>> +    if (!chap->key) {
>> +        dev_warn(ctrl->device, "qid %d: cannot select hash, no key\n",
>> +             chap->qid);
>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
>> +        crypto_free_shash(chap->shash_tfm);
> 
> Wouldn't it better to check this before allocating the tfm?
> 

Indeed. Will be changing it.

>> +        chap->shash_tfm = NULL;
>> +        return -EINVAL;
>> +    }
>> +    ret = crypto_shash_setkey(chap->shash_tfm, chap->key, 
>> chap->key_len);
>> +    if (ret) {
>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
>> +        crypto_free_shash(chap->shash_tfm);
>> +        chap->shash_tfm = NULL;
>> +        return ret;
>> +    }
>> +    dev_info(ctrl->device, "qid %d: DH-HMAC_CHAP: selected hash %s\n",
>> +         chap->qid, hash_name);
>> +    return 0;
>> +}
>> +
>> +static int nvme_auth_dhchap_host_response(struct nvme_ctrl *ctrl,
>> +                      struct nvme_dhchap_context *chap)
>> +{
>> +    SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
>> +    u8 buf[4], *challenge = chap->c1;
>> +    int ret;
>> +
>> +    dev_dbg(ctrl->device, "%s: qid %d host response seq %d 
>> transaction %d\n",
>> +        __func__, chap->qid, chap->s1, chap->transaction);
>> +    shash->tfm = chap->shash_tfm;
>> +    ret = crypto_shash_init(shash);
>> +    if (ret)
>> +        goto out;
>> +    ret = crypto_shash_update(shash, challenge, chap->hash_len);
>> +    if (ret)
>> +        goto out;
>> +    put_unaligned_le32(chap->s1, buf);
>> +    ret = crypto_shash_update(shash, buf, 4);
>> +    if (ret)
>> +        goto out;
>> +    put_unaligned_le16(chap->transaction, buf);
>> +    ret = crypto_shash_update(shash, buf, 2);
>> +    if (ret)
>> +        goto out;
>> +    memset(buf, 0, sizeof(buf));
>> +    ret = crypto_shash_update(shash, buf, 1);
>> +    if (ret)
>> +        goto out;
>> +    ret = crypto_shash_update(shash, "HostHost", 8);
> 
> HostHost ? Can you refer me to the specific section
> that talks about this?
> 

NVMe 2.0 section DH-HMAC-CHAP_Reply Message, paragraph Response Value.
HostHost.

> Would be good to have a comment on the format fed to the
> shash.
> 

Yes, will be doing so.

>> +    if (ret)
>> +        goto out;
>> +    ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
>> +                  strlen(ctrl->opts->host->nqn));
>> +    if (ret)
>> +        goto out;
>> +    ret = crypto_shash_update(shash, buf, 1);
>> +    if (ret)
>> +        goto out;
>> +    ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
>> +                strlen(ctrl->opts->subsysnqn));
>> +    if (ret)
>> +        goto out;
>> +    ret = crypto_shash_final(shash, chap->response);
>> +out:
>> +    return ret;
>> +}
>> +
>> +static int nvme_auth_dhchap_ctrl_response(struct nvme_ctrl *ctrl,
>> +                      struct nvme_dhchap_context *chap)
>> +{
>> +    SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
>> +    u8 buf[4], *challenge = chap->c2;
>> +    int ret;
>> +
>> +    dev_dbg(ctrl->device, "%s: qid %d host response seq %d 
>> transaction %d\n",
>> +        __func__, chap->qid, chap->s2, chap->transaction);
>> +    dev_dbg(ctrl->device, "%s: qid %d challenge %*ph\n",
>> +        __func__, chap->qid, chap->hash_len, challenge);
>> +    dev_dbg(ctrl->device, "%s: qid %d subsysnqn %s\n",
>> +        __func__, chap->qid, ctrl->opts->subsysnqn);
>> +    dev_dbg(ctrl->device, "%s: qid %d hostnqn %s\n",
>> +        __func__, chap->qid, ctrl->opts->host->nqn);
>> +    shash->tfm = chap->shash_tfm;
>> +    ret = crypto_shash_init(shash);
>> +    if (ret)
>> +        goto out;
>> +    ret = crypto_shash_update(shash, challenge, chap->hash_len);
>> +    if (ret)
>> +        goto out;
>> +    put_unaligned_le32(chap->s2, buf);
>> +    ret = crypto_shash_update(shash, buf, 4);
>> +    if (ret)
>> +        goto out;
>> +    put_unaligned_le16(chap->transaction, buf);
>> +    ret = crypto_shash_update(shash, buf, 2);
>> +    if (ret)
>> +        goto out;
>> +    memset(buf, 0, 4);
>> +    ret = crypto_shash_update(shash, buf, 1);
>> +    if (ret)
>> +        goto out;
>> +    ret = crypto_shash_update(shash, "Controller", 10);
>> +    if (ret)
>> +        goto out;
>> +    ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
>> +                  strlen(ctrl->opts->subsysnqn));
>> +    if (ret)
>> +        goto out;
>> +    ret = crypto_shash_update(shash, buf, 1);
>> +    if (ret)
>> +        goto out;
>> +    ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
>> +                  strlen(ctrl->opts->host->nqn));
>> +    if (ret)
>> +        goto out;
>> +    ret = crypto_shash_final(shash, chap->response);
>> +out:
>> +    return ret;
>> +}
>> +
>> +int nvme_auth_generate_key(struct nvme_ctrl *ctrl,
>> +               struct nvme_dhchap_context *chap)
>> +{
>> +    int ret;
>> +    u8 key_hash;
>> +    const char *hmac_name;
>> +    struct crypto_shash *key_tfm;
>> +
>> +    if (sscanf(ctrl->opts->dhchap_secret, "DHHC-1:%hhd:%*s:",
>> +           &key_hash) != 1)
>> +        return -EINVAL;
> 
> I'd expect that the user will pass in a secret key (as binary)
> the the driver will build the spec compliant formatted string no?
>  > Am I not reading this correctly?
> 

I'm under the impression that this is the format into which the 
User/Admin will get hold of the secret key.
Spec says:

'... all NVMe over Fabrics entities shall support the following ASCII
representation of secrets ...'

And as the userspace interface is the only way how the user/admin 
_could_ interact with the NVMe over Fabrics entities in Linux I guess 
we'll need to be able to parse it.

We sure could allow a binary secret, too, but then what would be the 
point in converting it into the secret representation?
The protocol revolves around the binary secret, not the transport 
representation.

>> +
>> +    chap->key = nvme_auth_extract_secret(ctrl->opts->dhchap_secret,
>> +                         &chap->key_len);
>> +    if (IS_ERR(chap->key)) {
>> +        ret = PTR_ERR(chap->key);
>> +        chap->key = NULL;
>> +        return ret;
>> +    }
>> +
>> +    if (key_hash == 0)
>> +        return 0;
>> +
>> +    hmac_name = nvme_auth_hmac_name(key_hash);
>> +    if (!hmac_name) {
>> +        pr_debug("Invalid key hash id %d\n", key_hash);
>> +        return -EKEYREJECTED;
>> +    }
> 
> Why does the user influence the hmac used? isn't that is driven
> by the susbsystem?
> 
> I don't think that the user should choose in this level.
> 

That is another weirdness of the spec.
The _secret_ will be hashed with a specific function, and that function 
is stated in the transport representation.
(Cf section "DH-HMAC-CHAP Security Requirements").
This is _not_ the hash function used by the authentication itself, which 
will be selected by the protocol.
So it's not the user here, but rather the transport specification of the 
key which selects the hash algorithm.

>> +
>> +    key_tfm = crypto_alloc_shash(hmac_name, 0, 0);
>> +    if (IS_ERR(key_tfm)) {
>> +        kfree(chap->key);
>> +        chap->key = NULL;
>> +        ret = PTR_ERR(key_tfm);
> 
> You set ret and later return 0? I think that the success
> path in the else clause is hard to read and error prone...
> 

Do I? Will need to fix it up.

>> +    } else {
>> +        SHASH_DESC_ON_STACK(shash, key_tfm);
>> +
>> +        shash->tfm = key_tfm;
>> +        ret = crypto_shash_setkey(key_tfm, chap->key,
>> +                      chap->key_len);
>> +        if (ret < 0) {
>> +            crypto_free_shash(key_tfm);
>> +            kfree(chap->key);
>> +            chap->key = NULL;
>> +            return ret;
>> +        }
>> +        crypto_shash_init(shash);
>> +        crypto_shash_update(shash, ctrl->opts->host->nqn,
>> +                    strlen(ctrl->opts->host->nqn));
>> +        crypto_shash_update(shash, "NVMe-over-Fabrics", 17);
>> +        crypto_shash_final(shash, chap->key);
>> +        crypto_free_shash(key_tfm);
> 
> Shouldn't these be done when preparing the dh-hmac-chap reply?
> 

By setting the hash here I avoid having to pass the required hash 
function for the secret transformation.
I could be doing the entire secret transformation thingie when preparing 
the reply; reason why I did it here is that _having_ a secret is the 
precondition to everything else, so I wanted to check upfront for that.
But I'll check what would happen if I move it.

>> +    }
>> +    return 0;
>> +}
>> +
>> +void nvme_auth_free(struct nvme_dhchap_context *chap)
>> +{
>> +    if (chap->shash_tfm)
>> +        crypto_free_shash(chap->shash_tfm);
>> +    if (chap->key)
>> +        kfree(chap->key);
>> +    if (chap->ctrl_key)
>> +        kfree(chap->ctrl_key);
>> +    if (chap->host_key)
>> +        kfree(chap->host_key);
>> +    if (chap->sess_key)
>> +        kfree(chap->sess_key);
> 
> No need to check null for kfree...
> 

Will be fixing it up.

>> +    kfree(chap);
>> +}
>> +
>> +int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid)
>> +{
>> +    struct nvme_dhchap_context *chap;
>> +    void *buf;
>> +    size_t buf_size, tl;
>> +    int ret = 0;
>> +
>> +    chap = kzalloc(sizeof(*chap), GFP_KERNEL);
>> +    if (!chap)
>> +        return -ENOMEM;
>> +    chap->qid = qid;
>> +    chap->transaction = ctrl->transaction++;
>> +
>> +    ret = nvme_auth_generate_key(ctrl, chap);
>> +    if (ret) {
>> +        dev_dbg(ctrl->device, "%s: failed to generate key, error %d\n",
>> +            __func__, ret);
>> +        nvme_auth_free(chap);
>> +        return ret;
>> +    }
>> +
>> +    /*
>> +     * Allocate a large enough buffer for the entire negotiation:
>> +     * 4k should be enough to ffdhe8192.
>> +     */
>> +    buf_size = 4096;
>> +    buf = kzalloc(buf_size, GFP_KERNEL);
>> +    if (!buf) {
>> +        ret = -ENOMEM;
>> +        goto out;
>> +    }
>> +
>> +    /* DH-HMAC-CHAP Step 1: send negotiate */
> 
> I'd consider breaking these into sub-routines.
> 

Which ones? The preparation step?
Sure, can do.

>> +    dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP negotiate\n",
>> +        __func__, qid);
>> +    ret = nvme_auth_dhchap_negotiate(ctrl, chap, buf, buf_size);
>> +    if (ret < 0)
>> +        goto out;
>> +    tl = ret;
>> +    ret = nvme_auth_send(ctrl, qid, buf, tl);
>> +    if (ret)
>> +        goto out;
>> +
>> +    memset(buf, 0, buf_size);
>> +    ret = nvme_auth_receive(ctrl, qid, buf, buf_size, chap->transaction,
>> +                NVME_AUTH_DHCHAP_MESSAGE_CHALLENGE);
>> +    if (ret < 0) {
>> +        dev_dbg(ctrl->device,
>> +            "%s: qid %d DH-HMAC-CHAP failed to receive challenge\n",
>> +            __func__, qid);
>> +        goto out;
>> +    }
>> +    if (ret > 0) {
>> +        chap->status = ret;
>> +        goto fail1;
>> +    }
>> +
>> +    /* DH-HMAC-CHAP Step 2: receive challenge */
>> +    dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP challenge\n",
>> +        __func__, qid);
>> +
>> +    ret = nvme_auth_dhchap_challenge(ctrl, chap, buf, buf_size);
>> +    if (ret) {
>> +        /* Invalid parameters for negotiate */
>> +        goto fail2;
>> +    }
>> +
>> +    dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP select hash\n",
>> +        __func__, qid);
>> +    ret = nvme_auth_select_hash(ctrl, chap);
>> +    if (ret)
>> +        goto fail2;
>> +
>> +    dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP host response\n",
>> +        __func__, qid);
>> +    ret = nvme_auth_dhchap_host_response(ctrl, chap);
>> +    if (ret)
>> +        goto fail2;
>> +
>> +    /* DH-HMAC-CHAP Step 3: send reply */
>> +    dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP reply\n",
>> +        __func__, qid);
>> +    ret = nvme_auth_dhchap_reply(ctrl, chap, buf, buf_size);
>> +    if (ret < 0)
>> +        goto fail2;
>> +
>> +    tl = ret;
>> +    ret = nvme_auth_send(ctrl, qid, buf, tl);
>> +    if (ret)
>> +        goto fail2;
>> +
>> +    memset(buf, 0, buf_size);
>> +    ret = nvme_auth_receive(ctrl, qid, buf, buf_size, chap->transaction,
>> +                NVME_AUTH_DHCHAP_MESSAGE_SUCCESS1);
>> +    if (ret < 0) {
>> +        dev_dbg(ctrl->device,
>> +            "%s: qid %d DH-HMAC-CHAP failed to receive success1\n",
>> +            __func__, qid);
>> +        goto out;
>> +    }
>> +    if (ret > 0) {
>> +        chap->status = ret;
>> +        goto fail1;
>> +    }
>> +
>> +    if (ctrl->opts->dhchap_auth) {
>> +        dev_dbg(ctrl->device,
>> +            "%s: qid %d DH-HMAC-CHAP controller response\n",
>> +            __func__, qid);
>> +        ret = nvme_auth_dhchap_ctrl_response(ctrl, chap);
>> +        if (ret)
>> +            goto fail2;
>> +    }
>> +
>> +    /* DH-HMAC-CHAP Step 4: receive success1 */
>> +    dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP success1\n",
>> +        __func__, qid);
>> +    ret = nvme_auth_dhchap_success1(ctrl, chap, buf, buf_size);
>> +    if (ret < 0) {
>> +        /* Controller authentication failed */
>> +        goto fail2;
>> +    }
>> +    tl = ret;
>> +    /* DH-HMAC-CHAP Step 5: send success2 */
>> +    dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP success2\n",
>> +        __func__, qid);
>> +    tl = nvme_auth_dhchap_success2(ctrl, chap, buf, buf_size);
>> +    ret = nvme_auth_send(ctrl, qid, buf, tl);
>> +    if (!ret)
>> +        goto out;
>> +
>> +fail1:
>> +    dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP failure1, status 
>> %x\n",
>> +        __func__, qid, chap->status);
>> +    goto out;
>> +
>> +fail2:
>> +    dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP failure2, status 
>> %x\n",
>> +        __func__, qid, chap->status);
>> +    tl = nvme_auth_dhchap_failure2(ctrl, chap, buf, buf_size);
>> +    ret = nvme_auth_send(ctrl, qid, buf, tl);
>> +
>> +out:
>> +    if (!ret && chap->status)
>> +        ret = -EPROTO;
>> +    if (!ret) {
>> +        ctrl->dhchap_hash = chap->hash_id;
>> +    }
>> +    kfree(buf);
>> +    nvme_auth_free(chap);
>> +    return ret;
>> +}
>> diff --git a/drivers/nvme/host/auth.h b/drivers/nvme/host/auth.h
>> new file mode 100644
>> index 000000000000..4950b1cb9470
>> --- /dev/null
>> +++ b/drivers/nvme/host/auth.h
>> @@ -0,0 +1,23 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +/*
>> + * Copyright (c) 2021 Hannes Reinecke, SUSE Software Solutions
>> + */
>> +
>> +#ifndef _NVME_AUTH_H
>> +#define _NVME_AUTH_H
>> +
>> +const char *nvme_auth_dhgroup_name(int dhgroup_id);
>> +int nvme_auth_dhgroup_pubkey_size(int dhgroup_id);
>> +int nvme_auth_dhgroup_privkey_size(int dhgroup_id);
>> +const char *nvme_auth_dhgroup_kpp(int dhgroup_id);
>> +int nvme_auth_dhgroup_id(const char *dhgroup_name);
>> +
>> +const char *nvme_auth_hmac_name(int hmac_id);
>> +const char *nvme_auth_digest_name(int hmac_id);
>> +int nvme_auth_hmac_id(const char *hmac_name);
>> +int nvme_auth_hmac_len(int hmac_len);
>> +
>> +unsigned char *nvme_auth_extract_secret(unsigned char *dhchap_secret,
>> +                    size_t *dhchap_key_len);
>> +
>> +#endif /* _NVME_AUTH_H */
>> diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
>> index 11779be42186..7ce9b666dc09 100644
>> --- a/drivers/nvme/host/core.c
>> +++ b/drivers/nvme/host/core.c
>> @@ -708,7 +708,9 @@ bool __nvme_check_ready(struct nvme_ctrl *ctrl, 
>> struct request *rq,
>>           switch (ctrl->state) {
>>           case NVME_CTRL_CONNECTING:
>>               if (blk_rq_is_passthrough(rq) && 
>> nvme_is_fabrics(req->cmd) &&
>> -                req->cmd->fabrics.fctype == nvme_fabrics_type_connect)
>> +                (req->cmd->fabrics.fctype == 
>> nvme_fabrics_type_connect ||
>> +                 req->cmd->fabrics.fctype == 
>> nvme_fabrics_type_auth_send ||
>> +                 req->cmd->fabrics.fctype == 
>> nvme_fabrics_type_auth_receive))
>>                   return true;
>>               break;
>>           default:
>> @@ -3426,6 +3428,66 @@ static ssize_t 
>> nvme_ctrl_fast_io_fail_tmo_store(struct device *dev,
>>   static DEVICE_ATTR(fast_io_fail_tmo, S_IRUGO | S_IWUSR,
>>       nvme_ctrl_fast_io_fail_tmo_show, nvme_ctrl_fast_io_fail_tmo_store);
>> +#ifdef CONFIG_NVME_AUTH
>> +struct nvmet_dhchap_hash_map {
>> +    int id;
>> +    const char name[15];
>> +} hash_map[] = {
>> +    {.id = NVME_AUTH_DHCHAP_HASH_SHA256,
>> +     .name = "hmac(sha256)", },
>> +    {.id = NVME_AUTH_DHCHAP_HASH_SHA384,
>> +     .name = "hmac(sha384)", },
>> +    {.id = NVME_AUTH_DHCHAP_HASH_SHA512,
>> +     .name = "hmac(sha512)", },
>> +};
>> +
>> +static ssize_t dhchap_hash_show(struct device *dev,
>> +    struct device_attribute *attr, char *buf)
>> +{
>> +    struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
>> +    int i;
>> +
>> +    for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
>> +        if (hash_map[i].id == ctrl->dhchap_hash)
>> +            return sprintf(buf, "%s\n", hash_map[i].name);
>> +    }
>> +    return sprintf(buf, "none\n");
>> +}
>> +DEVICE_ATTR_RO(dhchap_hash);
>> +
>> +struct nvmet_dhchap_group_map {
>> +    int id;
>> +    const char name[15];
>> +} dhgroup_map[] = {
>> +    {.id = NVME_AUTH_DHCHAP_DHGROUP_NULL,
>> +     .name = "NULL", },
>> +    {.id = NVME_AUTH_DHCHAP_DHGROUP_2048,
>> +     .name = "ffdhe2048", },
>> +    {.id = NVME_AUTH_DHCHAP_DHGROUP_3072,
>> +     .name = "ffdhe3072", },
>> +    {.id = NVME_AUTH_DHCHAP_DHGROUP_4096,
>> +     .name = "ffdhe4096", },
>> +    {.id = NVME_AUTH_DHCHAP_DHGROUP_6144,
>> +     .name = "ffdhe6144", },
>> +    {.id = NVME_AUTH_DHCHAP_DHGROUP_8192,
>> +     .name = "ffdhe8192", },
>> +};
>> +
>> +static ssize_t dhchap_dhgroup_show(struct device *dev,
>> +    struct device_attribute *attr, char *buf)
>> +{
>> +    struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
>> +    int i;
>> +
>> +    for (i = 0; i < ARRAY_SIZE(dhgroup_map); i++) {
>> +        if (hash_map[i].id == ctrl->dhchap_dhgroup)
>> +            return sprintf(buf, "%s\n", dhgroup_map[i].name);
>> +    }
>> +    return sprintf(buf, "none\n");
>> +}
>> +DEVICE_ATTR_RO(dhchap_dhgroup);
>> +#endif
>> +
>>   static struct attribute *nvme_dev_attrs[] = {
>>       &dev_attr_reset_controller.attr,
>>       &dev_attr_rescan_controller.attr,
>> @@ -3447,6 +3509,10 @@ static struct attribute *nvme_dev_attrs[] = {
>>       &dev_attr_reconnect_delay.attr,
>>       &dev_attr_fast_io_fail_tmo.attr,
>>       &dev_attr_kato.attr,
>> +#ifdef CONFIG_NVME_AUTH
>> +    &dev_attr_dhchap_hash.attr,
>> +    &dev_attr_dhchap_dhgroup.attr,
>> +#endif
>>       NULL
>>   };
>> @@ -3470,6 +3536,10 @@ static umode_t 
>> nvme_dev_attrs_are_visible(struct kobject *kobj,
>>           return 0;
>>       if (a == &dev_attr_fast_io_fail_tmo.attr && !ctrl->opts)
>>           return 0;
>> +#ifdef CONFIG_NVME_AUTH
>> +    if (a == &dev_attr_dhchap_hash.attr && !ctrl->opts)
>> +        return 0;
>> +#endif
>>       return a->mode;
>>   }
>> @@ -4581,6 +4651,11 @@ static inline void _nvme_check_size(void)
>>       BUILD_BUG_ON(sizeof(struct nvme_smart_log) != 512);
>>       BUILD_BUG_ON(sizeof(struct nvme_dbbuf) != 64);
>>       BUILD_BUG_ON(sizeof(struct nvme_directive_cmd) != 64);
>> +    BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_negotiate_data) != 8);
>> +    BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_challenge_data) != 16);
>> +    BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_reply_data) != 16);
>> +    BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_success1_data) != 16);
>> +    BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_success2_data) != 16);
>>   }
>> diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
>> index a5469fd9d4c3..6404ab9b604b 100644
>> --- a/drivers/nvme/host/fabrics.c
>> +++ b/drivers/nvme/host/fabrics.c
>> @@ -366,6 +366,7 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)
>>       union nvme_result res;
>>       struct nvmf_connect_data *data;
>>       int ret;
>> +    u32 result;
>>       cmd.connect.opcode = nvme_fabrics_command;
>>       cmd.connect.fctype = nvme_fabrics_type_connect;
>> @@ -398,8 +399,18 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)
>>           goto out_free_data;
>>       }
>> -    ctrl->cntlid = le16_to_cpu(res.u16);
>> -
>> +    result = le32_to_cpu(res.u32);
>> +    ctrl->cntlid = result & 0xFFFF;
>> +    if ((result >> 16) & 2) {
>> +        /* Authentication required */
>> +        ret = nvme_auth_negotiate(ctrl, NVME_QID_ANY);
>> +        if (ret)
>> +            dev_warn(ctrl->device,
>> +                 "qid 0: authentication failed\n");
>> +        else
>> +            dev_info(ctrl->device,
>> +                 "qid 0: authenticated\n");
> 
> info is too chatty.
> 

Hmm. I know I need to work on logging...

>> +    }
>>   out_free_data:
>>       kfree(data);
>>       return ret;
>> @@ -432,6 +443,7 @@ int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, 
>> u16 qid)
>>       struct nvmf_connect_data *data;
>>       union nvme_result res;
>>       int ret;
>> +    u32 result;
>>       cmd.connect.opcode = nvme_fabrics_command;
>>       cmd.connect.fctype = nvme_fabrics_type_connect;
>> @@ -457,6 +469,17 @@ int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, 
>> u16 qid)
>>           nvmf_log_connect_error(ctrl, ret, le32_to_cpu(res.u32),
>>                          &cmd, data);
>>       }
>> +    result = le32_to_cpu(res.u32);
>> +    if ((result >> 16) & 2) {
>> +        /* Authentication required */
>> +        ret = nvme_auth_negotiate(ctrl, qid);
>> +        if (ret)
>> +            dev_warn(ctrl->device,
>> +                 "qid %u: authentication failed\n", qid);
>> +        else
>> +            dev_info(ctrl->device,
>> +                 "qid %u: authenticated\n", qid);
>> +    }
>>       kfree(data);
>>       return ret;
>>   }
>> @@ -548,6 +571,9 @@ static const match_table_t opt_tokens = {
>>       { NVMF_OPT_NR_POLL_QUEUES,    "nr_poll_queues=%d"    },
>>       { NVMF_OPT_TOS,            "tos=%d"        },
>>       { NVMF_OPT_FAIL_FAST_TMO,    "fast_io_fail_tmo=%d"    },
>> +    { NVMF_OPT_DHCHAP_SECRET,    "dhchap_secret=%s"    },
>> +    { NVMF_OPT_DHCHAP_AUTH,        "authenticate"        },
>> +    { NVMF_OPT_DHCHAP_GROUP,    "dhchap_group=%s"    },
> 
> Isn't the group driven by the subsystem? also why is there a
> "authenticate" boolean? what is it good for?
> 
Ah. Right. Of course, the 'group' is pointless here.
And the 'authenticate' bool is the abovementioned bidirectional 
authentication.
I _do_ need to give it another name.

>>       { NVMF_OPT_ERR,            NULL            }
>>   };
>> @@ -824,6 +850,35 @@ static int nvmf_parse_options(struct 
>> nvmf_ctrl_options *opts,
>>               }
>>               opts->tos = token;
>>               break;
>> +        case NVMF_OPT_DHCHAP_SECRET:
>> +            p = match_strdup(args);
>> +            if (!p) {
>> +                ret = -ENOMEM;
>> +                goto out;
>> +            }
>> +            if (strncmp(p, "DHHC-1:00:", 10)) {
>> +                pr_err("Invalid DH-CHAP secret %s\n", p);
>> +                ret = -EINVAL;
>> +                goto out;
>> +            }
>> +            kfree(opts->dhchap_secret);
>> +            opts->dhchap_secret = p;
>> +            break;
>> +        case NVMF_OPT_DHCHAP_AUTH:
>> +            opts->dhchap_auth = true;
>> +            break;
>> +        case NVMF_OPT_DHCHAP_GROUP:
>> +            if (match_int(args, &token)) {
>> +                ret = -EINVAL;
>> +                goto out;
>> +            }
>> +            if (token <= 0) {
>> +                pr_err("Invalid dhchap_group %d\n", token);
>> +                ret = -EINVAL;
>> +                goto out;
>> +            }
>> +            opts->dhchap_group = token;
>> +            break;
>>           default:
>>               pr_warn("unknown parameter or missing value '%s' in ctrl 
>> creation request\n",
>>                   p);
>> @@ -942,6 +997,7 @@ void nvmf_free_options(struct nvmf_ctrl_options 
>> *opts)
>>       kfree(opts->subsysnqn);
>>       kfree(opts->host_traddr);
>>       kfree(opts->host_iface);
>> +    kfree(opts->dhchap_secret);
>>       kfree(opts);
>>   }
>>   EXPORT_SYMBOL_GPL(nvmf_free_options);
>> @@ -951,7 +1007,10 @@ EXPORT_SYMBOL_GPL(nvmf_free_options);
>>                    NVMF_OPT_KATO | NVMF_OPT_HOSTNQN | \
>>                    NVMF_OPT_HOST_ID | NVMF_OPT_DUP_CONNECT |\
>>                    NVMF_OPT_DISABLE_SQFLOW |\
>> -                 NVMF_OPT_FAIL_FAST_TMO)
>> +                 NVMF_OPT_CTRL_LOSS_TMO |\
>> +                 NVMF_OPT_FAIL_FAST_TMO |\
>> +                 NVMF_OPT_DHCHAP_SECRET |\
>> +                 NVMF_OPT_DHCHAP_AUTH | NVMF_OPT_DHCHAP_GROUP)
>>   static struct nvme_ctrl *
>>   nvmf_create_ctrl(struct device *dev, const char *buf)
>> diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h
>> index a146cb903869..535bc544f0f6 100644
>> --- a/drivers/nvme/host/fabrics.h
>> +++ b/drivers/nvme/host/fabrics.h
>> @@ -67,6 +67,9 @@ enum {
>>       NVMF_OPT_TOS        = 1 << 19,
>>       NVMF_OPT_FAIL_FAST_TMO    = 1 << 20,
>>       NVMF_OPT_HOST_IFACE    = 1 << 21,
>> +    NVMF_OPT_DHCHAP_SECRET    = 1 << 22,
>> +    NVMF_OPT_DHCHAP_AUTH    = 1 << 23,
>> +    NVMF_OPT_DHCHAP_GROUP    = 1 << 24,
>>   };
>>   /**
>> @@ -96,6 +99,8 @@ enum {
>>    * @max_reconnects: maximum number of allowed reconnect attempts 
>> before removing
>>    *              the controller, (-1) means reconnect forever, zero 
>> means remove
>>    *              immediately;
>> + * @dhchap_secret: DH-HMAC-CHAP secret
>> + * @dhchap_auth: DH-HMAC-CHAP authenticate controller
>>    * @disable_sqflow: disable controller sq flow control
>>    * @hdr_digest: generate/verify header digest (TCP)
>>    * @data_digest: generate/verify data digest (TCP)
>> @@ -120,6 +125,9 @@ struct nvmf_ctrl_options {
>>       unsigned int        kato;
>>       struct nvmf_host    *host;
>>       int            max_reconnects;
>> +    char            *dhchap_secret;
>> +    int            dhchap_group;
>> +    bool            dhchap_auth;
>>       bool            disable_sqflow;
>>       bool            hdr_digest;
>>       bool            data_digest;
>> diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
>> index 18ef8dd03a90..bcd5b8276c26 100644
>> --- a/drivers/nvme/host/nvme.h
>> +++ b/drivers/nvme/host/nvme.h
>> @@ -328,6 +328,12 @@ struct nvme_ctrl {
>>       struct work_struct ana_work;
>>   #endif
>> +#ifdef CONFIG_NVME_AUTH
>> +    u16 transaction;
>> +    u8 dhchap_hash;
>> +    u8 dhchap_dhgroup;
> 
> Do multiple controllers in the same subsystem have different
> params? no, so I think these should be lifted to subsys.
> 

It doesn't actually say in the spec; it always refers to the params as 
being set by the controller.
So it could be either; maybe we should ask for clafication at the fmds call.

>> +#endif
>> +
>>       /* Power saving configuration */
>>       u64 ps_max_latency_us;
>>       bool apst_enabled;
>> @@ -874,6 +880,15 @@ static inline bool nvme_ctrl_sgl_supported(struct 
>> nvme_ctrl *ctrl)
>>       return ctrl->sgls & ((1 << 0) | (1 << 1));
>>   }
>> +#ifdef CONFIG_NVME_AUTH
>> +int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid);
>> +#else
>> +static inline int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid)
>> +{
>> +    return -EPROTONOSUPPORT;
>> +}
>> +#endif
>> +
>>   u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
>>                u8 opcode);
>>   int nvme_execute_passthru_rq(struct request *rq);
>> diff --git a/drivers/nvme/host/trace.c b/drivers/nvme/host/trace.c
>> index 6543015b6121..66f75d8ea925 100644
>> --- a/drivers/nvme/host/trace.c
>> +++ b/drivers/nvme/host/trace.c
> 
> I'd split out the tracing logic.
> 
Okay.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer

Re: [PATCH 06/11] nvme: Implement In-Band authentication

[Hannes Reinecke]

On 7/17/21 9:22 AM, Sagi Grimberg wrote:
>> Implement NVMe-oF In-Band authentication. This patch adds two new
>> fabric options 'dhchap_key' to specify the PSK
> 
> pre-shared-key.
> 
> Also, we need a sysfs knob to rotate the key that will trigger
> re-authentication or even a simple controller(s-plural) reset, so this
> should go beyond just the connection string.
> 

Yeah, re-authentication currently is not implemented. I first wanted to 
get this patchset out such that we can settle on the userspace interface 
(both from host and target).
I'll have to think on how we should handle authentication; one of the 
really interesting cases would be when one malicious admin will _just_ 
send a 'negotiate' command to the controller. As per spec the controller 
will be waiting for an 'authentication receive' command to send a 
'challenge' payload back to the host. But that will never come, so as it 
stands currently the controller is required to abort the connection.
Not very nice.

> P.S. can you add also the nvme-cli code in the next go?
> 
Oh, sure. It's already sitting around in my local repo (surprise, 
surprise); will be ending it out next time.

>> and 'dhchap_authenticate'
>> to request bi-directional authentication of both the host and the 
>> controller.
> 
> bidirectional? not uni-directional?
> 

Yeah, that's a bit of a misnomer. When a PSK is specified, the 
controller will start the authentication protocol such that the 
_controller_ can validate the host. If the host wants to authenticate 
the controller is needs to set this flag.
Hence bi-directional authentication.
But I'm the first to admit that this is poor wording for the flag.

>>
>> Signed-off-by: Hannes Reinecke <hare@suse.de>
>> ---
>>   drivers/nvme/host/Kconfig   |  11 +
>>   drivers/nvme/host/Makefile  |   1 +
>>   drivers/nvme/host/auth.c    | 813 ++++++++++++++++++++++++++++++++++++
>>   drivers/nvme/host/auth.h    |  23 +
>>   drivers/nvme/host/core.c    |  77 +++-
>>   drivers/nvme/host/fabrics.c |  65 ++-
>>   drivers/nvme/host/fabrics.h |   8 +
>>   drivers/nvme/host/nvme.h    |  15 +
>>   drivers/nvme/host/trace.c   |  32 ++
>>   9 files changed, 1041 insertions(+), 4 deletions(-)
>>   create mode 100644 drivers/nvme/host/auth.c
>>   create mode 100644 drivers/nvme/host/auth.h
>>
>> diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig
>> index c3f3d77f1aac..853c546305e9 100644
>> --- a/drivers/nvme/host/Kconfig
>> +++ b/drivers/nvme/host/Kconfig
>> @@ -85,3 +85,14 @@ config NVME_TCP
>>         from https://github.com/linux-nvme/nvme-cli.
>>         If unsure, say N.
>> +
>> +config NVME_AUTH
>> +    bool "NVM Express over Fabrics In-Band Authentication"
>> +    depends on NVME_TCP
>> +    select CRYPTO_SHA256
>> +    select CRYPTO_SHA512
>> +    help
>> +      This provides support for NVMe over Fabrics In-Band Authentication
>> +      for the NVMe over TCP transport.
> 
> In this form, nothing is specific to nvme-tcp here afaict.
> 

Indeed. I guess we can leave out the nvme-tcp reference here.

>> +
>> +      If unsure, say N.
>> diff --git a/drivers/nvme/host/Makefile b/drivers/nvme/host/Makefile
>> index cbc509784b2e..03748a55a12b 100644
>> --- a/drivers/nvme/host/Makefile
>> +++ b/drivers/nvme/host/Makefile
>> @@ -20,6 +20,7 @@ nvme-core-$(CONFIG_NVME_HWMON)        += hwmon.o
>>   nvme-y                    += pci.o
>>   nvme-fabrics-y                += fabrics.o
>> +nvme-fabrics-$(CONFIG_NVME_AUTH)    += auth.o
>>   nvme-rdma-y                += rdma.o
>> diff --git a/drivers/nvme/host/auth.c b/drivers/nvme/host/auth.c
>> new file mode 100644
>> index 000000000000..448a3adebea6
>> --- /dev/null
>> +++ b/drivers/nvme/host/auth.c
>> @@ -0,0 +1,813 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * Copyright (c) 2020 Hannes Reinecke, SUSE Linux
>> + */
>> +
>> +#include <linux/crc32.h>
>> +#include <linux/base64.h>
>> +#include <asm/unaligned.h>
>> +#include <crypto/hash.h>
>> +#include <crypto/kpp.h>
>> +#include "nvme.h"
>> +#include "fabrics.h"
>> +#include "auth.h"
>> +
>> +static u32 nvme_dhchap_seqnum;
>> +
>> +struct nvme_dhchap_context {
> 
> Maybe nvme_dhchap_queue_context ?
> 
> I'm thinking that we should perhaps split
> it to host-wide, subsys-wide and queue specific
> auth contexts?
> 
> Let's see...
> 

Interestingly enough, that's what I did for the target side.
For the host side I found it easier that way, as then we'll have a 
single authentication context which can be deleted after authentication 
finished, and be sure that we removed _all_ information.
Security and all that.
Splitting it off would require to remove information on three different 
places, and observing life-time rules for them.
So more of a chance to mess things up :-)

>> +    struct crypto_shash *shash_tfm;
>> +    unsigned char *key;
>> +    size_t key_len;
>> +    int qid;
>> +    u32 s1;
>> +    u32 s2;
>> +    u16 transaction;
>> +    u8 status;
>> +    u8 hash_id;
>> +    u8 hash_len;
>> +    u8 c1[64];
>> +    u8 c2[64];
>> +    u8 response[64];
>> +    u8 *ctrl_key;
>> +    int ctrl_key_len;
>> +    u8 *host_key;
>> +    int host_key_len;
>> +    u8 *sess_key;
>> +    int sess_key_len;
>> +};
>> +
>> +struct nvmet_dhchap_hash_map {
> 
> nvmet?
> 

Yeah; originally I coded that for the target side, and only later moved 
it into the host side to have it usable for both.
Will be fixing it up.

>> +    int id;
>> +    int hash_len;
>> +    const char hmac[15];
>> +    const char digest[15];
>> +} hash_map[] = {
>> +    {.id = NVME_AUTH_DHCHAP_HASH_SHA256,
>> +     .hash_len = 32,
>> +     .hmac = "hmac(sha256)", .digest = "sha256" },
>> +    {.id = NVME_AUTH_DHCHAP_HASH_SHA384,
>> +     .hash_len = 48,
>> +     .hmac = "hmac(sha384)", .digest = "sha384" },
>> +    {.id = NVME_AUTH_DHCHAP_HASH_SHA512,
>> +     .hash_len = 64,
>> +     .hmac = "hmac(sha512)", .digest = "sha512" },
>> +};
>> +
>> +const char *nvme_auth_hmac_name(int hmac_id)
> 
> Should these arrays be static?
> 

Definitely.

>> +{
>> +    int i;
>> +
>> +    for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
>> +        if (hash_map[i].id == hmac_id)
>> +            return hash_map[i].hmac;
>> +    }
>> +    return NULL;
>> +}
>> +EXPORT_SYMBOL_GPL(nvme_auth_hmac_name);
>> +
>> +const char *nvme_auth_digest_name(int hmac_id)
>> +{
>> +    int i;
>> +
>> +    for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
>> +        if (hash_map[i].id == hmac_id)
>> +            return hash_map[i].digest;
>> +    }
>> +    return NULL;
>> +}
>> +EXPORT_SYMBOL_GPL(nvme_auth_digest_name);
>> +
>> +int nvme_auth_hmac_len(int hmac_id)
>> +{
>> +    int i;
>> +
>> +    for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
>> +        if (hash_map[i].id == hmac_id)
>> +            return hash_map[i].hash_len;
>> +    }
>> +    return -1;
>> +}
>> +EXPORT_SYMBOL_GPL(nvme_auth_hmac_len);
>> +
>> +int nvme_auth_hmac_id(const char *hmac_name)
>> +{
>> +    int i;
>> +
>> +    for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
>> +        if (!strncmp(hash_map[i].hmac, hmac_name,
>> +                 strlen(hash_map[i].hmac)))
>> +            return hash_map[i].id;
>> +    }
>> +    return -1;
>> +}
>> +EXPORT_SYMBOL_GPL(nvme_auth_hmac_id);
>> +
>> +unsigned char *nvme_auth_extract_secret(unsigned char *dhchap_secret,
>> +                    size_t *dhchap_key_len)
>> +{
>> +    unsigned char *dhchap_key;
>> +    u32 crc;
>> +    int key_len;
>> +    size_t allocated_len;
>> +
>> +    allocated_len = strlen(dhchap_secret) - 10;
> 
> the 10 feels like a magic here, should at least note this is the
> "DHHC-1:..." prefix.
> 

It _is_ magic. And it might even be better to just pass in the string 
_without_ the DHHC-1: prefix.

>> +    dhchap_key = kzalloc(allocated_len, GFP_KERNEL);
>> +    if (!dhchap_key)
>> +        return ERR_PTR(-ENOMEM);
>> +
>> +    key_len = base64_decode(dhchap_secret + 10,
>> +                allocated_len, dhchap_key);
>> +    if (key_len != 36 && key_len != 52 &&
>> +        key_len != 68) {
>> +        pr_debug("Invalid DH-HMAC-CHAP key len %d\n",
>> +             key_len);
>> +        kfree(dhchap_key);
>> +        return ERR_PTR(-EINVAL);
>> +    }
>> +    pr_debug("DH-HMAC-CHAP Key: %*ph\n",
>> +         (int)key_len, dhchap_key);
> 
> One can argue if even printing this is problematic..
> 

Debugging scaffolding. You wouldn't believe how many things can go wrong...

And yes, that should be removed.

>> +
>> +    /* The last four bytes is the CRC in little-endian format */
>> +    key_len -= 4;
>> +    /*
>> +     * The linux implementation doesn't do pre- and post-increments,
>> +     * so we have to do it manually.
>> +     */
>> +    crc = ~crc32(~0, dhchap_key, key_len);
>> +
>> +    if (get_unaligned_le32(dhchap_key + key_len) != crc) {
>> +        pr_debug("DH-HMAC-CHAP crc mismatch (key %08x, crc %08x)\n",
>> +               get_unaligned_le32(dhchap_key + key_len), crc);
>> +        kfree(dhchap_key);
>> +        return ERR_PTR(-EKEYREJECTED);
>> +    }
>> +    *dhchap_key_len = key_len;
>> +    return dhchap_key;
>> +}
>> +EXPORT_SYMBOL_GPL(nvme_auth_extract_secret);
>> +
>> +static int nvme_auth_send(struct nvme_ctrl *ctrl, int qid,
>> +              void *data, size_t tl)
>> +{
>> +    struct nvme_command cmd = {};
>> +    blk_mq_req_flags_t flags = qid == NVME_QID_ANY ?
>> +        0 : BLK_MQ_REQ_NOWAIT | BLK_MQ_REQ_RESERVED;
>> +    struct request_queue *q = qid == NVME_QID_ANY ?
>> +        ctrl->fabrics_q : ctrl->connect_q;
>> +    int ret;
>> +
>> +    cmd.auth_send.opcode = nvme_fabrics_command;
>> +    cmd.auth_send.fctype = nvme_fabrics_type_auth_send;
>> +    cmd.auth_send.secp = NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER;
>> +    cmd.auth_send.spsp0 = 0x01;
>> +    cmd.auth_send.spsp1 = 0x01;
>> +    cmd.auth_send.tl = tl;
>> +
>> +    ret = __nvme_submit_sync_cmd(q, &cmd, NULL, data, tl, 0, qid,
>> +                     0, flags);
>> +    if (ret)
>> +        dev_dbg(ctrl->device,
>> +            "%s: qid %d error %d\n", __func__, qid, ret);
> 
> Maybe a little more informative print rather than __func__ ?
> 

Yes, can do.

>> +    return ret;
>> +}
>> +
>> +static int nvme_auth_receive(struct nvme_ctrl *ctrl, int qid,
>> +                 void *buf, size_t al,
>> +                 u16 transaction, u8 expected_msg )
>> +{
>> +    struct nvme_command cmd = {};
>> +    struct nvmf_auth_dhchap_failure_data *data = buf;
>> +    blk_mq_req_flags_t flags = qid == NVME_QID_ANY ?
>> +        0 : BLK_MQ_REQ_NOWAIT | BLK_MQ_REQ_RESERVED;
>> +    struct request_queue *q = qid == NVME_QID_ANY ?
>> +        ctrl->fabrics_q : ctrl->connect_q;
>> +    int ret;
>> +
>> +    cmd.auth_receive.opcode = nvme_fabrics_command;
>> +    cmd.auth_receive.fctype = nvme_fabrics_type_auth_receive;
>> +    cmd.auth_receive.secp = NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER;
>> +    cmd.auth_receive.spsp0 = 0x01;
>> +    cmd.auth_receive.spsp1 = 0x01;
>> +    cmd.auth_receive.al = al;
>> +
>> +    ret = __nvme_submit_sync_cmd(q, &cmd, NULL, buf, al, 0, qid,
>> +                     0, flags);
>> +    if (ret > 0) {
>> +        dev_dbg(ctrl->device, "%s: qid %d nvme status %x\n",
>> +            __func__, qid, ret);
>> +        ret = -EIO;
>> +    }
>> +    if (ret < 0) {
>> +        dev_dbg(ctrl->device, "%s: qid %d error %d\n",
>> +            __func__, qid, ret);
>> +        return ret;
>> +    }
>> +    dev_dbg(ctrl->device, "%s: qid %d auth_type %d auth_id %x\n",
>> +        __func__, qid, data->auth_type, data->auth_id);
>> +    if (data->auth_type == NVME_AUTH_COMMON_MESSAGES &&
>> +        data->auth_id == NVME_AUTH_DHCHAP_MESSAGE_FAILURE1) {
>> +        return data->reason_code_explanation;
>> +    }
>> +    if (data->auth_type != NVME_AUTH_DHCHAP_MESSAGES ||
>> +        data->auth_id != expected_msg) {
>> +        dev_warn(ctrl->device,
>> +             "qid %d invalid message %02x/%02x\n",
>> +             qid, data->auth_type, data->auth_id);
>> +        return NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
>> +    }
>> +    if (le16_to_cpu(data->t_id) != transaction) {
>> +        dev_warn(ctrl->device,
>> +             "qid %d invalid transaction ID %d\n",
>> +             qid, le16_to_cpu(data->t_id));
>> +        return NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
>> +    }
>> +
>> +    return 0;
>> +}
>> +
>> +static int nvme_auth_dhchap_negotiate(struct nvme_ctrl *ctrl,
>> +                      struct nvme_dhchap_context *chap,
>> +                      void *buf, size_t buf_size)
> 
> Maybe nvme_auth_set_dhchap_negotiate_data ?
> 

These are the individual steps in the state machine later on, so I 
wanted to keep the names identical.
But I'm open to suggestions.

>> +{
>> +    struct nvmf_auth_dhchap_negotiate_data *data = buf;
>> +    size_t size = sizeof(*data) + sizeof(union nvmf_auth_protocol);
>> +
>> +    if (buf_size < size)
>> +        return -EINVAL;
>> +
>> +    memset((u8 *)buf, 0, size);
>> +    data->auth_type = NVME_AUTH_COMMON_MESSAGES;
>> +    data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE;
>> +    data->t_id = cpu_to_le16(chap->transaction);
>> +    data->sc_c = 0; /* No secure channel concatenation */
>> +    data->napd = 1;
>> +    data->auth_protocol[0].dhchap.authid = NVME_AUTH_DHCHAP_AUTH_ID;
>> +    data->auth_protocol[0].dhchap.halen = 3;
>> +    data->auth_protocol[0].dhchap.dhlen = 1;
>> +    data->auth_protocol[0].dhchap.idlist[0] = 
>> NVME_AUTH_DHCHAP_HASH_SHA256;
>> +    data->auth_protocol[0].dhchap.idlist[1] = 
>> NVME_AUTH_DHCHAP_HASH_SHA384;
>> +    data->auth_protocol[0].dhchap.idlist[2] = 
>> NVME_AUTH_DHCHAP_HASH_SHA512;
>> +    data->auth_protocol[0].dhchap.idlist[3] = 
>> NVME_AUTH_DHCHAP_DHGROUP_NULL;
> You should comment that this routine expects buf to have enough
> room for both negotiate and auth_proto structures.
> 
Hmm. I do a check for the overall size at the start, so I'm not sure 
what this will buy us.
And actually, anyone wanting to make sense of the implementation would 
need to look at the spec anyway.

>> +
>> +    return size;
>> +}
>> +
>> +static int nvme_auth_dhchap_challenge(struct nvme_ctrl *ctrl,
>> +                      struct nvme_dhchap_context *chap,
>> +                      void *buf, size_t buf_size)
> 
> Maybe nvme_auth_process_dhchap_challange ?
> 

See above. I'd rather have consistent names for the state machine.
But I can change them to 'nvme_process_chchap_<statename>'

>> +{
>> +    struct nvmf_auth_dhchap_challenge_data *data = buf;
>> +    size_t size = sizeof(*data) + data->hl + data->dhvlen;
>> +    const char *gid_name;
>> +
>> +    if (buf_size < size) {
>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
>> +        return -ENOMSG;
>> +    }
>> +
>> +    if (data->hashid != NVME_AUTH_DHCHAP_HASH_SHA256 &&
>> +        data->hashid != NVME_AUTH_DHCHAP_HASH_SHA384 &&
>> +        data->hashid != NVME_AUTH_DHCHAP_HASH_SHA512) {
>> +        dev_warn(ctrl->device,
>> +             "qid %d: DH-HMAC-CHAP: invalid HASH ID %d\n",
>> +             chap->qid, data->hashid);
>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
>> +        return -EPROTO;
>> +    }
>> +    switch (data->dhgid) {
>> +    case NVME_AUTH_DHCHAP_DHGROUP_NULL:
>> +        gid_name = "null";
>> +        break;
>> +    default:
>> +        gid_name = NULL;
>> +        break;
>> +    }
>> +    if (!gid_name) {
>> +        dev_warn(ctrl->device,
>> +             "qid %d: DH-HMAC-CHAP: invalid DH group id %d\n",
>> +             chap->qid, data->dhgid);
>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
>> +        return -EPROTO;
>> +    }
> 
> Maybe some spaces between condition blocks?
> 

Ok.

>> +    if (data->dhgid != NVME_AUTH_DHCHAP_DHGROUP_NULL) {
>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
>> +        return -EPROTO;
>> +    }
>> +    if (data->dhgid == NVME_AUTH_DHCHAP_DHGROUP_NULL && data->dhvlen 
>> != 0) {
>> +        dev_warn(ctrl->device,
>> +             "qid %d: DH-HMAC-CHAP: invalid DH value for NULL DH\n",
>> +            chap->qid);
>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
>> +        return -EPROTO;
>> +    }
>> +    dev_dbg(ctrl->device, "%s: qid %d requested hash id %d\n",
>> +        __func__, chap->qid, data->hashid);
>> +    if (nvme_auth_hmac_len(data->hashid) != data->hl) {
>> +        dev_warn(ctrl->device,
>> +             "qid %d: DH-HMAC-CHAP: invalid hash length\n",
>> +            chap->qid);
>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
>> +        return -EPROTO;
>> +    }
>> +    chap->hash_id = data->hashid;
>> +    chap->hash_len = data->hl;
>> +    chap->s1 = le32_to_cpu(data->seqnum);
>> +    memcpy(chap->c1, data->cval, chap->hash_len);
>> +
>> +    return 0;
>> +}
>> +
>> +static int nvme_auth_dhchap_reply(struct nvme_ctrl *ctrl,
>> +                  struct nvme_dhchap_context *chap,
>> +                  void *buf, size_t buf_size)
> 
> nvme_auth_set_dhchap_reply
> 

Ah. Now I see what you're getting at.
Okay, will be changing it.

>> +{
>> +    struct nvmf_auth_dhchap_reply_data *data = buf;
>> +    size_t size = sizeof(*data);
>> +
>> +    size += 2 * chap->hash_len;
>> +    if (ctrl->opts->dhchap_auth) {
> 
> The ctrl opts is not clear to me. what is dhchap_auth
> mean?
> 
As stated above, this is for bi-directional authentication.
And yes, it is poor wording.

'dhchap_bidirectional' ?

> Also shouldn't these params be lifted to the subsys?
> 

I kinda like to have it all encapsulated in a common per-queue 
structure; on the host side this one isn't even attached to anything, so 
any new authentication attempt will allocate a new one, with no chance 
of accidentally re-using existing values.
I thought this to be a rather nice property for a state-machine.

>> +        get_random_bytes(chap->c2, chap->hash_len);
>> +        chap->s2 = nvme_dhchap_seqnum++;
>> +    } else
>> +        memset(chap->c2, 0, chap->hash_len);
>> +
>> +    if (chap->host_key_len)
>> +        size += chap->host_key_len;
>> +
>> +    if (buf_size < size)
>> +        return -EINVAL;
>> +
>> +    memset(buf, 0, size);
>> +    data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
>> +    data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_REPLY;
>> +    data->t_id = cpu_to_le16(chap->transaction);
>> +    data->hl = chap->hash_len;
>> +    data->dhvlen = chap->host_key_len;
>> +    data->seqnum = cpu_to_le32(chap->s2);
>> +    memcpy(data->rval, chap->response, chap->hash_len);
>> +    if (ctrl->opts->dhchap_auth) {
>> +        dev_dbg(ctrl->device, "%s: qid %d ctrl challenge %*ph\n",
>> +            __func__, chap->qid,
>> +            chap->hash_len, chap->c2);
>> +        data->cvalid = 1;
>> +        memcpy(data->rval + chap->hash_len, chap->c2,
>> +               chap->hash_len);
>> +    }
>> +    if (chap->host_key_len)
>> +        memcpy(data->rval + 2 * chap->hash_len, chap->host_key,
>> +               chap->host_key_len);
>> +
>> +    return size;
>> +}
>> +
>> +static int nvme_auth_dhchap_success1(struct nvme_ctrl *ctrl,
>> +                     struct nvme_dhchap_context *chap,
>> +                     void *buf, size_t buf_size)
> 
> nvme_auth_process_dhchap_success1
> 

OK.

>> +{
>> +    struct nvmf_auth_dhchap_success1_data *data = buf;
>> +    size_t size = sizeof(*data);
>> +
>> +    if (ctrl->opts->dhchap_auth)
>> +        size += chap->hash_len;
>> +
>> +
>> +    if (buf_size < size) {
>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
>> +        return -ENOMSG;
>> +    }
>> +
>> +    if (data->hl != chap->hash_len) {
>> +        dev_warn(ctrl->device,
>> +             "qid %d: DH-HMAC-CHAP: invalid hash length %d\n",
>> +             chap->qid, data->hl);
>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
>> +        return -EPROTO;
>> +    }
>> +
>> +    if (!data->rvalid)
>> +        return 0;
>> +
>> +    /* Validate controller response */
>> +    if (memcmp(chap->response, data->rval, data->hl)) {
>> +        dev_dbg(ctrl->device, "%s: qid %d ctrl response %*ph\n",
>> +            __func__, chap->qid, chap->hash_len, data->rval);
>> +        dev_dbg(ctrl->device, "%s: qid %d host response %*ph\n",
>> +            __func__, chap->qid, chap->hash_len, chap->response);
>> +        dev_warn(ctrl->device,
>> +             "qid %d: DH-HMAC-CHAP: controller authentication failed\n",
>> +             chap->qid);
>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
>> +        return -EPROTO;
>> +    }
>> +    dev_info(ctrl->device,
>> +         "qid %d: DH-HMAC-CHAP: controller authenticated\n",
>> +        chap->qid);
>> +    return 0;
>> +}
>> +
>> +static int nvme_auth_dhchap_success2(struct nvme_ctrl *ctrl,
>> +                     struct nvme_dhchap_context *chap,
>> +                     void *buf, size_t buf_size)
> 
> same
> 
>> +{
>> +    struct nvmf_auth_dhchap_success2_data *data = buf;
>> +    size_t size = sizeof(*data);
>> +
>> +    memset(buf, 0, size);
>> +    data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
>> +    data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_SUCCESS2;
>> +    data->t_id = cpu_to_le16(chap->transaction);
>> +
>> +    return size;
>> +}
>> +
>> +static int nvme_auth_dhchap_failure2(struct nvme_ctrl *ctrl,
>> +                     struct nvme_dhchap_context *chap,
>> +                     void *buf, size_t buf_size)
> 
> same
> 
>> +{
>> +    struct nvmf_auth_dhchap_failure_data *data = buf;
>> +    size_t size = sizeof(*data);
>> +
>> +    memset(buf, 0, size);
>> +    data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
>> +    data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_FAILURE2;
>> +    data->t_id = cpu_to_le16(chap->transaction);
>> +    data->reason_code = 1;
>> +    data->reason_code_explanation = chap->status;
>> +
>> +    return size;
>> +}
>> +
>> +int nvme_auth_select_hash(struct nvme_ctrl *ctrl,
>> +              struct nvme_dhchap_context *chap)
> 
> Maybe _select_hf (hash function)? not a must, just sticks
> to the spec language.
> 

Hmm. Will be checking.

>> +{
>> +    char *hash_name;
>> +    int ret;
>> +
>> +    switch (chap->hash_id) {
>> +    case NVME_AUTH_DHCHAP_HASH_SHA256:
>> +        hash_name = "hmac(sha256)";
>> +        break;
>> +    case NVME_AUTH_DHCHAP_HASH_SHA384:
>> +        hash_name = "hmac(sha384)";
>> +        break;
>> +    case NVME_AUTH_DHCHAP_HASH_SHA512:
>> +        hash_name = "hmac(sha512)";
>> +        break;
>> +    default:
>> +        hash_name = NULL;
>> +        break;
>> +    }
>> +    if (!hash_name) {
>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
>> +        return -EPROTO;
>> +    }
>> +    chap->shash_tfm = crypto_alloc_shash(hash_name, 0,
>> +                         CRYPTO_ALG_ALLOCATES_MEMORY);
>> +    if (IS_ERR(chap->shash_tfm)) {
>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
>> +        chap->shash_tfm = NULL;
>> +        return -EPROTO;
>> +    }
>> +    if (!chap->key) {
>> +        dev_warn(ctrl->device, "qid %d: cannot select hash, no key\n",
>> +             chap->qid);
>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
>> +        crypto_free_shash(chap->shash_tfm);
> 
> Wouldn't it better to check this before allocating the tfm?
> 

Indeed. Will be changing it.

>> +        chap->shash_tfm = NULL;
>> +        return -EINVAL;
>> +    }
>> +    ret = crypto_shash_setkey(chap->shash_tfm, chap->key, 
>> chap->key_len);
>> +    if (ret) {
>> +        chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
>> +        crypto_free_shash(chap->shash_tfm);
>> +        chap->shash_tfm = NULL;
>> +        return ret;
>> +    }
>> +    dev_info(ctrl->device, "qid %d: DH-HMAC_CHAP: selected hash %s\n",
>> +         chap->qid, hash_name);
>> +    return 0;
>> +}
>> +
>> +static int nvme_auth_dhchap_host_response(struct nvme_ctrl *ctrl,
>> +                      struct nvme_dhchap_context *chap)
>> +{
>> +    SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
>> +    u8 buf[4], *challenge = chap->c1;
>> +    int ret;
>> +
>> +    dev_dbg(ctrl->device, "%s: qid %d host response seq %d 
>> transaction %d\n",
>> +        __func__, chap->qid, chap->s1, chap->transaction);
>> +    shash->tfm = chap->shash_tfm;
>> +    ret = crypto_shash_init(shash);
>> +    if (ret)
>> +        goto out;
>> +    ret = crypto_shash_update(shash, challenge, chap->hash_len);
>> +    if (ret)
>> +        goto out;
>> +    put_unaligned_le32(chap->s1, buf);
>> +    ret = crypto_shash_update(shash, buf, 4);
>> +    if (ret)
>> +        goto out;
>> +    put_unaligned_le16(chap->transaction, buf);
>> +    ret = crypto_shash_update(shash, buf, 2);
>> +    if (ret)
>> +        goto out;
>> +    memset(buf, 0, sizeof(buf));
>> +    ret = crypto_shash_update(shash, buf, 1);
>> +    if (ret)
>> +        goto out;
>> +    ret = crypto_shash_update(shash, "HostHost", 8);
> 
> HostHost ? Can you refer me to the specific section
> that talks about this?
> 

NVMe 2.0 section DH-HMAC-CHAP_Reply Message, paragraph Response Value.
HostHost.

> Would be good to have a comment on the format fed to the
> shash.
> 

Yes, will be doing so.

>> +    if (ret)
>> +        goto out;
>> +    ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
>> +                  strlen(ctrl->opts->host->nqn));
>> +    if (ret)
>> +        goto out;
>> +    ret = crypto_shash_update(shash, buf, 1);
>> +    if (ret)
>> +        goto out;
>> +    ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
>> +                strlen(ctrl->opts->subsysnqn));
>> +    if (ret)
>> +        goto out;
>> +    ret = crypto_shash_final(shash, chap->response);
>> +out:
>> +    return ret;
>> +}
>> +
>> +static int nvme_auth_dhchap_ctrl_response(struct nvme_ctrl *ctrl,
>> +                      struct nvme_dhchap_context *chap)
>> +{
>> +    SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
>> +    u8 buf[4], *challenge = chap->c2;
>> +    int ret;
>> +
>> +    dev_dbg(ctrl->device, "%s: qid %d host response seq %d 
>> transaction %d\n",
>> +        __func__, chap->qid, chap->s2, chap->transaction);
>> +    dev_dbg(ctrl->device, "%s: qid %d challenge %*ph\n",
>> +        __func__, chap->qid, chap->hash_len, challenge);
>> +    dev_dbg(ctrl->device, "%s: qid %d subsysnqn %s\n",
>> +        __func__, chap->qid, ctrl->opts->subsysnqn);
>> +    dev_dbg(ctrl->device, "%s: qid %d hostnqn %s\n",
>> +        __func__, chap->qid, ctrl->opts->host->nqn);
>> +    shash->tfm = chap->shash_tfm;
>> +    ret = crypto_shash_init(shash);
>> +    if (ret)
>> +        goto out;
>> +    ret = crypto_shash_update(shash, challenge, chap->hash_len);
>> +    if (ret)
>> +        goto out;
>> +    put_unaligned_le32(chap->s2, buf);
>> +    ret = crypto_shash_update(shash, buf, 4);
>> +    if (ret)
>> +        goto out;
>> +    put_unaligned_le16(chap->transaction, buf);
>> +    ret = crypto_shash_update(shash, buf, 2);
>> +    if (ret)
>> +        goto out;
>> +    memset(buf, 0, 4);
>> +    ret = crypto_shash_update(shash, buf, 1);
>> +    if (ret)
>> +        goto out;
>> +    ret = crypto_shash_update(shash, "Controller", 10);
>> +    if (ret)
>> +        goto out;
>> +    ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
>> +                  strlen(ctrl->opts->subsysnqn));
>> +    if (ret)
>> +        goto out;
>> +    ret = crypto_shash_update(shash, buf, 1);
>> +    if (ret)
>> +        goto out;
>> +    ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
>> +                  strlen(ctrl->opts->host->nqn));
>> +    if (ret)
>> +        goto out;
>> +    ret = crypto_shash_final(shash, chap->response);
>> +out:
>> +    return ret;
>> +}
>> +
>> +int nvme_auth_generate_key(struct nvme_ctrl *ctrl,
>> +               struct nvme_dhchap_context *chap)
>> +{
>> +    int ret;
>> +    u8 key_hash;
>> +    const char *hmac_name;
>> +    struct crypto_shash *key_tfm;
>> +
>> +    if (sscanf(ctrl->opts->dhchap_secret, "DHHC-1:%hhd:%*s:",
>> +           &key_hash) != 1)
>> +        return -EINVAL;
> 
> I'd expect that the user will pass in a secret key (as binary)
> the the driver will build the spec compliant formatted string no?
>  > Am I not reading this correctly?
> 

I'm under the impression that this is the format into which the 
User/Admin will get hold of the secret key.
Spec says:

'... all NVMe over Fabrics entities shall support the following ASCII
representation of secrets ...'

And as the userspace interface is the only way how the user/admin 
_could_ interact with the NVMe over Fabrics entities in Linux I guess 
we'll need to be able to parse it.

We sure could allow a binary secret, too, but then what would be the 
point in converting it into the secret representation?
The protocol revolves around the binary secret, not the transport 
representation.

>> +
>> +    chap->key = nvme_auth_extract_secret(ctrl->opts->dhchap_secret,
>> +                         &chap->key_len);
>> +    if (IS_ERR(chap->key)) {
>> +        ret = PTR_ERR(chap->key);
>> +        chap->key = NULL;
>> +        return ret;
>> +    }
>> +
>> +    if (key_hash == 0)
>> +        return 0;
>> +
>> +    hmac_name = nvme_auth_hmac_name(key_hash);
>> +    if (!hmac_name) {
>> +        pr_debug("Invalid key hash id %d\n", key_hash);
>> +        return -EKEYREJECTED;
>> +    }
> 
> Why does the user influence the hmac used? isn't that is driven
> by the susbsystem?
> 
> I don't think that the user should choose in this level.
> 

That is another weirdness of the spec.
The _secret_ will be hashed with a specific function, and that function 
is stated in the transport representation.
(Cf section "DH-HMAC-CHAP Security Requirements").
This is _not_ the hash function used by the authentication itself, which 
will be selected by the protocol.
So it's not the user here, but rather the transport specification of the 
key which selects the hash algorithm.

>> +
>> +    key_tfm = crypto_alloc_shash(hmac_name, 0, 0);
>> +    if (IS_ERR(key_tfm)) {
>> +        kfree(chap->key);
>> +        chap->key = NULL;
>> +        ret = PTR_ERR(key_tfm);
> 
> You set ret and later return 0? I think that the success
> path in the else clause is hard to read and error prone...
> 

Do I? Will need to fix it up.

>> +    } else {
>> +        SHASH_DESC_ON_STACK(shash, key_tfm);
>> +
>> +        shash->tfm = key_tfm;
>> +        ret = crypto_shash_setkey(key_tfm, chap->key,
>> +                      chap->key_len);
>> +        if (ret < 0) {
>> +            crypto_free_shash(key_tfm);
>> +            kfree(chap->key);
>> +            chap->key = NULL;
>> +            return ret;
>> +        }
>> +        crypto_shash_init(shash);
>> +        crypto_shash_update(shash, ctrl->opts->host->nqn,
>> +                    strlen(ctrl->opts->host->nqn));
>> +        crypto_shash_update(shash, "NVMe-over-Fabrics", 17);
>> +        crypto_shash_final(shash, chap->key);
>> +        crypto_free_shash(key_tfm);
> 
> Shouldn't these be done when preparing the dh-hmac-chap reply?
> 

By setting the hash here I avoid having to pass the required hash 
function for the secret transformation.
I could be doing the entire secret transformation thingie when preparing 
the reply; reason why I did it here is that _having_ a secret is the 
precondition to everything else, so I wanted to check upfront for that.
But I'll check what would happen if I move it.

>> +    }
>> +    return 0;
>> +}
>> +
>> +void nvme_auth_free(struct nvme_dhchap_context *chap)
>> +{
>> +    if (chap->shash_tfm)
>> +        crypto_free_shash(chap->shash_tfm);
>> +    if (chap->key)
>> +        kfree(chap->key);
>> +    if (chap->ctrl_key)
>> +        kfree(chap->ctrl_key);
>> +    if (chap->host_key)
>> +        kfree(chap->host_key);
>> +    if (chap->sess_key)
>> +        kfree(chap->sess_key);
> 
> No need to check null for kfree...
> 

Will be fixing it up.

>> +    kfree(chap);
>> +}
>> +
>> +int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid)
>> +{
>> +    struct nvme_dhchap_context *chap;
>> +    void *buf;
>> +    size_t buf_size, tl;
>> +    int ret = 0;
>> +
>> +    chap = kzalloc(sizeof(*chap), GFP_KERNEL);
>> +    if (!chap)
>> +        return -ENOMEM;
>> +    chap->qid = qid;
>> +    chap->transaction = ctrl->transaction++;
>> +
>> +    ret = nvme_auth_generate_key(ctrl, chap);
>> +    if (ret) {
>> +        dev_dbg(ctrl->device, "%s: failed to generate key, error %d\n",
>> +            __func__, ret);
>> +        nvme_auth_free(chap);
>> +        return ret;
>> +    }
>> +
>> +    /*
>> +     * Allocate a large enough buffer for the entire negotiation:
>> +     * 4k should be enough to ffdhe8192.
>> +     */
>> +    buf_size = 4096;
>> +    buf = kzalloc(buf_size, GFP_KERNEL);
>> +    if (!buf) {
>> +        ret = -ENOMEM;
>> +        goto out;
>> +    }
>> +
>> +    /* DH-HMAC-CHAP Step 1: send negotiate */
> 
> I'd consider breaking these into sub-routines.
> 

Which ones? The preparation step?
Sure, can do.

>> +    dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP negotiate\n",
>> +        __func__, qid);
>> +    ret = nvme_auth_dhchap_negotiate(ctrl, chap, buf, buf_size);
>> +    if (ret < 0)
>> +        goto out;
>> +    tl = ret;
>> +    ret = nvme_auth_send(ctrl, qid, buf, tl);
>> +    if (ret)
>> +        goto out;
>> +
>> +    memset(buf, 0, buf_size);
>> +    ret = nvme_auth_receive(ctrl, qid, buf, buf_size, chap->transaction,
>> +                NVME_AUTH_DHCHAP_MESSAGE_CHALLENGE);
>> +    if (ret < 0) {
>> +        dev_dbg(ctrl->device,
>> +            "%s: qid %d DH-HMAC-CHAP failed to receive challenge\n",
>> +            __func__, qid);
>> +        goto out;
>> +    }
>> +    if (ret > 0) {
>> +        chap->status = ret;
>> +        goto fail1;
>> +    }
>> +
>> +    /* DH-HMAC-CHAP Step 2: receive challenge */
>> +    dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP challenge\n",
>> +        __func__, qid);
>> +
>> +    ret = nvme_auth_dhchap_challenge(ctrl, chap, buf, buf_size);
>> +    if (ret) {
>> +        /* Invalid parameters for negotiate */
>> +        goto fail2;
>> +    }
>> +
>> +    dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP select hash\n",
>> +        __func__, qid);
>> +    ret = nvme_auth_select_hash(ctrl, chap);
>> +    if (ret)
>> +        goto fail2;
>> +
>> +    dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP host response\n",
>> +        __func__, qid);
>> +    ret = nvme_auth_dhchap_host_response(ctrl, chap);
>> +    if (ret)
>> +        goto fail2;
>> +
>> +    /* DH-HMAC-CHAP Step 3: send reply */
>> +    dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP reply\n",
>> +        __func__, qid);
>> +    ret = nvme_auth_dhchap_reply(ctrl, chap, buf, buf_size);
>> +    if (ret < 0)
>> +        goto fail2;
>> +
>> +    tl = ret;
>> +    ret = nvme_auth_send(ctrl, qid, buf, tl);
>> +    if (ret)
>> +        goto fail2;
>> +
>> +    memset(buf, 0, buf_size);
>> +    ret = nvme_auth_receive(ctrl, qid, buf, buf_size, chap->transaction,
>> +                NVME_AUTH_DHCHAP_MESSAGE_SUCCESS1);
>> +    if (ret < 0) {
>> +        dev_dbg(ctrl->device,
>> +            "%s: qid %d DH-HMAC-CHAP failed to receive success1\n",
>> +            __func__, qid);
>> +        goto out;
>> +    }
>> +    if (ret > 0) {
>> +        chap->status = ret;
>> +        goto fail1;
>> +    }
>> +
>> +    if (ctrl->opts->dhchap_auth) {
>> +        dev_dbg(ctrl->device,
>> +            "%s: qid %d DH-HMAC-CHAP controller response\n",
>> +            __func__, qid);
>> +        ret = nvme_auth_dhchap_ctrl_response(ctrl, chap);
>> +        if (ret)
>> +            goto fail2;
>> +    }
>> +
>> +    /* DH-HMAC-CHAP Step 4: receive success1 */
>> +    dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP success1\n",
>> +        __func__, qid);
>> +    ret = nvme_auth_dhchap_success1(ctrl, chap, buf, buf_size);
>> +    if (ret < 0) {
>> +        /* Controller authentication failed */
>> +        goto fail2;
>> +    }
>> +    tl = ret;
>> +    /* DH-HMAC-CHAP Step 5: send success2 */
>> +    dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP success2\n",
>> +        __func__, qid);
>> +    tl = nvme_auth_dhchap_success2(ctrl, chap, buf, buf_size);
>> +    ret = nvme_auth_send(ctrl, qid, buf, tl);
>> +    if (!ret)
>> +        goto out;
>> +
>> +fail1:
>> +    dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP failure1, status 
>> %x\n",
>> +        __func__, qid, chap->status);
>> +    goto out;
>> +
>> +fail2:
>> +    dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP failure2, status 
>> %x\n",
>> +        __func__, qid, chap->status);
>> +    tl = nvme_auth_dhchap_failure2(ctrl, chap, buf, buf_size);
>> +    ret = nvme_auth_send(ctrl, qid, buf, tl);
>> +
>> +out:
>> +    if (!ret && chap->status)
>> +        ret = -EPROTO;
>> +    if (!ret) {
>> +        ctrl->dhchap_hash = chap->hash_id;
>> +    }
>> +    kfree(buf);
>> +    nvme_auth_free(chap);
>> +    return ret;
>> +}
>> diff --git a/drivers/nvme/host/auth.h b/drivers/nvme/host/auth.h
>> new file mode 100644
>> index 000000000000..4950b1cb9470
>> --- /dev/null
>> +++ b/drivers/nvme/host/auth.h
>> @@ -0,0 +1,23 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +/*
>> + * Copyright (c) 2021 Hannes Reinecke, SUSE Software Solutions
>> + */
>> +
>> +#ifndef _NVME_AUTH_H
>> +#define _NVME_AUTH_H
>> +
>> +const char *nvme_auth_dhgroup_name(int dhgroup_id);
>> +int nvme_auth_dhgroup_pubkey_size(int dhgroup_id);
>> +int nvme_auth_dhgroup_privkey_size(int dhgroup_id);
>> +const char *nvme_auth_dhgroup_kpp(int dhgroup_id);
>> +int nvme_auth_dhgroup_id(const char *dhgroup_name);
>> +
>> +const char *nvme_auth_hmac_name(int hmac_id);
>> +const char *nvme_auth_digest_name(int hmac_id);
>> +int nvme_auth_hmac_id(const char *hmac_name);
>> +int nvme_auth_hmac_len(int hmac_len);
>> +
>> +unsigned char *nvme_auth_extract_secret(unsigned char *dhchap_secret,
>> +                    size_t *dhchap_key_len);
>> +
>> +#endif /* _NVME_AUTH_H */
>> diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
>> index 11779be42186..7ce9b666dc09 100644
>> --- a/drivers/nvme/host/core.c
>> +++ b/drivers/nvme/host/core.c
>> @@ -708,7 +708,9 @@ bool __nvme_check_ready(struct nvme_ctrl *ctrl, 
>> struct request *rq,
>>           switch (ctrl->state) {
>>           case NVME_CTRL_CONNECTING:
>>               if (blk_rq_is_passthrough(rq) && 
>> nvme_is_fabrics(req->cmd) &&
>> -                req->cmd->fabrics.fctype == nvme_fabrics_type_connect)
>> +                (req->cmd->fabrics.fctype == 
>> nvme_fabrics_type_connect ||
>> +                 req->cmd->fabrics.fctype == 
>> nvme_fabrics_type_auth_send ||
>> +                 req->cmd->fabrics.fctype == 
>> nvme_fabrics_type_auth_receive))
>>                   return true;
>>               break;
>>           default:
>> @@ -3426,6 +3428,66 @@ static ssize_t 
>> nvme_ctrl_fast_io_fail_tmo_store(struct device *dev,
>>   static DEVICE_ATTR(fast_io_fail_tmo, S_IRUGO | S_IWUSR,
>>       nvme_ctrl_fast_io_fail_tmo_show, nvme_ctrl_fast_io_fail_tmo_store);
>> +#ifdef CONFIG_NVME_AUTH
>> +struct nvmet_dhchap_hash_map {
>> +    int id;
>> +    const char name[15];
>> +} hash_map[] = {
>> +    {.id = NVME_AUTH_DHCHAP_HASH_SHA256,
>> +     .name = "hmac(sha256)", },
>> +    {.id = NVME_AUTH_DHCHAP_HASH_SHA384,
>> +     .name = "hmac(sha384)", },
>> +    {.id = NVME_AUTH_DHCHAP_HASH_SHA512,
>> +     .name = "hmac(sha512)", },
>> +};
>> +
>> +static ssize_t dhchap_hash_show(struct device *dev,
>> +    struct device_attribute *attr, char *buf)
>> +{
>> +    struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
>> +    int i;
>> +
>> +    for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
>> +        if (hash_map[i].id == ctrl->dhchap_hash)
>> +            return sprintf(buf, "%s\n", hash_map[i].name);
>> +    }
>> +    return sprintf(buf, "none\n");
>> +}
>> +DEVICE_ATTR_RO(dhchap_hash);
>> +
>> +struct nvmet_dhchap_group_map {
>> +    int id;
>> +    const char name[15];
>> +} dhgroup_map[] = {
>> +    {.id = NVME_AUTH_DHCHAP_DHGROUP_NULL,
>> +     .name = "NULL", },
>> +    {.id = NVME_AUTH_DHCHAP_DHGROUP_2048,
>> +     .name = "ffdhe2048", },
>> +    {.id = NVME_AUTH_DHCHAP_DHGROUP_3072,
>> +     .name = "ffdhe3072", },
>> +    {.id = NVME_AUTH_DHCHAP_DHGROUP_4096,
>> +     .name = "ffdhe4096", },
>> +    {.id = NVME_AUTH_DHCHAP_DHGROUP_6144,
>> +     .name = "ffdhe6144", },
>> +    {.id = NVME_AUTH_DHCHAP_DHGROUP_8192,
>> +     .name = "ffdhe8192", },
>> +};
>> +
>> +static ssize_t dhchap_dhgroup_show(struct device *dev,
>> +    struct device_attribute *attr, char *buf)
>> +{
>> +    struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
>> +    int i;
>> +
>> +    for (i = 0; i < ARRAY_SIZE(dhgroup_map); i++) {
>> +        if (hash_map[i].id == ctrl->dhchap_dhgroup)
>> +            return sprintf(buf, "%s\n", dhgroup_map[i].name);
>> +    }
>> +    return sprintf(buf, "none\n");
>> +}
>> +DEVICE_ATTR_RO(dhchap_dhgroup);
>> +#endif
>> +
>>   static struct attribute *nvme_dev_attrs[] = {
>>       &dev_attr_reset_controller.attr,
>>       &dev_attr_rescan_controller.attr,
>> @@ -3447,6 +3509,10 @@ static struct attribute *nvme_dev_attrs[] = {
>>       &dev_attr_reconnect_delay.attr,
>>       &dev_attr_fast_io_fail_tmo.attr,
>>       &dev_attr_kato.attr,
>> +#ifdef CONFIG_NVME_AUTH
>> +    &dev_attr_dhchap_hash.attr,
>> +    &dev_attr_dhchap_dhgroup.attr,
>> +#endif
>>       NULL
>>   };
>> @@ -3470,6 +3536,10 @@ static umode_t 
>> nvme_dev_attrs_are_visible(struct kobject *kobj,
>>           return 0;
>>       if (a == &dev_attr_fast_io_fail_tmo.attr && !ctrl->opts)
>>           return 0;
>> +#ifdef CONFIG_NVME_AUTH
>> +    if (a == &dev_attr_dhchap_hash.attr && !ctrl->opts)
>> +        return 0;
>> +#endif
>>       return a->mode;
>>   }
>> @@ -4581,6 +4651,11 @@ static inline void _nvme_check_size(void)
>>       BUILD_BUG_ON(sizeof(struct nvme_smart_log) != 512);
>>       BUILD_BUG_ON(sizeof(struct nvme_dbbuf) != 64);
>>       BUILD_BUG_ON(sizeof(struct nvme_directive_cmd) != 64);
>> +    BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_negotiate_data) != 8);
>> +    BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_challenge_data) != 16);
>> +    BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_reply_data) != 16);
>> +    BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_success1_data) != 16);
>> +    BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_success2_data) != 16);
>>   }
>> diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
>> index a5469fd9d4c3..6404ab9b604b 100644
>> --- a/drivers/nvme/host/fabrics.c
>> +++ b/drivers/nvme/host/fabrics.c
>> @@ -366,6 +366,7 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)
>>       union nvme_result res;
>>       struct nvmf_connect_data *data;
>>       int ret;
>> +    u32 result;
>>       cmd.connect.opcode = nvme_fabrics_command;
>>       cmd.connect.fctype = nvme_fabrics_type_connect;
>> @@ -398,8 +399,18 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)
>>           goto out_free_data;
>>       }
>> -    ctrl->cntlid = le16_to_cpu(res.u16);
>> -
>> +    result = le32_to_cpu(res.u32);
>> +    ctrl->cntlid = result & 0xFFFF;
>> +    if ((result >> 16) & 2) {
>> +        /* Authentication required */
>> +        ret = nvme_auth_negotiate(ctrl, NVME_QID_ANY);
>> +        if (ret)
>> +            dev_warn(ctrl->device,
>> +                 "qid 0: authentication failed\n");
>> +        else
>> +            dev_info(ctrl->device,
>> +                 "qid 0: authenticated\n");
> 
> info is too chatty.
> 

Hmm. I know I need to work on logging...

>> +    }
>>   out_free_data:
>>       kfree(data);
>>       return ret;
>> @@ -432,6 +443,7 @@ int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, 
>> u16 qid)
>>       struct nvmf_connect_data *data;
>>       union nvme_result res;
>>       int ret;
>> +    u32 result;
>>       cmd.connect.opcode = nvme_fabrics_command;
>>       cmd.connect.fctype = nvme_fabrics_type_connect;
>> @@ -457,6 +469,17 @@ int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, 
>> u16 qid)
>>           nvmf_log_connect_error(ctrl, ret, le32_to_cpu(res.u32),
>>                          &cmd, data);
>>       }
>> +    result = le32_to_cpu(res.u32);
>> +    if ((result >> 16) & 2) {
>> +        /* Authentication required */
>> +        ret = nvme_auth_negotiate(ctrl, qid);
>> +        if (ret)
>> +            dev_warn(ctrl->device,
>> +                 "qid %u: authentication failed\n", qid);
>> +        else
>> +            dev_info(ctrl->device,
>> +                 "qid %u: authenticated\n", qid);
>> +    }
>>       kfree(data);
>>       return ret;
>>   }
>> @@ -548,6 +571,9 @@ static const match_table_t opt_tokens = {
>>       { NVMF_OPT_NR_POLL_QUEUES,    "nr_poll_queues=%d"    },
>>       { NVMF_OPT_TOS,            "tos=%d"        },
>>       { NVMF_OPT_FAIL_FAST_TMO,    "fast_io_fail_tmo=%d"    },
>> +    { NVMF_OPT_DHCHAP_SECRET,    "dhchap_secret=%s"    },
>> +    { NVMF_OPT_DHCHAP_AUTH,        "authenticate"        },
>> +    { NVMF_OPT_DHCHAP_GROUP,    "dhchap_group=%s"    },
> 
> Isn't the group driven by the subsystem? also why is there a
> "authenticate" boolean? what is it good for?
> 
Ah. Right. Of course, the 'group' is pointless here.
And the 'authenticate' bool is the abovementioned bidirectional 
authentication.
I _do_ need to give it another name.

>>       { NVMF_OPT_ERR,            NULL            }
>>   };
>> @@ -824,6 +850,35 @@ static int nvmf_parse_options(struct 
>> nvmf_ctrl_options *opts,
>>               }
>>               opts->tos = token;
>>               break;
>> +        case NVMF_OPT_DHCHAP_SECRET:
>> +            p = match_strdup(args);
>> +            if (!p) {
>> +                ret = -ENOMEM;
>> +                goto out;
>> +            }
>> +            if (strncmp(p, "DHHC-1:00:", 10)) {
>> +                pr_err("Invalid DH-CHAP secret %s\n", p);
>> +                ret = -EINVAL;
>> +                goto out;
>> +            }
>> +            kfree(opts->dhchap_secret);
>> +            opts->dhchap_secret = p;
>> +            break;
>> +        case NVMF_OPT_DHCHAP_AUTH:
>> +            opts->dhchap_auth = true;
>> +            break;
>> +        case NVMF_OPT_DHCHAP_GROUP:
>> +            if (match_int(args, &token)) {
>> +                ret = -EINVAL;
>> +                goto out;
>> +            }
>> +            if (token <= 0) {
>> +                pr_err("Invalid dhchap_group %d\n", token);
>> +                ret = -EINVAL;
>> +                goto out;
>> +            }
>> +            opts->dhchap_group = token;
>> +            break;
>>           default:
>>               pr_warn("unknown parameter or missing value '%s' in ctrl 
>> creation request\n",
>>                   p);
>> @@ -942,6 +997,7 @@ void nvmf_free_options(struct nvmf_ctrl_options 
>> *opts)
>>       kfree(opts->subsysnqn);
>>       kfree(opts->host_traddr);
>>       kfree(opts->host_iface);
>> +    kfree(opts->dhchap_secret);
>>       kfree(opts);
>>   }
>>   EXPORT_SYMBOL_GPL(nvmf_free_options);
>> @@ -951,7 +1007,10 @@ EXPORT_SYMBOL_GPL(nvmf_free_options);
>>                    NVMF_OPT_KATO | NVMF_OPT_HOSTNQN | \
>>                    NVMF_OPT_HOST_ID | NVMF_OPT_DUP_CONNECT |\
>>                    NVMF_OPT_DISABLE_SQFLOW |\
>> -                 NVMF_OPT_FAIL_FAST_TMO)
>> +                 NVMF_OPT_CTRL_LOSS_TMO |\
>> +                 NVMF_OPT_FAIL_FAST_TMO |\
>> +                 NVMF_OPT_DHCHAP_SECRET |\
>> +                 NVMF_OPT_DHCHAP_AUTH | NVMF_OPT_DHCHAP_GROUP)
>>   static struct nvme_ctrl *
>>   nvmf_create_ctrl(struct device *dev, const char *buf)
>> diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h
>> index a146cb903869..535bc544f0f6 100644
>> --- a/drivers/nvme/host/fabrics.h
>> +++ b/drivers/nvme/host/fabrics.h
>> @@ -67,6 +67,9 @@ enum {
>>       NVMF_OPT_TOS        = 1 << 19,
>>       NVMF_OPT_FAIL_FAST_TMO    = 1 << 20,
>>       NVMF_OPT_HOST_IFACE    = 1 << 21,
>> +    NVMF_OPT_DHCHAP_SECRET    = 1 << 22,
>> +    NVMF_OPT_DHCHAP_AUTH    = 1 << 23,
>> +    NVMF_OPT_DHCHAP_GROUP    = 1 << 24,
>>   };
>>   /**
>> @@ -96,6 +99,8 @@ enum {
>>    * @max_reconnects: maximum number of allowed reconnect attempts 
>> before removing
>>    *              the controller, (-1) means reconnect forever, zero 
>> means remove
>>    *              immediately;
>> + * @dhchap_secret: DH-HMAC-CHAP secret
>> + * @dhchap_auth: DH-HMAC-CHAP authenticate controller
>>    * @disable_sqflow: disable controller sq flow control
>>    * @hdr_digest: generate/verify header digest (TCP)
>>    * @data_digest: generate/verify data digest (TCP)
>> @@ -120,6 +125,9 @@ struct nvmf_ctrl_options {
>>       unsigned int        kato;
>>       struct nvmf_host    *host;
>>       int            max_reconnects;
>> +    char            *dhchap_secret;
>> +    int            dhchap_group;
>> +    bool            dhchap_auth;
>>       bool            disable_sqflow;
>>       bool            hdr_digest;
>>       bool            data_digest;
>> diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
>> index 18ef8dd03a90..bcd5b8276c26 100644
>> --- a/drivers/nvme/host/nvme.h
>> +++ b/drivers/nvme/host/nvme.h
>> @@ -328,6 +328,12 @@ struct nvme_ctrl {
>>       struct work_struct ana_work;
>>   #endif
>> +#ifdef CONFIG_NVME_AUTH
>> +    u16 transaction;
>> +    u8 dhchap_hash;
>> +    u8 dhchap_dhgroup;
> 
> Do multiple controllers in the same subsystem have different
> params? no, so I think these should be lifted to subsys.
> 

It doesn't actually say in the spec; it always refers to the params as 
being set by the controller.
So it could be either; maybe we should ask for clafication at the fmds call.

>> +#endif
>> +
>>       /* Power saving configuration */
>>       u64 ps_max_latency_us;
>>       bool apst_enabled;
>> @@ -874,6 +880,15 @@ static inline bool nvme_ctrl_sgl_supported(struct 
>> nvme_ctrl *ctrl)
>>       return ctrl->sgls & ((1 << 0) | (1 << 1));
>>   }
>> +#ifdef CONFIG_NVME_AUTH
>> +int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid);
>> +#else
>> +static inline int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid)
>> +{
>> +    return -EPROTONOSUPPORT;
>> +}
>> +#endif
>> +
>>   u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
>>                u8 opcode);
>>   int nvme_execute_passthru_rq(struct request *rq);
>> diff --git a/drivers/nvme/host/trace.c b/drivers/nvme/host/trace.c
>> index 6543015b6121..66f75d8ea925 100644
>> --- a/drivers/nvme/host/trace.c
>> +++ b/drivers/nvme/host/trace.c
> 
> I'd split out the tracing logic.
> 
Okay.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 06/11] nvme: Implement In-Band authentication

[Stephan Müller]

Am Freitag, 16. Juli 2021, 13:04:23 CEST schrieb Hannes Reinecke:

Hi Hannes,

> Implement NVMe-oF In-Band authentication. This patch adds two new
> fabric options 'dhchap_key' to specify the PSK and 'dhchap_authenticate'
> to request bi-directional authentication of both the host and the
> controller.
> 
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>  drivers/nvme/host/Kconfig   |  11 +
>  drivers/nvme/host/Makefile  |   1 +
>  drivers/nvme/host/auth.c    | 813 ++++++++++++++++++++++++++++++++++++
>  drivers/nvme/host/auth.h    |  23 +
>  drivers/nvme/host/core.c    |  77 +++-
>  drivers/nvme/host/fabrics.c |  65 ++-
>  drivers/nvme/host/fabrics.h |   8 +
>  drivers/nvme/host/nvme.h    |  15 +
>  drivers/nvme/host/trace.c   |  32 ++
>  9 files changed, 1041 insertions(+), 4 deletions(-)
>  create mode 100644 drivers/nvme/host/auth.c
>  create mode 100644 drivers/nvme/host/auth.h
> 
> diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig
> index c3f3d77f1aac..853c546305e9 100644
> --- a/drivers/nvme/host/Kconfig
> +++ b/drivers/nvme/host/Kconfig
> @@ -85,3 +85,14 @@ config NVME_TCP
>  	  from https://github.com/linux-nvme/nvme-cli.
> 
>  	  If unsure, say N.
> +
> +config NVME_AUTH
> +	bool "NVM Express over Fabrics In-Band Authentication"
> +	depends on NVME_TCP
> +	select CRYPTO_SHA256
> +	select CRYPTO_SHA512

What about adding CRYPTO_HMAC here?

> +	help
> +	  This provides support for NVMe over Fabrics In-Band Authentication
> +	  for the NVMe over TCP transport.
> +
> +	  If unsure, say N.
> diff --git a/drivers/nvme/host/Makefile b/drivers/nvme/host/Makefile
> index cbc509784b2e..03748a55a12b 100644
> --- a/drivers/nvme/host/Makefile
> +++ b/drivers/nvme/host/Makefile
> @@ -20,6 +20,7 @@ nvme-core-$(CONFIG_NVME_HWMON)		+= hwmon.o
>  nvme-y					+= pci.o
> 
>  nvme-fabrics-y				+= fabrics.o
> +nvme-fabrics-$(CONFIG_NVME_AUTH)	+= auth.o
> 
>  nvme-rdma-y				+= rdma.o
> 
> diff --git a/drivers/nvme/host/auth.c b/drivers/nvme/host/auth.c
> new file mode 100644
> index 000000000000..448a3adebea6
> --- /dev/null
> +++ b/drivers/nvme/host/auth.c
> @@ -0,0 +1,813 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Copyright (c) 2020 Hannes Reinecke, SUSE Linux
> + */
> +
> +#include <linux/crc32.h>
> +#include <linux/base64.h>
> +#include <asm/unaligned.h>
> +#include <crypto/hash.h>
> +#include <crypto/kpp.h>
> +#include "nvme.h"
> +#include "fabrics.h"
> +#include "auth.h"
> +
> +static u32 nvme_dhchap_seqnum;
> +
> +struct nvme_dhchap_context {
> +	struct crypto_shash *shash_tfm;
> +	unsigned char *key;
> +	size_t key_len;
> +	int qid;
> +	u32 s1;
> +	u32 s2;
> +	u16 transaction;
> +	u8 status;
> +	u8 hash_id;
> +	u8 hash_len;
> +	u8 c1[64];
> +	u8 c2[64];
> +	u8 response[64];
> +	u8 *ctrl_key;
> +	int ctrl_key_len;
> +	u8 *host_key;
> +	int host_key_len;
> +	u8 *sess_key;
> +	int sess_key_len;
> +};
> +
> +struct nvmet_dhchap_hash_map {
> +	int id;
> +	int hash_len;
> +	const char hmac[15];
> +	const char digest[15];
> +} hash_map[] = {
> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA256,
> +	 .hash_len = 32,
> +	 .hmac = "hmac(sha256)", .digest = "sha256" },
> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA384,
> +	 .hash_len = 48,
> +	 .hmac = "hmac(sha384)", .digest = "sha384" },
> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA512,
> +	 .hash_len = 64,
> +	 .hmac = "hmac(sha512)", .digest = "sha512" },
> +};
> +
> +const char *nvme_auth_hmac_name(int hmac_id)
> +{
> +	int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
> +		if (hash_map[i].id == hmac_id)
> +			return hash_map[i].hmac;
> +	}
> +	return NULL;
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_hmac_name);
> +
> +const char *nvme_auth_digest_name(int hmac_id)
> +{
> +	int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
> +		if (hash_map[i].id == hmac_id)
> +			return hash_map[i].digest;
> +	}
> +	return NULL;
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_digest_name);
> +
> +int nvme_auth_hmac_len(int hmac_id)
> +{
> +	int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
> +		if (hash_map[i].id == hmac_id)
> +			return hash_map[i].hash_len;
> +	}
> +	return -1;
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_hmac_len);
> +
> +int nvme_auth_hmac_id(const char *hmac_name)
> +{
> +	int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
> +		if (!strncmp(hash_map[i].hmac, hmac_name,
> +			     strlen(hash_map[i].hmac)))
> +			return hash_map[i].id;
> +	}
> +	return -1;
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_hmac_id);
> +
> +unsigned char *nvme_auth_extract_secret(unsigned char *dhchap_secret,
> +					size_t *dhchap_key_len)
> +{
> +	unsigned char *dhchap_key;
> +	u32 crc;
> +	int key_len;
> +	size_t allocated_len;
> +
> +	allocated_len = strlen(dhchap_secret) - 10;

Are you sure that the string is always at least 10 bytes long? If so, can you 
please add a comment to it?

Also, is it guaranteed that we have an ASCII string? Note, a secret sounds to 
be like a binary string which may contain \0 as an appropriate value.

> +	dhchap_key = kzalloc(allocated_len, GFP_KERNEL);

What about aligning it to CRYPTO_MINALIGN_ATTR to save a memcpy in 
shash_final?

> +	if (!dhchap_key)
> +		return ERR_PTR(-ENOMEM);
> +
> +	key_len = base64_decode(dhchap_secret + 10,
> +				allocated_len, dhchap_key);
> +	if (key_len != 36 && key_len != 52 &&
> +	    key_len != 68) {
> +		pr_debug("Invalid DH-HMAC-CHAP key len %d\n",
> +			 key_len);
> +		kfree(dhchap_key);
> +		return ERR_PTR(-EINVAL);
> +	}
> +	pr_debug("DH-HMAC-CHAP Key: %*ph\n",
> +		 (int)key_len, dhchap_key);
> +
> +	/* The last four bytes is the CRC in little-endian format */
> +	key_len -= 4;
> +	/*
> +	 * The linux implementation doesn't do pre- and post-increments,
> +	 * so we have to do it manually.
> +	 */
> +	crc = ~crc32(~0, dhchap_key, key_len);
> +
> +	if (get_unaligned_le32(dhchap_key + key_len) != crc) {
> +		pr_debug("DH-HMAC-CHAP crc mismatch (key %08x, crc %08x)\n",
> +		       get_unaligned_le32(dhchap_key + key_len), crc);
> +		kfree(dhchap_key);
> +		return ERR_PTR(-EKEYREJECTED);
> +	}
> +	*dhchap_key_len = key_len;
> +	return dhchap_key;
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_extract_secret);
> +
> +static int nvme_auth_send(struct nvme_ctrl *ctrl, int qid,
> +			  void *data, size_t tl)
> +{
> +	struct nvme_command cmd = {};
> +	blk_mq_req_flags_t flags = qid == NVME_QID_ANY ?
> +		0 : BLK_MQ_REQ_NOWAIT | BLK_MQ_REQ_RESERVED;
> +	struct request_queue *q = qid == NVME_QID_ANY ?
> +		ctrl->fabrics_q : ctrl->connect_q;
> +	int ret;
> +
> +	cmd.auth_send.opcode = nvme_fabrics_command;
> +	cmd.auth_send.fctype = nvme_fabrics_type_auth_send;
> +	cmd.auth_send.secp = NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER;
> +	cmd.auth_send.spsp0 = 0x01;
> +	cmd.auth_send.spsp1 = 0x01;
> +	cmd.auth_send.tl = tl;
> +
> +	ret = __nvme_submit_sync_cmd(q, &cmd, NULL, data, tl, 0, qid,
> +				     0, flags);
> +	if (ret)
> +		dev_dbg(ctrl->device,
> +			"%s: qid %d error %d\n", __func__, qid, ret);
> +	return ret;
> +}
> +
> +static int nvme_auth_receive(struct nvme_ctrl *ctrl, int qid,
> +			     void *buf, size_t al,
> +			     u16 transaction, u8 expected_msg )
> +{
> +	struct nvme_command cmd = {};
> +	struct nvmf_auth_dhchap_failure_data *data = buf;
> +	blk_mq_req_flags_t flags = qid == NVME_QID_ANY ?
> +		0 : BLK_MQ_REQ_NOWAIT | BLK_MQ_REQ_RESERVED;
> +	struct request_queue *q = qid == NVME_QID_ANY ?
> +		ctrl->fabrics_q : ctrl->connect_q;
> +	int ret;
> +
> +	cmd.auth_receive.opcode = nvme_fabrics_command;
> +	cmd.auth_receive.fctype = nvme_fabrics_type_auth_receive;
> +	cmd.auth_receive.secp = NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER;
> +	cmd.auth_receive.spsp0 = 0x01;
> +	cmd.auth_receive.spsp1 = 0x01;
> +	cmd.auth_receive.al = al;
> +
> +	ret = __nvme_submit_sync_cmd(q, &cmd, NULL, buf, al, 0, qid,
> +				     0, flags);
> +	if (ret > 0) {
> +		dev_dbg(ctrl->device, "%s: qid %d nvme status %x\n",
> +			__func__, qid, ret);
> +		ret = -EIO;
> +	}
> +	if (ret < 0) {
> +		dev_dbg(ctrl->device, "%s: qid %d error %d\n",
> +			__func__, qid, ret);
> +		return ret;
> +	}
> +	dev_dbg(ctrl->device, "%s: qid %d auth_type %d auth_id %x\n",
> +		__func__, qid, data->auth_type, data->auth_id);
> +	if (data->auth_type == NVME_AUTH_COMMON_MESSAGES &&
> +	    data->auth_id == NVME_AUTH_DHCHAP_MESSAGE_FAILURE1) {
> +		return data->reason_code_explanation;
> +	}
> +	if (data->auth_type != NVME_AUTH_DHCHAP_MESSAGES ||
> +	    data->auth_id != expected_msg) {
> +		dev_warn(ctrl->device,
> +			 "qid %d invalid message %02x/%02x\n",
> +			 qid, data->auth_type, data->auth_id);
> +		return NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
> +	}
> +	if (le16_to_cpu(data->t_id) != transaction) {
> +		dev_warn(ctrl->device,
> +			 "qid %d invalid transaction ID %d\n",
> +			 qid, le16_to_cpu(data->t_id));
> +		return NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
> +	}
> +
> +	return 0;
> +}
> +
> +static int nvme_auth_dhchap_negotiate(struct nvme_ctrl *ctrl,
> +				      struct nvme_dhchap_context *chap,
> +				      void *buf, size_t buf_size)
> +{
> +	struct nvmf_auth_dhchap_negotiate_data *data = buf;
> +	size_t size = sizeof(*data) + sizeof(union nvmf_auth_protocol);
> +
> +	if (buf_size < size)
> +		return -EINVAL;
> +
> +	memset((u8 *)buf, 0, size);
> +	data->auth_type = NVME_AUTH_COMMON_MESSAGES;
> +	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE;
> +	data->t_id = cpu_to_le16(chap->transaction);
> +	data->sc_c = 0; /* No secure channel concatenation */
> +	data->napd = 1;
> +	data->auth_protocol[0].dhchap.authid = NVME_AUTH_DHCHAP_AUTH_ID;
> +	data->auth_protocol[0].dhchap.halen = 3;
> +	data->auth_protocol[0].dhchap.dhlen = 1;
> +	data->auth_protocol[0].dhchap.idlist[0] = NVME_AUTH_DHCHAP_HASH_SHA256;
> +	data->auth_protocol[0].dhchap.idlist[1] = NVME_AUTH_DHCHAP_HASH_SHA384;
> +	data->auth_protocol[0].dhchap.idlist[2] = NVME_AUTH_DHCHAP_HASH_SHA512;
> +	data->auth_protocol[0].dhchap.idlist[3] = NVME_AUTH_DHCHAP_DHGROUP_NULL;
> +
> +	return size;
> +}
> +
> +static int nvme_auth_dhchap_challenge(struct nvme_ctrl *ctrl,
> +				      struct nvme_dhchap_context *chap,
> +				      void *buf, size_t buf_size)
> +{
> +	struct nvmf_auth_dhchap_challenge_data *data = buf;
> +	size_t size = sizeof(*data) + data->hl + data->dhvlen;
> +	const char *gid_name;
> +
> +	if (buf_size < size) {
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
> +		return -ENOMSG;
> +	}
> +
> +	if (data->hashid != NVME_AUTH_DHCHAP_HASH_SHA256 &&
> +	    data->hashid != NVME_AUTH_DHCHAP_HASH_SHA384 &&
> +	    data->hashid != NVME_AUTH_DHCHAP_HASH_SHA512) {
> +		dev_warn(ctrl->device,
> +			 "qid %d: DH-HMAC-CHAP: invalid HASH ID %d\n",
> +			 chap->qid, data->hashid);
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
> +		return -EPROTO;
> +	}
> +	switch (data->dhgid) {
> +	case NVME_AUTH_DHCHAP_DHGROUP_NULL:
> +		gid_name = "null";
> +		break;
> +	default:
> +		gid_name = NULL;
> +		break;
> +	}
> +	if (!gid_name) {
> +		dev_warn(ctrl->device,
> +			 "qid %d: DH-HMAC-CHAP: invalid DH group id %d\n",
> +			 chap->qid, data->dhgid);
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
> +		return -EPROTO;
> +	}
> +	if (data->dhgid != NVME_AUTH_DHCHAP_DHGROUP_NULL) {
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
> +		return -EPROTO;
> +	}
> +	if (data->dhgid == NVME_AUTH_DHCHAP_DHGROUP_NULL && data->dhvlen != 0) {
> +		dev_warn(ctrl->device,
> +			 "qid %d: DH-HMAC-CHAP: invalid DH value for NULL DH\n",
> +			chap->qid);
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
> +		return -EPROTO;
> +	}
> +	dev_dbg(ctrl->device, "%s: qid %d requested hash id %d\n",
> +		__func__, chap->qid, data->hashid);
> +	if (nvme_auth_hmac_len(data->hashid) != data->hl) {
> +		dev_warn(ctrl->device,
> +			 "qid %d: DH-HMAC-CHAP: invalid hash length\n",
> +			chap->qid);
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
> +		return -EPROTO;
> +	}
> +	chap->hash_id = data->hashid;
> +	chap->hash_len = data->hl;
> +	chap->s1 = le32_to_cpu(data->seqnum);
> +	memcpy(chap->c1, data->cval, chap->hash_len);
> +
> +	return 0;
> +}
> +
> +static int nvme_auth_dhchap_reply(struct nvme_ctrl *ctrl,
> +				  struct nvme_dhchap_context *chap,
> +				  void *buf, size_t buf_size)
> +{
> +	struct nvmf_auth_dhchap_reply_data *data = buf;
> +	size_t size = sizeof(*data);
> +
> +	size += 2 * chap->hash_len;
> +	if (ctrl->opts->dhchap_auth) {
> +		get_random_bytes(chap->c2, chap->hash_len);

Why are you using CRYPTO_RNG_DEFAULT when you are using get_random_bytes here?

> +		chap->s2 = nvme_dhchap_seqnum++;
> +	} else
> +		memset(chap->c2, 0, chap->hash_len);
> +
> +	if (chap->host_key_len)
> +		size += chap->host_key_len;
> +
> +	if (buf_size < size)
> +		return -EINVAL;
> +
> +	memset(buf, 0, size);
> +	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
> +	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_REPLY;
> +	data->t_id = cpu_to_le16(chap->transaction);
> +	data->hl = chap->hash_len;
> +	data->dhvlen = chap->host_key_len;
> +	data->seqnum = cpu_to_le32(chap->s2);
> +	memcpy(data->rval, chap->response, chap->hash_len);
> +	if (ctrl->opts->dhchap_auth) {
> +		dev_dbg(ctrl->device, "%s: qid %d ctrl challenge %*ph\n",
> +			__func__, chap->qid,
> +			chap->hash_len, chap->c2);
> +		data->cvalid = 1;
> +		memcpy(data->rval + chap->hash_len, chap->c2,
> +		       chap->hash_len);
> +	}
> +	if (chap->host_key_len)
> +		memcpy(data->rval + 2 * chap->hash_len, chap->host_key,
> +		       chap->host_key_len);
> +
> +	return size;
> +}
> +
> +static int nvme_auth_dhchap_success1(struct nvme_ctrl *ctrl,
> +				     struct nvme_dhchap_context *chap,
> +				     void *buf, size_t buf_size)
> +{
> +	struct nvmf_auth_dhchap_success1_data *data = buf;
> +	size_t size = sizeof(*data);
> +
> +	if (ctrl->opts->dhchap_auth)
> +		size += chap->hash_len;
> +
> +
> +	if (buf_size < size) {
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
> +		return -ENOMSG;
> +	}
> +
> +	if (data->hl != chap->hash_len) {
> +		dev_warn(ctrl->device,
> +			 "qid %d: DH-HMAC-CHAP: invalid hash length %d\n",
> +			 chap->qid, data->hl);
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
> +		return -EPROTO;
> +	}
> +
> +	if (!data->rvalid)
> +		return 0;
> +
> +	/* Validate controller response */
> +	if (memcmp(chap->response, data->rval, data->hl)) {
> +		dev_dbg(ctrl->device, "%s: qid %d ctrl response %*ph\n",
> +			__func__, chap->qid, chap->hash_len, data->rval);
> +		dev_dbg(ctrl->device, "%s: qid %d host response %*ph\n",
> +			__func__, chap->qid, chap->hash_len, chap->response);
> +		dev_warn(ctrl->device,
> +			 "qid %d: DH-HMAC-CHAP: controller authentication failed\n",
> +			 chap->qid);
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
> +		return -EPROTO;
> +	}
> +	dev_info(ctrl->device,
> +		 "qid %d: DH-HMAC-CHAP: controller authenticated\n",
> +		chap->qid);
> +	return 0;
> +}
> +
> +static int nvme_auth_dhchap_success2(struct nvme_ctrl *ctrl,
> +				     struct nvme_dhchap_context *chap,
> +				     void *buf, size_t buf_size)
> +{
> +	struct nvmf_auth_dhchap_success2_data *data = buf;
> +	size_t size = sizeof(*data);
> +
> +	memset(buf, 0, size);
> +	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
> +	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_SUCCESS2;
> +	data->t_id = cpu_to_le16(chap->transaction);
> +
> +	return size;
> +}
> +
> +static int nvme_auth_dhchap_failure2(struct nvme_ctrl *ctrl,
> +				     struct nvme_dhchap_context *chap,
> +				     void *buf, size_t buf_size)
> +{
> +	struct nvmf_auth_dhchap_failure_data *data = buf;
> +	size_t size = sizeof(*data);
> +
> +	memset(buf, 0, size);
> +	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
> +	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_FAILURE2;
> +	data->t_id = cpu_to_le16(chap->transaction);
> +	data->reason_code = 1;
> +	data->reason_code_explanation = chap->status;
> +
> +	return size;
> +}
> +
> +int nvme_auth_select_hash(struct nvme_ctrl *ctrl,
> +			  struct nvme_dhchap_context *chap)
> +{
> +	char *hash_name;
> +	int ret;
> +
> +	switch (chap->hash_id) {
> +	case NVME_AUTH_DHCHAP_HASH_SHA256:
> +		hash_name = "hmac(sha256)";
> +		break;
> +	case NVME_AUTH_DHCHAP_HASH_SHA384:
> +		hash_name = "hmac(sha384)";
> +		break;
> +	case NVME_AUTH_DHCHAP_HASH_SHA512:
> +		hash_name = "hmac(sha512)";
> +		break;
> +	default:
> +		hash_name = NULL;
> +		break;
> +	}
> +	if (!hash_name) {
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
> +		return -EPROTO;
> +	}
> +	chap->shash_tfm = crypto_alloc_shash(hash_name, 0,
> +					     CRYPTO_ALG_ALLOCATES_MEMORY);
> +	if (IS_ERR(chap->shash_tfm)) {
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
> +		chap->shash_tfm = NULL;
> +		return -EPROTO;
> +	}
> +	if (!chap->key) {
> +		dev_warn(ctrl->device, "qid %d: cannot select hash, no key\n",
> +			 chap->qid);
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
> +		crypto_free_shash(chap->shash_tfm);
> +		chap->shash_tfm = NULL;
> +		return -EINVAL;
> +	}
> +	ret = crypto_shash_setkey(chap->shash_tfm, chap->key, chap->key_len);
> +	if (ret) {
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
> +		crypto_free_shash(chap->shash_tfm);
> +		chap->shash_tfm = NULL;
> +		return ret;
> +	}
> +	dev_info(ctrl->device, "qid %d: DH-HMAC_CHAP: selected hash %s\n",
> +		 chap->qid, hash_name);
> +	return 0;
> +}
> +
> +static int nvme_auth_dhchap_host_response(struct nvme_ctrl *ctrl,
> +					  struct nvme_dhchap_context *chap)
> +{
> +	SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
> +	u8 buf[4], *challenge = chap->c1;
> +	int ret;
> +
> +	dev_dbg(ctrl->device, "%s: qid %d host response seq %d transaction 
%d\n",
> +		__func__, chap->qid, chap->s1, chap->transaction);
> +	shash->tfm = chap->shash_tfm;
> +	ret = crypto_shash_init(shash);
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, challenge, chap->hash_len);
> +	if (ret)
> +		goto out;
> +	put_unaligned_le32(chap->s1, buf);
> +	ret = crypto_shash_update(shash, buf, 4);
> +	if (ret)
> +		goto out;
> +	put_unaligned_le16(chap->transaction, buf);
> +	ret = crypto_shash_update(shash, buf, 2);
> +	if (ret)
> +		goto out;
> +	memset(buf, 0, sizeof(buf));
> +	ret = crypto_shash_update(shash, buf, 1);
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, "HostHost", 8);
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
> +				  strlen(ctrl->opts->host->nqn));
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, buf, 1);
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
> +			    strlen(ctrl->opts->subsysnqn));
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_final(shash, chap->response);
> +out:
> +	return ret;
> +}
> +
> +static int nvme_auth_dhchap_ctrl_response(struct nvme_ctrl *ctrl,
> +					  struct nvme_dhchap_context *chap)
> +{
> +	SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
> +	u8 buf[4], *challenge = chap->c2;
> +	int ret;
> +
> +	dev_dbg(ctrl->device, "%s: qid %d host response seq %d transaction 
%d\n",
> +		__func__, chap->qid, chap->s2, chap->transaction);
> +	dev_dbg(ctrl->device, "%s: qid %d challenge %*ph\n",
> +		__func__, chap->qid, chap->hash_len, challenge);
> +	dev_dbg(ctrl->device, "%s: qid %d subsysnqn %s\n",
> +		__func__, chap->qid, ctrl->opts->subsysnqn);
> +	dev_dbg(ctrl->device, "%s: qid %d hostnqn %s\n",
> +		__func__, chap->qid, ctrl->opts->host->nqn);
> +	shash->tfm = chap->shash_tfm;
> +	ret = crypto_shash_init(shash);
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, challenge, chap->hash_len);
> +	if (ret)
> +		goto out;
> +	put_unaligned_le32(chap->s2, buf);
> +	ret = crypto_shash_update(shash, buf, 4);
> +	if (ret)
> +		goto out;
> +	put_unaligned_le16(chap->transaction, buf);
> +	ret = crypto_shash_update(shash, buf, 2);
> +	if (ret)
> +		goto out;
> +	memset(buf, 0, 4);
> +	ret = crypto_shash_update(shash, buf, 1);
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, "Controller", 10);
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
> +				  strlen(ctrl->opts->subsysnqn));
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, buf, 1);
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
> +				  strlen(ctrl->opts->host->nqn));
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_final(shash, chap->response);
> +out:
> +	return ret;
> +}
> +
> +int nvme_auth_generate_key(struct nvme_ctrl *ctrl,
> +			   struct nvme_dhchap_context *chap)
> +{
> +	int ret;
> +	u8 key_hash;
> +	const char *hmac_name;
> +	struct crypto_shash *key_tfm;
> +
> +	if (sscanf(ctrl->opts->dhchap_secret, "DHHC-1:%hhd:%*s:",
> +		   &key_hash) != 1)
> +		return -EINVAL;
> +
> +	chap->key = nvme_auth_extract_secret(ctrl->opts->dhchap_secret,
> +					     &chap->key_len);
> +	if (IS_ERR(chap->key)) {
> +		ret = PTR_ERR(chap->key);
> +		chap->key = NULL;
> +		return ret;
> +	}
> +
> +	if (key_hash == 0)
> +		return 0;
> +
> +	hmac_name = nvme_auth_hmac_name(key_hash);
> +	if (!hmac_name) {
> +		pr_debug("Invalid key hash id %d\n", key_hash);
> +		return -EKEYREJECTED;
> +	}
> +
> +	key_tfm = crypto_alloc_shash(hmac_name, 0, 0);
> +	if (IS_ERR(key_tfm)) {
> +		kfree(chap->key);
> +		chap->key = NULL;
> +		ret = PTR_ERR(key_tfm);
> +	} else {
> +		SHASH_DESC_ON_STACK(shash, key_tfm);
> +
> +		shash->tfm = key_tfm;
> +		ret = crypto_shash_setkey(key_tfm, chap->key,
> +					  chap->key_len);
> +		if (ret < 0) {
> +			crypto_free_shash(key_tfm);
> +			kfree(chap->key);
> +			chap->key = NULL;
> +			return ret;
> +		}
> +		crypto_shash_init(shash);
> +		crypto_shash_update(shash, ctrl->opts->host->nqn,
> +				    strlen(ctrl->opts->host->nqn));
> +		crypto_shash_update(shash, "NVMe-over-Fabrics", 17);
> +		crypto_shash_final(shash, chap->key);
> +		crypto_free_shash(key_tfm);
> +	}
> +	return 0;
> +}
> +
> +void nvme_auth_free(struct nvme_dhchap_context *chap)
> +{
> +	if (chap->shash_tfm)
> +		crypto_free_shash(chap->shash_tfm);
> +	if (chap->key)
> +		kfree(chap->key);
> +	if (chap->ctrl_key)
> +		kfree(chap->ctrl_key);
> +	if (chap->host_key)
> +		kfree(chap->host_key);
> +	if (chap->sess_key)
> +		kfree(chap->sess_key);
> +	kfree(chap);

kfree_sensitive in all cases as all buffers have sensitive data?

> +}
> +
> +int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid)
> +{
> +	struct nvme_dhchap_context *chap;
> +	void *buf;
> +	size_t buf_size, tl;
> +	int ret = 0;
> +
> +	chap = kzalloc(sizeof(*chap), GFP_KERNEL);

Suggestion: make sure that chap->response is aligned to CRYPTO_MINALIGN_ATTR - 
then you would save a memcpy in crypto_shash_final

> +	if (!chap)
> +		return -ENOMEM;
> +	chap->qid = qid;
> +	chap->transaction = ctrl->transaction++;
> +
> +	ret = nvme_auth_generate_key(ctrl, chap);
> +	if (ret) {
> +		dev_dbg(ctrl->device, "%s: failed to generate key, error %d\n",
> +			__func__, ret);
> +		nvme_auth_free(chap);
> +		return ret;
> +	}
> +
> +	/*
> +	 * Allocate a large enough buffer for the entire negotiation:
> +	 * 4k should be enough to ffdhe8192.
> +	 */
> +	buf_size = 4096;
> +	buf = kzalloc(buf_size, GFP_KERNEL);
> +	if (!buf) {
> +		ret = -ENOMEM;
> +		goto out;
> +	}
> +
> +	/* DH-HMAC-CHAP Step 1: send negotiate */
> +	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP negotiate\n",
> +		__func__, qid);
> +	ret = nvme_auth_dhchap_negotiate(ctrl, chap, buf, buf_size);
> +	if (ret < 0)
> +		goto out;
> +	tl = ret;
> +	ret = nvme_auth_send(ctrl, qid, buf, tl);
> +	if (ret)
> +		goto out;
> +
> +	memset(buf, 0, buf_size);
> +	ret = nvme_auth_receive(ctrl, qid, buf, buf_size, chap->transaction,
> +				NVME_AUTH_DHCHAP_MESSAGE_CHALLENGE);
> +	if (ret < 0) {
> +		dev_dbg(ctrl->device,
> +			"%s: qid %d DH-HMAC-CHAP failed to receive challenge\n",
> +			__func__, qid);
> +		goto out;
> +	}
> +	if (ret > 0) {
> +		chap->status = ret;
> +		goto fail1;
> +	}
> +
> +	/* DH-HMAC-CHAP Step 2: receive challenge */
> +	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP challenge\n",
> +		__func__, qid);
> +
> +	ret = nvme_auth_dhchap_challenge(ctrl, chap, buf, buf_size);
> +	if (ret) {
> +		/* Invalid parameters for negotiate */
> +		goto fail2;
> +	}
> +
> +	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP select hash\n",
> +		__func__, qid);
> +	ret = nvme_auth_select_hash(ctrl, chap);
> +	if (ret)
> +		goto fail2;
> +
> +	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP host response\n",
> +		__func__, qid);
> +	ret = nvme_auth_dhchap_host_response(ctrl, chap);
> +	if (ret)
> +		goto fail2;
> +
> +	/* DH-HMAC-CHAP Step 3: send reply */
> +	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP reply\n",
> +		__func__, qid);
> +	ret = nvme_auth_dhchap_reply(ctrl, chap, buf, buf_size);
> +	if (ret < 0)
> +		goto fail2;
> +
> +	tl = ret;
> +	ret = nvme_auth_send(ctrl, qid, buf, tl);
> +	if (ret)
> +		goto fail2;
> +
> +	memset(buf, 0, buf_size);
> +	ret = nvme_auth_receive(ctrl, qid, buf, buf_size, chap->transaction,
> +				NVME_AUTH_DHCHAP_MESSAGE_SUCCESS1);
> +	if (ret < 0) {
> +		dev_dbg(ctrl->device,
> +			"%s: qid %d DH-HMAC-CHAP failed to receive success1\n",
> +			__func__, qid);
> +		goto out;
> +	}
> +	if (ret > 0) {
> +		chap->status = ret;
> +		goto fail1;
> +	}
> +
> +	if (ctrl->opts->dhchap_auth) {
> +		dev_dbg(ctrl->device,
> +			"%s: qid %d DH-HMAC-CHAP controller response\n",
> +			__func__, qid);
> +		ret = nvme_auth_dhchap_ctrl_response(ctrl, chap);
> +		if (ret)
> +			goto fail2;
> +	}
> +
> +	/* DH-HMAC-CHAP Step 4: receive success1 */
> +	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP success1\n",
> +		__func__, qid);
> +	ret = nvme_auth_dhchap_success1(ctrl, chap, buf, buf_size);
> +	if (ret < 0) {
> +		/* Controller authentication failed */
> +		goto fail2;
> +	}
> +	tl = ret;
> +	/* DH-HMAC-CHAP Step 5: send success2 */
> +	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP success2\n",
> +		__func__, qid);
> +	tl = nvme_auth_dhchap_success2(ctrl, chap, buf, buf_size);
> +	ret = nvme_auth_send(ctrl, qid, buf, tl);
> +	if (!ret)
> +		goto out;
> +
> +fail1:
> +	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP failure1, status %x\n",
> +		__func__, qid, chap->status);
> +	goto out;
> +
> +fail2:
> +	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP failure2, status %x\n",
> +		__func__, qid, chap->status);
> +	tl = nvme_auth_dhchap_failure2(ctrl, chap, buf, buf_size);
> +	ret = nvme_auth_send(ctrl, qid, buf, tl);
> +
> +out:
> +	if (!ret && chap->status)
> +		ret = -EPROTO;
> +	if (!ret) {
> +		ctrl->dhchap_hash = chap->hash_id;
> +	}
> +	kfree(buf);
> +	nvme_auth_free(chap);
> +	return ret;
> +}
> diff --git a/drivers/nvme/host/auth.h b/drivers/nvme/host/auth.h
> new file mode 100644
> index 000000000000..4950b1cb9470
> --- /dev/null
> +++ b/drivers/nvme/host/auth.h
> @@ -0,0 +1,23 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Copyright (c) 2021 Hannes Reinecke, SUSE Software Solutions
> + */
> +
> +#ifndef _NVME_AUTH_H
> +#define _NVME_AUTH_H
> +
> +const char *nvme_auth_dhgroup_name(int dhgroup_id);
> +int nvme_auth_dhgroup_pubkey_size(int dhgroup_id);
> +int nvme_auth_dhgroup_privkey_size(int dhgroup_id);
> +const char *nvme_auth_dhgroup_kpp(int dhgroup_id);
> +int nvme_auth_dhgroup_id(const char *dhgroup_name);
> +
> +const char *nvme_auth_hmac_name(int hmac_id);
> +const char *nvme_auth_digest_name(int hmac_id);
> +int nvme_auth_hmac_id(const char *hmac_name);
> +int nvme_auth_hmac_len(int hmac_len);
> +
> +unsigned char *nvme_auth_extract_secret(unsigned char *dhchap_secret,
> +					size_t *dhchap_key_len);
> +
> +#endif /* _NVME_AUTH_H */
> diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
> index 11779be42186..7ce9b666dc09 100644
> --- a/drivers/nvme/host/core.c
> +++ b/drivers/nvme/host/core.c
> @@ -708,7 +708,9 @@ bool __nvme_check_ready(struct nvme_ctrl *ctrl, struct
> request *rq, switch (ctrl->state) {
>  		case NVME_CTRL_CONNECTING:
>  			if (blk_rq_is_passthrough(rq) && nvme_is_fabrics(req->cmd) 
&&
> -			    req->cmd->fabrics.fctype == nvme_fabrics_type_connect)
> +			    (req->cmd->fabrics.fctype == nvme_fabrics_type_connect 
||
> +			     req->cmd->fabrics.fctype == nvme_fabrics_type_auth_send 
||
> +			     req->cmd->fabrics.fctype == 
nvme_fabrics_type_auth_receive))
>  				return true;
>  			break;
>  		default:
> @@ -3426,6 +3428,66 @@ static ssize_t
> nvme_ctrl_fast_io_fail_tmo_store(struct device *dev, static
> DEVICE_ATTR(fast_io_fail_tmo, S_IRUGO | S_IWUSR,
>  	nvme_ctrl_fast_io_fail_tmo_show, nvme_ctrl_fast_io_fail_tmo_store);
> 
> +#ifdef CONFIG_NVME_AUTH
> +struct nvmet_dhchap_hash_map {
> +	int id;
> +	const char name[15];
> +} hash_map[] = {
> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA256,
> +	 .name = "hmac(sha256)", },
> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA384,
> +	 .name = "hmac(sha384)", },
> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA512,
> +	 .name = "hmac(sha512)", },
> +};
> +
> +static ssize_t dhchap_hash_show(struct device *dev,
> +	struct device_attribute *attr, char *buf)
> +{
> +	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
> +	int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
> +		if (hash_map[i].id == ctrl->dhchap_hash)
> +			return sprintf(buf, "%s\n", hash_map[i].name);
> +	}
> +	return sprintf(buf, "none\n");
> +}
> +DEVICE_ATTR_RO(dhchap_hash);
> +
> +struct nvmet_dhchap_group_map {
> +	int id;
> +	const char name[15];
> +} dhgroup_map[] = {
> +	{.id = NVME_AUTH_DHCHAP_DHGROUP_NULL,
> +	 .name = "NULL", },
> +	{.id = NVME_AUTH_DHCHAP_DHGROUP_2048,
> +	 .name = "ffdhe2048", },
> +	{.id = NVME_AUTH_DHCHAP_DHGROUP_3072,
> +	 .name = "ffdhe3072", },
> +	{.id = NVME_AUTH_DHCHAP_DHGROUP_4096,
> +	 .name = "ffdhe4096", },
> +	{.id = NVME_AUTH_DHCHAP_DHGROUP_6144,
> +	 .name = "ffdhe6144", },
> +	{.id = NVME_AUTH_DHCHAP_DHGROUP_8192,
> +	 .name = "ffdhe8192", },
> +};
> +
> +static ssize_t dhchap_dhgroup_show(struct device *dev,
> +	struct device_attribute *attr, char *buf)
> +{
> +	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
> +	int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(dhgroup_map); i++) {
> +		if (hash_map[i].id == ctrl->dhchap_dhgroup)
> +			return sprintf(buf, "%s\n", dhgroup_map[i].name);
> +	}
> +	return sprintf(buf, "none\n");
> +}
> +DEVICE_ATTR_RO(dhchap_dhgroup);
> +#endif
> +
>  static struct attribute *nvme_dev_attrs[] = {
>  	&dev_attr_reset_controller.attr,
>  	&dev_attr_rescan_controller.attr,
> @@ -3447,6 +3509,10 @@ static struct attribute *nvme_dev_attrs[] = {
>  	&dev_attr_reconnect_delay.attr,
>  	&dev_attr_fast_io_fail_tmo.attr,
>  	&dev_attr_kato.attr,
> +#ifdef CONFIG_NVME_AUTH
> +	&dev_attr_dhchap_hash.attr,
> +	&dev_attr_dhchap_dhgroup.attr,
> +#endif
>  	NULL
>  };
> 
> @@ -3470,6 +3536,10 @@ static umode_t nvme_dev_attrs_are_visible(struct
> kobject *kobj, return 0;
>  	if (a == &dev_attr_fast_io_fail_tmo.attr && !ctrl->opts)
>  		return 0;
> +#ifdef CONFIG_NVME_AUTH
> +	if (a == &dev_attr_dhchap_hash.attr && !ctrl->opts)
> +		return 0;
> +#endif
> 
>  	return a->mode;
>  }
> @@ -4581,6 +4651,11 @@ static inline void _nvme_check_size(void)
>  	BUILD_BUG_ON(sizeof(struct nvme_smart_log) != 512);
>  	BUILD_BUG_ON(sizeof(struct nvme_dbbuf) != 64);
>  	BUILD_BUG_ON(sizeof(struct nvme_directive_cmd) != 64);
> +	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_negotiate_data) != 8);
> +	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_challenge_data) != 16);
> +	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_reply_data) != 16);
> +	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_success1_data) != 16);
> +	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_success2_data) != 16);
>  }
> 
> 
> diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
> index a5469fd9d4c3..6404ab9b604b 100644
> --- a/drivers/nvme/host/fabrics.c
> +++ b/drivers/nvme/host/fabrics.c
> @@ -366,6 +366,7 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)
>  	union nvme_result res;
>  	struct nvmf_connect_data *data;
>  	int ret;
> +	u32 result;
> 
>  	cmd.connect.opcode = nvme_fabrics_command;
>  	cmd.connect.fctype = nvme_fabrics_type_connect;
> @@ -398,8 +399,18 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)
>  		goto out_free_data;
>  	}
> 
> -	ctrl->cntlid = le16_to_cpu(res.u16);
> -
> +	result = le32_to_cpu(res.u32);
> +	ctrl->cntlid = result & 0xFFFF;
> +	if ((result >> 16) & 2) {
> +		/* Authentication required */
> +		ret = nvme_auth_negotiate(ctrl, NVME_QID_ANY);
> +		if (ret)
> +			dev_warn(ctrl->device,
> +				 "qid 0: authentication failed\n");
> +		else
> +			dev_info(ctrl->device,
> +				 "qid 0: authenticated\n");
> +	}
>  out_free_data:
>  	kfree(data);
>  	return ret;
> @@ -432,6 +443,7 @@ int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, u16
> qid) struct nvmf_connect_data *data;
>  	union nvme_result res;
>  	int ret;
> +	u32 result;
> 
>  	cmd.connect.opcode = nvme_fabrics_command;
>  	cmd.connect.fctype = nvme_fabrics_type_connect;
> @@ -457,6 +469,17 @@ int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, u16
> qid) nvmf_log_connect_error(ctrl, ret, le32_to_cpu(res.u32),
>  				       &cmd, data);
>  	}
> +	result = le32_to_cpu(res.u32);
> +	if ((result >> 16) & 2) {
> +		/* Authentication required */
> +		ret = nvme_auth_negotiate(ctrl, qid);
> +		if (ret)
> +			dev_warn(ctrl->device,
> +				 "qid %u: authentication failed\n", qid);
> +		else
> +			dev_info(ctrl->device,
> +				 "qid %u: authenticated\n", qid);
> +	}
>  	kfree(data);
>  	return ret;
>  }
> @@ -548,6 +571,9 @@ static const match_table_t opt_tokens = {
>  	{ NVMF_OPT_NR_POLL_QUEUES,	"nr_poll_queues=%d"	},
>  	{ NVMF_OPT_TOS,			"tos=%d"		},
>  	{ NVMF_OPT_FAIL_FAST_TMO,	"fast_io_fail_tmo=%d"	},
> +	{ NVMF_OPT_DHCHAP_SECRET,	"dhchap_secret=%s"	},
> +	{ NVMF_OPT_DHCHAP_AUTH,		"authenticate"		},
> +	{ NVMF_OPT_DHCHAP_GROUP,	"dhchap_group=%s"	},
>  	{ NVMF_OPT_ERR,			NULL			}
>  };
> 
> @@ -824,6 +850,35 @@ static int nvmf_parse_options(struct nvmf_ctrl_options
> *opts, }
>  			opts->tos = token;
>  			break;
> +		case NVMF_OPT_DHCHAP_SECRET:
> +			p = match_strdup(args);
> +			if (!p) {
> +				ret = -ENOMEM;
> +				goto out;
> +			}
> +			if (strncmp(p, "DHHC-1:00:", 10)) {
> +				pr_err("Invalid DH-CHAP secret %s\n", p);
> +				ret = -EINVAL;
> +				goto out;
> +			}
> +			kfree(opts->dhchap_secret);
> +			opts->dhchap_secret = p;
> +			break;
> +		case NVMF_OPT_DHCHAP_AUTH:
> +			opts->dhchap_auth = true;
> +			break;
> +		case NVMF_OPT_DHCHAP_GROUP:
> +			if (match_int(args, &token)) {
> +				ret = -EINVAL;
> +				goto out;
> +			}
> +			if (token <= 0) {
> +				pr_err("Invalid dhchap_group %d\n", token);
> +				ret = -EINVAL;
> +				goto out;
> +			}
> +			opts->dhchap_group = token;
> +			break;
>  		default:
>  			pr_warn("unknown parameter or missing value '%s' in ctrl 
creation
> request\n", p);
> @@ -942,6 +997,7 @@ void nvmf_free_options(struct nvmf_ctrl_options *opts)
>  	kfree(opts->subsysnqn);
>  	kfree(opts->host_traddr);
>  	kfree(opts->host_iface);
> +	kfree(opts->dhchap_secret);
>  	kfree(opts);
>  }
>  EXPORT_SYMBOL_GPL(nvmf_free_options);
> @@ -951,7 +1007,10 @@ EXPORT_SYMBOL_GPL(nvmf_free_options);
>  				 NVMF_OPT_KATO | NVMF_OPT_HOSTNQN | \
>  				 NVMF_OPT_HOST_ID | NVMF_OPT_DUP_CONNECT |\
>  				 NVMF_OPT_DISABLE_SQFLOW |\
> -				 NVMF_OPT_FAIL_FAST_TMO)
> +				 NVMF_OPT_CTRL_LOSS_TMO |\
> +				 NVMF_OPT_FAIL_FAST_TMO |\
> +				 NVMF_OPT_DHCHAP_SECRET |\
> +				 NVMF_OPT_DHCHAP_AUTH | NVMF_OPT_DHCHAP_GROUP)
> 
>  static struct nvme_ctrl *
>  nvmf_create_ctrl(struct device *dev, const char *buf)
> diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h
> index a146cb903869..535bc544f0f6 100644
> --- a/drivers/nvme/host/fabrics.h
> +++ b/drivers/nvme/host/fabrics.h
> @@ -67,6 +67,9 @@ enum {
>  	NVMF_OPT_TOS		= 1 << 19,
>  	NVMF_OPT_FAIL_FAST_TMO	= 1 << 20,
>  	NVMF_OPT_HOST_IFACE	= 1 << 21,
> +	NVMF_OPT_DHCHAP_SECRET	= 1 << 22,
> +	NVMF_OPT_DHCHAP_AUTH	= 1 << 23,
> +	NVMF_OPT_DHCHAP_GROUP	= 1 << 24,
>  };
> 
>  /**
> @@ -96,6 +99,8 @@ enum {
>   * @max_reconnects: maximum number of allowed reconnect attempts before
> removing *              the controller, (-1) means reconnect forever, zero
> means remove *              immediately;
> + * @dhchap_secret: DH-HMAC-CHAP secret
> + * @dhchap_auth: DH-HMAC-CHAP authenticate controller
>   * @disable_sqflow: disable controller sq flow control
>   * @hdr_digest: generate/verify header digest (TCP)
>   * @data_digest: generate/verify data digest (TCP)
> @@ -120,6 +125,9 @@ struct nvmf_ctrl_options {
>  	unsigned int		kato;
>  	struct nvmf_host	*host;
>  	int			max_reconnects;
> +	char			*dhchap_secret;
> +	int			dhchap_group;
> +	bool			dhchap_auth;
>  	bool			disable_sqflow;
>  	bool			hdr_digest;
>  	bool			data_digest;
> diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
> index 18ef8dd03a90..bcd5b8276c26 100644
> --- a/drivers/nvme/host/nvme.h
> +++ b/drivers/nvme/host/nvme.h
> @@ -328,6 +328,12 @@ struct nvme_ctrl {
>  	struct work_struct ana_work;
>  #endif
> 
> +#ifdef CONFIG_NVME_AUTH
> +	u16 transaction;
> +	u8 dhchap_hash;
> +	u8 dhchap_dhgroup;
> +#endif
> +
>  	/* Power saving configuration */
>  	u64 ps_max_latency_us;
>  	bool apst_enabled;
> @@ -874,6 +880,15 @@ static inline bool nvme_ctrl_sgl_supported(struct
> nvme_ctrl *ctrl) return ctrl->sgls & ((1 << 0) | (1 << 1));
>  }
> 
> +#ifdef CONFIG_NVME_AUTH
> +int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid);
> +#else
> +static inline int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid)
> +{
> +	return -EPROTONOSUPPORT;
> +}
> +#endif
> +
>  u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
>  			 u8 opcode);
>  int nvme_execute_passthru_rq(struct request *rq);
> diff --git a/drivers/nvme/host/trace.c b/drivers/nvme/host/trace.c
> index 6543015b6121..66f75d8ea925 100644
> --- a/drivers/nvme/host/trace.c
> +++ b/drivers/nvme/host/trace.c
> @@ -271,6 +271,34 @@ static const char
> *nvme_trace_fabrics_property_get(struct trace_seq *p, u8 *spc) return ret;
>  }
> 
> +static const char *nvme_trace_fabrics_auth_send(struct trace_seq *p, u8
> *spc) +{
> +	const char *ret = trace_seq_buffer_ptr(p);
> +	u8 spsp0 = spc[1];
> +	u8 spsp1 = spc[2];
> +	u8 secp = spc[3];
> +	u32 tl = get_unaligned_le32(spc + 4);
> +
> +	trace_seq_printf(p, "spsp0=%02x, spsp1=%02x, secp=%02x, tl=%u",
> +			 spsp0, spsp1, secp, tl);
> +	trace_seq_putc(p, 0);
> +	return ret;
> +}
> +
> +static const char *nvme_trace_fabrics_auth_receive(struct trace_seq *p, u8
> *spc) +{
> +	const char *ret = trace_seq_buffer_ptr(p);
> +	u8 spsp0 = spc[1];
> +	u8 spsp1 = spc[2];
> +	u8 secp = spc[3];
> +	u32 al = get_unaligned_le32(spc + 4);
> +
> +	trace_seq_printf(p, "spsp0=%02x, spsp1=%02x, secp=%02x, al=%u",
> +			 spsp0, spsp1, secp, al);
> +	trace_seq_putc(p, 0);
> +	return ret;
> +}
> +
>  static const char *nvme_trace_fabrics_common(struct trace_seq *p, u8 *spc)
>  {
>  	const char *ret = trace_seq_buffer_ptr(p);
> @@ -290,6 +318,10 @@ const char *nvme_trace_parse_fabrics_cmd(struct
> trace_seq *p, return nvme_trace_fabrics_connect(p, spc);
>  	case nvme_fabrics_type_property_get:
>  		return nvme_trace_fabrics_property_get(p, spc);
> +	case nvme_fabrics_type_auth_send:
> +		return nvme_trace_fabrics_auth_send(p, spc);
> +	case nvme_fabrics_type_auth_receive:
> +		return nvme_trace_fabrics_auth_receive(p, spc);
>  	default:
>  		return nvme_trace_fabrics_common(p, spc);
>  	}


Ciao
Stephan

Re: [PATCH 06/11] nvme: Implement In-Band authentication

[Stephan Müller]

Am Freitag, 16. Juli 2021, 13:04:23 CEST schrieb Hannes Reinecke:

Hi Hannes,

> Implement NVMe-oF In-Band authentication. This patch adds two new
> fabric options 'dhchap_key' to specify the PSK and 'dhchap_authenticate'
> to request bi-directional authentication of both the host and the
> controller.
> 
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>  drivers/nvme/host/Kconfig   |  11 +
>  drivers/nvme/host/Makefile  |   1 +
>  drivers/nvme/host/auth.c    | 813 ++++++++++++++++++++++++++++++++++++
>  drivers/nvme/host/auth.h    |  23 +
>  drivers/nvme/host/core.c    |  77 +++-
>  drivers/nvme/host/fabrics.c |  65 ++-
>  drivers/nvme/host/fabrics.h |   8 +
>  drivers/nvme/host/nvme.h    |  15 +
>  drivers/nvme/host/trace.c   |  32 ++
>  9 files changed, 1041 insertions(+), 4 deletions(-)
>  create mode 100644 drivers/nvme/host/auth.c
>  create mode 100644 drivers/nvme/host/auth.h
> 
> diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig
> index c3f3d77f1aac..853c546305e9 100644
> --- a/drivers/nvme/host/Kconfig
> +++ b/drivers/nvme/host/Kconfig
> @@ -85,3 +85,14 @@ config NVME_TCP
>  	  from https://github.com/linux-nvme/nvme-cli.
> 
>  	  If unsure, say N.
> +
> +config NVME_AUTH
> +	bool "NVM Express over Fabrics In-Band Authentication"
> +	depends on NVME_TCP
> +	select CRYPTO_SHA256
> +	select CRYPTO_SHA512

What about adding CRYPTO_HMAC here?

> +	help
> +	  This provides support for NVMe over Fabrics In-Band Authentication
> +	  for the NVMe over TCP transport.
> +
> +	  If unsure, say N.
> diff --git a/drivers/nvme/host/Makefile b/drivers/nvme/host/Makefile
> index cbc509784b2e..03748a55a12b 100644
> --- a/drivers/nvme/host/Makefile
> +++ b/drivers/nvme/host/Makefile
> @@ -20,6 +20,7 @@ nvme-core-$(CONFIG_NVME_HWMON)		+= hwmon.o
>  nvme-y					+= pci.o
> 
>  nvme-fabrics-y				+= fabrics.o
> +nvme-fabrics-$(CONFIG_NVME_AUTH)	+= auth.o
> 
>  nvme-rdma-y				+= rdma.o
> 
> diff --git a/drivers/nvme/host/auth.c b/drivers/nvme/host/auth.c
> new file mode 100644
> index 000000000000..448a3adebea6
> --- /dev/null
> +++ b/drivers/nvme/host/auth.c
> @@ -0,0 +1,813 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Copyright (c) 2020 Hannes Reinecke, SUSE Linux
> + */
> +
> +#include <linux/crc32.h>
> +#include <linux/base64.h>
> +#include <asm/unaligned.h>
> +#include <crypto/hash.h>
> +#include <crypto/kpp.h>
> +#include "nvme.h"
> +#include "fabrics.h"
> +#include "auth.h"
> +
> +static u32 nvme_dhchap_seqnum;
> +
> +struct nvme_dhchap_context {
> +	struct crypto_shash *shash_tfm;
> +	unsigned char *key;
> +	size_t key_len;
> +	int qid;
> +	u32 s1;
> +	u32 s2;
> +	u16 transaction;
> +	u8 status;
> +	u8 hash_id;
> +	u8 hash_len;
> +	u8 c1[64];
> +	u8 c2[64];
> +	u8 response[64];
> +	u8 *ctrl_key;
> +	int ctrl_key_len;
> +	u8 *host_key;
> +	int host_key_len;
> +	u8 *sess_key;
> +	int sess_key_len;
> +};
> +
> +struct nvmet_dhchap_hash_map {
> +	int id;
> +	int hash_len;
> +	const char hmac[15];
> +	const char digest[15];
> +} hash_map[] = {
> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA256,
> +	 .hash_len = 32,
> +	 .hmac = "hmac(sha256)", .digest = "sha256" },
> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA384,
> +	 .hash_len = 48,
> +	 .hmac = "hmac(sha384)", .digest = "sha384" },
> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA512,
> +	 .hash_len = 64,
> +	 .hmac = "hmac(sha512)", .digest = "sha512" },
> +};
> +
> +const char *nvme_auth_hmac_name(int hmac_id)
> +{
> +	int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
> +		if (hash_map[i].id == hmac_id)
> +			return hash_map[i].hmac;
> +	}
> +	return NULL;
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_hmac_name);
> +
> +const char *nvme_auth_digest_name(int hmac_id)
> +{
> +	int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
> +		if (hash_map[i].id == hmac_id)
> +			return hash_map[i].digest;
> +	}
> +	return NULL;
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_digest_name);
> +
> +int nvme_auth_hmac_len(int hmac_id)
> +{
> +	int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
> +		if (hash_map[i].id == hmac_id)
> +			return hash_map[i].hash_len;
> +	}
> +	return -1;
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_hmac_len);
> +
> +int nvme_auth_hmac_id(const char *hmac_name)
> +{
> +	int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
> +		if (!strncmp(hash_map[i].hmac, hmac_name,
> +			     strlen(hash_map[i].hmac)))
> +			return hash_map[i].id;
> +	}
> +	return -1;
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_hmac_id);
> +
> +unsigned char *nvme_auth_extract_secret(unsigned char *dhchap_secret,
> +					size_t *dhchap_key_len)
> +{
> +	unsigned char *dhchap_key;
> +	u32 crc;
> +	int key_len;
> +	size_t allocated_len;
> +
> +	allocated_len = strlen(dhchap_secret) - 10;

Are you sure that the string is always at least 10 bytes long? If so, can you 
please add a comment to it?

Also, is it guaranteed that we have an ASCII string? Note, a secret sounds to 
be like a binary string which may contain \0 as an appropriate value.

> +	dhchap_key = kzalloc(allocated_len, GFP_KERNEL);

What about aligning it to CRYPTO_MINALIGN_ATTR to save a memcpy in 
shash_final?

> +	if (!dhchap_key)
> +		return ERR_PTR(-ENOMEM);
> +
> +	key_len = base64_decode(dhchap_secret + 10,
> +				allocated_len, dhchap_key);
> +	if (key_len != 36 && key_len != 52 &&
> +	    key_len != 68) {
> +		pr_debug("Invalid DH-HMAC-CHAP key len %d\n",
> +			 key_len);
> +		kfree(dhchap_key);
> +		return ERR_PTR(-EINVAL);
> +	}
> +	pr_debug("DH-HMAC-CHAP Key: %*ph\n",
> +		 (int)key_len, dhchap_key);
> +
> +	/* The last four bytes is the CRC in little-endian format */
> +	key_len -= 4;
> +	/*
> +	 * The linux implementation doesn't do pre- and post-increments,
> +	 * so we have to do it manually.
> +	 */
> +	crc = ~crc32(~0, dhchap_key, key_len);
> +
> +	if (get_unaligned_le32(dhchap_key + key_len) != crc) {
> +		pr_debug("DH-HMAC-CHAP crc mismatch (key %08x, crc %08x)\n",
> +		       get_unaligned_le32(dhchap_key + key_len), crc);
> +		kfree(dhchap_key);
> +		return ERR_PTR(-EKEYREJECTED);
> +	}
> +	*dhchap_key_len = key_len;
> +	return dhchap_key;
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_extract_secret);
> +
> +static int nvme_auth_send(struct nvme_ctrl *ctrl, int qid,
> +			  void *data, size_t tl)
> +{
> +	struct nvme_command cmd = {};
> +	blk_mq_req_flags_t flags = qid == NVME_QID_ANY ?
> +		0 : BLK_MQ_REQ_NOWAIT | BLK_MQ_REQ_RESERVED;
> +	struct request_queue *q = qid == NVME_QID_ANY ?
> +		ctrl->fabrics_q : ctrl->connect_q;
> +	int ret;
> +
> +	cmd.auth_send.opcode = nvme_fabrics_command;
> +	cmd.auth_send.fctype = nvme_fabrics_type_auth_send;
> +	cmd.auth_send.secp = NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER;
> +	cmd.auth_send.spsp0 = 0x01;
> +	cmd.auth_send.spsp1 = 0x01;
> +	cmd.auth_send.tl = tl;
> +
> +	ret = __nvme_submit_sync_cmd(q, &cmd, NULL, data, tl, 0, qid,
> +				     0, flags);
> +	if (ret)
> +		dev_dbg(ctrl->device,
> +			"%s: qid %d error %d\n", __func__, qid, ret);
> +	return ret;
> +}
> +
> +static int nvme_auth_receive(struct nvme_ctrl *ctrl, int qid,
> +			     void *buf, size_t al,
> +			     u16 transaction, u8 expected_msg )
> +{
> +	struct nvme_command cmd = {};
> +	struct nvmf_auth_dhchap_failure_data *data = buf;
> +	blk_mq_req_flags_t flags = qid == NVME_QID_ANY ?
> +		0 : BLK_MQ_REQ_NOWAIT | BLK_MQ_REQ_RESERVED;
> +	struct request_queue *q = qid == NVME_QID_ANY ?
> +		ctrl->fabrics_q : ctrl->connect_q;
> +	int ret;
> +
> +	cmd.auth_receive.opcode = nvme_fabrics_command;
> +	cmd.auth_receive.fctype = nvme_fabrics_type_auth_receive;
> +	cmd.auth_receive.secp = NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER;
> +	cmd.auth_receive.spsp0 = 0x01;
> +	cmd.auth_receive.spsp1 = 0x01;
> +	cmd.auth_receive.al = al;
> +
> +	ret = __nvme_submit_sync_cmd(q, &cmd, NULL, buf, al, 0, qid,
> +				     0, flags);
> +	if (ret > 0) {
> +		dev_dbg(ctrl->device, "%s: qid %d nvme status %x\n",
> +			__func__, qid, ret);
> +		ret = -EIO;
> +	}
> +	if (ret < 0) {
> +		dev_dbg(ctrl->device, "%s: qid %d error %d\n",
> +			__func__, qid, ret);
> +		return ret;
> +	}
> +	dev_dbg(ctrl->device, "%s: qid %d auth_type %d auth_id %x\n",
> +		__func__, qid, data->auth_type, data->auth_id);
> +	if (data->auth_type == NVME_AUTH_COMMON_MESSAGES &&
> +	    data->auth_id == NVME_AUTH_DHCHAP_MESSAGE_FAILURE1) {
> +		return data->reason_code_explanation;
> +	}
> +	if (data->auth_type != NVME_AUTH_DHCHAP_MESSAGES ||
> +	    data->auth_id != expected_msg) {
> +		dev_warn(ctrl->device,
> +			 "qid %d invalid message %02x/%02x\n",
> +			 qid, data->auth_type, data->auth_id);
> +		return NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
> +	}
> +	if (le16_to_cpu(data->t_id) != transaction) {
> +		dev_warn(ctrl->device,
> +			 "qid %d invalid transaction ID %d\n",
> +			 qid, le16_to_cpu(data->t_id));
> +		return NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
> +	}
> +
> +	return 0;
> +}
> +
> +static int nvme_auth_dhchap_negotiate(struct nvme_ctrl *ctrl,
> +				      struct nvme_dhchap_context *chap,
> +				      void *buf, size_t buf_size)
> +{
> +	struct nvmf_auth_dhchap_negotiate_data *data = buf;
> +	size_t size = sizeof(*data) + sizeof(union nvmf_auth_protocol);
> +
> +	if (buf_size < size)
> +		return -EINVAL;
> +
> +	memset((u8 *)buf, 0, size);
> +	data->auth_type = NVME_AUTH_COMMON_MESSAGES;
> +	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE;
> +	data->t_id = cpu_to_le16(chap->transaction);
> +	data->sc_c = 0; /* No secure channel concatenation */
> +	data->napd = 1;
> +	data->auth_protocol[0].dhchap.authid = NVME_AUTH_DHCHAP_AUTH_ID;
> +	data->auth_protocol[0].dhchap.halen = 3;
> +	data->auth_protocol[0].dhchap.dhlen = 1;
> +	data->auth_protocol[0].dhchap.idlist[0] = NVME_AUTH_DHCHAP_HASH_SHA256;
> +	data->auth_protocol[0].dhchap.idlist[1] = NVME_AUTH_DHCHAP_HASH_SHA384;
> +	data->auth_protocol[0].dhchap.idlist[2] = NVME_AUTH_DHCHAP_HASH_SHA512;
> +	data->auth_protocol[0].dhchap.idlist[3] = NVME_AUTH_DHCHAP_DHGROUP_NULL;
> +
> +	return size;
> +}
> +
> +static int nvme_auth_dhchap_challenge(struct nvme_ctrl *ctrl,
> +				      struct nvme_dhchap_context *chap,
> +				      void *buf, size_t buf_size)
> +{
> +	struct nvmf_auth_dhchap_challenge_data *data = buf;
> +	size_t size = sizeof(*data) + data->hl + data->dhvlen;
> +	const char *gid_name;
> +
> +	if (buf_size < size) {
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
> +		return -ENOMSG;
> +	}
> +
> +	if (data->hashid != NVME_AUTH_DHCHAP_HASH_SHA256 &&
> +	    data->hashid != NVME_AUTH_DHCHAP_HASH_SHA384 &&
> +	    data->hashid != NVME_AUTH_DHCHAP_HASH_SHA512) {
> +		dev_warn(ctrl->device,
> +			 "qid %d: DH-HMAC-CHAP: invalid HASH ID %d\n",
> +			 chap->qid, data->hashid);
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
> +		return -EPROTO;
> +	}
> +	switch (data->dhgid) {
> +	case NVME_AUTH_DHCHAP_DHGROUP_NULL:
> +		gid_name = "null";
> +		break;
> +	default:
> +		gid_name = NULL;
> +		break;
> +	}
> +	if (!gid_name) {
> +		dev_warn(ctrl->device,
> +			 "qid %d: DH-HMAC-CHAP: invalid DH group id %d\n",
> +			 chap->qid, data->dhgid);
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
> +		return -EPROTO;
> +	}
> +	if (data->dhgid != NVME_AUTH_DHCHAP_DHGROUP_NULL) {
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
> +		return -EPROTO;
> +	}
> +	if (data->dhgid == NVME_AUTH_DHCHAP_DHGROUP_NULL && data->dhvlen != 0) {
> +		dev_warn(ctrl->device,
> +			 "qid %d: DH-HMAC-CHAP: invalid DH value for NULL DH\n",
> +			chap->qid);
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
> +		return -EPROTO;
> +	}
> +	dev_dbg(ctrl->device, "%s: qid %d requested hash id %d\n",
> +		__func__, chap->qid, data->hashid);
> +	if (nvme_auth_hmac_len(data->hashid) != data->hl) {
> +		dev_warn(ctrl->device,
> +			 "qid %d: DH-HMAC-CHAP: invalid hash length\n",
> +			chap->qid);
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
> +		return -EPROTO;
> +	}
> +	chap->hash_id = data->hashid;
> +	chap->hash_len = data->hl;
> +	chap->s1 = le32_to_cpu(data->seqnum);
> +	memcpy(chap->c1, data->cval, chap->hash_len);
> +
> +	return 0;
> +}
> +
> +static int nvme_auth_dhchap_reply(struct nvme_ctrl *ctrl,
> +				  struct nvme_dhchap_context *chap,
> +				  void *buf, size_t buf_size)
> +{
> +	struct nvmf_auth_dhchap_reply_data *data = buf;
> +	size_t size = sizeof(*data);
> +
> +	size += 2 * chap->hash_len;
> +	if (ctrl->opts->dhchap_auth) {
> +		get_random_bytes(chap->c2, chap->hash_len);

Why are you using CRYPTO_RNG_DEFAULT when you are using get_random_bytes here?

> +		chap->s2 = nvme_dhchap_seqnum++;
> +	} else
> +		memset(chap->c2, 0, chap->hash_len);
> +
> +	if (chap->host_key_len)
> +		size += chap->host_key_len;
> +
> +	if (buf_size < size)
> +		return -EINVAL;
> +
> +	memset(buf, 0, size);
> +	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
> +	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_REPLY;
> +	data->t_id = cpu_to_le16(chap->transaction);
> +	data->hl = chap->hash_len;
> +	data->dhvlen = chap->host_key_len;
> +	data->seqnum = cpu_to_le32(chap->s2);
> +	memcpy(data->rval, chap->response, chap->hash_len);
> +	if (ctrl->opts->dhchap_auth) {
> +		dev_dbg(ctrl->device, "%s: qid %d ctrl challenge %*ph\n",
> +			__func__, chap->qid,
> +			chap->hash_len, chap->c2);
> +		data->cvalid = 1;
> +		memcpy(data->rval + chap->hash_len, chap->c2,
> +		       chap->hash_len);
> +	}
> +	if (chap->host_key_len)
> +		memcpy(data->rval + 2 * chap->hash_len, chap->host_key,
> +		       chap->host_key_len);
> +
> +	return size;
> +}
> +
> +static int nvme_auth_dhchap_success1(struct nvme_ctrl *ctrl,
> +				     struct nvme_dhchap_context *chap,
> +				     void *buf, size_t buf_size)
> +{
> +	struct nvmf_auth_dhchap_success1_data *data = buf;
> +	size_t size = sizeof(*data);
> +
> +	if (ctrl->opts->dhchap_auth)
> +		size += chap->hash_len;
> +
> +
> +	if (buf_size < size) {
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
> +		return -ENOMSG;
> +	}
> +
> +	if (data->hl != chap->hash_len) {
> +		dev_warn(ctrl->device,
> +			 "qid %d: DH-HMAC-CHAP: invalid hash length %d\n",
> +			 chap->qid, data->hl);
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
> +		return -EPROTO;
> +	}
> +
> +	if (!data->rvalid)
> +		return 0;
> +
> +	/* Validate controller response */
> +	if (memcmp(chap->response, data->rval, data->hl)) {
> +		dev_dbg(ctrl->device, "%s: qid %d ctrl response %*ph\n",
> +			__func__, chap->qid, chap->hash_len, data->rval);
> +		dev_dbg(ctrl->device, "%s: qid %d host response %*ph\n",
> +			__func__, chap->qid, chap->hash_len, chap->response);
> +		dev_warn(ctrl->device,
> +			 "qid %d: DH-HMAC-CHAP: controller authentication failed\n",
> +			 chap->qid);
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
> +		return -EPROTO;
> +	}
> +	dev_info(ctrl->device,
> +		 "qid %d: DH-HMAC-CHAP: controller authenticated\n",
> +		chap->qid);
> +	return 0;
> +}
> +
> +static int nvme_auth_dhchap_success2(struct nvme_ctrl *ctrl,
> +				     struct nvme_dhchap_context *chap,
> +				     void *buf, size_t buf_size)
> +{
> +	struct nvmf_auth_dhchap_success2_data *data = buf;
> +	size_t size = sizeof(*data);
> +
> +	memset(buf, 0, size);
> +	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
> +	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_SUCCESS2;
> +	data->t_id = cpu_to_le16(chap->transaction);
> +
> +	return size;
> +}
> +
> +static int nvme_auth_dhchap_failure2(struct nvme_ctrl *ctrl,
> +				     struct nvme_dhchap_context *chap,
> +				     void *buf, size_t buf_size)
> +{
> +	struct nvmf_auth_dhchap_failure_data *data = buf;
> +	size_t size = sizeof(*data);
> +
> +	memset(buf, 0, size);
> +	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
> +	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_FAILURE2;
> +	data->t_id = cpu_to_le16(chap->transaction);
> +	data->reason_code = 1;
> +	data->reason_code_explanation = chap->status;
> +
> +	return size;
> +}
> +
> +int nvme_auth_select_hash(struct nvme_ctrl *ctrl,
> +			  struct nvme_dhchap_context *chap)
> +{
> +	char *hash_name;
> +	int ret;
> +
> +	switch (chap->hash_id) {
> +	case NVME_AUTH_DHCHAP_HASH_SHA256:
> +		hash_name = "hmac(sha256)";
> +		break;
> +	case NVME_AUTH_DHCHAP_HASH_SHA384:
> +		hash_name = "hmac(sha384)";
> +		break;
> +	case NVME_AUTH_DHCHAP_HASH_SHA512:
> +		hash_name = "hmac(sha512)";
> +		break;
> +	default:
> +		hash_name = NULL;
> +		break;
> +	}
> +	if (!hash_name) {
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
> +		return -EPROTO;
> +	}
> +	chap->shash_tfm = crypto_alloc_shash(hash_name, 0,
> +					     CRYPTO_ALG_ALLOCATES_MEMORY);
> +	if (IS_ERR(chap->shash_tfm)) {
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
> +		chap->shash_tfm = NULL;
> +		return -EPROTO;
> +	}
> +	if (!chap->key) {
> +		dev_warn(ctrl->device, "qid %d: cannot select hash, no key\n",
> +			 chap->qid);
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
> +		crypto_free_shash(chap->shash_tfm);
> +		chap->shash_tfm = NULL;
> +		return -EINVAL;
> +	}
> +	ret = crypto_shash_setkey(chap->shash_tfm, chap->key, chap->key_len);
> +	if (ret) {
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
> +		crypto_free_shash(chap->shash_tfm);
> +		chap->shash_tfm = NULL;
> +		return ret;
> +	}
> +	dev_info(ctrl->device, "qid %d: DH-HMAC_CHAP: selected hash %s\n",
> +		 chap->qid, hash_name);
> +	return 0;
> +}
> +
> +static int nvme_auth_dhchap_host_response(struct nvme_ctrl *ctrl,
> +					  struct nvme_dhchap_context *chap)
> +{
> +	SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
> +	u8 buf[4], *challenge = chap->c1;
> +	int ret;
> +
> +	dev_dbg(ctrl->device, "%s: qid %d host response seq %d transaction 
%d\n",
> +		__func__, chap->qid, chap->s1, chap->transaction);
> +	shash->tfm = chap->shash_tfm;
> +	ret = crypto_shash_init(shash);
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, challenge, chap->hash_len);
> +	if (ret)
> +		goto out;
> +	put_unaligned_le32(chap->s1, buf);
> +	ret = crypto_shash_update(shash, buf, 4);
> +	if (ret)
> +		goto out;
> +	put_unaligned_le16(chap->transaction, buf);
> +	ret = crypto_shash_update(shash, buf, 2);
> +	if (ret)
> +		goto out;
> +	memset(buf, 0, sizeof(buf));
> +	ret = crypto_shash_update(shash, buf, 1);
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, "HostHost", 8);
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
> +				  strlen(ctrl->opts->host->nqn));
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, buf, 1);
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
> +			    strlen(ctrl->opts->subsysnqn));
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_final(shash, chap->response);
> +out:
> +	return ret;
> +}
> +
> +static int nvme_auth_dhchap_ctrl_response(struct nvme_ctrl *ctrl,
> +					  struct nvme_dhchap_context *chap)
> +{
> +	SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
> +	u8 buf[4], *challenge = chap->c2;
> +	int ret;
> +
> +	dev_dbg(ctrl->device, "%s: qid %d host response seq %d transaction 
%d\n",
> +		__func__, chap->qid, chap->s2, chap->transaction);
> +	dev_dbg(ctrl->device, "%s: qid %d challenge %*ph\n",
> +		__func__, chap->qid, chap->hash_len, challenge);
> +	dev_dbg(ctrl->device, "%s: qid %d subsysnqn %s\n",
> +		__func__, chap->qid, ctrl->opts->subsysnqn);
> +	dev_dbg(ctrl->device, "%s: qid %d hostnqn %s\n",
> +		__func__, chap->qid, ctrl->opts->host->nqn);
> +	shash->tfm = chap->shash_tfm;
> +	ret = crypto_shash_init(shash);
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, challenge, chap->hash_len);
> +	if (ret)
> +		goto out;
> +	put_unaligned_le32(chap->s2, buf);
> +	ret = crypto_shash_update(shash, buf, 4);
> +	if (ret)
> +		goto out;
> +	put_unaligned_le16(chap->transaction, buf);
> +	ret = crypto_shash_update(shash, buf, 2);
> +	if (ret)
> +		goto out;
> +	memset(buf, 0, 4);
> +	ret = crypto_shash_update(shash, buf, 1);
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, "Controller", 10);
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
> +				  strlen(ctrl->opts->subsysnqn));
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, buf, 1);
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
> +				  strlen(ctrl->opts->host->nqn));
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_final(shash, chap->response);
> +out:
> +	return ret;
> +}
> +
> +int nvme_auth_generate_key(struct nvme_ctrl *ctrl,
> +			   struct nvme_dhchap_context *chap)
> +{
> +	int ret;
> +	u8 key_hash;
> +	const char *hmac_name;
> +	struct crypto_shash *key_tfm;
> +
> +	if (sscanf(ctrl->opts->dhchap_secret, "DHHC-1:%hhd:%*s:",
> +		   &key_hash) != 1)
> +		return -EINVAL;
> +
> +	chap->key = nvme_auth_extract_secret(ctrl->opts->dhchap_secret,
> +					     &chap->key_len);
> +	if (IS_ERR(chap->key)) {
> +		ret = PTR_ERR(chap->key);
> +		chap->key = NULL;
> +		return ret;
> +	}
> +
> +	if (key_hash == 0)
> +		return 0;
> +
> +	hmac_name = nvme_auth_hmac_name(key_hash);
> +	if (!hmac_name) {
> +		pr_debug("Invalid key hash id %d\n", key_hash);
> +		return -EKEYREJECTED;
> +	}
> +
> +	key_tfm = crypto_alloc_shash(hmac_name, 0, 0);
> +	if (IS_ERR(key_tfm)) {
> +		kfree(chap->key);
> +		chap->key = NULL;
> +		ret = PTR_ERR(key_tfm);
> +	} else {
> +		SHASH_DESC_ON_STACK(shash, key_tfm);
> +
> +		shash->tfm = key_tfm;
> +		ret = crypto_shash_setkey(key_tfm, chap->key,
> +					  chap->key_len);
> +		if (ret < 0) {
> +			crypto_free_shash(key_tfm);
> +			kfree(chap->key);
> +			chap->key = NULL;
> +			return ret;
> +		}
> +		crypto_shash_init(shash);
> +		crypto_shash_update(shash, ctrl->opts->host->nqn,
> +				    strlen(ctrl->opts->host->nqn));
> +		crypto_shash_update(shash, "NVMe-over-Fabrics", 17);
> +		crypto_shash_final(shash, chap->key);
> +		crypto_free_shash(key_tfm);
> +	}
> +	return 0;
> +}
> +
> +void nvme_auth_free(struct nvme_dhchap_context *chap)
> +{
> +	if (chap->shash_tfm)
> +		crypto_free_shash(chap->shash_tfm);
> +	if (chap->key)
> +		kfree(chap->key);
> +	if (chap->ctrl_key)
> +		kfree(chap->ctrl_key);
> +	if (chap->host_key)
> +		kfree(chap->host_key);
> +	if (chap->sess_key)
> +		kfree(chap->sess_key);
> +	kfree(chap);

kfree_sensitive in all cases as all buffers have sensitive data?

> +}
> +
> +int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid)
> +{
> +	struct nvme_dhchap_context *chap;
> +	void *buf;
> +	size_t buf_size, tl;
> +	int ret = 0;
> +
> +	chap = kzalloc(sizeof(*chap), GFP_KERNEL);

Suggestion: make sure that chap->response is aligned to CRYPTO_MINALIGN_ATTR - 
then you would save a memcpy in crypto_shash_final

> +	if (!chap)
> +		return -ENOMEM;
> +	chap->qid = qid;
> +	chap->transaction = ctrl->transaction++;
> +
> +	ret = nvme_auth_generate_key(ctrl, chap);
> +	if (ret) {
> +		dev_dbg(ctrl->device, "%s: failed to generate key, error %d\n",
> +			__func__, ret);
> +		nvme_auth_free(chap);
> +		return ret;
> +	}
> +
> +	/*
> +	 * Allocate a large enough buffer for the entire negotiation:
> +	 * 4k should be enough to ffdhe8192.
> +	 */
> +	buf_size = 4096;
> +	buf = kzalloc(buf_size, GFP_KERNEL);
> +	if (!buf) {
> +		ret = -ENOMEM;
> +		goto out;
> +	}
> +
> +	/* DH-HMAC-CHAP Step 1: send negotiate */
> +	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP negotiate\n",
> +		__func__, qid);
> +	ret = nvme_auth_dhchap_negotiate(ctrl, chap, buf, buf_size);
> +	if (ret < 0)
> +		goto out;
> +	tl = ret;
> +	ret = nvme_auth_send(ctrl, qid, buf, tl);
> +	if (ret)
> +		goto out;
> +
> +	memset(buf, 0, buf_size);
> +	ret = nvme_auth_receive(ctrl, qid, buf, buf_size, chap->transaction,
> +				NVME_AUTH_DHCHAP_MESSAGE_CHALLENGE);
> +	if (ret < 0) {
> +		dev_dbg(ctrl->device,
> +			"%s: qid %d DH-HMAC-CHAP failed to receive challenge\n",
> +			__func__, qid);
> +		goto out;
> +	}
> +	if (ret > 0) {
> +		chap->status = ret;
> +		goto fail1;
> +	}
> +
> +	/* DH-HMAC-CHAP Step 2: receive challenge */
> +	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP challenge\n",
> +		__func__, qid);
> +
> +	ret = nvme_auth_dhchap_challenge(ctrl, chap, buf, buf_size);
> +	if (ret) {
> +		/* Invalid parameters for negotiate */
> +		goto fail2;
> +	}
> +
> +	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP select hash\n",
> +		__func__, qid);
> +	ret = nvme_auth_select_hash(ctrl, chap);
> +	if (ret)
> +		goto fail2;
> +
> +	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP host response\n",
> +		__func__, qid);
> +	ret = nvme_auth_dhchap_host_response(ctrl, chap);
> +	if (ret)
> +		goto fail2;
> +
> +	/* DH-HMAC-CHAP Step 3: send reply */
> +	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP reply\n",
> +		__func__, qid);
> +	ret = nvme_auth_dhchap_reply(ctrl, chap, buf, buf_size);
> +	if (ret < 0)
> +		goto fail2;
> +
> +	tl = ret;
> +	ret = nvme_auth_send(ctrl, qid, buf, tl);
> +	if (ret)
> +		goto fail2;
> +
> +	memset(buf, 0, buf_size);
> +	ret = nvme_auth_receive(ctrl, qid, buf, buf_size, chap->transaction,
> +				NVME_AUTH_DHCHAP_MESSAGE_SUCCESS1);
> +	if (ret < 0) {
> +		dev_dbg(ctrl->device,
> +			"%s: qid %d DH-HMAC-CHAP failed to receive success1\n",
> +			__func__, qid);
> +		goto out;
> +	}
> +	if (ret > 0) {
> +		chap->status = ret;
> +		goto fail1;
> +	}
> +
> +	if (ctrl->opts->dhchap_auth) {
> +		dev_dbg(ctrl->device,
> +			"%s: qid %d DH-HMAC-CHAP controller response\n",
> +			__func__, qid);
> +		ret = nvme_auth_dhchap_ctrl_response(ctrl, chap);
> +		if (ret)
> +			goto fail2;
> +	}
> +
> +	/* DH-HMAC-CHAP Step 4: receive success1 */
> +	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP success1\n",
> +		__func__, qid);
> +	ret = nvme_auth_dhchap_success1(ctrl, chap, buf, buf_size);
> +	if (ret < 0) {
> +		/* Controller authentication failed */
> +		goto fail2;
> +	}
> +	tl = ret;
> +	/* DH-HMAC-CHAP Step 5: send success2 */
> +	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP success2\n",
> +		__func__, qid);
> +	tl = nvme_auth_dhchap_success2(ctrl, chap, buf, buf_size);
> +	ret = nvme_auth_send(ctrl, qid, buf, tl);
> +	if (!ret)
> +		goto out;
> +
> +fail1:
> +	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP failure1, status %x\n",
> +		__func__, qid, chap->status);
> +	goto out;
> +
> +fail2:
> +	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP failure2, status %x\n",
> +		__func__, qid, chap->status);
> +	tl = nvme_auth_dhchap_failure2(ctrl, chap, buf, buf_size);
> +	ret = nvme_auth_send(ctrl, qid, buf, tl);
> +
> +out:
> +	if (!ret && chap->status)
> +		ret = -EPROTO;
> +	if (!ret) {
> +		ctrl->dhchap_hash = chap->hash_id;
> +	}
> +	kfree(buf);
> +	nvme_auth_free(chap);
> +	return ret;
> +}
> diff --git a/drivers/nvme/host/auth.h b/drivers/nvme/host/auth.h
> new file mode 100644
> index 000000000000..4950b1cb9470
> --- /dev/null
> +++ b/drivers/nvme/host/auth.h
> @@ -0,0 +1,23 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Copyright (c) 2021 Hannes Reinecke, SUSE Software Solutions
> + */
> +
> +#ifndef _NVME_AUTH_H
> +#define _NVME_AUTH_H
> +
> +const char *nvme_auth_dhgroup_name(int dhgroup_id);
> +int nvme_auth_dhgroup_pubkey_size(int dhgroup_id);
> +int nvme_auth_dhgroup_privkey_size(int dhgroup_id);
> +const char *nvme_auth_dhgroup_kpp(int dhgroup_id);
> +int nvme_auth_dhgroup_id(const char *dhgroup_name);
> +
> +const char *nvme_auth_hmac_name(int hmac_id);
> +const char *nvme_auth_digest_name(int hmac_id);
> +int nvme_auth_hmac_id(const char *hmac_name);
> +int nvme_auth_hmac_len(int hmac_len);
> +
> +unsigned char *nvme_auth_extract_secret(unsigned char *dhchap_secret,
> +					size_t *dhchap_key_len);
> +
> +#endif /* _NVME_AUTH_H */
> diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
> index 11779be42186..7ce9b666dc09 100644
> --- a/drivers/nvme/host/core.c
> +++ b/drivers/nvme/host/core.c
> @@ -708,7 +708,9 @@ bool __nvme_check_ready(struct nvme_ctrl *ctrl, struct
> request *rq, switch (ctrl->state) {
>  		case NVME_CTRL_CONNECTING:
>  			if (blk_rq_is_passthrough(rq) && nvme_is_fabrics(req->cmd) 
&&
> -			    req->cmd->fabrics.fctype == nvme_fabrics_type_connect)
> +			    (req->cmd->fabrics.fctype == nvme_fabrics_type_connect 
||
> +			     req->cmd->fabrics.fctype == nvme_fabrics_type_auth_send 
||
> +			     req->cmd->fabrics.fctype == 
nvme_fabrics_type_auth_receive))
>  				return true;
>  			break;
>  		default:
> @@ -3426,6 +3428,66 @@ static ssize_t
> nvme_ctrl_fast_io_fail_tmo_store(struct device *dev, static
> DEVICE_ATTR(fast_io_fail_tmo, S_IRUGO | S_IWUSR,
>  	nvme_ctrl_fast_io_fail_tmo_show, nvme_ctrl_fast_io_fail_tmo_store);
> 
> +#ifdef CONFIG_NVME_AUTH
> +struct nvmet_dhchap_hash_map {
> +	int id;
> +	const char name[15];
> +} hash_map[] = {
> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA256,
> +	 .name = "hmac(sha256)", },
> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA384,
> +	 .name = "hmac(sha384)", },
> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA512,
> +	 .name = "hmac(sha512)", },
> +};
> +
> +static ssize_t dhchap_hash_show(struct device *dev,
> +	struct device_attribute *attr, char *buf)
> +{
> +	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
> +	int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
> +		if (hash_map[i].id == ctrl->dhchap_hash)
> +			return sprintf(buf, "%s\n", hash_map[i].name);
> +	}
> +	return sprintf(buf, "none\n");
> +}
> +DEVICE_ATTR_RO(dhchap_hash);
> +
> +struct nvmet_dhchap_group_map {
> +	int id;
> +	const char name[15];
> +} dhgroup_map[] = {
> +	{.id = NVME_AUTH_DHCHAP_DHGROUP_NULL,
> +	 .name = "NULL", },
> +	{.id = NVME_AUTH_DHCHAP_DHGROUP_2048,
> +	 .name = "ffdhe2048", },
> +	{.id = NVME_AUTH_DHCHAP_DHGROUP_3072,
> +	 .name = "ffdhe3072", },
> +	{.id = NVME_AUTH_DHCHAP_DHGROUP_4096,
> +	 .name = "ffdhe4096", },
> +	{.id = NVME_AUTH_DHCHAP_DHGROUP_6144,
> +	 .name = "ffdhe6144", },
> +	{.id = NVME_AUTH_DHCHAP_DHGROUP_8192,
> +	 .name = "ffdhe8192", },
> +};
> +
> +static ssize_t dhchap_dhgroup_show(struct device *dev,
> +	struct device_attribute *attr, char *buf)
> +{
> +	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
> +	int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(dhgroup_map); i++) {
> +		if (hash_map[i].id == ctrl->dhchap_dhgroup)
> +			return sprintf(buf, "%s\n", dhgroup_map[i].name);
> +	}
> +	return sprintf(buf, "none\n");
> +}
> +DEVICE_ATTR_RO(dhchap_dhgroup);
> +#endif
> +
>  static struct attribute *nvme_dev_attrs[] = {
>  	&dev_attr_reset_controller.attr,
>  	&dev_attr_rescan_controller.attr,
> @@ -3447,6 +3509,10 @@ static struct attribute *nvme_dev_attrs[] = {
>  	&dev_attr_reconnect_delay.attr,
>  	&dev_attr_fast_io_fail_tmo.attr,
>  	&dev_attr_kato.attr,
> +#ifdef CONFIG_NVME_AUTH
> +	&dev_attr_dhchap_hash.attr,
> +	&dev_attr_dhchap_dhgroup.attr,
> +#endif
>  	NULL
>  };
> 
> @@ -3470,6 +3536,10 @@ static umode_t nvme_dev_attrs_are_visible(struct
> kobject *kobj, return 0;
>  	if (a == &dev_attr_fast_io_fail_tmo.attr && !ctrl->opts)
>  		return 0;
> +#ifdef CONFIG_NVME_AUTH
> +	if (a == &dev_attr_dhchap_hash.attr && !ctrl->opts)
> +		return 0;
> +#endif
> 
>  	return a->mode;
>  }
> @@ -4581,6 +4651,11 @@ static inline void _nvme_check_size(void)
>  	BUILD_BUG_ON(sizeof(struct nvme_smart_log) != 512);
>  	BUILD_BUG_ON(sizeof(struct nvme_dbbuf) != 64);
>  	BUILD_BUG_ON(sizeof(struct nvme_directive_cmd) != 64);
> +	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_negotiate_data) != 8);
> +	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_challenge_data) != 16);
> +	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_reply_data) != 16);
> +	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_success1_data) != 16);
> +	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_success2_data) != 16);
>  }
> 
> 
> diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
> index a5469fd9d4c3..6404ab9b604b 100644
> --- a/drivers/nvme/host/fabrics.c
> +++ b/drivers/nvme/host/fabrics.c
> @@ -366,6 +366,7 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)
>  	union nvme_result res;
>  	struct nvmf_connect_data *data;
>  	int ret;
> +	u32 result;
> 
>  	cmd.connect.opcode = nvme_fabrics_command;
>  	cmd.connect.fctype = nvme_fabrics_type_connect;
> @@ -398,8 +399,18 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)
>  		goto out_free_data;
>  	}
> 
> -	ctrl->cntlid = le16_to_cpu(res.u16);
> -
> +	result = le32_to_cpu(res.u32);
> +	ctrl->cntlid = result & 0xFFFF;
> +	if ((result >> 16) & 2) {
> +		/* Authentication required */
> +		ret = nvme_auth_negotiate(ctrl, NVME_QID_ANY);
> +		if (ret)
> +			dev_warn(ctrl->device,
> +				 "qid 0: authentication failed\n");
> +		else
> +			dev_info(ctrl->device,
> +				 "qid 0: authenticated\n");
> +	}
>  out_free_data:
>  	kfree(data);
>  	return ret;
> @@ -432,6 +443,7 @@ int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, u16
> qid) struct nvmf_connect_data *data;
>  	union nvme_result res;
>  	int ret;
> +	u32 result;
> 
>  	cmd.connect.opcode = nvme_fabrics_command;
>  	cmd.connect.fctype = nvme_fabrics_type_connect;
> @@ -457,6 +469,17 @@ int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, u16
> qid) nvmf_log_connect_error(ctrl, ret, le32_to_cpu(res.u32),
>  				       &cmd, data);
>  	}
> +	result = le32_to_cpu(res.u32);
> +	if ((result >> 16) & 2) {
> +		/* Authentication required */
> +		ret = nvme_auth_negotiate(ctrl, qid);
> +		if (ret)
> +			dev_warn(ctrl->device,
> +				 "qid %u: authentication failed\n", qid);
> +		else
> +			dev_info(ctrl->device,
> +				 "qid %u: authenticated\n", qid);
> +	}
>  	kfree(data);
>  	return ret;
>  }
> @@ -548,6 +571,9 @@ static const match_table_t opt_tokens = {
>  	{ NVMF_OPT_NR_POLL_QUEUES,	"nr_poll_queues=%d"	},
>  	{ NVMF_OPT_TOS,			"tos=%d"		},
>  	{ NVMF_OPT_FAIL_FAST_TMO,	"fast_io_fail_tmo=%d"	},
> +	{ NVMF_OPT_DHCHAP_SECRET,	"dhchap_secret=%s"	},
> +	{ NVMF_OPT_DHCHAP_AUTH,		"authenticate"		},
> +	{ NVMF_OPT_DHCHAP_GROUP,	"dhchap_group=%s"	},
>  	{ NVMF_OPT_ERR,			NULL			}
>  };
> 
> @@ -824,6 +850,35 @@ static int nvmf_parse_options(struct nvmf_ctrl_options
> *opts, }
>  			opts->tos = token;
>  			break;
> +		case NVMF_OPT_DHCHAP_SECRET:
> +			p = match_strdup(args);
> +			if (!p) {
> +				ret = -ENOMEM;
> +				goto out;
> +			}
> +			if (strncmp(p, "DHHC-1:00:", 10)) {
> +				pr_err("Invalid DH-CHAP secret %s\n", p);
> +				ret = -EINVAL;
> +				goto out;
> +			}
> +			kfree(opts->dhchap_secret);
> +			opts->dhchap_secret = p;
> +			break;
> +		case NVMF_OPT_DHCHAP_AUTH:
> +			opts->dhchap_auth = true;
> +			break;
> +		case NVMF_OPT_DHCHAP_GROUP:
> +			if (match_int(args, &token)) {
> +				ret = -EINVAL;
> +				goto out;
> +			}
> +			if (token <= 0) {
> +				pr_err("Invalid dhchap_group %d\n", token);
> +				ret = -EINVAL;
> +				goto out;
> +			}
> +			opts->dhchap_group = token;
> +			break;
>  		default:
>  			pr_warn("unknown parameter or missing value '%s' in ctrl 
creation
> request\n", p);
> @@ -942,6 +997,7 @@ void nvmf_free_options(struct nvmf_ctrl_options *opts)
>  	kfree(opts->subsysnqn);
>  	kfree(opts->host_traddr);
>  	kfree(opts->host_iface);
> +	kfree(opts->dhchap_secret);
>  	kfree(opts);
>  }
>  EXPORT_SYMBOL_GPL(nvmf_free_options);
> @@ -951,7 +1007,10 @@ EXPORT_SYMBOL_GPL(nvmf_free_options);
>  				 NVMF_OPT_KATO | NVMF_OPT_HOSTNQN | \
>  				 NVMF_OPT_HOST_ID | NVMF_OPT_DUP_CONNECT |\
>  				 NVMF_OPT_DISABLE_SQFLOW |\
> -				 NVMF_OPT_FAIL_FAST_TMO)
> +				 NVMF_OPT_CTRL_LOSS_TMO |\
> +				 NVMF_OPT_FAIL_FAST_TMO |\
> +				 NVMF_OPT_DHCHAP_SECRET |\
> +				 NVMF_OPT_DHCHAP_AUTH | NVMF_OPT_DHCHAP_GROUP)
> 
>  static struct nvme_ctrl *
>  nvmf_create_ctrl(struct device *dev, const char *buf)
> diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h
> index a146cb903869..535bc544f0f6 100644
> --- a/drivers/nvme/host/fabrics.h
> +++ b/drivers/nvme/host/fabrics.h
> @@ -67,6 +67,9 @@ enum {
>  	NVMF_OPT_TOS		= 1 << 19,
>  	NVMF_OPT_FAIL_FAST_TMO	= 1 << 20,
>  	NVMF_OPT_HOST_IFACE	= 1 << 21,
> +	NVMF_OPT_DHCHAP_SECRET	= 1 << 22,
> +	NVMF_OPT_DHCHAP_AUTH	= 1 << 23,
> +	NVMF_OPT_DHCHAP_GROUP	= 1 << 24,
>  };
> 
>  /**
> @@ -96,6 +99,8 @@ enum {
>   * @max_reconnects: maximum number of allowed reconnect attempts before
> removing *              the controller, (-1) means reconnect forever, zero
> means remove *              immediately;
> + * @dhchap_secret: DH-HMAC-CHAP secret
> + * @dhchap_auth: DH-HMAC-CHAP authenticate controller
>   * @disable_sqflow: disable controller sq flow control
>   * @hdr_digest: generate/verify header digest (TCP)
>   * @data_digest: generate/verify data digest (TCP)
> @@ -120,6 +125,9 @@ struct nvmf_ctrl_options {
>  	unsigned int		kato;
>  	struct nvmf_host	*host;
>  	int			max_reconnects;
> +	char			*dhchap_secret;
> +	int			dhchap_group;
> +	bool			dhchap_auth;
>  	bool			disable_sqflow;
>  	bool			hdr_digest;
>  	bool			data_digest;
> diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
> index 18ef8dd03a90..bcd5b8276c26 100644
> --- a/drivers/nvme/host/nvme.h
> +++ b/drivers/nvme/host/nvme.h
> @@ -328,6 +328,12 @@ struct nvme_ctrl {
>  	struct work_struct ana_work;
>  #endif
> 
> +#ifdef CONFIG_NVME_AUTH
> +	u16 transaction;
> +	u8 dhchap_hash;
> +	u8 dhchap_dhgroup;
> +#endif
> +
>  	/* Power saving configuration */
>  	u64 ps_max_latency_us;
>  	bool apst_enabled;
> @@ -874,6 +880,15 @@ static inline bool nvme_ctrl_sgl_supported(struct
> nvme_ctrl *ctrl) return ctrl->sgls & ((1 << 0) | (1 << 1));
>  }
> 
> +#ifdef CONFIG_NVME_AUTH
> +int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid);
> +#else
> +static inline int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid)
> +{
> +	return -EPROTONOSUPPORT;
> +}
> +#endif
> +
>  u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
>  			 u8 opcode);
>  int nvme_execute_passthru_rq(struct request *rq);
> diff --git a/drivers/nvme/host/trace.c b/drivers/nvme/host/trace.c
> index 6543015b6121..66f75d8ea925 100644
> --- a/drivers/nvme/host/trace.c
> +++ b/drivers/nvme/host/trace.c
> @@ -271,6 +271,34 @@ static const char
> *nvme_trace_fabrics_property_get(struct trace_seq *p, u8 *spc) return ret;
>  }
> 
> +static const char *nvme_trace_fabrics_auth_send(struct trace_seq *p, u8
> *spc) +{
> +	const char *ret = trace_seq_buffer_ptr(p);
> +	u8 spsp0 = spc[1];
> +	u8 spsp1 = spc[2];
> +	u8 secp = spc[3];
> +	u32 tl = get_unaligned_le32(spc + 4);
> +
> +	trace_seq_printf(p, "spsp0=%02x, spsp1=%02x, secp=%02x, tl=%u",
> +			 spsp0, spsp1, secp, tl);
> +	trace_seq_putc(p, 0);
> +	return ret;
> +}
> +
> +static const char *nvme_trace_fabrics_auth_receive(struct trace_seq *p, u8
> *spc) +{
> +	const char *ret = trace_seq_buffer_ptr(p);
> +	u8 spsp0 = spc[1];
> +	u8 spsp1 = spc[2];
> +	u8 secp = spc[3];
> +	u32 al = get_unaligned_le32(spc + 4);
> +
> +	trace_seq_printf(p, "spsp0=%02x, spsp1=%02x, secp=%02x, al=%u",
> +			 spsp0, spsp1, secp, al);
> +	trace_seq_putc(p, 0);
> +	return ret;
> +}
> +
>  static const char *nvme_trace_fabrics_common(struct trace_seq *p, u8 *spc)
>  {
>  	const char *ret = trace_seq_buffer_ptr(p);
> @@ -290,6 +318,10 @@ const char *nvme_trace_parse_fabrics_cmd(struct
> trace_seq *p, return nvme_trace_fabrics_connect(p, spc);
>  	case nvme_fabrics_type_property_get:
>  		return nvme_trace_fabrics_property_get(p, spc);
> +	case nvme_fabrics_type_auth_send:
> +		return nvme_trace_fabrics_auth_send(p, spc);
> +	case nvme_fabrics_type_auth_receive:
> +		return nvme_trace_fabrics_auth_receive(p, spc);
>  	default:
>  		return nvme_trace_fabrics_common(p, spc);
>  	}


Ciao
Stephan



_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 06/11] nvme: Implement In-Band authentication

[Sagi Grimberg]

> Implement NVMe-oF In-Band authentication. This patch adds two new
> fabric options 'dhchap_key' to specify the PSK

pre-shared-key.

Also, we need a sysfs knob to rotate the key that will trigger
re-authentication or even a simple controller(s-plural) reset, so this
should go beyond just the connection string.

P.S. can you add also the nvme-cli code in the next go?

> and 'dhchap_authenticate'
> to request bi-directional authentication of both the host and the controller.

bidirectional? not uni-directional?

> 
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>   drivers/nvme/host/Kconfig   |  11 +
>   drivers/nvme/host/Makefile  |   1 +
>   drivers/nvme/host/auth.c    | 813 ++++++++++++++++++++++++++++++++++++
>   drivers/nvme/host/auth.h    |  23 +
>   drivers/nvme/host/core.c    |  77 +++-
>   drivers/nvme/host/fabrics.c |  65 ++-
>   drivers/nvme/host/fabrics.h |   8 +
>   drivers/nvme/host/nvme.h    |  15 +
>   drivers/nvme/host/trace.c   |  32 ++
>   9 files changed, 1041 insertions(+), 4 deletions(-)
>   create mode 100644 drivers/nvme/host/auth.c
>   create mode 100644 drivers/nvme/host/auth.h
> 
> diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig
> index c3f3d77f1aac..853c546305e9 100644
> --- a/drivers/nvme/host/Kconfig
> +++ b/drivers/nvme/host/Kconfig
> @@ -85,3 +85,14 @@ config NVME_TCP
>   	  from https://github.com/linux-nvme/nvme-cli.
>   
>   	  If unsure, say N.
> +
> +config NVME_AUTH
> +	bool "NVM Express over Fabrics In-Band Authentication"
> +	depends on NVME_TCP
> +	select CRYPTO_SHA256
> +	select CRYPTO_SHA512
> +	help
> +	  This provides support for NVMe over Fabrics In-Band Authentication
> +	  for the NVMe over TCP transport.

In this form, nothing is specific to nvme-tcp here afaict.

> +
> +	  If unsure, say N.
> diff --git a/drivers/nvme/host/Makefile b/drivers/nvme/host/Makefile
> index cbc509784b2e..03748a55a12b 100644
> --- a/drivers/nvme/host/Makefile
> +++ b/drivers/nvme/host/Makefile
> @@ -20,6 +20,7 @@ nvme-core-$(CONFIG_NVME_HWMON)		+= hwmon.o
>   nvme-y					+= pci.o
>   
>   nvme-fabrics-y				+= fabrics.o
> +nvme-fabrics-$(CONFIG_NVME_AUTH)	+= auth.o
>   
>   nvme-rdma-y				+= rdma.o
>   
> diff --git a/drivers/nvme/host/auth.c b/drivers/nvme/host/auth.c
> new file mode 100644
> index 000000000000..448a3adebea6
> --- /dev/null
> +++ b/drivers/nvme/host/auth.c
> @@ -0,0 +1,813 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Copyright (c) 2020 Hannes Reinecke, SUSE Linux
> + */
> +
> +#include <linux/crc32.h>
> +#include <linux/base64.h>
> +#include <asm/unaligned.h>
> +#include <crypto/hash.h>
> +#include <crypto/kpp.h>
> +#include "nvme.h"
> +#include "fabrics.h"
> +#include "auth.h"
> +
> +static u32 nvme_dhchap_seqnum;
> +
> +struct nvme_dhchap_context {

Maybe nvme_dhchap_queue_context ?

I'm thinking that we should perhaps split
it to host-wide, subsys-wide and queue specific
auth contexts?

Let's see...

> +	struct crypto_shash *shash_tfm;
> +	unsigned char *key;
> +	size_t key_len;
> +	int qid;
> +	u32 s1;
> +	u32 s2;
> +	u16 transaction;
> +	u8 status;
> +	u8 hash_id;
> +	u8 hash_len;
> +	u8 c1[64];
> +	u8 c2[64];
> +	u8 response[64];
> +	u8 *ctrl_key;
> +	int ctrl_key_len;
> +	u8 *host_key;
> +	int host_key_len;
> +	u8 *sess_key;
> +	int sess_key_len;
> +};
> +
> +struct nvmet_dhchap_hash_map {

nvmet?

> +	int id;
> +	int hash_len;
> +	const char hmac[15];
> +	const char digest[15];
> +} hash_map[] = {
> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA256,
> +	 .hash_len = 32,
> +	 .hmac = "hmac(sha256)", .digest = "sha256" },
> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA384,
> +	 .hash_len = 48,
> +	 .hmac = "hmac(sha384)", .digest = "sha384" },
> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA512,
> +	 .hash_len = 64,
> +	 .hmac = "hmac(sha512)", .digest = "sha512" },
> +};
> +
> +const char *nvme_auth_hmac_name(int hmac_id)

Should these arrays be static?

> +{
> +	int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
> +		if (hash_map[i].id == hmac_id)
> +			return hash_map[i].hmac;
> +	}
> +	return NULL;
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_hmac_name);
> +
> +const char *nvme_auth_digest_name(int hmac_id)
> +{
> +	int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
> +		if (hash_map[i].id == hmac_id)
> +			return hash_map[i].digest;
> +	}
> +	return NULL;
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_digest_name);
> +
> +int nvme_auth_hmac_len(int hmac_id)
> +{
> +	int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
> +		if (hash_map[i].id == hmac_id)
> +			return hash_map[i].hash_len;
> +	}
> +	return -1;
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_hmac_len);
> +
> +int nvme_auth_hmac_id(const char *hmac_name)
> +{
> +	int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
> +		if (!strncmp(hash_map[i].hmac, hmac_name,
> +			     strlen(hash_map[i].hmac)))
> +			return hash_map[i].id;
> +	}
> +	return -1;
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_hmac_id);
> +
> +unsigned char *nvme_auth_extract_secret(unsigned char *dhchap_secret,
> +					size_t *dhchap_key_len)
> +{
> +	unsigned char *dhchap_key;
> +	u32 crc;
> +	int key_len;
> +	size_t allocated_len;
> +
> +	allocated_len = strlen(dhchap_secret) - 10;

the 10 feels like a magic here, should at least note this is the
"DHHC-1:..." prefix.

> +	dhchap_key = kzalloc(allocated_len, GFP_KERNEL);
> +	if (!dhchap_key)
> +		return ERR_PTR(-ENOMEM);
> +
> +	key_len = base64_decode(dhchap_secret + 10,
> +				allocated_len, dhchap_key);
> +	if (key_len != 36 && key_len != 52 &&
> +	    key_len != 68) {
> +		pr_debug("Invalid DH-HMAC-CHAP key len %d\n",
> +			 key_len);
> +		kfree(dhchap_key);
> +		return ERR_PTR(-EINVAL);
> +	}
> +	pr_debug("DH-HMAC-CHAP Key: %*ph\n",
> +		 (int)key_len, dhchap_key);

One can argue if even printing this is problematic..

> +
> +	/* The last four bytes is the CRC in little-endian format */
> +	key_len -= 4;
> +	/*
> +	 * The linux implementation doesn't do pre- and post-increments,
> +	 * so we have to do it manually.
> +	 */
> +	crc = ~crc32(~0, dhchap_key, key_len);
> +
> +	if (get_unaligned_le32(dhchap_key + key_len) != crc) {
> +		pr_debug("DH-HMAC-CHAP crc mismatch (key %08x, crc %08x)\n",
> +		       get_unaligned_le32(dhchap_key + key_len), crc);
> +		kfree(dhchap_key);
> +		return ERR_PTR(-EKEYREJECTED);
> +	}
> +	*dhchap_key_len = key_len;
> +	return dhchap_key;
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_extract_secret);
> +
> +static int nvme_auth_send(struct nvme_ctrl *ctrl, int qid,
> +			  void *data, size_t tl)
> +{
> +	struct nvme_command cmd = {};
> +	blk_mq_req_flags_t flags = qid == NVME_QID_ANY ?
> +		0 : BLK_MQ_REQ_NOWAIT | BLK_MQ_REQ_RESERVED;
> +	struct request_queue *q = qid == NVME_QID_ANY ?
> +		ctrl->fabrics_q : ctrl->connect_q;
> +	int ret;
> +
> +	cmd.auth_send.opcode = nvme_fabrics_command;
> +	cmd.auth_send.fctype = nvme_fabrics_type_auth_send;
> +	cmd.auth_send.secp = NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER;
> +	cmd.auth_send.spsp0 = 0x01;
> +	cmd.auth_send.spsp1 = 0x01;
> +	cmd.auth_send.tl = tl;
> +
> +	ret = __nvme_submit_sync_cmd(q, &cmd, NULL, data, tl, 0, qid,
> +				     0, flags);
> +	if (ret)
> +		dev_dbg(ctrl->device,
> +			"%s: qid %d error %d\n", __func__, qid, ret);

Maybe a little more informative print rather than __func__ ?

> +	return ret;
> +}
> +
> +static int nvme_auth_receive(struct nvme_ctrl *ctrl, int qid,
> +			     void *buf, size_t al,
> +			     u16 transaction, u8 expected_msg )
> +{
> +	struct nvme_command cmd = {};
> +	struct nvmf_auth_dhchap_failure_data *data = buf;
> +	blk_mq_req_flags_t flags = qid == NVME_QID_ANY ?
> +		0 : BLK_MQ_REQ_NOWAIT | BLK_MQ_REQ_RESERVED;
> +	struct request_queue *q = qid == NVME_QID_ANY ?
> +		ctrl->fabrics_q : ctrl->connect_q;
> +	int ret;
> +
> +	cmd.auth_receive.opcode = nvme_fabrics_command;
> +	cmd.auth_receive.fctype = nvme_fabrics_type_auth_receive;
> +	cmd.auth_receive.secp = NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER;
> +	cmd.auth_receive.spsp0 = 0x01;
> +	cmd.auth_receive.spsp1 = 0x01;
> +	cmd.auth_receive.al = al;
> +
> +	ret = __nvme_submit_sync_cmd(q, &cmd, NULL, buf, al, 0, qid,
> +				     0, flags);
> +	if (ret > 0) {
> +		dev_dbg(ctrl->device, "%s: qid %d nvme status %x\n",
> +			__func__, qid, ret);
> +		ret = -EIO;
> +	}
> +	if (ret < 0) {
> +		dev_dbg(ctrl->device, "%s: qid %d error %d\n",
> +			__func__, qid, ret);
> +		return ret;
> +	}
> +	dev_dbg(ctrl->device, "%s: qid %d auth_type %d auth_id %x\n",
> +		__func__, qid, data->auth_type, data->auth_id);
> +	if (data->auth_type == NVME_AUTH_COMMON_MESSAGES &&
> +	    data->auth_id == NVME_AUTH_DHCHAP_MESSAGE_FAILURE1) {
> +		return data->reason_code_explanation;
> +	}
> +	if (data->auth_type != NVME_AUTH_DHCHAP_MESSAGES ||
> +	    data->auth_id != expected_msg) {
> +		dev_warn(ctrl->device,
> +			 "qid %d invalid message %02x/%02x\n",
> +			 qid, data->auth_type, data->auth_id);
> +		return NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
> +	}
> +	if (le16_to_cpu(data->t_id) != transaction) {
> +		dev_warn(ctrl->device,
> +			 "qid %d invalid transaction ID %d\n",
> +			 qid, le16_to_cpu(data->t_id));
> +		return NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
> +	}
> +
> +	return 0;
> +}
> +
> +static int nvme_auth_dhchap_negotiate(struct nvme_ctrl *ctrl,
> +				      struct nvme_dhchap_context *chap,
> +				      void *buf, size_t buf_size)

Maybe nvme_auth_set_dhchap_negotiate_data ?

> +{
> +	struct nvmf_auth_dhchap_negotiate_data *data = buf;
> +	size_t size = sizeof(*data) + sizeof(union nvmf_auth_protocol);
> +
> +	if (buf_size < size)
> +		return -EINVAL;
> +
> +	memset((u8 *)buf, 0, size);
> +	data->auth_type = NVME_AUTH_COMMON_MESSAGES;
> +	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE;
> +	data->t_id = cpu_to_le16(chap->transaction);
> +	data->sc_c = 0; /* No secure channel concatenation */
> +	data->napd = 1;
> +	data->auth_protocol[0].dhchap.authid = NVME_AUTH_DHCHAP_AUTH_ID;
> +	data->auth_protocol[0].dhchap.halen = 3;
> +	data->auth_protocol[0].dhchap.dhlen = 1;
> +	data->auth_protocol[0].dhchap.idlist[0] = NVME_AUTH_DHCHAP_HASH_SHA256;
> +	data->auth_protocol[0].dhchap.idlist[1] = NVME_AUTH_DHCHAP_HASH_SHA384;
> +	data->auth_protocol[0].dhchap.idlist[2] = NVME_AUTH_DHCHAP_HASH_SHA512;
> +	data->auth_protocol[0].dhchap.idlist[3] = NVME_AUTH_DHCHAP_DHGROUP_NULL;
You should comment that this routine expects buf to have enough
room for both negotiate and auth_proto structures.

> +
> +	return size;
> +}
> +
> +static int nvme_auth_dhchap_challenge(struct nvme_ctrl *ctrl,
> +				      struct nvme_dhchap_context *chap,
> +				      void *buf, size_t buf_size)

Maybe nvme_auth_process_dhchap_challange ?

> +{
> +	struct nvmf_auth_dhchap_challenge_data *data = buf;
> +	size_t size = sizeof(*data) + data->hl + data->dhvlen;
> +	const char *gid_name;
> +
> +	if (buf_size < size) {
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
> +		return -ENOMSG;
> +	}
> +
> +	if (data->hashid != NVME_AUTH_DHCHAP_HASH_SHA256 &&
> +	    data->hashid != NVME_AUTH_DHCHAP_HASH_SHA384 &&
> +	    data->hashid != NVME_AUTH_DHCHAP_HASH_SHA512) {
> +		dev_warn(ctrl->device,
> +			 "qid %d: DH-HMAC-CHAP: invalid HASH ID %d\n",
> +			 chap->qid, data->hashid);
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
> +		return -EPROTO;
> +	}
> +	switch (data->dhgid) {
> +	case NVME_AUTH_DHCHAP_DHGROUP_NULL:
> +		gid_name = "null";
> +		break;
> +	default:
> +		gid_name = NULL;
> +		break;
> +	}
> +	if (!gid_name) {
> +		dev_warn(ctrl->device,
> +			 "qid %d: DH-HMAC-CHAP: invalid DH group id %d\n",
> +			 chap->qid, data->dhgid);
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
> +		return -EPROTO;
> +	}

Maybe some spaces between condition blocks?

> +	if (data->dhgid != NVME_AUTH_DHCHAP_DHGROUP_NULL) {
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
> +		return -EPROTO;
> +	}
> +	if (data->dhgid == NVME_AUTH_DHCHAP_DHGROUP_NULL && data->dhvlen != 0) {
> +		dev_warn(ctrl->device,
> +			 "qid %d: DH-HMAC-CHAP: invalid DH value for NULL DH\n",
> +			chap->qid);
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
> +		return -EPROTO;
> +	}
> +	dev_dbg(ctrl->device, "%s: qid %d requested hash id %d\n",
> +		__func__, chap->qid, data->hashid);
> +	if (nvme_auth_hmac_len(data->hashid) != data->hl) {
> +		dev_warn(ctrl->device,
> +			 "qid %d: DH-HMAC-CHAP: invalid hash length\n",
> +			chap->qid);
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
> +		return -EPROTO;
> +	}
> +	chap->hash_id = data->hashid;
> +	chap->hash_len = data->hl;
> +	chap->s1 = le32_to_cpu(data->seqnum);
> +	memcpy(chap->c1, data->cval, chap->hash_len);
> +
> +	return 0;
> +}
> +
> +static int nvme_auth_dhchap_reply(struct nvme_ctrl *ctrl,
> +				  struct nvme_dhchap_context *chap,
> +				  void *buf, size_t buf_size)

nvme_auth_set_dhchap_reply

> +{
> +	struct nvmf_auth_dhchap_reply_data *data = buf;
> +	size_t size = sizeof(*data);
> +
> +	size += 2 * chap->hash_len;
> +	if (ctrl->opts->dhchap_auth) {

The ctrl opts is not clear to me. what is dhchap_auth
mean?

Also shouldn't these params be lifted to the subsys?

> +		get_random_bytes(chap->c2, chap->hash_len);
> +		chap->s2 = nvme_dhchap_seqnum++;
> +	} else
> +		memset(chap->c2, 0, chap->hash_len);
> +
> +	if (chap->host_key_len)
> +		size += chap->host_key_len;
> +
> +	if (buf_size < size)
> +		return -EINVAL;
> +
> +	memset(buf, 0, size);
> +	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
> +	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_REPLY;
> +	data->t_id = cpu_to_le16(chap->transaction);
> +	data->hl = chap->hash_len;
> +	data->dhvlen = chap->host_key_len;
> +	data->seqnum = cpu_to_le32(chap->s2);
> +	memcpy(data->rval, chap->response, chap->hash_len);
> +	if (ctrl->opts->dhchap_auth) {
> +		dev_dbg(ctrl->device, "%s: qid %d ctrl challenge %*ph\n",
> +			__func__, chap->qid,
> +			chap->hash_len, chap->c2);
> +		data->cvalid = 1;
> +		memcpy(data->rval + chap->hash_len, chap->c2,
> +		       chap->hash_len);
> +	}
> +	if (chap->host_key_len)
> +		memcpy(data->rval + 2 * chap->hash_len, chap->host_key,
> +		       chap->host_key_len);
> +
> +	return size;
> +}
> +
> +static int nvme_auth_dhchap_success1(struct nvme_ctrl *ctrl,
> +				     struct nvme_dhchap_context *chap,
> +				     void *buf, size_t buf_size)

nvme_auth_process_dhchap_success1

> +{
> +	struct nvmf_auth_dhchap_success1_data *data = buf;
> +	size_t size = sizeof(*data);
> +
> +	if (ctrl->opts->dhchap_auth)
> +		size += chap->hash_len;
> +
> +
> +	if (buf_size < size) {
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
> +		return -ENOMSG;
> +	}
> +
> +	if (data->hl != chap->hash_len) {
> +		dev_warn(ctrl->device,
> +			 "qid %d: DH-HMAC-CHAP: invalid hash length %d\n",
> +			 chap->qid, data->hl);
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
> +		return -EPROTO;
> +	}
> +
> +	if (!data->rvalid)
> +		return 0;
> +
> +	/* Validate controller response */
> +	if (memcmp(chap->response, data->rval, data->hl)) {
> +		dev_dbg(ctrl->device, "%s: qid %d ctrl response %*ph\n",
> +			__func__, chap->qid, chap->hash_len, data->rval);
> +		dev_dbg(ctrl->device, "%s: qid %d host response %*ph\n",
> +			__func__, chap->qid, chap->hash_len, chap->response);
> +		dev_warn(ctrl->device,
> +			 "qid %d: DH-HMAC-CHAP: controller authentication failed\n",
> +			 chap->qid);
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
> +		return -EPROTO;
> +	}
> +	dev_info(ctrl->device,
> +		 "qid %d: DH-HMAC-CHAP: controller authenticated\n",
> +		chap->qid);
> +	return 0;
> +}
> +
> +static int nvme_auth_dhchap_success2(struct nvme_ctrl *ctrl,
> +				     struct nvme_dhchap_context *chap,
> +				     void *buf, size_t buf_size)

same

> +{
> +	struct nvmf_auth_dhchap_success2_data *data = buf;
> +	size_t size = sizeof(*data);
> +
> +	memset(buf, 0, size);
> +	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
> +	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_SUCCESS2;
> +	data->t_id = cpu_to_le16(chap->transaction);
> +
> +	return size;
> +}
> +
> +static int nvme_auth_dhchap_failure2(struct nvme_ctrl *ctrl,
> +				     struct nvme_dhchap_context *chap,
> +				     void *buf, size_t buf_size)

same

> +{
> +	struct nvmf_auth_dhchap_failure_data *data = buf;
> +	size_t size = sizeof(*data);
> +
> +	memset(buf, 0, size);
> +	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
> +	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_FAILURE2;
> +	data->t_id = cpu_to_le16(chap->transaction);
> +	data->reason_code = 1;
> +	data->reason_code_explanation = chap->status;
> +
> +	return size;
> +}
> +
> +int nvme_auth_select_hash(struct nvme_ctrl *ctrl,
> +			  struct nvme_dhchap_context *chap)

Maybe _select_hf (hash function)? not a must, just sticks
to the spec language.

> +{
> +	char *hash_name;
> +	int ret;
> +
> +	switch (chap->hash_id) {
> +	case NVME_AUTH_DHCHAP_HASH_SHA256:
> +		hash_name = "hmac(sha256)";
> +		break;
> +	case NVME_AUTH_DHCHAP_HASH_SHA384:
> +		hash_name = "hmac(sha384)";
> +		break;
> +	case NVME_AUTH_DHCHAP_HASH_SHA512:
> +		hash_name = "hmac(sha512)";
> +		break;
> +	default:
> +		hash_name = NULL;
> +		break;
> +	}
> +	if (!hash_name) {
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
> +		return -EPROTO;
> +	}
> +	chap->shash_tfm = crypto_alloc_shash(hash_name, 0,
> +					     CRYPTO_ALG_ALLOCATES_MEMORY);
> +	if (IS_ERR(chap->shash_tfm)) {
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
> +		chap->shash_tfm = NULL;
> +		return -EPROTO;
> +	}
> +	if (!chap->key) {
> +		dev_warn(ctrl->device, "qid %d: cannot select hash, no key\n",
> +			 chap->qid);
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
> +		crypto_free_shash(chap->shash_tfm);

Wouldn't it better to check this before allocating the tfm?

> +		chap->shash_tfm = NULL;
> +		return -EINVAL;
> +	}
> +	ret = crypto_shash_setkey(chap->shash_tfm, chap->key, chap->key_len);
> +	if (ret) {
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
> +		crypto_free_shash(chap->shash_tfm);
> +		chap->shash_tfm = NULL;
> +		return ret;
> +	}
> +	dev_info(ctrl->device, "qid %d: DH-HMAC_CHAP: selected hash %s\n",
> +		 chap->qid, hash_name);
> +	return 0;
> +}
> +
> +static int nvme_auth_dhchap_host_response(struct nvme_ctrl *ctrl,
> +					  struct nvme_dhchap_context *chap)
> +{
> +	SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
> +	u8 buf[4], *challenge = chap->c1;
> +	int ret;
> +
> +	dev_dbg(ctrl->device, "%s: qid %d host response seq %d transaction %d\n",
> +		__func__, chap->qid, chap->s1, chap->transaction);
> +	shash->tfm = chap->shash_tfm;
> +	ret = crypto_shash_init(shash);
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, challenge, chap->hash_len);
> +	if (ret)
> +		goto out;
> +	put_unaligned_le32(chap->s1, buf);
> +	ret = crypto_shash_update(shash, buf, 4);
> +	if (ret)
> +		goto out;
> +	put_unaligned_le16(chap->transaction, buf);
> +	ret = crypto_shash_update(shash, buf, 2);
> +	if (ret)
> +		goto out;
> +	memset(buf, 0, sizeof(buf));
> +	ret = crypto_shash_update(shash, buf, 1);
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, "HostHost", 8);

HostHost ? Can you refer me to the specific section
that talks about this?

Would be good to have a comment on the format fed to the
shash.

> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
> +				  strlen(ctrl->opts->host->nqn));
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, buf, 1);
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
> +			    strlen(ctrl->opts->subsysnqn));
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_final(shash, chap->response);
> +out:
> +	return ret;
> +}
> +
> +static int nvme_auth_dhchap_ctrl_response(struct nvme_ctrl *ctrl,
> +					  struct nvme_dhchap_context *chap)
> +{
> +	SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
> +	u8 buf[4], *challenge = chap->c2;
> +	int ret;
> +
> +	dev_dbg(ctrl->device, "%s: qid %d host response seq %d transaction %d\n",
> +		__func__, chap->qid, chap->s2, chap->transaction);
> +	dev_dbg(ctrl->device, "%s: qid %d challenge %*ph\n",
> +		__func__, chap->qid, chap->hash_len, challenge);
> +	dev_dbg(ctrl->device, "%s: qid %d subsysnqn %s\n",
> +		__func__, chap->qid, ctrl->opts->subsysnqn);
> +	dev_dbg(ctrl->device, "%s: qid %d hostnqn %s\n",
> +		__func__, chap->qid, ctrl->opts->host->nqn);
> +	shash->tfm = chap->shash_tfm;
> +	ret = crypto_shash_init(shash);
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, challenge, chap->hash_len);
> +	if (ret)
> +		goto out;
> +	put_unaligned_le32(chap->s2, buf);
> +	ret = crypto_shash_update(shash, buf, 4);
> +	if (ret)
> +		goto out;
> +	put_unaligned_le16(chap->transaction, buf);
> +	ret = crypto_shash_update(shash, buf, 2);
> +	if (ret)
> +		goto out;
> +	memset(buf, 0, 4);
> +	ret = crypto_shash_update(shash, buf, 1);
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, "Controller", 10);
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
> +				  strlen(ctrl->opts->subsysnqn));
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, buf, 1);
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
> +				  strlen(ctrl->opts->host->nqn));
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_final(shash, chap->response);
> +out:
> +	return ret;
> +}
> +
> +int nvme_auth_generate_key(struct nvme_ctrl *ctrl,
> +			   struct nvme_dhchap_context *chap)
> +{
> +	int ret;
> +	u8 key_hash;
> +	const char *hmac_name;
> +	struct crypto_shash *key_tfm;
> +
> +	if (sscanf(ctrl->opts->dhchap_secret, "DHHC-1:%hhd:%*s:",
> +		   &key_hash) != 1)
> +		return -EINVAL;

I'd expect that the user will pass in a secret key (as binary)
the the driver will build the spec compliant formatted string no?

Am I not reading this correctly?

> +
> +	chap->key = nvme_auth_extract_secret(ctrl->opts->dhchap_secret,
> +					     &chap->key_len);
> +	if (IS_ERR(chap->key)) {
> +		ret = PTR_ERR(chap->key);
> +		chap->key = NULL;
> +		return ret;
> +	}
> +
> +	if (key_hash == 0)
> +		return 0;
> +
> +	hmac_name = nvme_auth_hmac_name(key_hash);
> +	if (!hmac_name) {
> +		pr_debug("Invalid key hash id %d\n", key_hash);
> +		return -EKEYREJECTED;
> +	}

Why does the user influence the hmac used? isn't that is driven
by the susbsystem?

I don't think that the user should choose in this level.

> +
> +	key_tfm = crypto_alloc_shash(hmac_name, 0, 0);
> +	if (IS_ERR(key_tfm)) {
> +		kfree(chap->key);
> +		chap->key = NULL;
> +		ret = PTR_ERR(key_tfm);

You set ret and later return 0? I think that the success
path in the else clause is hard to read and error prone...

> +	} else {
> +		SHASH_DESC_ON_STACK(shash, key_tfm);
> +
> +		shash->tfm = key_tfm;
> +		ret = crypto_shash_setkey(key_tfm, chap->key,
> +					  chap->key_len);
> +		if (ret < 0) {
> +			crypto_free_shash(key_tfm);
> +			kfree(chap->key);
> +			chap->key = NULL;
> +			return ret;
> +		}
> +		crypto_shash_init(shash);
> +		crypto_shash_update(shash, ctrl->opts->host->nqn,
> +				    strlen(ctrl->opts->host->nqn));
> +		crypto_shash_update(shash, "NVMe-over-Fabrics", 17);
> +		crypto_shash_final(shash, chap->key);
> +		crypto_free_shash(key_tfm);

Shouldn't these be done when preparing the dh-hmac-chap reply?

> +	}
> +	return 0;
> +}
> +
> +void nvme_auth_free(struct nvme_dhchap_context *chap)
> +{
> +	if (chap->shash_tfm)
> +		crypto_free_shash(chap->shash_tfm);
> +	if (chap->key)
> +		kfree(chap->key);
> +	if (chap->ctrl_key)
> +		kfree(chap->ctrl_key);
> +	if (chap->host_key)
> +		kfree(chap->host_key);
> +	if (chap->sess_key)
> +		kfree(chap->sess_key);

No need to check null for kfree...

> +	kfree(chap);
> +}
> +
> +int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid)
> +{
> +	struct nvme_dhchap_context *chap;
> +	void *buf;
> +	size_t buf_size, tl;
> +	int ret = 0;
> +
> +	chap = kzalloc(sizeof(*chap), GFP_KERNEL);
> +	if (!chap)
> +		return -ENOMEM;
> +	chap->qid = qid;
> +	chap->transaction = ctrl->transaction++;
> +
> +	ret = nvme_auth_generate_key(ctrl, chap);
> +	if (ret) {
> +		dev_dbg(ctrl->device, "%s: failed to generate key, error %d\n",
> +			__func__, ret);
> +		nvme_auth_free(chap);
> +		return ret;
> +	}
> +
> +	/*
> +	 * Allocate a large enough buffer for the entire negotiation:
> +	 * 4k should be enough to ffdhe8192.
> +	 */
> +	buf_size = 4096;
> +	buf = kzalloc(buf_size, GFP_KERNEL);
> +	if (!buf) {
> +		ret = -ENOMEM;
> +		goto out;
> +	}
> +
> +	/* DH-HMAC-CHAP Step 1: send negotiate */

I'd consider breaking these into sub-routines.

> +	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP negotiate\n",
> +		__func__, qid);
> +	ret = nvme_auth_dhchap_negotiate(ctrl, chap, buf, buf_size);
> +	if (ret < 0)
> +		goto out;
> +	tl = ret;
> +	ret = nvme_auth_send(ctrl, qid, buf, tl);
> +	if (ret)
> +		goto out;
> +
> +	memset(buf, 0, buf_size);
> +	ret = nvme_auth_receive(ctrl, qid, buf, buf_size, chap->transaction,
> +				NVME_AUTH_DHCHAP_MESSAGE_CHALLENGE);
> +	if (ret < 0) {
> +		dev_dbg(ctrl->device,
> +			"%s: qid %d DH-HMAC-CHAP failed to receive challenge\n",
> +			__func__, qid);
> +		goto out;
> +	}
> +	if (ret > 0) {
> +		chap->status = ret;
> +		goto fail1;
> +	}
> +
> +	/* DH-HMAC-CHAP Step 2: receive challenge */
> +	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP challenge\n",
> +		__func__, qid);
> +
> +	ret = nvme_auth_dhchap_challenge(ctrl, chap, buf, buf_size);
> +	if (ret) {
> +		/* Invalid parameters for negotiate */
> +		goto fail2;
> +	}
> +
> +	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP select hash\n",
> +		__func__, qid);
> +	ret = nvme_auth_select_hash(ctrl, chap);
> +	if (ret)
> +		goto fail2;
> +
> +	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP host response\n",
> +		__func__, qid);
> +	ret = nvme_auth_dhchap_host_response(ctrl, chap);
> +	if (ret)
> +		goto fail2;
> +
> +	/* DH-HMAC-CHAP Step 3: send reply */
> +	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP reply\n",
> +		__func__, qid);
> +	ret = nvme_auth_dhchap_reply(ctrl, chap, buf, buf_size);
> +	if (ret < 0)
> +		goto fail2;
> +
> +	tl = ret;
> +	ret = nvme_auth_send(ctrl, qid, buf, tl);
> +	if (ret)
> +		goto fail2;
> +
> +	memset(buf, 0, buf_size);
> +	ret = nvme_auth_receive(ctrl, qid, buf, buf_size, chap->transaction,
> +				NVME_AUTH_DHCHAP_MESSAGE_SUCCESS1);
> +	if (ret < 0) {
> +		dev_dbg(ctrl->device,
> +			"%s: qid %d DH-HMAC-CHAP failed to receive success1\n",
> +			__func__, qid);
> +		goto out;
> +	}
> +	if (ret > 0) {
> +		chap->status = ret;
> +		goto fail1;
> +	}
> +
> +	if (ctrl->opts->dhchap_auth) {
> +		dev_dbg(ctrl->device,
> +			"%s: qid %d DH-HMAC-CHAP controller response\n",
> +			__func__, qid);
> +		ret = nvme_auth_dhchap_ctrl_response(ctrl, chap);
> +		if (ret)
> +			goto fail2;
> +	}
> +
> +	/* DH-HMAC-CHAP Step 4: receive success1 */
> +	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP success1\n",
> +		__func__, qid);
> +	ret = nvme_auth_dhchap_success1(ctrl, chap, buf, buf_size);
> +	if (ret < 0) {
> +		/* Controller authentication failed */
> +		goto fail2;
> +	}
> +	tl = ret;
> +	/* DH-HMAC-CHAP Step 5: send success2 */
> +	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP success2\n",
> +		__func__, qid);
> +	tl = nvme_auth_dhchap_success2(ctrl, chap, buf, buf_size);
> +	ret = nvme_auth_send(ctrl, qid, buf, tl);
> +	if (!ret)
> +		goto out;
> +
> +fail1:
> +	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP failure1, status %x\n",
> +		__func__, qid, chap->status);
> +	goto out;
> +
> +fail2:
> +	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP failure2, status %x\n",
> +		__func__, qid, chap->status);
> +	tl = nvme_auth_dhchap_failure2(ctrl, chap, buf, buf_size);
> +	ret = nvme_auth_send(ctrl, qid, buf, tl);
> +
> +out:
> +	if (!ret && chap->status)
> +		ret = -EPROTO;
> +	if (!ret) {
> +		ctrl->dhchap_hash = chap->hash_id;
> +	}
> +	kfree(buf);
> +	nvme_auth_free(chap);
> +	return ret;
> +}
> diff --git a/drivers/nvme/host/auth.h b/drivers/nvme/host/auth.h
> new file mode 100644
> index 000000000000..4950b1cb9470
> --- /dev/null
> +++ b/drivers/nvme/host/auth.h
> @@ -0,0 +1,23 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Copyright (c) 2021 Hannes Reinecke, SUSE Software Solutions
> + */
> +
> +#ifndef _NVME_AUTH_H
> +#define _NVME_AUTH_H
> +
> +const char *nvme_auth_dhgroup_name(int dhgroup_id);
> +int nvme_auth_dhgroup_pubkey_size(int dhgroup_id);
> +int nvme_auth_dhgroup_privkey_size(int dhgroup_id);
> +const char *nvme_auth_dhgroup_kpp(int dhgroup_id);
> +int nvme_auth_dhgroup_id(const char *dhgroup_name);
> +
> +const char *nvme_auth_hmac_name(int hmac_id);
> +const char *nvme_auth_digest_name(int hmac_id);
> +int nvme_auth_hmac_id(const char *hmac_name);
> +int nvme_auth_hmac_len(int hmac_len);
> +
> +unsigned char *nvme_auth_extract_secret(unsigned char *dhchap_secret,
> +					size_t *dhchap_key_len);
> +
> +#endif /* _NVME_AUTH_H */
> diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
> index 11779be42186..7ce9b666dc09 100644
> --- a/drivers/nvme/host/core.c
> +++ b/drivers/nvme/host/core.c
> @@ -708,7 +708,9 @@ bool __nvme_check_ready(struct nvme_ctrl *ctrl, struct request *rq,
>   		switch (ctrl->state) {
>   		case NVME_CTRL_CONNECTING:
>   			if (blk_rq_is_passthrough(rq) && nvme_is_fabrics(req->cmd) &&
> -			    req->cmd->fabrics.fctype == nvme_fabrics_type_connect)
> +			    (req->cmd->fabrics.fctype == nvme_fabrics_type_connect ||
> +			     req->cmd->fabrics.fctype == nvme_fabrics_type_auth_send ||
> +			     req->cmd->fabrics.fctype == nvme_fabrics_type_auth_receive))
>   				return true;
>   			break;
>   		default:
> @@ -3426,6 +3428,66 @@ static ssize_t nvme_ctrl_fast_io_fail_tmo_store(struct device *dev,
>   static DEVICE_ATTR(fast_io_fail_tmo, S_IRUGO | S_IWUSR,
>   	nvme_ctrl_fast_io_fail_tmo_show, nvme_ctrl_fast_io_fail_tmo_store);
>   
> +#ifdef CONFIG_NVME_AUTH
> +struct nvmet_dhchap_hash_map {
> +	int id;
> +	const char name[15];
> +} hash_map[] = {
> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA256,
> +	 .name = "hmac(sha256)", },
> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA384,
> +	 .name = "hmac(sha384)", },
> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA512,
> +	 .name = "hmac(sha512)", },
> +};
> +
> +static ssize_t dhchap_hash_show(struct device *dev,
> +	struct device_attribute *attr, char *buf)
> +{
> +	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
> +	int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
> +		if (hash_map[i].id == ctrl->dhchap_hash)
> +			return sprintf(buf, "%s\n", hash_map[i].name);
> +	}
> +	return sprintf(buf, "none\n");
> +}
> +DEVICE_ATTR_RO(dhchap_hash);
> +
> +struct nvmet_dhchap_group_map {
> +	int id;
> +	const char name[15];
> +} dhgroup_map[] = {
> +	{.id = NVME_AUTH_DHCHAP_DHGROUP_NULL,
> +	 .name = "NULL", },
> +	{.id = NVME_AUTH_DHCHAP_DHGROUP_2048,
> +	 .name = "ffdhe2048", },
> +	{.id = NVME_AUTH_DHCHAP_DHGROUP_3072,
> +	 .name = "ffdhe3072", },
> +	{.id = NVME_AUTH_DHCHAP_DHGROUP_4096,
> +	 .name = "ffdhe4096", },
> +	{.id = NVME_AUTH_DHCHAP_DHGROUP_6144,
> +	 .name = "ffdhe6144", },
> +	{.id = NVME_AUTH_DHCHAP_DHGROUP_8192,
> +	 .name = "ffdhe8192", },
> +};
> +
> +static ssize_t dhchap_dhgroup_show(struct device *dev,
> +	struct device_attribute *attr, char *buf)
> +{
> +	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
> +	int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(dhgroup_map); i++) {
> +		if (hash_map[i].id == ctrl->dhchap_dhgroup)
> +			return sprintf(buf, "%s\n", dhgroup_map[i].name);
> +	}
> +	return sprintf(buf, "none\n");
> +}
> +DEVICE_ATTR_RO(dhchap_dhgroup);
> +#endif
> +
>   static struct attribute *nvme_dev_attrs[] = {
>   	&dev_attr_reset_controller.attr,
>   	&dev_attr_rescan_controller.attr,
> @@ -3447,6 +3509,10 @@ static struct attribute *nvme_dev_attrs[] = {
>   	&dev_attr_reconnect_delay.attr,
>   	&dev_attr_fast_io_fail_tmo.attr,
>   	&dev_attr_kato.attr,
> +#ifdef CONFIG_NVME_AUTH
> +	&dev_attr_dhchap_hash.attr,
> +	&dev_attr_dhchap_dhgroup.attr,
> +#endif
>   	NULL
>   };
>   
> @@ -3470,6 +3536,10 @@ static umode_t nvme_dev_attrs_are_visible(struct kobject *kobj,
>   		return 0;
>   	if (a == &dev_attr_fast_io_fail_tmo.attr && !ctrl->opts)
>   		return 0;
> +#ifdef CONFIG_NVME_AUTH
> +	if (a == &dev_attr_dhchap_hash.attr && !ctrl->opts)
> +		return 0;
> +#endif
>   
>   	return a->mode;
>   }
> @@ -4581,6 +4651,11 @@ static inline void _nvme_check_size(void)
>   	BUILD_BUG_ON(sizeof(struct nvme_smart_log) != 512);
>   	BUILD_BUG_ON(sizeof(struct nvme_dbbuf) != 64);
>   	BUILD_BUG_ON(sizeof(struct nvme_directive_cmd) != 64);
> +	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_negotiate_data) != 8);
> +	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_challenge_data) != 16);
> +	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_reply_data) != 16);
> +	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_success1_data) != 16);
> +	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_success2_data) != 16);
>   }
>   
>   
> diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
> index a5469fd9d4c3..6404ab9b604b 100644
> --- a/drivers/nvme/host/fabrics.c
> +++ b/drivers/nvme/host/fabrics.c
> @@ -366,6 +366,7 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)
>   	union nvme_result res;
>   	struct nvmf_connect_data *data;
>   	int ret;
> +	u32 result;
>   
>   	cmd.connect.opcode = nvme_fabrics_command;
>   	cmd.connect.fctype = nvme_fabrics_type_connect;
> @@ -398,8 +399,18 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)
>   		goto out_free_data;
>   	}
>   
> -	ctrl->cntlid = le16_to_cpu(res.u16);
> -
> +	result = le32_to_cpu(res.u32);
> +	ctrl->cntlid = result & 0xFFFF;
> +	if ((result >> 16) & 2) {
> +		/* Authentication required */
> +		ret = nvme_auth_negotiate(ctrl, NVME_QID_ANY);
> +		if (ret)
> +			dev_warn(ctrl->device,
> +				 "qid 0: authentication failed\n");
> +		else
> +			dev_info(ctrl->device,
> +				 "qid 0: authenticated\n");

info is too chatty.

> +	}
>   out_free_data:
>   	kfree(data);
>   	return ret;
> @@ -432,6 +443,7 @@ int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, u16 qid)
>   	struct nvmf_connect_data *data;
>   	union nvme_result res;
>   	int ret;
> +	u32 result;
>   
>   	cmd.connect.opcode = nvme_fabrics_command;
>   	cmd.connect.fctype = nvme_fabrics_type_connect;
> @@ -457,6 +469,17 @@ int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, u16 qid)
>   		nvmf_log_connect_error(ctrl, ret, le32_to_cpu(res.u32),
>   				       &cmd, data);
>   	}
> +	result = le32_to_cpu(res.u32);
> +	if ((result >> 16) & 2) {
> +		/* Authentication required */
> +		ret = nvme_auth_negotiate(ctrl, qid);
> +		if (ret)
> +			dev_warn(ctrl->device,
> +				 "qid %u: authentication failed\n", qid);
> +		else
> +			dev_info(ctrl->device,
> +				 "qid %u: authenticated\n", qid);
> +	}
>   	kfree(data);
>   	return ret;
>   }
> @@ -548,6 +571,9 @@ static const match_table_t opt_tokens = {
>   	{ NVMF_OPT_NR_POLL_QUEUES,	"nr_poll_queues=%d"	},
>   	{ NVMF_OPT_TOS,			"tos=%d"		},
>   	{ NVMF_OPT_FAIL_FAST_TMO,	"fast_io_fail_tmo=%d"	},
> +	{ NVMF_OPT_DHCHAP_SECRET,	"dhchap_secret=%s"	},
> +	{ NVMF_OPT_DHCHAP_AUTH,		"authenticate"		},
> +	{ NVMF_OPT_DHCHAP_GROUP,	"dhchap_group=%s"	},

Isn't the group driven by the subsystem? also why is there a
"authenticate" boolean? what is it good for?

>   	{ NVMF_OPT_ERR,			NULL			}
>   };
>   
> @@ -824,6 +850,35 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
>   			}
>   			opts->tos = token;
>   			break;
> +		case NVMF_OPT_DHCHAP_SECRET:
> +			p = match_strdup(args);
> +			if (!p) {
> +				ret = -ENOMEM;
> +				goto out;
> +			}
> +			if (strncmp(p, "DHHC-1:00:", 10)) {
> +				pr_err("Invalid DH-CHAP secret %s\n", p);
> +				ret = -EINVAL;
> +				goto out;
> +			}
> +			kfree(opts->dhchap_secret);
> +			opts->dhchap_secret = p;
> +			break;
> +		case NVMF_OPT_DHCHAP_AUTH:
> +			opts->dhchap_auth = true;
> +			break;
> +		case NVMF_OPT_DHCHAP_GROUP:
> +			if (match_int(args, &token)) {
> +				ret = -EINVAL;
> +				goto out;
> +			}
> +			if (token <= 0) {
> +				pr_err("Invalid dhchap_group %d\n", token);
> +				ret = -EINVAL;
> +				goto out;
> +			}
> +			opts->dhchap_group = token;
> +			break;
>   		default:
>   			pr_warn("unknown parameter or missing value '%s' in ctrl creation request\n",
>   				p);
> @@ -942,6 +997,7 @@ void nvmf_free_options(struct nvmf_ctrl_options *opts)
>   	kfree(opts->subsysnqn);
>   	kfree(opts->host_traddr);
>   	kfree(opts->host_iface);
> +	kfree(opts->dhchap_secret);
>   	kfree(opts);
>   }
>   EXPORT_SYMBOL_GPL(nvmf_free_options);
> @@ -951,7 +1007,10 @@ EXPORT_SYMBOL_GPL(nvmf_free_options);
>   				 NVMF_OPT_KATO | NVMF_OPT_HOSTNQN | \
>   				 NVMF_OPT_HOST_ID | NVMF_OPT_DUP_CONNECT |\
>   				 NVMF_OPT_DISABLE_SQFLOW |\
> -				 NVMF_OPT_FAIL_FAST_TMO)
> +				 NVMF_OPT_CTRL_LOSS_TMO |\
> +				 NVMF_OPT_FAIL_FAST_TMO |\
> +				 NVMF_OPT_DHCHAP_SECRET |\
> +				 NVMF_OPT_DHCHAP_AUTH | NVMF_OPT_DHCHAP_GROUP)
>   
>   static struct nvme_ctrl *
>   nvmf_create_ctrl(struct device *dev, const char *buf)
> diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h
> index a146cb903869..535bc544f0f6 100644
> --- a/drivers/nvme/host/fabrics.h
> +++ b/drivers/nvme/host/fabrics.h
> @@ -67,6 +67,9 @@ enum {
>   	NVMF_OPT_TOS		= 1 << 19,
>   	NVMF_OPT_FAIL_FAST_TMO	= 1 << 20,
>   	NVMF_OPT_HOST_IFACE	= 1 << 21,
> +	NVMF_OPT_DHCHAP_SECRET	= 1 << 22,
> +	NVMF_OPT_DHCHAP_AUTH	= 1 << 23,
> +	NVMF_OPT_DHCHAP_GROUP	= 1 << 24,
>   };
>   
>   /**
> @@ -96,6 +99,8 @@ enum {
>    * @max_reconnects: maximum number of allowed reconnect attempts before removing
>    *              the controller, (-1) means reconnect forever, zero means remove
>    *              immediately;
> + * @dhchap_secret: DH-HMAC-CHAP secret
> + * @dhchap_auth: DH-HMAC-CHAP authenticate controller
>    * @disable_sqflow: disable controller sq flow control
>    * @hdr_digest: generate/verify header digest (TCP)
>    * @data_digest: generate/verify data digest (TCP)
> @@ -120,6 +125,9 @@ struct nvmf_ctrl_options {
>   	unsigned int		kato;
>   	struct nvmf_host	*host;
>   	int			max_reconnects;
> +	char			*dhchap_secret;
> +	int			dhchap_group;
> +	bool			dhchap_auth;
>   	bool			disable_sqflow;
>   	bool			hdr_digest;
>   	bool			data_digest;
> diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
> index 18ef8dd03a90..bcd5b8276c26 100644
> --- a/drivers/nvme/host/nvme.h
> +++ b/drivers/nvme/host/nvme.h
> @@ -328,6 +328,12 @@ struct nvme_ctrl {
>   	struct work_struct ana_work;
>   #endif
>   
> +#ifdef CONFIG_NVME_AUTH
> +	u16 transaction;
> +	u8 dhchap_hash;
> +	u8 dhchap_dhgroup;

Do multiple controllers in the same subsystem have different
params? no, so I think these should be lifted to subsys.

> +#endif
> +
>   	/* Power saving configuration */
>   	u64 ps_max_latency_us;
>   	bool apst_enabled;
> @@ -874,6 +880,15 @@ static inline bool nvme_ctrl_sgl_supported(struct nvme_ctrl *ctrl)
>   	return ctrl->sgls & ((1 << 0) | (1 << 1));
>   }
>   
> +#ifdef CONFIG_NVME_AUTH
> +int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid);
> +#else
> +static inline int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid)
> +{
> +	return -EPROTONOSUPPORT;
> +}
> +#endif
> +
>   u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
>   			 u8 opcode);
>   int nvme_execute_passthru_rq(struct request *rq);
> diff --git a/drivers/nvme/host/trace.c b/drivers/nvme/host/trace.c
> index 6543015b6121..66f75d8ea925 100644
> --- a/drivers/nvme/host/trace.c
> +++ b/drivers/nvme/host/trace.c

I'd split out the tracing logic.

> @@ -271,6 +271,34 @@ static const char *nvme_trace_fabrics_property_get(struct trace_seq *p, u8 *spc)
>   	return ret;
>   }
>   
> +static const char *nvme_trace_fabrics_auth_send(struct trace_seq *p, u8 *spc)
> +{
> +	const char *ret = trace_seq_buffer_ptr(p);
> +	u8 spsp0 = spc[1];
> +	u8 spsp1 = spc[2];
> +	u8 secp = spc[3];
> +	u32 tl = get_unaligned_le32(spc + 4);
> +
> +	trace_seq_printf(p, "spsp0=%02x, spsp1=%02x, secp=%02x, tl=%u",
> +			 spsp0, spsp1, secp, tl);
> +	trace_seq_putc(p, 0);
> +	return ret;
> +}
> +
> +static const char *nvme_trace_fabrics_auth_receive(struct trace_seq *p, u8 *spc)
> +{
> +	const char *ret = trace_seq_buffer_ptr(p);
> +	u8 spsp0 = spc[1];
> +	u8 spsp1 = spc[2];
> +	u8 secp = spc[3];
> +	u32 al = get_unaligned_le32(spc + 4);
> +
> +	trace_seq_printf(p, "spsp0=%02x, spsp1=%02x, secp=%02x, al=%u",
> +			 spsp0, spsp1, secp, al);
> +	trace_seq_putc(p, 0);
> +	return ret;
> +}
> +
>   static const char *nvme_trace_fabrics_common(struct trace_seq *p, u8 *spc)
>   {
>   	const char *ret = trace_seq_buffer_ptr(p);
> @@ -290,6 +318,10 @@ const char *nvme_trace_parse_fabrics_cmd(struct trace_seq *p,
>   		return nvme_trace_fabrics_connect(p, spc);
>   	case nvme_fabrics_type_property_get:
>   		return nvme_trace_fabrics_property_get(p, spc);
> +	case nvme_fabrics_type_auth_send:
> +		return nvme_trace_fabrics_auth_send(p, spc);
> +	case nvme_fabrics_type_auth_receive:
> +		return nvme_trace_fabrics_auth_receive(p, spc);
>   	default:
>   		return nvme_trace_fabrics_common(p, spc);
>   	}
> 

Re: [PATCH 06/11] nvme: Implement In-Band authentication

[Sagi Grimberg]

> Implement NVMe-oF In-Band authentication. This patch adds two new
> fabric options 'dhchap_key' to specify the PSK

pre-shared-key.

Also, we need a sysfs knob to rotate the key that will trigger
re-authentication or even a simple controller(s-plural) reset, so this
should go beyond just the connection string.

P.S. can you add also the nvme-cli code in the next go?

> and 'dhchap_authenticate'
> to request bi-directional authentication of both the host and the controller.

bidirectional? not uni-directional?

> 
> Signed-off-by: Hannes Reinecke <hare@suse.de>
> ---
>   drivers/nvme/host/Kconfig   |  11 +
>   drivers/nvme/host/Makefile  |   1 +
>   drivers/nvme/host/auth.c    | 813 ++++++++++++++++++++++++++++++++++++
>   drivers/nvme/host/auth.h    |  23 +
>   drivers/nvme/host/core.c    |  77 +++-
>   drivers/nvme/host/fabrics.c |  65 ++-
>   drivers/nvme/host/fabrics.h |   8 +
>   drivers/nvme/host/nvme.h    |  15 +
>   drivers/nvme/host/trace.c   |  32 ++
>   9 files changed, 1041 insertions(+), 4 deletions(-)
>   create mode 100644 drivers/nvme/host/auth.c
>   create mode 100644 drivers/nvme/host/auth.h
> 
> diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig
> index c3f3d77f1aac..853c546305e9 100644
> --- a/drivers/nvme/host/Kconfig
> +++ b/drivers/nvme/host/Kconfig
> @@ -85,3 +85,14 @@ config NVME_TCP
>   	  from https://github.com/linux-nvme/nvme-cli.
>   
>   	  If unsure, say N.
> +
> +config NVME_AUTH
> +	bool "NVM Express over Fabrics In-Band Authentication"
> +	depends on NVME_TCP
> +	select CRYPTO_SHA256
> +	select CRYPTO_SHA512
> +	help
> +	  This provides support for NVMe over Fabrics In-Band Authentication
> +	  for the NVMe over TCP transport.

In this form, nothing is specific to nvme-tcp here afaict.

> +
> +	  If unsure, say N.
> diff --git a/drivers/nvme/host/Makefile b/drivers/nvme/host/Makefile
> index cbc509784b2e..03748a55a12b 100644
> --- a/drivers/nvme/host/Makefile
> +++ b/drivers/nvme/host/Makefile
> @@ -20,6 +20,7 @@ nvme-core-$(CONFIG_NVME_HWMON)		+= hwmon.o
>   nvme-y					+= pci.o
>   
>   nvme-fabrics-y				+= fabrics.o
> +nvme-fabrics-$(CONFIG_NVME_AUTH)	+= auth.o
>   
>   nvme-rdma-y				+= rdma.o
>   
> diff --git a/drivers/nvme/host/auth.c b/drivers/nvme/host/auth.c
> new file mode 100644
> index 000000000000..448a3adebea6
> --- /dev/null
> +++ b/drivers/nvme/host/auth.c
> @@ -0,0 +1,813 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Copyright (c) 2020 Hannes Reinecke, SUSE Linux
> + */
> +
> +#include <linux/crc32.h>
> +#include <linux/base64.h>
> +#include <asm/unaligned.h>
> +#include <crypto/hash.h>
> +#include <crypto/kpp.h>
> +#include "nvme.h"
> +#include "fabrics.h"
> +#include "auth.h"
> +
> +static u32 nvme_dhchap_seqnum;
> +
> +struct nvme_dhchap_context {

Maybe nvme_dhchap_queue_context ?

I'm thinking that we should perhaps split
it to host-wide, subsys-wide and queue specific
auth contexts?

Let's see...

> +	struct crypto_shash *shash_tfm;
> +	unsigned char *key;
> +	size_t key_len;
> +	int qid;
> +	u32 s1;
> +	u32 s2;
> +	u16 transaction;
> +	u8 status;
> +	u8 hash_id;
> +	u8 hash_len;
> +	u8 c1[64];
> +	u8 c2[64];
> +	u8 response[64];
> +	u8 *ctrl_key;
> +	int ctrl_key_len;
> +	u8 *host_key;
> +	int host_key_len;
> +	u8 *sess_key;
> +	int sess_key_len;
> +};
> +
> +struct nvmet_dhchap_hash_map {

nvmet?

> +	int id;
> +	int hash_len;
> +	const char hmac[15];
> +	const char digest[15];
> +} hash_map[] = {
> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA256,
> +	 .hash_len = 32,
> +	 .hmac = "hmac(sha256)", .digest = "sha256" },
> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA384,
> +	 .hash_len = 48,
> +	 .hmac = "hmac(sha384)", .digest = "sha384" },
> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA512,
> +	 .hash_len = 64,
> +	 .hmac = "hmac(sha512)", .digest = "sha512" },
> +};
> +
> +const char *nvme_auth_hmac_name(int hmac_id)

Should these arrays be static?

> +{
> +	int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
> +		if (hash_map[i].id == hmac_id)
> +			return hash_map[i].hmac;
> +	}
> +	return NULL;
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_hmac_name);
> +
> +const char *nvme_auth_digest_name(int hmac_id)
> +{
> +	int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
> +		if (hash_map[i].id == hmac_id)
> +			return hash_map[i].digest;
> +	}
> +	return NULL;
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_digest_name);
> +
> +int nvme_auth_hmac_len(int hmac_id)
> +{
> +	int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
> +		if (hash_map[i].id == hmac_id)
> +			return hash_map[i].hash_len;
> +	}
> +	return -1;
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_hmac_len);
> +
> +int nvme_auth_hmac_id(const char *hmac_name)
> +{
> +	int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
> +		if (!strncmp(hash_map[i].hmac, hmac_name,
> +			     strlen(hash_map[i].hmac)))
> +			return hash_map[i].id;
> +	}
> +	return -1;
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_hmac_id);
> +
> +unsigned char *nvme_auth_extract_secret(unsigned char *dhchap_secret,
> +					size_t *dhchap_key_len)
> +{
> +	unsigned char *dhchap_key;
> +	u32 crc;
> +	int key_len;
> +	size_t allocated_len;
> +
> +	allocated_len = strlen(dhchap_secret) - 10;

the 10 feels like a magic here, should at least note this is the
"DHHC-1:..." prefix.

> +	dhchap_key = kzalloc(allocated_len, GFP_KERNEL);
> +	if (!dhchap_key)
> +		return ERR_PTR(-ENOMEM);
> +
> +	key_len = base64_decode(dhchap_secret + 10,
> +				allocated_len, dhchap_key);
> +	if (key_len != 36 && key_len != 52 &&
> +	    key_len != 68) {
> +		pr_debug("Invalid DH-HMAC-CHAP key len %d\n",
> +			 key_len);
> +		kfree(dhchap_key);
> +		return ERR_PTR(-EINVAL);
> +	}
> +	pr_debug("DH-HMAC-CHAP Key: %*ph\n",
> +		 (int)key_len, dhchap_key);

One can argue if even printing this is problematic..

> +
> +	/* The last four bytes is the CRC in little-endian format */
> +	key_len -= 4;
> +	/*
> +	 * The linux implementation doesn't do pre- and post-increments,
> +	 * so we have to do it manually.
> +	 */
> +	crc = ~crc32(~0, dhchap_key, key_len);
> +
> +	if (get_unaligned_le32(dhchap_key + key_len) != crc) {
> +		pr_debug("DH-HMAC-CHAP crc mismatch (key %08x, crc %08x)\n",
> +		       get_unaligned_le32(dhchap_key + key_len), crc);
> +		kfree(dhchap_key);
> +		return ERR_PTR(-EKEYREJECTED);
> +	}
> +	*dhchap_key_len = key_len;
> +	return dhchap_key;
> +}
> +EXPORT_SYMBOL_GPL(nvme_auth_extract_secret);
> +
> +static int nvme_auth_send(struct nvme_ctrl *ctrl, int qid,
> +			  void *data, size_t tl)
> +{
> +	struct nvme_command cmd = {};
> +	blk_mq_req_flags_t flags = qid == NVME_QID_ANY ?
> +		0 : BLK_MQ_REQ_NOWAIT | BLK_MQ_REQ_RESERVED;
> +	struct request_queue *q = qid == NVME_QID_ANY ?
> +		ctrl->fabrics_q : ctrl->connect_q;
> +	int ret;
> +
> +	cmd.auth_send.opcode = nvme_fabrics_command;
> +	cmd.auth_send.fctype = nvme_fabrics_type_auth_send;
> +	cmd.auth_send.secp = NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER;
> +	cmd.auth_send.spsp0 = 0x01;
> +	cmd.auth_send.spsp1 = 0x01;
> +	cmd.auth_send.tl = tl;
> +
> +	ret = __nvme_submit_sync_cmd(q, &cmd, NULL, data, tl, 0, qid,
> +				     0, flags);
> +	if (ret)
> +		dev_dbg(ctrl->device,
> +			"%s: qid %d error %d\n", __func__, qid, ret);

Maybe a little more informative print rather than __func__ ?

> +	return ret;
> +}
> +
> +static int nvme_auth_receive(struct nvme_ctrl *ctrl, int qid,
> +			     void *buf, size_t al,
> +			     u16 transaction, u8 expected_msg )
> +{
> +	struct nvme_command cmd = {};
> +	struct nvmf_auth_dhchap_failure_data *data = buf;
> +	blk_mq_req_flags_t flags = qid == NVME_QID_ANY ?
> +		0 : BLK_MQ_REQ_NOWAIT | BLK_MQ_REQ_RESERVED;
> +	struct request_queue *q = qid == NVME_QID_ANY ?
> +		ctrl->fabrics_q : ctrl->connect_q;
> +	int ret;
> +
> +	cmd.auth_receive.opcode = nvme_fabrics_command;
> +	cmd.auth_receive.fctype = nvme_fabrics_type_auth_receive;
> +	cmd.auth_receive.secp = NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER;
> +	cmd.auth_receive.spsp0 = 0x01;
> +	cmd.auth_receive.spsp1 = 0x01;
> +	cmd.auth_receive.al = al;
> +
> +	ret = __nvme_submit_sync_cmd(q, &cmd, NULL, buf, al, 0, qid,
> +				     0, flags);
> +	if (ret > 0) {
> +		dev_dbg(ctrl->device, "%s: qid %d nvme status %x\n",
> +			__func__, qid, ret);
> +		ret = -EIO;
> +	}
> +	if (ret < 0) {
> +		dev_dbg(ctrl->device, "%s: qid %d error %d\n",
> +			__func__, qid, ret);
> +		return ret;
> +	}
> +	dev_dbg(ctrl->device, "%s: qid %d auth_type %d auth_id %x\n",
> +		__func__, qid, data->auth_type, data->auth_id);
> +	if (data->auth_type == NVME_AUTH_COMMON_MESSAGES &&
> +	    data->auth_id == NVME_AUTH_DHCHAP_MESSAGE_FAILURE1) {
> +		return data->reason_code_explanation;
> +	}
> +	if (data->auth_type != NVME_AUTH_DHCHAP_MESSAGES ||
> +	    data->auth_id != expected_msg) {
> +		dev_warn(ctrl->device,
> +			 "qid %d invalid message %02x/%02x\n",
> +			 qid, data->auth_type, data->auth_id);
> +		return NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
> +	}
> +	if (le16_to_cpu(data->t_id) != transaction) {
> +		dev_warn(ctrl->device,
> +			 "qid %d invalid transaction ID %d\n",
> +			 qid, le16_to_cpu(data->t_id));
> +		return NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
> +	}
> +
> +	return 0;
> +}
> +
> +static int nvme_auth_dhchap_negotiate(struct nvme_ctrl *ctrl,
> +				      struct nvme_dhchap_context *chap,
> +				      void *buf, size_t buf_size)

Maybe nvme_auth_set_dhchap_negotiate_data ?

> +{
> +	struct nvmf_auth_dhchap_negotiate_data *data = buf;
> +	size_t size = sizeof(*data) + sizeof(union nvmf_auth_protocol);
> +
> +	if (buf_size < size)
> +		return -EINVAL;
> +
> +	memset((u8 *)buf, 0, size);
> +	data->auth_type = NVME_AUTH_COMMON_MESSAGES;
> +	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE;
> +	data->t_id = cpu_to_le16(chap->transaction);
> +	data->sc_c = 0; /* No secure channel concatenation */
> +	data->napd = 1;
> +	data->auth_protocol[0].dhchap.authid = NVME_AUTH_DHCHAP_AUTH_ID;
> +	data->auth_protocol[0].dhchap.halen = 3;
> +	data->auth_protocol[0].dhchap.dhlen = 1;
> +	data->auth_protocol[0].dhchap.idlist[0] = NVME_AUTH_DHCHAP_HASH_SHA256;
> +	data->auth_protocol[0].dhchap.idlist[1] = NVME_AUTH_DHCHAP_HASH_SHA384;
> +	data->auth_protocol[0].dhchap.idlist[2] = NVME_AUTH_DHCHAP_HASH_SHA512;
> +	data->auth_protocol[0].dhchap.idlist[3] = NVME_AUTH_DHCHAP_DHGROUP_NULL;
You should comment that this routine expects buf to have enough
room for both negotiate and auth_proto structures.

> +
> +	return size;
> +}
> +
> +static int nvme_auth_dhchap_challenge(struct nvme_ctrl *ctrl,
> +				      struct nvme_dhchap_context *chap,
> +				      void *buf, size_t buf_size)

Maybe nvme_auth_process_dhchap_challange ?

> +{
> +	struct nvmf_auth_dhchap_challenge_data *data = buf;
> +	size_t size = sizeof(*data) + data->hl + data->dhvlen;
> +	const char *gid_name;
> +
> +	if (buf_size < size) {
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
> +		return -ENOMSG;
> +	}
> +
> +	if (data->hashid != NVME_AUTH_DHCHAP_HASH_SHA256 &&
> +	    data->hashid != NVME_AUTH_DHCHAP_HASH_SHA384 &&
> +	    data->hashid != NVME_AUTH_DHCHAP_HASH_SHA512) {
> +		dev_warn(ctrl->device,
> +			 "qid %d: DH-HMAC-CHAP: invalid HASH ID %d\n",
> +			 chap->qid, data->hashid);
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
> +		return -EPROTO;
> +	}
> +	switch (data->dhgid) {
> +	case NVME_AUTH_DHCHAP_DHGROUP_NULL:
> +		gid_name = "null";
> +		break;
> +	default:
> +		gid_name = NULL;
> +		break;
> +	}
> +	if (!gid_name) {
> +		dev_warn(ctrl->device,
> +			 "qid %d: DH-HMAC-CHAP: invalid DH group id %d\n",
> +			 chap->qid, data->dhgid);
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
> +		return -EPROTO;
> +	}

Maybe some spaces between condition blocks?

> +	if (data->dhgid != NVME_AUTH_DHCHAP_DHGROUP_NULL) {
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
> +		return -EPROTO;
> +	}
> +	if (data->dhgid == NVME_AUTH_DHCHAP_DHGROUP_NULL && data->dhvlen != 0) {
> +		dev_warn(ctrl->device,
> +			 "qid %d: DH-HMAC-CHAP: invalid DH value for NULL DH\n",
> +			chap->qid);
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
> +		return -EPROTO;
> +	}
> +	dev_dbg(ctrl->device, "%s: qid %d requested hash id %d\n",
> +		__func__, chap->qid, data->hashid);
> +	if (nvme_auth_hmac_len(data->hashid) != data->hl) {
> +		dev_warn(ctrl->device,
> +			 "qid %d: DH-HMAC-CHAP: invalid hash length\n",
> +			chap->qid);
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
> +		return -EPROTO;
> +	}
> +	chap->hash_id = data->hashid;
> +	chap->hash_len = data->hl;
> +	chap->s1 = le32_to_cpu(data->seqnum);
> +	memcpy(chap->c1, data->cval, chap->hash_len);
> +
> +	return 0;
> +}
> +
> +static int nvme_auth_dhchap_reply(struct nvme_ctrl *ctrl,
> +				  struct nvme_dhchap_context *chap,
> +				  void *buf, size_t buf_size)

nvme_auth_set_dhchap_reply

> +{
> +	struct nvmf_auth_dhchap_reply_data *data = buf;
> +	size_t size = sizeof(*data);
> +
> +	size += 2 * chap->hash_len;
> +	if (ctrl->opts->dhchap_auth) {

The ctrl opts is not clear to me. what is dhchap_auth
mean?

Also shouldn't these params be lifted to the subsys?

> +		get_random_bytes(chap->c2, chap->hash_len);
> +		chap->s2 = nvme_dhchap_seqnum++;
> +	} else
> +		memset(chap->c2, 0, chap->hash_len);
> +
> +	if (chap->host_key_len)
> +		size += chap->host_key_len;
> +
> +	if (buf_size < size)
> +		return -EINVAL;
> +
> +	memset(buf, 0, size);
> +	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
> +	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_REPLY;
> +	data->t_id = cpu_to_le16(chap->transaction);
> +	data->hl = chap->hash_len;
> +	data->dhvlen = chap->host_key_len;
> +	data->seqnum = cpu_to_le32(chap->s2);
> +	memcpy(data->rval, chap->response, chap->hash_len);
> +	if (ctrl->opts->dhchap_auth) {
> +		dev_dbg(ctrl->device, "%s: qid %d ctrl challenge %*ph\n",
> +			__func__, chap->qid,
> +			chap->hash_len, chap->c2);
> +		data->cvalid = 1;
> +		memcpy(data->rval + chap->hash_len, chap->c2,
> +		       chap->hash_len);
> +	}
> +	if (chap->host_key_len)
> +		memcpy(data->rval + 2 * chap->hash_len, chap->host_key,
> +		       chap->host_key_len);
> +
> +	return size;
> +}
> +
> +static int nvme_auth_dhchap_success1(struct nvme_ctrl *ctrl,
> +				     struct nvme_dhchap_context *chap,
> +				     void *buf, size_t buf_size)

nvme_auth_process_dhchap_success1

> +{
> +	struct nvmf_auth_dhchap_success1_data *data = buf;
> +	size_t size = sizeof(*data);
> +
> +	if (ctrl->opts->dhchap_auth)
> +		size += chap->hash_len;
> +
> +
> +	if (buf_size < size) {
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
> +		return -ENOMSG;
> +	}
> +
> +	if (data->hl != chap->hash_len) {
> +		dev_warn(ctrl->device,
> +			 "qid %d: DH-HMAC-CHAP: invalid hash length %d\n",
> +			 chap->qid, data->hl);
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
> +		return -EPROTO;
> +	}
> +
> +	if (!data->rvalid)
> +		return 0;
> +
> +	/* Validate controller response */
> +	if (memcmp(chap->response, data->rval, data->hl)) {
> +		dev_dbg(ctrl->device, "%s: qid %d ctrl response %*ph\n",
> +			__func__, chap->qid, chap->hash_len, data->rval);
> +		dev_dbg(ctrl->device, "%s: qid %d host response %*ph\n",
> +			__func__, chap->qid, chap->hash_len, chap->response);
> +		dev_warn(ctrl->device,
> +			 "qid %d: DH-HMAC-CHAP: controller authentication failed\n",
> +			 chap->qid);
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
> +		return -EPROTO;
> +	}
> +	dev_info(ctrl->device,
> +		 "qid %d: DH-HMAC-CHAP: controller authenticated\n",
> +		chap->qid);
> +	return 0;
> +}
> +
> +static int nvme_auth_dhchap_success2(struct nvme_ctrl *ctrl,
> +				     struct nvme_dhchap_context *chap,
> +				     void *buf, size_t buf_size)

same

> +{
> +	struct nvmf_auth_dhchap_success2_data *data = buf;
> +	size_t size = sizeof(*data);
> +
> +	memset(buf, 0, size);
> +	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
> +	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_SUCCESS2;
> +	data->t_id = cpu_to_le16(chap->transaction);
> +
> +	return size;
> +}
> +
> +static int nvme_auth_dhchap_failure2(struct nvme_ctrl *ctrl,
> +				     struct nvme_dhchap_context *chap,
> +				     void *buf, size_t buf_size)

same

> +{
> +	struct nvmf_auth_dhchap_failure_data *data = buf;
> +	size_t size = sizeof(*data);
> +
> +	memset(buf, 0, size);
> +	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
> +	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_FAILURE2;
> +	data->t_id = cpu_to_le16(chap->transaction);
> +	data->reason_code = 1;
> +	data->reason_code_explanation = chap->status;
> +
> +	return size;
> +}
> +
> +int nvme_auth_select_hash(struct nvme_ctrl *ctrl,
> +			  struct nvme_dhchap_context *chap)

Maybe _select_hf (hash function)? not a must, just sticks
to the spec language.

> +{
> +	char *hash_name;
> +	int ret;
> +
> +	switch (chap->hash_id) {
> +	case NVME_AUTH_DHCHAP_HASH_SHA256:
> +		hash_name = "hmac(sha256)";
> +		break;
> +	case NVME_AUTH_DHCHAP_HASH_SHA384:
> +		hash_name = "hmac(sha384)";
> +		break;
> +	case NVME_AUTH_DHCHAP_HASH_SHA512:
> +		hash_name = "hmac(sha512)";
> +		break;
> +	default:
> +		hash_name = NULL;
> +		break;
> +	}
> +	if (!hash_name) {
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
> +		return -EPROTO;
> +	}
> +	chap->shash_tfm = crypto_alloc_shash(hash_name, 0,
> +					     CRYPTO_ALG_ALLOCATES_MEMORY);
> +	if (IS_ERR(chap->shash_tfm)) {
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
> +		chap->shash_tfm = NULL;
> +		return -EPROTO;
> +	}
> +	if (!chap->key) {
> +		dev_warn(ctrl->device, "qid %d: cannot select hash, no key\n",
> +			 chap->qid);
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
> +		crypto_free_shash(chap->shash_tfm);

Wouldn't it better to check this before allocating the tfm?

> +		chap->shash_tfm = NULL;
> +		return -EINVAL;
> +	}
> +	ret = crypto_shash_setkey(chap->shash_tfm, chap->key, chap->key_len);
> +	if (ret) {
> +		chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
> +		crypto_free_shash(chap->shash_tfm);
> +		chap->shash_tfm = NULL;
> +		return ret;
> +	}
> +	dev_info(ctrl->device, "qid %d: DH-HMAC_CHAP: selected hash %s\n",
> +		 chap->qid, hash_name);
> +	return 0;
> +}
> +
> +static int nvme_auth_dhchap_host_response(struct nvme_ctrl *ctrl,
> +					  struct nvme_dhchap_context *chap)
> +{
> +	SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
> +	u8 buf[4], *challenge = chap->c1;
> +	int ret;
> +
> +	dev_dbg(ctrl->device, "%s: qid %d host response seq %d transaction %d\n",
> +		__func__, chap->qid, chap->s1, chap->transaction);
> +	shash->tfm = chap->shash_tfm;
> +	ret = crypto_shash_init(shash);
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, challenge, chap->hash_len);
> +	if (ret)
> +		goto out;
> +	put_unaligned_le32(chap->s1, buf);
> +	ret = crypto_shash_update(shash, buf, 4);
> +	if (ret)
> +		goto out;
> +	put_unaligned_le16(chap->transaction, buf);
> +	ret = crypto_shash_update(shash, buf, 2);
> +	if (ret)
> +		goto out;
> +	memset(buf, 0, sizeof(buf));
> +	ret = crypto_shash_update(shash, buf, 1);
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, "HostHost", 8);

HostHost ? Can you refer me to the specific section
that talks about this?

Would be good to have a comment on the format fed to the
shash.

> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
> +				  strlen(ctrl->opts->host->nqn));
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, buf, 1);
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
> +			    strlen(ctrl->opts->subsysnqn));
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_final(shash, chap->response);
> +out:
> +	return ret;
> +}
> +
> +static int nvme_auth_dhchap_ctrl_response(struct nvme_ctrl *ctrl,
> +					  struct nvme_dhchap_context *chap)
> +{
> +	SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
> +	u8 buf[4], *challenge = chap->c2;
> +	int ret;
> +
> +	dev_dbg(ctrl->device, "%s: qid %d host response seq %d transaction %d\n",
> +		__func__, chap->qid, chap->s2, chap->transaction);
> +	dev_dbg(ctrl->device, "%s: qid %d challenge %*ph\n",
> +		__func__, chap->qid, chap->hash_len, challenge);
> +	dev_dbg(ctrl->device, "%s: qid %d subsysnqn %s\n",
> +		__func__, chap->qid, ctrl->opts->subsysnqn);
> +	dev_dbg(ctrl->device, "%s: qid %d hostnqn %s\n",
> +		__func__, chap->qid, ctrl->opts->host->nqn);
> +	shash->tfm = chap->shash_tfm;
> +	ret = crypto_shash_init(shash);
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, challenge, chap->hash_len);
> +	if (ret)
> +		goto out;
> +	put_unaligned_le32(chap->s2, buf);
> +	ret = crypto_shash_update(shash, buf, 4);
> +	if (ret)
> +		goto out;
> +	put_unaligned_le16(chap->transaction, buf);
> +	ret = crypto_shash_update(shash, buf, 2);
> +	if (ret)
> +		goto out;
> +	memset(buf, 0, 4);
> +	ret = crypto_shash_update(shash, buf, 1);
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, "Controller", 10);
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
> +				  strlen(ctrl->opts->subsysnqn));
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, buf, 1);
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
> +				  strlen(ctrl->opts->host->nqn));
> +	if (ret)
> +		goto out;
> +	ret = crypto_shash_final(shash, chap->response);
> +out:
> +	return ret;
> +}
> +
> +int nvme_auth_generate_key(struct nvme_ctrl *ctrl,
> +			   struct nvme_dhchap_context *chap)
> +{
> +	int ret;
> +	u8 key_hash;
> +	const char *hmac_name;
> +	struct crypto_shash *key_tfm;
> +
> +	if (sscanf(ctrl->opts->dhchap_secret, "DHHC-1:%hhd:%*s:",
> +		   &key_hash) != 1)
> +		return -EINVAL;

I'd expect that the user will pass in a secret key (as binary)
the the driver will build the spec compliant formatted string no?

Am I not reading this correctly?

> +
> +	chap->key = nvme_auth_extract_secret(ctrl->opts->dhchap_secret,
> +					     &chap->key_len);
> +	if (IS_ERR(chap->key)) {
> +		ret = PTR_ERR(chap->key);
> +		chap->key = NULL;
> +		return ret;
> +	}
> +
> +	if (key_hash == 0)
> +		return 0;
> +
> +	hmac_name = nvme_auth_hmac_name(key_hash);
> +	if (!hmac_name) {
> +		pr_debug("Invalid key hash id %d\n", key_hash);
> +		return -EKEYREJECTED;
> +	}

Why does the user influence the hmac used? isn't that is driven
by the susbsystem?

I don't think that the user should choose in this level.

> +
> +	key_tfm = crypto_alloc_shash(hmac_name, 0, 0);
> +	if (IS_ERR(key_tfm)) {
> +		kfree(chap->key);
> +		chap->key = NULL;
> +		ret = PTR_ERR(key_tfm);

You set ret and later return 0? I think that the success
path in the else clause is hard to read and error prone...

> +	} else {
> +		SHASH_DESC_ON_STACK(shash, key_tfm);
> +
> +		shash->tfm = key_tfm;
> +		ret = crypto_shash_setkey(key_tfm, chap->key,
> +					  chap->key_len);
> +		if (ret < 0) {
> +			crypto_free_shash(key_tfm);
> +			kfree(chap->key);
> +			chap->key = NULL;
> +			return ret;
> +		}
> +		crypto_shash_init(shash);
> +		crypto_shash_update(shash, ctrl->opts->host->nqn,
> +				    strlen(ctrl->opts->host->nqn));
> +		crypto_shash_update(shash, "NVMe-over-Fabrics", 17);
> +		crypto_shash_final(shash, chap->key);
> +		crypto_free_shash(key_tfm);

Shouldn't these be done when preparing the dh-hmac-chap reply?

> +	}
> +	return 0;
> +}
> +
> +void nvme_auth_free(struct nvme_dhchap_context *chap)
> +{
> +	if (chap->shash_tfm)
> +		crypto_free_shash(chap->shash_tfm);
> +	if (chap->key)
> +		kfree(chap->key);
> +	if (chap->ctrl_key)
> +		kfree(chap->ctrl_key);
> +	if (chap->host_key)
> +		kfree(chap->host_key);
> +	if (chap->sess_key)
> +		kfree(chap->sess_key);

No need to check null for kfree...

> +	kfree(chap);
> +}
> +
> +int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid)
> +{
> +	struct nvme_dhchap_context *chap;
> +	void *buf;
> +	size_t buf_size, tl;
> +	int ret = 0;
> +
> +	chap = kzalloc(sizeof(*chap), GFP_KERNEL);
> +	if (!chap)
> +		return -ENOMEM;
> +	chap->qid = qid;
> +	chap->transaction = ctrl->transaction++;
> +
> +	ret = nvme_auth_generate_key(ctrl, chap);
> +	if (ret) {
> +		dev_dbg(ctrl->device, "%s: failed to generate key, error %d\n",
> +			__func__, ret);
> +		nvme_auth_free(chap);
> +		return ret;
> +	}
> +
> +	/*
> +	 * Allocate a large enough buffer for the entire negotiation:
> +	 * 4k should be enough to ffdhe8192.
> +	 */
> +	buf_size = 4096;
> +	buf = kzalloc(buf_size, GFP_KERNEL);
> +	if (!buf) {
> +		ret = -ENOMEM;
> +		goto out;
> +	}
> +
> +	/* DH-HMAC-CHAP Step 1: send negotiate */

I'd consider breaking these into sub-routines.

> +	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP negotiate\n",
> +		__func__, qid);
> +	ret = nvme_auth_dhchap_negotiate(ctrl, chap, buf, buf_size);
> +	if (ret < 0)
> +		goto out;
> +	tl = ret;
> +	ret = nvme_auth_send(ctrl, qid, buf, tl);
> +	if (ret)
> +		goto out;
> +
> +	memset(buf, 0, buf_size);
> +	ret = nvme_auth_receive(ctrl, qid, buf, buf_size, chap->transaction,
> +				NVME_AUTH_DHCHAP_MESSAGE_CHALLENGE);
> +	if (ret < 0) {
> +		dev_dbg(ctrl->device,
> +			"%s: qid %d DH-HMAC-CHAP failed to receive challenge\n",
> +			__func__, qid);
> +		goto out;
> +	}
> +	if (ret > 0) {
> +		chap->status = ret;
> +		goto fail1;
> +	}
> +
> +	/* DH-HMAC-CHAP Step 2: receive challenge */
> +	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP challenge\n",
> +		__func__, qid);
> +
> +	ret = nvme_auth_dhchap_challenge(ctrl, chap, buf, buf_size);
> +	if (ret) {
> +		/* Invalid parameters for negotiate */
> +		goto fail2;
> +	}
> +
> +	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP select hash\n",
> +		__func__, qid);
> +	ret = nvme_auth_select_hash(ctrl, chap);
> +	if (ret)
> +		goto fail2;
> +
> +	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP host response\n",
> +		__func__, qid);
> +	ret = nvme_auth_dhchap_host_response(ctrl, chap);
> +	if (ret)
> +		goto fail2;
> +
> +	/* DH-HMAC-CHAP Step 3: send reply */
> +	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP reply\n",
> +		__func__, qid);
> +	ret = nvme_auth_dhchap_reply(ctrl, chap, buf, buf_size);
> +	if (ret < 0)
> +		goto fail2;
> +
> +	tl = ret;
> +	ret = nvme_auth_send(ctrl, qid, buf, tl);
> +	if (ret)
> +		goto fail2;
> +
> +	memset(buf, 0, buf_size);
> +	ret = nvme_auth_receive(ctrl, qid, buf, buf_size, chap->transaction,
> +				NVME_AUTH_DHCHAP_MESSAGE_SUCCESS1);
> +	if (ret < 0) {
> +		dev_dbg(ctrl->device,
> +			"%s: qid %d DH-HMAC-CHAP failed to receive success1\n",
> +			__func__, qid);
> +		goto out;
> +	}
> +	if (ret > 0) {
> +		chap->status = ret;
> +		goto fail1;
> +	}
> +
> +	if (ctrl->opts->dhchap_auth) {
> +		dev_dbg(ctrl->device,
> +			"%s: qid %d DH-HMAC-CHAP controller response\n",
> +			__func__, qid);
> +		ret = nvme_auth_dhchap_ctrl_response(ctrl, chap);
> +		if (ret)
> +			goto fail2;
> +	}
> +
> +	/* DH-HMAC-CHAP Step 4: receive success1 */
> +	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP success1\n",
> +		__func__, qid);
> +	ret = nvme_auth_dhchap_success1(ctrl, chap, buf, buf_size);
> +	if (ret < 0) {
> +		/* Controller authentication failed */
> +		goto fail2;
> +	}
> +	tl = ret;
> +	/* DH-HMAC-CHAP Step 5: send success2 */
> +	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP success2\n",
> +		__func__, qid);
> +	tl = nvme_auth_dhchap_success2(ctrl, chap, buf, buf_size);
> +	ret = nvme_auth_send(ctrl, qid, buf, tl);
> +	if (!ret)
> +		goto out;
> +
> +fail1:
> +	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP failure1, status %x\n",
> +		__func__, qid, chap->status);
> +	goto out;
> +
> +fail2:
> +	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP failure2, status %x\n",
> +		__func__, qid, chap->status);
> +	tl = nvme_auth_dhchap_failure2(ctrl, chap, buf, buf_size);
> +	ret = nvme_auth_send(ctrl, qid, buf, tl);
> +
> +out:
> +	if (!ret && chap->status)
> +		ret = -EPROTO;
> +	if (!ret) {
> +		ctrl->dhchap_hash = chap->hash_id;
> +	}
> +	kfree(buf);
> +	nvme_auth_free(chap);
> +	return ret;
> +}
> diff --git a/drivers/nvme/host/auth.h b/drivers/nvme/host/auth.h
> new file mode 100644
> index 000000000000..4950b1cb9470
> --- /dev/null
> +++ b/drivers/nvme/host/auth.h
> @@ -0,0 +1,23 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Copyright (c) 2021 Hannes Reinecke, SUSE Software Solutions
> + */
> +
> +#ifndef _NVME_AUTH_H
> +#define _NVME_AUTH_H
> +
> +const char *nvme_auth_dhgroup_name(int dhgroup_id);
> +int nvme_auth_dhgroup_pubkey_size(int dhgroup_id);
> +int nvme_auth_dhgroup_privkey_size(int dhgroup_id);
> +const char *nvme_auth_dhgroup_kpp(int dhgroup_id);
> +int nvme_auth_dhgroup_id(const char *dhgroup_name);
> +
> +const char *nvme_auth_hmac_name(int hmac_id);
> +const char *nvme_auth_digest_name(int hmac_id);
> +int nvme_auth_hmac_id(const char *hmac_name);
> +int nvme_auth_hmac_len(int hmac_len);
> +
> +unsigned char *nvme_auth_extract_secret(unsigned char *dhchap_secret,
> +					size_t *dhchap_key_len);
> +
> +#endif /* _NVME_AUTH_H */
> diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
> index 11779be42186..7ce9b666dc09 100644
> --- a/drivers/nvme/host/core.c
> +++ b/drivers/nvme/host/core.c
> @@ -708,7 +708,9 @@ bool __nvme_check_ready(struct nvme_ctrl *ctrl, struct request *rq,
>   		switch (ctrl->state) {
>   		case NVME_CTRL_CONNECTING:
>   			if (blk_rq_is_passthrough(rq) && nvme_is_fabrics(req->cmd) &&
> -			    req->cmd->fabrics.fctype == nvme_fabrics_type_connect)
> +			    (req->cmd->fabrics.fctype == nvme_fabrics_type_connect ||
> +			     req->cmd->fabrics.fctype == nvme_fabrics_type_auth_send ||
> +			     req->cmd->fabrics.fctype == nvme_fabrics_type_auth_receive))
>   				return true;
>   			break;
>   		default:
> @@ -3426,6 +3428,66 @@ static ssize_t nvme_ctrl_fast_io_fail_tmo_store(struct device *dev,
>   static DEVICE_ATTR(fast_io_fail_tmo, S_IRUGO | S_IWUSR,
>   	nvme_ctrl_fast_io_fail_tmo_show, nvme_ctrl_fast_io_fail_tmo_store);
>   
> +#ifdef CONFIG_NVME_AUTH
> +struct nvmet_dhchap_hash_map {
> +	int id;
> +	const char name[15];
> +} hash_map[] = {
> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA256,
> +	 .name = "hmac(sha256)", },
> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA384,
> +	 .name = "hmac(sha384)", },
> +	{.id = NVME_AUTH_DHCHAP_HASH_SHA512,
> +	 .name = "hmac(sha512)", },
> +};
> +
> +static ssize_t dhchap_hash_show(struct device *dev,
> +	struct device_attribute *attr, char *buf)
> +{
> +	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
> +	int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
> +		if (hash_map[i].id == ctrl->dhchap_hash)
> +			return sprintf(buf, "%s\n", hash_map[i].name);
> +	}
> +	return sprintf(buf, "none\n");
> +}
> +DEVICE_ATTR_RO(dhchap_hash);
> +
> +struct nvmet_dhchap_group_map {
> +	int id;
> +	const char name[15];
> +} dhgroup_map[] = {
> +	{.id = NVME_AUTH_DHCHAP_DHGROUP_NULL,
> +	 .name = "NULL", },
> +	{.id = NVME_AUTH_DHCHAP_DHGROUP_2048,
> +	 .name = "ffdhe2048", },
> +	{.id = NVME_AUTH_DHCHAP_DHGROUP_3072,
> +	 .name = "ffdhe3072", },
> +	{.id = NVME_AUTH_DHCHAP_DHGROUP_4096,
> +	 .name = "ffdhe4096", },
> +	{.id = NVME_AUTH_DHCHAP_DHGROUP_6144,
> +	 .name = "ffdhe6144", },
> +	{.id = NVME_AUTH_DHCHAP_DHGROUP_8192,
> +	 .name = "ffdhe8192", },
> +};
> +
> +static ssize_t dhchap_dhgroup_show(struct device *dev,
> +	struct device_attribute *attr, char *buf)
> +{
> +	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
> +	int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(dhgroup_map); i++) {
> +		if (hash_map[i].id == ctrl->dhchap_dhgroup)
> +			return sprintf(buf, "%s\n", dhgroup_map[i].name);
> +	}
> +	return sprintf(buf, "none\n");
> +}
> +DEVICE_ATTR_RO(dhchap_dhgroup);
> +#endif
> +
>   static struct attribute *nvme_dev_attrs[] = {
>   	&dev_attr_reset_controller.attr,
>   	&dev_attr_rescan_controller.attr,
> @@ -3447,6 +3509,10 @@ static struct attribute *nvme_dev_attrs[] = {
>   	&dev_attr_reconnect_delay.attr,
>   	&dev_attr_fast_io_fail_tmo.attr,
>   	&dev_attr_kato.attr,
> +#ifdef CONFIG_NVME_AUTH
> +	&dev_attr_dhchap_hash.attr,
> +	&dev_attr_dhchap_dhgroup.attr,
> +#endif
>   	NULL
>   };
>   
> @@ -3470,6 +3536,10 @@ static umode_t nvme_dev_attrs_are_visible(struct kobject *kobj,
>   		return 0;
>   	if (a == &dev_attr_fast_io_fail_tmo.attr && !ctrl->opts)
>   		return 0;
> +#ifdef CONFIG_NVME_AUTH
> +	if (a == &dev_attr_dhchap_hash.attr && !ctrl->opts)
> +		return 0;
> +#endif
>   
>   	return a->mode;
>   }
> @@ -4581,6 +4651,11 @@ static inline void _nvme_check_size(void)
>   	BUILD_BUG_ON(sizeof(struct nvme_smart_log) != 512);
>   	BUILD_BUG_ON(sizeof(struct nvme_dbbuf) != 64);
>   	BUILD_BUG_ON(sizeof(struct nvme_directive_cmd) != 64);
> +	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_negotiate_data) != 8);
> +	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_challenge_data) != 16);
> +	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_reply_data) != 16);
> +	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_success1_data) != 16);
> +	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_success2_data) != 16);
>   }
>   
>   
> diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
> index a5469fd9d4c3..6404ab9b604b 100644
> --- a/drivers/nvme/host/fabrics.c
> +++ b/drivers/nvme/host/fabrics.c
> @@ -366,6 +366,7 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)
>   	union nvme_result res;
>   	struct nvmf_connect_data *data;
>   	int ret;
> +	u32 result;
>   
>   	cmd.connect.opcode = nvme_fabrics_command;
>   	cmd.connect.fctype = nvme_fabrics_type_connect;
> @@ -398,8 +399,18 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)
>   		goto out_free_data;
>   	}
>   
> -	ctrl->cntlid = le16_to_cpu(res.u16);
> -
> +	result = le32_to_cpu(res.u32);
> +	ctrl->cntlid = result & 0xFFFF;
> +	if ((result >> 16) & 2) {
> +		/* Authentication required */
> +		ret = nvme_auth_negotiate(ctrl, NVME_QID_ANY);
> +		if (ret)
> +			dev_warn(ctrl->device,
> +				 "qid 0: authentication failed\n");
> +		else
> +			dev_info(ctrl->device,
> +				 "qid 0: authenticated\n");

info is too chatty.

> +	}
>   out_free_data:
>   	kfree(data);
>   	return ret;
> @@ -432,6 +443,7 @@ int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, u16 qid)
>   	struct nvmf_connect_data *data;
>   	union nvme_result res;
>   	int ret;
> +	u32 result;
>   
>   	cmd.connect.opcode = nvme_fabrics_command;
>   	cmd.connect.fctype = nvme_fabrics_type_connect;
> @@ -457,6 +469,17 @@ int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, u16 qid)
>   		nvmf_log_connect_error(ctrl, ret, le32_to_cpu(res.u32),
>   				       &cmd, data);
>   	}
> +	result = le32_to_cpu(res.u32);
> +	if ((result >> 16) & 2) {
> +		/* Authentication required */
> +		ret = nvme_auth_negotiate(ctrl, qid);
> +		if (ret)
> +			dev_warn(ctrl->device,
> +				 "qid %u: authentication failed\n", qid);
> +		else
> +			dev_info(ctrl->device,
> +				 "qid %u: authenticated\n", qid);
> +	}
>   	kfree(data);
>   	return ret;
>   }
> @@ -548,6 +571,9 @@ static const match_table_t opt_tokens = {
>   	{ NVMF_OPT_NR_POLL_QUEUES,	"nr_poll_queues=%d"	},
>   	{ NVMF_OPT_TOS,			"tos=%d"		},
>   	{ NVMF_OPT_FAIL_FAST_TMO,	"fast_io_fail_tmo=%d"	},
> +	{ NVMF_OPT_DHCHAP_SECRET,	"dhchap_secret=%s"	},
> +	{ NVMF_OPT_DHCHAP_AUTH,		"authenticate"		},
> +	{ NVMF_OPT_DHCHAP_GROUP,	"dhchap_group=%s"	},

Isn't the group driven by the subsystem? also why is there a
"authenticate" boolean? what is it good for?

>   	{ NVMF_OPT_ERR,			NULL			}
>   };
>   
> @@ -824,6 +850,35 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
>   			}
>   			opts->tos = token;
>   			break;
> +		case NVMF_OPT_DHCHAP_SECRET:
> +			p = match_strdup(args);
> +			if (!p) {
> +				ret = -ENOMEM;
> +				goto out;
> +			}
> +			if (strncmp(p, "DHHC-1:00:", 10)) {
> +				pr_err("Invalid DH-CHAP secret %s\n", p);
> +				ret = -EINVAL;
> +				goto out;
> +			}
> +			kfree(opts->dhchap_secret);
> +			opts->dhchap_secret = p;
> +			break;
> +		case NVMF_OPT_DHCHAP_AUTH:
> +			opts->dhchap_auth = true;
> +			break;
> +		case NVMF_OPT_DHCHAP_GROUP:
> +			if (match_int(args, &token)) {
> +				ret = -EINVAL;
> +				goto out;
> +			}
> +			if (token <= 0) {
> +				pr_err("Invalid dhchap_group %d\n", token);
> +				ret = -EINVAL;
> +				goto out;
> +			}
> +			opts->dhchap_group = token;
> +			break;
>   		default:
>   			pr_warn("unknown parameter or missing value '%s' in ctrl creation request\n",
>   				p);
> @@ -942,6 +997,7 @@ void nvmf_free_options(struct nvmf_ctrl_options *opts)
>   	kfree(opts->subsysnqn);
>   	kfree(opts->host_traddr);
>   	kfree(opts->host_iface);
> +	kfree(opts->dhchap_secret);
>   	kfree(opts);
>   }
>   EXPORT_SYMBOL_GPL(nvmf_free_options);
> @@ -951,7 +1007,10 @@ EXPORT_SYMBOL_GPL(nvmf_free_options);
>   				 NVMF_OPT_KATO | NVMF_OPT_HOSTNQN | \
>   				 NVMF_OPT_HOST_ID | NVMF_OPT_DUP_CONNECT |\
>   				 NVMF_OPT_DISABLE_SQFLOW |\
> -				 NVMF_OPT_FAIL_FAST_TMO)
> +				 NVMF_OPT_CTRL_LOSS_TMO |\
> +				 NVMF_OPT_FAIL_FAST_TMO |\
> +				 NVMF_OPT_DHCHAP_SECRET |\
> +				 NVMF_OPT_DHCHAP_AUTH | NVMF_OPT_DHCHAP_GROUP)
>   
>   static struct nvme_ctrl *
>   nvmf_create_ctrl(struct device *dev, const char *buf)
> diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h
> index a146cb903869..535bc544f0f6 100644
> --- a/drivers/nvme/host/fabrics.h
> +++ b/drivers/nvme/host/fabrics.h
> @@ -67,6 +67,9 @@ enum {
>   	NVMF_OPT_TOS		= 1 << 19,
>   	NVMF_OPT_FAIL_FAST_TMO	= 1 << 20,
>   	NVMF_OPT_HOST_IFACE	= 1 << 21,
> +	NVMF_OPT_DHCHAP_SECRET	= 1 << 22,
> +	NVMF_OPT_DHCHAP_AUTH	= 1 << 23,
> +	NVMF_OPT_DHCHAP_GROUP	= 1 << 24,
>   };
>   
>   /**
> @@ -96,6 +99,8 @@ enum {
>    * @max_reconnects: maximum number of allowed reconnect attempts before removing
>    *              the controller, (-1) means reconnect forever, zero means remove
>    *              immediately;
> + * @dhchap_secret: DH-HMAC-CHAP secret
> + * @dhchap_auth: DH-HMAC-CHAP authenticate controller
>    * @disable_sqflow: disable controller sq flow control
>    * @hdr_digest: generate/verify header digest (TCP)
>    * @data_digest: generate/verify data digest (TCP)
> @@ -120,6 +125,9 @@ struct nvmf_ctrl_options {
>   	unsigned int		kato;
>   	struct nvmf_host	*host;
>   	int			max_reconnects;
> +	char			*dhchap_secret;
> +	int			dhchap_group;
> +	bool			dhchap_auth;
>   	bool			disable_sqflow;
>   	bool			hdr_digest;
>   	bool			data_digest;
> diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
> index 18ef8dd03a90..bcd5b8276c26 100644
> --- a/drivers/nvme/host/nvme.h
> +++ b/drivers/nvme/host/nvme.h
> @@ -328,6 +328,12 @@ struct nvme_ctrl {
>   	struct work_struct ana_work;
>   #endif
>   
> +#ifdef CONFIG_NVME_AUTH
> +	u16 transaction;
> +	u8 dhchap_hash;
> +	u8 dhchap_dhgroup;

Do multiple controllers in the same subsystem have different
params? no, so I think these should be lifted to subsys.

> +#endif
> +
>   	/* Power saving configuration */
>   	u64 ps_max_latency_us;
>   	bool apst_enabled;
> @@ -874,6 +880,15 @@ static inline bool nvme_ctrl_sgl_supported(struct nvme_ctrl *ctrl)
>   	return ctrl->sgls & ((1 << 0) | (1 << 1));
>   }
>   
> +#ifdef CONFIG_NVME_AUTH
> +int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid);
> +#else
> +static inline int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid)
> +{
> +	return -EPROTONOSUPPORT;
> +}
> +#endif
> +
>   u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
>   			 u8 opcode);
>   int nvme_execute_passthru_rq(struct request *rq);
> diff --git a/drivers/nvme/host/trace.c b/drivers/nvme/host/trace.c
> index 6543015b6121..66f75d8ea925 100644
> --- a/drivers/nvme/host/trace.c
> +++ b/drivers/nvme/host/trace.c

I'd split out the tracing logic.

> @@ -271,6 +271,34 @@ static const char *nvme_trace_fabrics_property_get(struct trace_seq *p, u8 *spc)
>   	return ret;
>   }
>   
> +static const char *nvme_trace_fabrics_auth_send(struct trace_seq *p, u8 *spc)
> +{
> +	const char *ret = trace_seq_buffer_ptr(p);
> +	u8 spsp0 = spc[1];
> +	u8 spsp1 = spc[2];
> +	u8 secp = spc[3];
> +	u32 tl = get_unaligned_le32(spc + 4);
> +
> +	trace_seq_printf(p, "spsp0=%02x, spsp1=%02x, secp=%02x, tl=%u",
> +			 spsp0, spsp1, secp, tl);
> +	trace_seq_putc(p, 0);
> +	return ret;
> +}
> +
> +static const char *nvme_trace_fabrics_auth_receive(struct trace_seq *p, u8 *spc)
> +{
> +	const char *ret = trace_seq_buffer_ptr(p);
> +	u8 spsp0 = spc[1];
> +	u8 spsp1 = spc[2];
> +	u8 secp = spc[3];
> +	u32 al = get_unaligned_le32(spc + 4);
> +
> +	trace_seq_printf(p, "spsp0=%02x, spsp1=%02x, secp=%02x, al=%u",
> +			 spsp0, spsp1, secp, al);
> +	trace_seq_putc(p, 0);
> +	return ret;
> +}
> +
>   static const char *nvme_trace_fabrics_common(struct trace_seq *p, u8 *spc)
>   {
>   	const char *ret = trace_seq_buffer_ptr(p);
> @@ -290,6 +318,10 @@ const char *nvme_trace_parse_fabrics_cmd(struct trace_seq *p,
>   		return nvme_trace_fabrics_connect(p, spc);
>   	case nvme_fabrics_type_property_get:
>   		return nvme_trace_fabrics_property_get(p, spc);
> +	case nvme_fabrics_type_auth_send:
> +		return nvme_trace_fabrics_auth_send(p, spc);
> +	case nvme_fabrics_type_auth_receive:
> +		return nvme_trace_fabrics_auth_receive(p, spc);
>   	default:
>   		return nvme_trace_fabrics_common(p, spc);
>   	}
> 

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

[PATCH 06/11] nvme: Implement In-Band authentication

[Hannes Reinecke]

Implement NVMe-oF In-Band authentication. This patch adds two new
fabric options 'dhchap_key' to specify the PSK and 'dhchap_authenticate'
to request bi-directional authentication of both the host and the controller.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/nvme/host/Kconfig   |  11 +
 drivers/nvme/host/Makefile  |   1 +
 drivers/nvme/host/auth.c    | 813 ++++++++++++++++++++++++++++++++++++
 drivers/nvme/host/auth.h    |  23 +
 drivers/nvme/host/core.c    |  77 +++-
 drivers/nvme/host/fabrics.c |  65 ++-
 drivers/nvme/host/fabrics.h |   8 +
 drivers/nvme/host/nvme.h    |  15 +
 drivers/nvme/host/trace.c   |  32 ++
 9 files changed, 1041 insertions(+), 4 deletions(-)
 create mode 100644 drivers/nvme/host/auth.c
 create mode 100644 drivers/nvme/host/auth.h

diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig
index c3f3d77f1aac..853c546305e9 100644
--- a/drivers/nvme/host/Kconfig
+++ b/drivers/nvme/host/Kconfig
@@ -85,3 +85,14 @@ config NVME_TCP
 	  from https://github.com/linux-nvme/nvme-cli.
 
 	  If unsure, say N.
+
+config NVME_AUTH
+	bool "NVM Express over Fabrics In-Band Authentication"
+	depends on NVME_TCP
+	select CRYPTO_SHA256
+	select CRYPTO_SHA512
+	help
+	  This provides support for NVMe over Fabrics In-Band Authentication
+	  for the NVMe over TCP transport.
+
+	  If unsure, say N.
diff --git a/drivers/nvme/host/Makefile b/drivers/nvme/host/Makefile
index cbc509784b2e..03748a55a12b 100644
--- a/drivers/nvme/host/Makefile
+++ b/drivers/nvme/host/Makefile
@@ -20,6 +20,7 @@ nvme-core-$(CONFIG_NVME_HWMON)		+= hwmon.o
 nvme-y					+= pci.o
 
 nvme-fabrics-y				+= fabrics.o
+nvme-fabrics-$(CONFIG_NVME_AUTH)	+= auth.o
 
 nvme-rdma-y				+= rdma.o
 
diff --git a/drivers/nvme/host/auth.c b/drivers/nvme/host/auth.c
new file mode 100644
index 000000000000..448a3adebea6
--- /dev/null
+++ b/drivers/nvme/host/auth.c
@@ -0,0 +1,813 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2020 Hannes Reinecke, SUSE Linux
+ */
+
+#include <linux/crc32.h>
+#include <linux/base64.h>
+#include <asm/unaligned.h>
+#include <crypto/hash.h>
+#include <crypto/kpp.h>
+#include "nvme.h"
+#include "fabrics.h"
+#include "auth.h"
+
+static u32 nvme_dhchap_seqnum;
+
+struct nvme_dhchap_context {
+	struct crypto_shash *shash_tfm;
+	unsigned char *key;
+	size_t key_len;
+	int qid;
+	u32 s1;
+	u32 s2;
+	u16 transaction;
+	u8 status;
+	u8 hash_id;
+	u8 hash_len;
+	u8 c1[64];
+	u8 c2[64];
+	u8 response[64];
+	u8 *ctrl_key;
+	int ctrl_key_len;
+	u8 *host_key;
+	int host_key_len;
+	u8 *sess_key;
+	int sess_key_len;
+};
+
+struct nvmet_dhchap_hash_map {
+	int id;
+	int hash_len;
+	const char hmac[15];
+	const char digest[15];
+} hash_map[] = {
+	{.id = NVME_AUTH_DHCHAP_HASH_SHA256,
+	 .hash_len = 32,
+	 .hmac = "hmac(sha256)", .digest = "sha256" },
+	{.id = NVME_AUTH_DHCHAP_HASH_SHA384,
+	 .hash_len = 48,
+	 .hmac = "hmac(sha384)", .digest = "sha384" },
+	{.id = NVME_AUTH_DHCHAP_HASH_SHA512,
+	 .hash_len = 64,
+	 .hmac = "hmac(sha512)", .digest = "sha512" },
+};
+
+const char *nvme_auth_hmac_name(int hmac_id)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
+		if (hash_map[i].id == hmac_id)
+			return hash_map[i].hmac;
+	}
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_hmac_name);
+
+const char *nvme_auth_digest_name(int hmac_id)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
+		if (hash_map[i].id == hmac_id)
+			return hash_map[i].digest;
+	}
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_digest_name);
+
+int nvme_auth_hmac_len(int hmac_id)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
+		if (hash_map[i].id == hmac_id)
+			return hash_map[i].hash_len;
+	}
+	return -1;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_hmac_len);
+
+int nvme_auth_hmac_id(const char *hmac_name)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
+		if (!strncmp(hash_map[i].hmac, hmac_name,
+			     strlen(hash_map[i].hmac)))
+			return hash_map[i].id;
+	}
+	return -1;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_hmac_id);
+
+unsigned char *nvme_auth_extract_secret(unsigned char *dhchap_secret,
+					size_t *dhchap_key_len)
+{
+	unsigned char *dhchap_key;
+	u32 crc;
+	int key_len;
+	size_t allocated_len;
+
+	allocated_len = strlen(dhchap_secret) - 10;
+	dhchap_key = kzalloc(allocated_len, GFP_KERNEL);
+	if (!dhchap_key)
+		return ERR_PTR(-ENOMEM);
+
+	key_len = base64_decode(dhchap_secret + 10,
+				allocated_len, dhchap_key);
+	if (key_len != 36 && key_len != 52 &&
+	    key_len != 68) {
+		pr_debug("Invalid DH-HMAC-CHAP key len %d\n",
+			 key_len);
+		kfree(dhchap_key);
+		return ERR_PTR(-EINVAL);
+	}
+	pr_debug("DH-HMAC-CHAP Key: %*ph\n",
+		 (int)key_len, dhchap_key);
+
+	/* The last four bytes is the CRC in little-endian format */
+	key_len -= 4;
+	/*
+	 * The linux implementation doesn't do pre- and post-increments,
+	 * so we have to do it manually.
+	 */
+	crc = ~crc32(~0, dhchap_key, key_len);
+
+	if (get_unaligned_le32(dhchap_key + key_len) != crc) {
+		pr_debug("DH-HMAC-CHAP crc mismatch (key %08x, crc %08x)\n",
+		       get_unaligned_le32(dhchap_key + key_len), crc);
+		kfree(dhchap_key);
+		return ERR_PTR(-EKEYREJECTED);
+	}
+	*dhchap_key_len = key_len;
+	return dhchap_key;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_extract_secret);
+
+static int nvme_auth_send(struct nvme_ctrl *ctrl, int qid,
+			  void *data, size_t tl)
+{
+	struct nvme_command cmd = {};
+	blk_mq_req_flags_t flags = qid == NVME_QID_ANY ?
+		0 : BLK_MQ_REQ_NOWAIT | BLK_MQ_REQ_RESERVED;
+	struct request_queue *q = qid == NVME_QID_ANY ?
+		ctrl->fabrics_q : ctrl->connect_q;
+	int ret;
+
+	cmd.auth_send.opcode = nvme_fabrics_command;
+	cmd.auth_send.fctype = nvme_fabrics_type_auth_send;
+	cmd.auth_send.secp = NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER;
+	cmd.auth_send.spsp0 = 0x01;
+	cmd.auth_send.spsp1 = 0x01;
+	cmd.auth_send.tl = tl;
+
+	ret = __nvme_submit_sync_cmd(q, &cmd, NULL, data, tl, 0, qid,
+				     0, flags);
+	if (ret)
+		dev_dbg(ctrl->device,
+			"%s: qid %d error %d\n", __func__, qid, ret);
+	return ret;
+}
+
+static int nvme_auth_receive(struct nvme_ctrl *ctrl, int qid,
+			     void *buf, size_t al,
+			     u16 transaction, u8 expected_msg )
+{
+	struct nvme_command cmd = {};
+	struct nvmf_auth_dhchap_failure_data *data = buf;
+	blk_mq_req_flags_t flags = qid == NVME_QID_ANY ?
+		0 : BLK_MQ_REQ_NOWAIT | BLK_MQ_REQ_RESERVED;
+	struct request_queue *q = qid == NVME_QID_ANY ?
+		ctrl->fabrics_q : ctrl->connect_q;
+	int ret;
+
+	cmd.auth_receive.opcode = nvme_fabrics_command;
+	cmd.auth_receive.fctype = nvme_fabrics_type_auth_receive;
+	cmd.auth_receive.secp = NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER;
+	cmd.auth_receive.spsp0 = 0x01;
+	cmd.auth_receive.spsp1 = 0x01;
+	cmd.auth_receive.al = al;
+
+	ret = __nvme_submit_sync_cmd(q, &cmd, NULL, buf, al, 0, qid,
+				     0, flags);
+	if (ret > 0) {
+		dev_dbg(ctrl->device, "%s: qid %d nvme status %x\n",
+			__func__, qid, ret);
+		ret = -EIO;
+	}
+	if (ret < 0) {
+		dev_dbg(ctrl->device, "%s: qid %d error %d\n",
+			__func__, qid, ret);
+		return ret;
+	}
+	dev_dbg(ctrl->device, "%s: qid %d auth_type %d auth_id %x\n",
+		__func__, qid, data->auth_type, data->auth_id);
+	if (data->auth_type == NVME_AUTH_COMMON_MESSAGES &&
+	    data->auth_id == NVME_AUTH_DHCHAP_MESSAGE_FAILURE1) {
+		return data->reason_code_explanation;
+	}
+	if (data->auth_type != NVME_AUTH_DHCHAP_MESSAGES ||
+	    data->auth_id != expected_msg) {
+		dev_warn(ctrl->device,
+			 "qid %d invalid message %02x/%02x\n",
+			 qid, data->auth_type, data->auth_id);
+		return NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
+	}
+	if (le16_to_cpu(data->t_id) != transaction) {
+		dev_warn(ctrl->device,
+			 "qid %d invalid transaction ID %d\n",
+			 qid, le16_to_cpu(data->t_id));
+		return NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
+	}
+
+	return 0;
+}
+
+static int nvme_auth_dhchap_negotiate(struct nvme_ctrl *ctrl,
+				      struct nvme_dhchap_context *chap,
+				      void *buf, size_t buf_size)
+{
+	struct nvmf_auth_dhchap_negotiate_data *data = buf;
+	size_t size = sizeof(*data) + sizeof(union nvmf_auth_protocol);
+
+	if (buf_size < size)
+		return -EINVAL;
+
+	memset((u8 *)buf, 0, size);
+	data->auth_type = NVME_AUTH_COMMON_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE;
+	data->t_id = cpu_to_le16(chap->transaction);
+	data->sc_c = 0; /* No secure channel concatenation */
+	data->napd = 1;
+	data->auth_protocol[0].dhchap.authid = NVME_AUTH_DHCHAP_AUTH_ID;
+	data->auth_protocol[0].dhchap.halen = 3;
+	data->auth_protocol[0].dhchap.dhlen = 1;
+	data->auth_protocol[0].dhchap.idlist[0] = NVME_AUTH_DHCHAP_HASH_SHA256;
+	data->auth_protocol[0].dhchap.idlist[1] = NVME_AUTH_DHCHAP_HASH_SHA384;
+	data->auth_protocol[0].dhchap.idlist[2] = NVME_AUTH_DHCHAP_HASH_SHA512;
+	data->auth_protocol[0].dhchap.idlist[3] = NVME_AUTH_DHCHAP_DHGROUP_NULL;
+
+	return size;
+}
+
+static int nvme_auth_dhchap_challenge(struct nvme_ctrl *ctrl,
+				      struct nvme_dhchap_context *chap,
+				      void *buf, size_t buf_size)
+{
+	struct nvmf_auth_dhchap_challenge_data *data = buf;
+	size_t size = sizeof(*data) + data->hl + data->dhvlen;
+	const char *gid_name;
+
+	if (buf_size < size) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
+		return -ENOMSG;
+	}
+
+	if (data->hashid != NVME_AUTH_DHCHAP_HASH_SHA256 &&
+	    data->hashid != NVME_AUTH_DHCHAP_HASH_SHA384 &&
+	    data->hashid != NVME_AUTH_DHCHAP_HASH_SHA512) {
+		dev_warn(ctrl->device,
+			 "qid %d: DH-HMAC-CHAP: invalid HASH ID %d\n",
+			 chap->qid, data->hashid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
+		return -EPROTO;
+	}
+	switch (data->dhgid) {
+	case NVME_AUTH_DHCHAP_DHGROUP_NULL:
+		gid_name = "null";
+		break;
+	default:
+		gid_name = NULL;
+		break;
+	}
+	if (!gid_name) {
+		dev_warn(ctrl->device,
+			 "qid %d: DH-HMAC-CHAP: invalid DH group id %d\n",
+			 chap->qid, data->dhgid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
+		return -EPROTO;
+	}
+	if (data->dhgid != NVME_AUTH_DHCHAP_DHGROUP_NULL) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
+		return -EPROTO;
+	}
+	if (data->dhgid == NVME_AUTH_DHCHAP_DHGROUP_NULL && data->dhvlen != 0) {
+		dev_warn(ctrl->device,
+			 "qid %d: DH-HMAC-CHAP: invalid DH value for NULL DH\n",
+			chap->qid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
+		return -EPROTO;
+	}
+	dev_dbg(ctrl->device, "%s: qid %d requested hash id %d\n",
+		__func__, chap->qid, data->hashid);
+	if (nvme_auth_hmac_len(data->hashid) != data->hl) {
+		dev_warn(ctrl->device,
+			 "qid %d: DH-HMAC-CHAP: invalid hash length\n",
+			chap->qid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
+		return -EPROTO;
+	}
+	chap->hash_id = data->hashid;
+	chap->hash_len = data->hl;
+	chap->s1 = le32_to_cpu(data->seqnum);
+	memcpy(chap->c1, data->cval, chap->hash_len);
+
+	return 0;
+}
+
+static int nvme_auth_dhchap_reply(struct nvme_ctrl *ctrl,
+				  struct nvme_dhchap_context *chap,
+				  void *buf, size_t buf_size)
+{
+	struct nvmf_auth_dhchap_reply_data *data = buf;
+	size_t size = sizeof(*data);
+
+	size += 2 * chap->hash_len;
+	if (ctrl->opts->dhchap_auth) {
+		get_random_bytes(chap->c2, chap->hash_len);
+		chap->s2 = nvme_dhchap_seqnum++;
+	} else
+		memset(chap->c2, 0, chap->hash_len);
+
+	if (chap->host_key_len)
+		size += chap->host_key_len;
+
+	if (buf_size < size)
+		return -EINVAL;
+
+	memset(buf, 0, size);
+	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_REPLY;
+	data->t_id = cpu_to_le16(chap->transaction);
+	data->hl = chap->hash_len;
+	data->dhvlen = chap->host_key_len;
+	data->seqnum = cpu_to_le32(chap->s2);
+	memcpy(data->rval, chap->response, chap->hash_len);
+	if (ctrl->opts->dhchap_auth) {
+		dev_dbg(ctrl->device, "%s: qid %d ctrl challenge %*ph\n",
+			__func__, chap->qid,
+			chap->hash_len, chap->c2);
+		data->cvalid = 1;
+		memcpy(data->rval + chap->hash_len, chap->c2,
+		       chap->hash_len);
+	}
+	if (chap->host_key_len)
+		memcpy(data->rval + 2 * chap->hash_len, chap->host_key,
+		       chap->host_key_len);
+
+	return size;
+}
+
+static int nvme_auth_dhchap_success1(struct nvme_ctrl *ctrl,
+				     struct nvme_dhchap_context *chap,
+				     void *buf, size_t buf_size)
+{
+	struct nvmf_auth_dhchap_success1_data *data = buf;
+	size_t size = sizeof(*data);
+
+	if (ctrl->opts->dhchap_auth)
+		size += chap->hash_len;
+
+
+	if (buf_size < size) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
+		return -ENOMSG;
+	}
+
+	if (data->hl != chap->hash_len) {
+		dev_warn(ctrl->device,
+			 "qid %d: DH-HMAC-CHAP: invalid hash length %d\n",
+			 chap->qid, data->hl);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
+		return -EPROTO;
+	}
+
+	if (!data->rvalid)
+		return 0;
+
+	/* Validate controller response */
+	if (memcmp(chap->response, data->rval, data->hl)) {
+		dev_dbg(ctrl->device, "%s: qid %d ctrl response %*ph\n",
+			__func__, chap->qid, chap->hash_len, data->rval);
+		dev_dbg(ctrl->device, "%s: qid %d host response %*ph\n",
+			__func__, chap->qid, chap->hash_len, chap->response);
+		dev_warn(ctrl->device,
+			 "qid %d: DH-HMAC-CHAP: controller authentication failed\n",
+			 chap->qid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
+		return -EPROTO;
+	}
+	dev_info(ctrl->device,
+		 "qid %d: DH-HMAC-CHAP: controller authenticated\n",
+		chap->qid);
+	return 0;
+}
+
+static int nvme_auth_dhchap_success2(struct nvme_ctrl *ctrl,
+				     struct nvme_dhchap_context *chap,
+				     void *buf, size_t buf_size)
+{
+	struct nvmf_auth_dhchap_success2_data *data = buf;
+	size_t size = sizeof(*data);
+
+	memset(buf, 0, size);
+	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_SUCCESS2;
+	data->t_id = cpu_to_le16(chap->transaction);
+
+	return size;
+}
+
+static int nvme_auth_dhchap_failure2(struct nvme_ctrl *ctrl,
+				     struct nvme_dhchap_context *chap,
+				     void *buf, size_t buf_size)
+{
+	struct nvmf_auth_dhchap_failure_data *data = buf;
+	size_t size = sizeof(*data);
+
+	memset(buf, 0, size);
+	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_FAILURE2;
+	data->t_id = cpu_to_le16(chap->transaction);
+	data->reason_code = 1;
+	data->reason_code_explanation = chap->status;
+
+	return size;
+}
+
+int nvme_auth_select_hash(struct nvme_ctrl *ctrl,
+			  struct nvme_dhchap_context *chap)
+{
+	char *hash_name;
+	int ret;
+
+	switch (chap->hash_id) {
+	case NVME_AUTH_DHCHAP_HASH_SHA256:
+		hash_name = "hmac(sha256)";
+		break;
+	case NVME_AUTH_DHCHAP_HASH_SHA384:
+		hash_name = "hmac(sha384)";
+		break;
+	case NVME_AUTH_DHCHAP_HASH_SHA512:
+		hash_name = "hmac(sha512)";
+		break;
+	default:
+		hash_name = NULL;
+		break;
+	}
+	if (!hash_name) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
+		return -EPROTO;
+	}
+	chap->shash_tfm = crypto_alloc_shash(hash_name, 0,
+					     CRYPTO_ALG_ALLOCATES_MEMORY);
+	if (IS_ERR(chap->shash_tfm)) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
+		chap->shash_tfm = NULL;
+		return -EPROTO;
+	}
+	if (!chap->key) {
+		dev_warn(ctrl->device, "qid %d: cannot select hash, no key\n",
+			 chap->qid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
+		crypto_free_shash(chap->shash_tfm);
+		chap->shash_tfm = NULL;
+		return -EINVAL;
+	}
+	ret = crypto_shash_setkey(chap->shash_tfm, chap->key, chap->key_len);
+	if (ret) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
+		crypto_free_shash(chap->shash_tfm);
+		chap->shash_tfm = NULL;
+		return ret;
+	}
+	dev_info(ctrl->device, "qid %d: DH-HMAC_CHAP: selected hash %s\n",
+		 chap->qid, hash_name);
+	return 0;
+}
+
+static int nvme_auth_dhchap_host_response(struct nvme_ctrl *ctrl,
+					  struct nvme_dhchap_context *chap)
+{
+	SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
+	u8 buf[4], *challenge = chap->c1;
+	int ret;
+
+	dev_dbg(ctrl->device, "%s: qid %d host response seq %d transaction %d\n",
+		__func__, chap->qid, chap->s1, chap->transaction);
+	shash->tfm = chap->shash_tfm;
+	ret = crypto_shash_init(shash);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, challenge, chap->hash_len);
+	if (ret)
+		goto out;
+	put_unaligned_le32(chap->s1, buf);
+	ret = crypto_shash_update(shash, buf, 4);
+	if (ret)
+		goto out;
+	put_unaligned_le16(chap->transaction, buf);
+	ret = crypto_shash_update(shash, buf, 2);
+	if (ret)
+		goto out;
+	memset(buf, 0, sizeof(buf));
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, "HostHost", 8);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
+				  strlen(ctrl->opts->host->nqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
+			    strlen(ctrl->opts->subsysnqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_final(shash, chap->response);
+out:
+	return ret;
+}
+
+static int nvme_auth_dhchap_ctrl_response(struct nvme_ctrl *ctrl,
+					  struct nvme_dhchap_context *chap)
+{
+	SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
+	u8 buf[4], *challenge = chap->c2;
+	int ret;
+
+	dev_dbg(ctrl->device, "%s: qid %d host response seq %d transaction %d\n",
+		__func__, chap->qid, chap->s2, chap->transaction);
+	dev_dbg(ctrl->device, "%s: qid %d challenge %*ph\n",
+		__func__, chap->qid, chap->hash_len, challenge);
+	dev_dbg(ctrl->device, "%s: qid %d subsysnqn %s\n",
+		__func__, chap->qid, ctrl->opts->subsysnqn);
+	dev_dbg(ctrl->device, "%s: qid %d hostnqn %s\n",
+		__func__, chap->qid, ctrl->opts->host->nqn);
+	shash->tfm = chap->shash_tfm;
+	ret = crypto_shash_init(shash);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, challenge, chap->hash_len);
+	if (ret)
+		goto out;
+	put_unaligned_le32(chap->s2, buf);
+	ret = crypto_shash_update(shash, buf, 4);
+	if (ret)
+		goto out;
+	put_unaligned_le16(chap->transaction, buf);
+	ret = crypto_shash_update(shash, buf, 2);
+	if (ret)
+		goto out;
+	memset(buf, 0, 4);
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, "Controller", 10);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
+				  strlen(ctrl->opts->subsysnqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
+				  strlen(ctrl->opts->host->nqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_final(shash, chap->response);
+out:
+	return ret;
+}
+
+int nvme_auth_generate_key(struct nvme_ctrl *ctrl,
+			   struct nvme_dhchap_context *chap)
+{
+	int ret;
+	u8 key_hash;
+	const char *hmac_name;
+	struct crypto_shash *key_tfm;
+
+	if (sscanf(ctrl->opts->dhchap_secret, "DHHC-1:%hhd:%*s:",
+		   &key_hash) != 1)
+		return -EINVAL;
+
+	chap->key = nvme_auth_extract_secret(ctrl->opts->dhchap_secret,
+					     &chap->key_len);
+	if (IS_ERR(chap->key)) {
+		ret = PTR_ERR(chap->key);
+		chap->key = NULL;
+		return ret;
+	}
+
+	if (key_hash == 0)
+		return 0;
+
+	hmac_name = nvme_auth_hmac_name(key_hash);
+	if (!hmac_name) {
+		pr_debug("Invalid key hash id %d\n", key_hash);
+		return -EKEYREJECTED;
+	}
+
+	key_tfm = crypto_alloc_shash(hmac_name, 0, 0);
+	if (IS_ERR(key_tfm)) {
+		kfree(chap->key);
+		chap->key = NULL;
+		ret = PTR_ERR(key_tfm);
+	} else {
+		SHASH_DESC_ON_STACK(shash, key_tfm);
+
+		shash->tfm = key_tfm;
+		ret = crypto_shash_setkey(key_tfm, chap->key,
+					  chap->key_len);
+		if (ret < 0) {
+			crypto_free_shash(key_tfm);
+			kfree(chap->key);
+			chap->key = NULL;
+			return ret;
+		}
+		crypto_shash_init(shash);
+		crypto_shash_update(shash, ctrl->opts->host->nqn,
+				    strlen(ctrl->opts->host->nqn));
+		crypto_shash_update(shash, "NVMe-over-Fabrics", 17);
+		crypto_shash_final(shash, chap->key);
+		crypto_free_shash(key_tfm);
+	}
+	return 0;
+}
+
+void nvme_auth_free(struct nvme_dhchap_context *chap)
+{
+	if (chap->shash_tfm)
+		crypto_free_shash(chap->shash_tfm);
+	if (chap->key)
+		kfree(chap->key);
+	if (chap->ctrl_key)
+		kfree(chap->ctrl_key);
+	if (chap->host_key)
+		kfree(chap->host_key);
+	if (chap->sess_key)
+		kfree(chap->sess_key);
+	kfree(chap);
+}
+
+int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid)
+{
+	struct nvme_dhchap_context *chap;
+	void *buf;
+	size_t buf_size, tl;
+	int ret = 0;
+
+	chap = kzalloc(sizeof(*chap), GFP_KERNEL);
+	if (!chap)
+		return -ENOMEM;
+	chap->qid = qid;
+	chap->transaction = ctrl->transaction++;
+
+	ret = nvme_auth_generate_key(ctrl, chap);
+	if (ret) {
+		dev_dbg(ctrl->device, "%s: failed to generate key, error %d\n",
+			__func__, ret);
+		nvme_auth_free(chap);
+		return ret;
+	}
+
+	/*
+	 * Allocate a large enough buffer for the entire negotiation:
+	 * 4k should be enough to ffdhe8192.
+	 */
+	buf_size = 4096;
+	buf = kzalloc(buf_size, GFP_KERNEL);
+	if (!buf) {
+		ret = -ENOMEM;
+		goto out;
+	}
+
+	/* DH-HMAC-CHAP Step 1: send negotiate */
+	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP negotiate\n",
+		__func__, qid);
+	ret = nvme_auth_dhchap_negotiate(ctrl, chap, buf, buf_size);
+	if (ret < 0)
+		goto out;
+	tl = ret;
+	ret = nvme_auth_send(ctrl, qid, buf, tl);
+	if (ret)
+		goto out;
+
+	memset(buf, 0, buf_size);
+	ret = nvme_auth_receive(ctrl, qid, buf, buf_size, chap->transaction,
+				NVME_AUTH_DHCHAP_MESSAGE_CHALLENGE);
+	if (ret < 0) {
+		dev_dbg(ctrl->device,
+			"%s: qid %d DH-HMAC-CHAP failed to receive challenge\n",
+			__func__, qid);
+		goto out;
+	}
+	if (ret > 0) {
+		chap->status = ret;
+		goto fail1;
+	}
+
+	/* DH-HMAC-CHAP Step 2: receive challenge */
+	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP challenge\n",
+		__func__, qid);
+
+	ret = nvme_auth_dhchap_challenge(ctrl, chap, buf, buf_size);
+	if (ret) {
+		/* Invalid parameters for negotiate */
+		goto fail2;
+	}
+
+	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP select hash\n",
+		__func__, qid);
+	ret = nvme_auth_select_hash(ctrl, chap);
+	if (ret)
+		goto fail2;
+
+	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP host response\n",
+		__func__, qid);
+	ret = nvme_auth_dhchap_host_response(ctrl, chap);
+	if (ret)
+		goto fail2;
+
+	/* DH-HMAC-CHAP Step 3: send reply */
+	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP reply\n",
+		__func__, qid);
+	ret = nvme_auth_dhchap_reply(ctrl, chap, buf, buf_size);
+	if (ret < 0)
+		goto fail2;
+
+	tl = ret;
+	ret = nvme_auth_send(ctrl, qid, buf, tl);
+	if (ret)
+		goto fail2;
+
+	memset(buf, 0, buf_size);
+	ret = nvme_auth_receive(ctrl, qid, buf, buf_size, chap->transaction,
+				NVME_AUTH_DHCHAP_MESSAGE_SUCCESS1);
+	if (ret < 0) {
+		dev_dbg(ctrl->device,
+			"%s: qid %d DH-HMAC-CHAP failed to receive success1\n",
+			__func__, qid);
+		goto out;
+	}
+	if (ret > 0) {
+		chap->status = ret;
+		goto fail1;
+	}
+
+	if (ctrl->opts->dhchap_auth) {
+		dev_dbg(ctrl->device,
+			"%s: qid %d DH-HMAC-CHAP controller response\n",
+			__func__, qid);
+		ret = nvme_auth_dhchap_ctrl_response(ctrl, chap);
+		if (ret)
+			goto fail2;
+	}
+
+	/* DH-HMAC-CHAP Step 4: receive success1 */
+	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP success1\n",
+		__func__, qid);
+	ret = nvme_auth_dhchap_success1(ctrl, chap, buf, buf_size);
+	if (ret < 0) {
+		/* Controller authentication failed */
+		goto fail2;
+	}
+	tl = ret;
+	/* DH-HMAC-CHAP Step 5: send success2 */
+	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP success2\n",
+		__func__, qid);
+	tl = nvme_auth_dhchap_success2(ctrl, chap, buf, buf_size);
+	ret = nvme_auth_send(ctrl, qid, buf, tl);
+	if (!ret)
+		goto out;
+
+fail1:
+	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP failure1, status %x\n",
+		__func__, qid, chap->status);
+	goto out;
+
+fail2:
+	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP failure2, status %x\n",
+		__func__, qid, chap->status);
+	tl = nvme_auth_dhchap_failure2(ctrl, chap, buf, buf_size);
+	ret = nvme_auth_send(ctrl, qid, buf, tl);
+
+out:
+	if (!ret && chap->status)
+		ret = -EPROTO;
+	if (!ret) {
+		ctrl->dhchap_hash = chap->hash_id;
+	}
+	kfree(buf);
+	nvme_auth_free(chap);
+	return ret;
+}
diff --git a/drivers/nvme/host/auth.h b/drivers/nvme/host/auth.h
new file mode 100644
index 000000000000..4950b1cb9470
--- /dev/null
+++ b/drivers/nvme/host/auth.h
@@ -0,0 +1,23 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2021 Hannes Reinecke, SUSE Software Solutions
+ */
+
+#ifndef _NVME_AUTH_H
+#define _NVME_AUTH_H
+
+const char *nvme_auth_dhgroup_name(int dhgroup_id);
+int nvme_auth_dhgroup_pubkey_size(int dhgroup_id);
+int nvme_auth_dhgroup_privkey_size(int dhgroup_id);
+const char *nvme_auth_dhgroup_kpp(int dhgroup_id);
+int nvme_auth_dhgroup_id(const char *dhgroup_name);
+
+const char *nvme_auth_hmac_name(int hmac_id);
+const char *nvme_auth_digest_name(int hmac_id);
+int nvme_auth_hmac_id(const char *hmac_name);
+int nvme_auth_hmac_len(int hmac_len);
+
+unsigned char *nvme_auth_extract_secret(unsigned char *dhchap_secret,
+					size_t *dhchap_key_len);
+
+#endif /* _NVME_AUTH_H */
diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 11779be42186..7ce9b666dc09 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -708,7 +708,9 @@ bool __nvme_check_ready(struct nvme_ctrl *ctrl, struct request *rq,
 		switch (ctrl->state) {
 		case NVME_CTRL_CONNECTING:
 			if (blk_rq_is_passthrough(rq) && nvme_is_fabrics(req->cmd) &&
-			    req->cmd->fabrics.fctype == nvme_fabrics_type_connect)
+			    (req->cmd->fabrics.fctype == nvme_fabrics_type_connect ||
+			     req->cmd->fabrics.fctype == nvme_fabrics_type_auth_send ||
+			     req->cmd->fabrics.fctype == nvme_fabrics_type_auth_receive))
 				return true;
 			break;
 		default:
@@ -3426,6 +3428,66 @@ static ssize_t nvme_ctrl_fast_io_fail_tmo_store(struct device *dev,
 static DEVICE_ATTR(fast_io_fail_tmo, S_IRUGO | S_IWUSR,
 	nvme_ctrl_fast_io_fail_tmo_show, nvme_ctrl_fast_io_fail_tmo_store);
 
+#ifdef CONFIG_NVME_AUTH
+struct nvmet_dhchap_hash_map {
+	int id;
+	const char name[15];
+} hash_map[] = {
+	{.id = NVME_AUTH_DHCHAP_HASH_SHA256,
+	 .name = "hmac(sha256)", },
+	{.id = NVME_AUTH_DHCHAP_HASH_SHA384,
+	 .name = "hmac(sha384)", },
+	{.id = NVME_AUTH_DHCHAP_HASH_SHA512,
+	 .name = "hmac(sha512)", },
+};
+
+static ssize_t dhchap_hash_show(struct device *dev,
+	struct device_attribute *attr, char *buf)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
+		if (hash_map[i].id == ctrl->dhchap_hash)
+			return sprintf(buf, "%s\n", hash_map[i].name);
+	}
+	return sprintf(buf, "none\n");
+}
+DEVICE_ATTR_RO(dhchap_hash);
+
+struct nvmet_dhchap_group_map {
+	int id;
+	const char name[15];
+} dhgroup_map[] = {
+	{.id = NVME_AUTH_DHCHAP_DHGROUP_NULL,
+	 .name = "NULL", },
+	{.id = NVME_AUTH_DHCHAP_DHGROUP_2048,
+	 .name = "ffdhe2048", },
+	{.id = NVME_AUTH_DHCHAP_DHGROUP_3072,
+	 .name = "ffdhe3072", },
+	{.id = NVME_AUTH_DHCHAP_DHGROUP_4096,
+	 .name = "ffdhe4096", },
+	{.id = NVME_AUTH_DHCHAP_DHGROUP_6144,
+	 .name = "ffdhe6144", },
+	{.id = NVME_AUTH_DHCHAP_DHGROUP_8192,
+	 .name = "ffdhe8192", },
+};
+
+static ssize_t dhchap_dhgroup_show(struct device *dev,
+	struct device_attribute *attr, char *buf)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(dhgroup_map); i++) {
+		if (hash_map[i].id == ctrl->dhchap_dhgroup)
+			return sprintf(buf, "%s\n", dhgroup_map[i].name);
+	}
+	return sprintf(buf, "none\n");
+}
+DEVICE_ATTR_RO(dhchap_dhgroup);
+#endif
+
 static struct attribute *nvme_dev_attrs[] = {
 	&dev_attr_reset_controller.attr,
 	&dev_attr_rescan_controller.attr,
@@ -3447,6 +3509,10 @@ static struct attribute *nvme_dev_attrs[] = {
 	&dev_attr_reconnect_delay.attr,
 	&dev_attr_fast_io_fail_tmo.attr,
 	&dev_attr_kato.attr,
+#ifdef CONFIG_NVME_AUTH
+	&dev_attr_dhchap_hash.attr,
+	&dev_attr_dhchap_dhgroup.attr,
+#endif
 	NULL
 };
 
@@ -3470,6 +3536,10 @@ static umode_t nvme_dev_attrs_are_visible(struct kobject *kobj,
 		return 0;
 	if (a == &dev_attr_fast_io_fail_tmo.attr && !ctrl->opts)
 		return 0;
+#ifdef CONFIG_NVME_AUTH
+	if (a == &dev_attr_dhchap_hash.attr && !ctrl->opts)
+		return 0;
+#endif
 
 	return a->mode;
 }
@@ -4581,6 +4651,11 @@ static inline void _nvme_check_size(void)
 	BUILD_BUG_ON(sizeof(struct nvme_smart_log) != 512);
 	BUILD_BUG_ON(sizeof(struct nvme_dbbuf) != 64);
 	BUILD_BUG_ON(sizeof(struct nvme_directive_cmd) != 64);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_negotiate_data) != 8);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_challenge_data) != 16);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_reply_data) != 16);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_success1_data) != 16);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_success2_data) != 16);
 }
 
 
diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index a5469fd9d4c3..6404ab9b604b 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -366,6 +366,7 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)
 	union nvme_result res;
 	struct nvmf_connect_data *data;
 	int ret;
+	u32 result;
 
 	cmd.connect.opcode = nvme_fabrics_command;
 	cmd.connect.fctype = nvme_fabrics_type_connect;
@@ -398,8 +399,18 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)
 		goto out_free_data;
 	}
 
-	ctrl->cntlid = le16_to_cpu(res.u16);
-
+	result = le32_to_cpu(res.u32);
+	ctrl->cntlid = result & 0xFFFF;
+	if ((result >> 16) & 2) {
+		/* Authentication required */
+		ret = nvme_auth_negotiate(ctrl, NVME_QID_ANY);
+		if (ret)
+			dev_warn(ctrl->device,
+				 "qid 0: authentication failed\n");
+		else
+			dev_info(ctrl->device,
+				 "qid 0: authenticated\n");
+	}
 out_free_data:
 	kfree(data);
 	return ret;
@@ -432,6 +443,7 @@ int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, u16 qid)
 	struct nvmf_connect_data *data;
 	union nvme_result res;
 	int ret;
+	u32 result;
 
 	cmd.connect.opcode = nvme_fabrics_command;
 	cmd.connect.fctype = nvme_fabrics_type_connect;
@@ -457,6 +469,17 @@ int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, u16 qid)
 		nvmf_log_connect_error(ctrl, ret, le32_to_cpu(res.u32),
 				       &cmd, data);
 	}
+	result = le32_to_cpu(res.u32);
+	if ((result >> 16) & 2) {
+		/* Authentication required */
+		ret = nvme_auth_negotiate(ctrl, qid);
+		if (ret)
+			dev_warn(ctrl->device,
+				 "qid %u: authentication failed\n", qid);
+		else
+			dev_info(ctrl->device,
+				 "qid %u: authenticated\n", qid);
+	}
 	kfree(data);
 	return ret;
 }
@@ -548,6 +571,9 @@ static const match_table_t opt_tokens = {
 	{ NVMF_OPT_NR_POLL_QUEUES,	"nr_poll_queues=%d"	},
 	{ NVMF_OPT_TOS,			"tos=%d"		},
 	{ NVMF_OPT_FAIL_FAST_TMO,	"fast_io_fail_tmo=%d"	},
+	{ NVMF_OPT_DHCHAP_SECRET,	"dhchap_secret=%s"	},
+	{ NVMF_OPT_DHCHAP_AUTH,		"authenticate"		},
+	{ NVMF_OPT_DHCHAP_GROUP,	"dhchap_group=%s"	},
 	{ NVMF_OPT_ERR,			NULL			}
 };
 
@@ -824,6 +850,35 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
 			}
 			opts->tos = token;
 			break;
+		case NVMF_OPT_DHCHAP_SECRET:
+			p = match_strdup(args);
+			if (!p) {
+				ret = -ENOMEM;
+				goto out;
+			}
+			if (strncmp(p, "DHHC-1:00:", 10)) {
+				pr_err("Invalid DH-CHAP secret %s\n", p);
+				ret = -EINVAL;
+				goto out;
+			}
+			kfree(opts->dhchap_secret);
+			opts->dhchap_secret = p;
+			break;
+		case NVMF_OPT_DHCHAP_AUTH:
+			opts->dhchap_auth = true;
+			break;
+		case NVMF_OPT_DHCHAP_GROUP:
+			if (match_int(args, &token)) {
+				ret = -EINVAL;
+				goto out;
+			}
+			if (token <= 0) {
+				pr_err("Invalid dhchap_group %d\n", token);
+				ret = -EINVAL;
+				goto out;
+			}
+			opts->dhchap_group = token;
+			break;
 		default:
 			pr_warn("unknown parameter or missing value '%s' in ctrl creation request\n",
 				p);
@@ -942,6 +997,7 @@ void nvmf_free_options(struct nvmf_ctrl_options *opts)
 	kfree(opts->subsysnqn);
 	kfree(opts->host_traddr);
 	kfree(opts->host_iface);
+	kfree(opts->dhchap_secret);
 	kfree(opts);
 }
 EXPORT_SYMBOL_GPL(nvmf_free_options);
@@ -951,7 +1007,10 @@ EXPORT_SYMBOL_GPL(nvmf_free_options);
 				 NVMF_OPT_KATO | NVMF_OPT_HOSTNQN | \
 				 NVMF_OPT_HOST_ID | NVMF_OPT_DUP_CONNECT |\
 				 NVMF_OPT_DISABLE_SQFLOW |\
-				 NVMF_OPT_FAIL_FAST_TMO)
+				 NVMF_OPT_CTRL_LOSS_TMO |\
+				 NVMF_OPT_FAIL_FAST_TMO |\
+				 NVMF_OPT_DHCHAP_SECRET |\
+				 NVMF_OPT_DHCHAP_AUTH | NVMF_OPT_DHCHAP_GROUP)
 
 static struct nvme_ctrl *
 nvmf_create_ctrl(struct device *dev, const char *buf)
diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h
index a146cb903869..535bc544f0f6 100644
--- a/drivers/nvme/host/fabrics.h
+++ b/drivers/nvme/host/fabrics.h
@@ -67,6 +67,9 @@ enum {
 	NVMF_OPT_TOS		= 1 << 19,
 	NVMF_OPT_FAIL_FAST_TMO	= 1 << 20,
 	NVMF_OPT_HOST_IFACE	= 1 << 21,
+	NVMF_OPT_DHCHAP_SECRET	= 1 << 22,
+	NVMF_OPT_DHCHAP_AUTH	= 1 << 23,
+	NVMF_OPT_DHCHAP_GROUP	= 1 << 24,
 };
 
 /**
@@ -96,6 +99,8 @@ enum {
  * @max_reconnects: maximum number of allowed reconnect attempts before removing
  *              the controller, (-1) means reconnect forever, zero means remove
  *              immediately;
+ * @dhchap_secret: DH-HMAC-CHAP secret
+ * @dhchap_auth: DH-HMAC-CHAP authenticate controller
  * @disable_sqflow: disable controller sq flow control
  * @hdr_digest: generate/verify header digest (TCP)
  * @data_digest: generate/verify data digest (TCP)
@@ -120,6 +125,9 @@ struct nvmf_ctrl_options {
 	unsigned int		kato;
 	struct nvmf_host	*host;
 	int			max_reconnects;
+	char			*dhchap_secret;
+	int			dhchap_group;
+	bool			dhchap_auth;
 	bool			disable_sqflow;
 	bool			hdr_digest;
 	bool			data_digest;
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 18ef8dd03a90..bcd5b8276c26 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -328,6 +328,12 @@ struct nvme_ctrl {
 	struct work_struct ana_work;
 #endif
 
+#ifdef CONFIG_NVME_AUTH
+	u16 transaction;
+	u8 dhchap_hash;
+	u8 dhchap_dhgroup;
+#endif
+
 	/* Power saving configuration */
 	u64 ps_max_latency_us;
 	bool apst_enabled;
@@ -874,6 +880,15 @@ static inline bool nvme_ctrl_sgl_supported(struct nvme_ctrl *ctrl)
 	return ctrl->sgls & ((1 << 0) | (1 << 1));
 }
 
+#ifdef CONFIG_NVME_AUTH
+int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid);
+#else
+static inline int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid)
+{
+	return -EPROTONOSUPPORT;
+}
+#endif
+
 u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 			 u8 opcode);
 int nvme_execute_passthru_rq(struct request *rq);
diff --git a/drivers/nvme/host/trace.c b/drivers/nvme/host/trace.c
index 6543015b6121..66f75d8ea925 100644
--- a/drivers/nvme/host/trace.c
+++ b/drivers/nvme/host/trace.c
@@ -271,6 +271,34 @@ static const char *nvme_trace_fabrics_property_get(struct trace_seq *p, u8 *spc)
 	return ret;
 }
 
+static const char *nvme_trace_fabrics_auth_send(struct trace_seq *p, u8 *spc)
+{
+	const char *ret = trace_seq_buffer_ptr(p);
+	u8 spsp0 = spc[1];
+	u8 spsp1 = spc[2];
+	u8 secp = spc[3];
+	u32 tl = get_unaligned_le32(spc + 4);
+
+	trace_seq_printf(p, "spsp0=%02x, spsp1=%02x, secp=%02x, tl=%u",
+			 spsp0, spsp1, secp, tl);
+	trace_seq_putc(p, 0);
+	return ret;
+}
+
+static const char *nvme_trace_fabrics_auth_receive(struct trace_seq *p, u8 *spc)
+{
+	const char *ret = trace_seq_buffer_ptr(p);
+	u8 spsp0 = spc[1];
+	u8 spsp1 = spc[2];
+	u8 secp = spc[3];
+	u32 al = get_unaligned_le32(spc + 4);
+
+	trace_seq_printf(p, "spsp0=%02x, spsp1=%02x, secp=%02x, al=%u",
+			 spsp0, spsp1, secp, al);
+	trace_seq_putc(p, 0);
+	return ret;
+}
+
 static const char *nvme_trace_fabrics_common(struct trace_seq *p, u8 *spc)
 {
 	const char *ret = trace_seq_buffer_ptr(p);
@@ -290,6 +318,10 @@ const char *nvme_trace_parse_fabrics_cmd(struct trace_seq *p,
 		return nvme_trace_fabrics_connect(p, spc);
 	case nvme_fabrics_type_property_get:
 		return nvme_trace_fabrics_property_get(p, spc);
+	case nvme_fabrics_type_auth_send:
+		return nvme_trace_fabrics_auth_send(p, spc);
+	case nvme_fabrics_type_auth_receive:
+		return nvme_trace_fabrics_auth_receive(p, spc);
 	default:
 		return nvme_trace_fabrics_common(p, spc);
 	}
-- 
2.29.2

[PATCH 06/11] nvme: Implement In-Band authentication

[Hannes Reinecke]

Implement NVMe-oF In-Band authentication. This patch adds two new
fabric options 'dhchap_key' to specify the PSK and 'dhchap_authenticate'
to request bi-directional authentication of both the host and the controller.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/nvme/host/Kconfig   |  11 +
 drivers/nvme/host/Makefile  |   1 +
 drivers/nvme/host/auth.c    | 813 ++++++++++++++++++++++++++++++++++++
 drivers/nvme/host/auth.h    |  23 +
 drivers/nvme/host/core.c    |  77 +++-
 drivers/nvme/host/fabrics.c |  65 ++-
 drivers/nvme/host/fabrics.h |   8 +
 drivers/nvme/host/nvme.h    |  15 +
 drivers/nvme/host/trace.c   |  32 ++
 9 files changed, 1041 insertions(+), 4 deletions(-)
 create mode 100644 drivers/nvme/host/auth.c
 create mode 100644 drivers/nvme/host/auth.h

diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig
index c3f3d77f1aac..853c546305e9 100644
--- a/drivers/nvme/host/Kconfig
+++ b/drivers/nvme/host/Kconfig
@@ -85,3 +85,14 @@ config NVME_TCP
 	  from https://github.com/linux-nvme/nvme-cli.
 
 	  If unsure, say N.
+
+config NVME_AUTH
+	bool "NVM Express over Fabrics In-Band Authentication"
+	depends on NVME_TCP
+	select CRYPTO_SHA256
+	select CRYPTO_SHA512
+	help
+	  This provides support for NVMe over Fabrics In-Band Authentication
+	  for the NVMe over TCP transport.
+
+	  If unsure, say N.
diff --git a/drivers/nvme/host/Makefile b/drivers/nvme/host/Makefile
index cbc509784b2e..03748a55a12b 100644
--- a/drivers/nvme/host/Makefile
+++ b/drivers/nvme/host/Makefile
@@ -20,6 +20,7 @@ nvme-core-$(CONFIG_NVME_HWMON)		+= hwmon.o
 nvme-y					+= pci.o
 
 nvme-fabrics-y				+= fabrics.o
+nvme-fabrics-$(CONFIG_NVME_AUTH)	+= auth.o
 
 nvme-rdma-y				+= rdma.o
 
diff --git a/drivers/nvme/host/auth.c b/drivers/nvme/host/auth.c
new file mode 100644
index 000000000000..448a3adebea6
--- /dev/null
+++ b/drivers/nvme/host/auth.c
@@ -0,0 +1,813 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2020 Hannes Reinecke, SUSE Linux
+ */
+
+#include <linux/crc32.h>
+#include <linux/base64.h>
+#include <asm/unaligned.h>
+#include <crypto/hash.h>
+#include <crypto/kpp.h>
+#include "nvme.h"
+#include "fabrics.h"
+#include "auth.h"
+
+static u32 nvme_dhchap_seqnum;
+
+struct nvme_dhchap_context {
+	struct crypto_shash *shash_tfm;
+	unsigned char *key;
+	size_t key_len;
+	int qid;
+	u32 s1;
+	u32 s2;
+	u16 transaction;
+	u8 status;
+	u8 hash_id;
+	u8 hash_len;
+	u8 c1[64];
+	u8 c2[64];
+	u8 response[64];
+	u8 *ctrl_key;
+	int ctrl_key_len;
+	u8 *host_key;
+	int host_key_len;
+	u8 *sess_key;
+	int sess_key_len;
+};
+
+struct nvmet_dhchap_hash_map {
+	int id;
+	int hash_len;
+	const char hmac[15];
+	const char digest[15];
+} hash_map[] = {
+	{.id = NVME_AUTH_DHCHAP_HASH_SHA256,
+	 .hash_len = 32,
+	 .hmac = "hmac(sha256)", .digest = "sha256" },
+	{.id = NVME_AUTH_DHCHAP_HASH_SHA384,
+	 .hash_len = 48,
+	 .hmac = "hmac(sha384)", .digest = "sha384" },
+	{.id = NVME_AUTH_DHCHAP_HASH_SHA512,
+	 .hash_len = 64,
+	 .hmac = "hmac(sha512)", .digest = "sha512" },
+};
+
+const char *nvme_auth_hmac_name(int hmac_id)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
+		if (hash_map[i].id == hmac_id)
+			return hash_map[i].hmac;
+	}
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_hmac_name);
+
+const char *nvme_auth_digest_name(int hmac_id)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
+		if (hash_map[i].id == hmac_id)
+			return hash_map[i].digest;
+	}
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_digest_name);
+
+int nvme_auth_hmac_len(int hmac_id)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
+		if (hash_map[i].id == hmac_id)
+			return hash_map[i].hash_len;
+	}
+	return -1;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_hmac_len);
+
+int nvme_auth_hmac_id(const char *hmac_name)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
+		if (!strncmp(hash_map[i].hmac, hmac_name,
+			     strlen(hash_map[i].hmac)))
+			return hash_map[i].id;
+	}
+	return -1;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_hmac_id);
+
+unsigned char *nvme_auth_extract_secret(unsigned char *dhchap_secret,
+					size_t *dhchap_key_len)
+{
+	unsigned char *dhchap_key;
+	u32 crc;
+	int key_len;
+	size_t allocated_len;
+
+	allocated_len = strlen(dhchap_secret) - 10;
+	dhchap_key = kzalloc(allocated_len, GFP_KERNEL);
+	if (!dhchap_key)
+		return ERR_PTR(-ENOMEM);
+
+	key_len = base64_decode(dhchap_secret + 10,
+				allocated_len, dhchap_key);
+	if (key_len != 36 && key_len != 52 &&
+	    key_len != 68) {
+		pr_debug("Invalid DH-HMAC-CHAP key len %d\n",
+			 key_len);
+		kfree(dhchap_key);
+		return ERR_PTR(-EINVAL);
+	}
+	pr_debug("DH-HMAC-CHAP Key: %*ph\n",
+		 (int)key_len, dhchap_key);
+
+	/* The last four bytes is the CRC in little-endian format */
+	key_len -= 4;
+	/*
+	 * The linux implementation doesn't do pre- and post-increments,
+	 * so we have to do it manually.
+	 */
+	crc = ~crc32(~0, dhchap_key, key_len);
+
+	if (get_unaligned_le32(dhchap_key + key_len) != crc) {
+		pr_debug("DH-HMAC-CHAP crc mismatch (key %08x, crc %08x)\n",
+		       get_unaligned_le32(dhchap_key + key_len), crc);
+		kfree(dhchap_key);
+		return ERR_PTR(-EKEYREJECTED);
+	}
+	*dhchap_key_len = key_len;
+	return dhchap_key;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_extract_secret);
+
+static int nvme_auth_send(struct nvme_ctrl *ctrl, int qid,
+			  void *data, size_t tl)
+{
+	struct nvme_command cmd = {};
+	blk_mq_req_flags_t flags = qid == NVME_QID_ANY ?
+		0 : BLK_MQ_REQ_NOWAIT | BLK_MQ_REQ_RESERVED;
+	struct request_queue *q = qid == NVME_QID_ANY ?
+		ctrl->fabrics_q : ctrl->connect_q;
+	int ret;
+
+	cmd.auth_send.opcode = nvme_fabrics_command;
+	cmd.auth_send.fctype = nvme_fabrics_type_auth_send;
+	cmd.auth_send.secp = NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER;
+	cmd.auth_send.spsp0 = 0x01;
+	cmd.auth_send.spsp1 = 0x01;
+	cmd.auth_send.tl = tl;
+
+	ret = __nvme_submit_sync_cmd(q, &cmd, NULL, data, tl, 0, qid,
+				     0, flags);
+	if (ret)
+		dev_dbg(ctrl->device,
+			"%s: qid %d error %d\n", __func__, qid, ret);
+	return ret;
+}
+
+static int nvme_auth_receive(struct nvme_ctrl *ctrl, int qid,
+			     void *buf, size_t al,
+			     u16 transaction, u8 expected_msg )
+{
+	struct nvme_command cmd = {};
+	struct nvmf_auth_dhchap_failure_data *data = buf;
+	blk_mq_req_flags_t flags = qid == NVME_QID_ANY ?
+		0 : BLK_MQ_REQ_NOWAIT | BLK_MQ_REQ_RESERVED;
+	struct request_queue *q = qid == NVME_QID_ANY ?
+		ctrl->fabrics_q : ctrl->connect_q;
+	int ret;
+
+	cmd.auth_receive.opcode = nvme_fabrics_command;
+	cmd.auth_receive.fctype = nvme_fabrics_type_auth_receive;
+	cmd.auth_receive.secp = NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER;
+	cmd.auth_receive.spsp0 = 0x01;
+	cmd.auth_receive.spsp1 = 0x01;
+	cmd.auth_receive.al = al;
+
+	ret = __nvme_submit_sync_cmd(q, &cmd, NULL, buf, al, 0, qid,
+				     0, flags);
+	if (ret > 0) {
+		dev_dbg(ctrl->device, "%s: qid %d nvme status %x\n",
+			__func__, qid, ret);
+		ret = -EIO;
+	}
+	if (ret < 0) {
+		dev_dbg(ctrl->device, "%s: qid %d error %d\n",
+			__func__, qid, ret);
+		return ret;
+	}
+	dev_dbg(ctrl->device, "%s: qid %d auth_type %d auth_id %x\n",
+		__func__, qid, data->auth_type, data->auth_id);
+	if (data->auth_type == NVME_AUTH_COMMON_MESSAGES &&
+	    data->auth_id == NVME_AUTH_DHCHAP_MESSAGE_FAILURE1) {
+		return data->reason_code_explanation;
+	}
+	if (data->auth_type != NVME_AUTH_DHCHAP_MESSAGES ||
+	    data->auth_id != expected_msg) {
+		dev_warn(ctrl->device,
+			 "qid %d invalid message %02x/%02x\n",
+			 qid, data->auth_type, data->auth_id);
+		return NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
+	}
+	if (le16_to_cpu(data->t_id) != transaction) {
+		dev_warn(ctrl->device,
+			 "qid %d invalid transaction ID %d\n",
+			 qid, le16_to_cpu(data->t_id));
+		return NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
+	}
+
+	return 0;
+}
+
+static int nvme_auth_dhchap_negotiate(struct nvme_ctrl *ctrl,
+				      struct nvme_dhchap_context *chap,
+				      void *buf, size_t buf_size)
+{
+	struct nvmf_auth_dhchap_negotiate_data *data = buf;
+	size_t size = sizeof(*data) + sizeof(union nvmf_auth_protocol);
+
+	if (buf_size < size)
+		return -EINVAL;
+
+	memset((u8 *)buf, 0, size);
+	data->auth_type = NVME_AUTH_COMMON_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE;
+	data->t_id = cpu_to_le16(chap->transaction);
+	data->sc_c = 0; /* No secure channel concatenation */
+	data->napd = 1;
+	data->auth_protocol[0].dhchap.authid = NVME_AUTH_DHCHAP_AUTH_ID;
+	data->auth_protocol[0].dhchap.halen = 3;
+	data->auth_protocol[0].dhchap.dhlen = 1;
+	data->auth_protocol[0].dhchap.idlist[0] = NVME_AUTH_DHCHAP_HASH_SHA256;
+	data->auth_protocol[0].dhchap.idlist[1] = NVME_AUTH_DHCHAP_HASH_SHA384;
+	data->auth_protocol[0].dhchap.idlist[2] = NVME_AUTH_DHCHAP_HASH_SHA512;
+	data->auth_protocol[0].dhchap.idlist[3] = NVME_AUTH_DHCHAP_DHGROUP_NULL;
+
+	return size;
+}
+
+static int nvme_auth_dhchap_challenge(struct nvme_ctrl *ctrl,
+				      struct nvme_dhchap_context *chap,
+				      void *buf, size_t buf_size)
+{
+	struct nvmf_auth_dhchap_challenge_data *data = buf;
+	size_t size = sizeof(*data) + data->hl + data->dhvlen;
+	const char *gid_name;
+
+	if (buf_size < size) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
+		return -ENOMSG;
+	}
+
+	if (data->hashid != NVME_AUTH_DHCHAP_HASH_SHA256 &&
+	    data->hashid != NVME_AUTH_DHCHAP_HASH_SHA384 &&
+	    data->hashid != NVME_AUTH_DHCHAP_HASH_SHA512) {
+		dev_warn(ctrl->device,
+			 "qid %d: DH-HMAC-CHAP: invalid HASH ID %d\n",
+			 chap->qid, data->hashid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
+		return -EPROTO;
+	}
+	switch (data->dhgid) {
+	case NVME_AUTH_DHCHAP_DHGROUP_NULL:
+		gid_name = "null";
+		break;
+	default:
+		gid_name = NULL;
+		break;
+	}
+	if (!gid_name) {
+		dev_warn(ctrl->device,
+			 "qid %d: DH-HMAC-CHAP: invalid DH group id %d\n",
+			 chap->qid, data->dhgid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
+		return -EPROTO;
+	}
+	if (data->dhgid != NVME_AUTH_DHCHAP_DHGROUP_NULL) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
+		return -EPROTO;
+	}
+	if (data->dhgid == NVME_AUTH_DHCHAP_DHGROUP_NULL && data->dhvlen != 0) {
+		dev_warn(ctrl->device,
+			 "qid %d: DH-HMAC-CHAP: invalid DH value for NULL DH\n",
+			chap->qid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
+		return -EPROTO;
+	}
+	dev_dbg(ctrl->device, "%s: qid %d requested hash id %d\n",
+		__func__, chap->qid, data->hashid);
+	if (nvme_auth_hmac_len(data->hashid) != data->hl) {
+		dev_warn(ctrl->device,
+			 "qid %d: DH-HMAC-CHAP: invalid hash length\n",
+			chap->qid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
+		return -EPROTO;
+	}
+	chap->hash_id = data->hashid;
+	chap->hash_len = data->hl;
+	chap->s1 = le32_to_cpu(data->seqnum);
+	memcpy(chap->c1, data->cval, chap->hash_len);
+
+	return 0;
+}
+
+static int nvme_auth_dhchap_reply(struct nvme_ctrl *ctrl,
+				  struct nvme_dhchap_context *chap,
+				  void *buf, size_t buf_size)
+{
+	struct nvmf_auth_dhchap_reply_data *data = buf;
+	size_t size = sizeof(*data);
+
+	size += 2 * chap->hash_len;
+	if (ctrl->opts->dhchap_auth) {
+		get_random_bytes(chap->c2, chap->hash_len);
+		chap->s2 = nvme_dhchap_seqnum++;
+	} else
+		memset(chap->c2, 0, chap->hash_len);
+
+	if (chap->host_key_len)
+		size += chap->host_key_len;
+
+	if (buf_size < size)
+		return -EINVAL;
+
+	memset(buf, 0, size);
+	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_REPLY;
+	data->t_id = cpu_to_le16(chap->transaction);
+	data->hl = chap->hash_len;
+	data->dhvlen = chap->host_key_len;
+	data->seqnum = cpu_to_le32(chap->s2);
+	memcpy(data->rval, chap->response, chap->hash_len);
+	if (ctrl->opts->dhchap_auth) {
+		dev_dbg(ctrl->device, "%s: qid %d ctrl challenge %*ph\n",
+			__func__, chap->qid,
+			chap->hash_len, chap->c2);
+		data->cvalid = 1;
+		memcpy(data->rval + chap->hash_len, chap->c2,
+		       chap->hash_len);
+	}
+	if (chap->host_key_len)
+		memcpy(data->rval + 2 * chap->hash_len, chap->host_key,
+		       chap->host_key_len);
+
+	return size;
+}
+
+static int nvme_auth_dhchap_success1(struct nvme_ctrl *ctrl,
+				     struct nvme_dhchap_context *chap,
+				     void *buf, size_t buf_size)
+{
+	struct nvmf_auth_dhchap_success1_data *data = buf;
+	size_t size = sizeof(*data);
+
+	if (ctrl->opts->dhchap_auth)
+		size += chap->hash_len;
+
+
+	if (buf_size < size) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
+		return -ENOMSG;
+	}
+
+	if (data->hl != chap->hash_len) {
+		dev_warn(ctrl->device,
+			 "qid %d: DH-HMAC-CHAP: invalid hash length %d\n",
+			 chap->qid, data->hl);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE;
+		return -EPROTO;
+	}
+
+	if (!data->rvalid)
+		return 0;
+
+	/* Validate controller response */
+	if (memcmp(chap->response, data->rval, data->hl)) {
+		dev_dbg(ctrl->device, "%s: qid %d ctrl response %*ph\n",
+			__func__, chap->qid, chap->hash_len, data->rval);
+		dev_dbg(ctrl->device, "%s: qid %d host response %*ph\n",
+			__func__, chap->qid, chap->hash_len, chap->response);
+		dev_warn(ctrl->device,
+			 "qid %d: DH-HMAC-CHAP: controller authentication failed\n",
+			 chap->qid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_INVALID_PAYLOAD;
+		return -EPROTO;
+	}
+	dev_info(ctrl->device,
+		 "qid %d: DH-HMAC-CHAP: controller authenticated\n",
+		chap->qid);
+	return 0;
+}
+
+static int nvme_auth_dhchap_success2(struct nvme_ctrl *ctrl,
+				     struct nvme_dhchap_context *chap,
+				     void *buf, size_t buf_size)
+{
+	struct nvmf_auth_dhchap_success2_data *data = buf;
+	size_t size = sizeof(*data);
+
+	memset(buf, 0, size);
+	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_SUCCESS2;
+	data->t_id = cpu_to_le16(chap->transaction);
+
+	return size;
+}
+
+static int nvme_auth_dhchap_failure2(struct nvme_ctrl *ctrl,
+				     struct nvme_dhchap_context *chap,
+				     void *buf, size_t buf_size)
+{
+	struct nvmf_auth_dhchap_failure_data *data = buf;
+	size_t size = sizeof(*data);
+
+	memset(buf, 0, size);
+	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
+	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_FAILURE2;
+	data->t_id = cpu_to_le16(chap->transaction);
+	data->reason_code = 1;
+	data->reason_code_explanation = chap->status;
+
+	return size;
+}
+
+int nvme_auth_select_hash(struct nvme_ctrl *ctrl,
+			  struct nvme_dhchap_context *chap)
+{
+	char *hash_name;
+	int ret;
+
+	switch (chap->hash_id) {
+	case NVME_AUTH_DHCHAP_HASH_SHA256:
+		hash_name = "hmac(sha256)";
+		break;
+	case NVME_AUTH_DHCHAP_HASH_SHA384:
+		hash_name = "hmac(sha384)";
+		break;
+	case NVME_AUTH_DHCHAP_HASH_SHA512:
+		hash_name = "hmac(sha512)";
+		break;
+	default:
+		hash_name = NULL;
+		break;
+	}
+	if (!hash_name) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
+		return -EPROTO;
+	}
+	chap->shash_tfm = crypto_alloc_shash(hash_name, 0,
+					     CRYPTO_ALG_ALLOCATES_MEMORY);
+	if (IS_ERR(chap->shash_tfm)) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
+		chap->shash_tfm = NULL;
+		return -EPROTO;
+	}
+	if (!chap->key) {
+		dev_warn(ctrl->device, "qid %d: cannot select hash, no key\n",
+			 chap->qid);
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
+		crypto_free_shash(chap->shash_tfm);
+		chap->shash_tfm = NULL;
+		return -EINVAL;
+	}
+	ret = crypto_shash_setkey(chap->shash_tfm, chap->key, chap->key_len);
+	if (ret) {
+		chap->status = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
+		crypto_free_shash(chap->shash_tfm);
+		chap->shash_tfm = NULL;
+		return ret;
+	}
+	dev_info(ctrl->device, "qid %d: DH-HMAC_CHAP: selected hash %s\n",
+		 chap->qid, hash_name);
+	return 0;
+}
+
+static int nvme_auth_dhchap_host_response(struct nvme_ctrl *ctrl,
+					  struct nvme_dhchap_context *chap)
+{
+	SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
+	u8 buf[4], *challenge = chap->c1;
+	int ret;
+
+	dev_dbg(ctrl->device, "%s: qid %d host response seq %d transaction %d\n",
+		__func__, chap->qid, chap->s1, chap->transaction);
+	shash->tfm = chap->shash_tfm;
+	ret = crypto_shash_init(shash);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, challenge, chap->hash_len);
+	if (ret)
+		goto out;
+	put_unaligned_le32(chap->s1, buf);
+	ret = crypto_shash_update(shash, buf, 4);
+	if (ret)
+		goto out;
+	put_unaligned_le16(chap->transaction, buf);
+	ret = crypto_shash_update(shash, buf, 2);
+	if (ret)
+		goto out;
+	memset(buf, 0, sizeof(buf));
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, "HostHost", 8);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
+				  strlen(ctrl->opts->host->nqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
+			    strlen(ctrl->opts->subsysnqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_final(shash, chap->response);
+out:
+	return ret;
+}
+
+static int nvme_auth_dhchap_ctrl_response(struct nvme_ctrl *ctrl,
+					  struct nvme_dhchap_context *chap)
+{
+	SHASH_DESC_ON_STACK(shash, chap->shash_tfm);
+	u8 buf[4], *challenge = chap->c2;
+	int ret;
+
+	dev_dbg(ctrl->device, "%s: qid %d host response seq %d transaction %d\n",
+		__func__, chap->qid, chap->s2, chap->transaction);
+	dev_dbg(ctrl->device, "%s: qid %d challenge %*ph\n",
+		__func__, chap->qid, chap->hash_len, challenge);
+	dev_dbg(ctrl->device, "%s: qid %d subsysnqn %s\n",
+		__func__, chap->qid, ctrl->opts->subsysnqn);
+	dev_dbg(ctrl->device, "%s: qid %d hostnqn %s\n",
+		__func__, chap->qid, ctrl->opts->host->nqn);
+	shash->tfm = chap->shash_tfm;
+	ret = crypto_shash_init(shash);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, challenge, chap->hash_len);
+	if (ret)
+		goto out;
+	put_unaligned_le32(chap->s2, buf);
+	ret = crypto_shash_update(shash, buf, 4);
+	if (ret)
+		goto out;
+	put_unaligned_le16(chap->transaction, buf);
+	ret = crypto_shash_update(shash, buf, 2);
+	if (ret)
+		goto out;
+	memset(buf, 0, 4);
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, "Controller", 10);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->opts->subsysnqn,
+				  strlen(ctrl->opts->subsysnqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, buf, 1);
+	if (ret)
+		goto out;
+	ret = crypto_shash_update(shash, ctrl->opts->host->nqn,
+				  strlen(ctrl->opts->host->nqn));
+	if (ret)
+		goto out;
+	ret = crypto_shash_final(shash, chap->response);
+out:
+	return ret;
+}
+
+int nvme_auth_generate_key(struct nvme_ctrl *ctrl,
+			   struct nvme_dhchap_context *chap)
+{
+	int ret;
+	u8 key_hash;
+	const char *hmac_name;
+	struct crypto_shash *key_tfm;
+
+	if (sscanf(ctrl->opts->dhchap_secret, "DHHC-1:%hhd:%*s:",
+		   &key_hash) != 1)
+		return -EINVAL;
+
+	chap->key = nvme_auth_extract_secret(ctrl->opts->dhchap_secret,
+					     &chap->key_len);
+	if (IS_ERR(chap->key)) {
+		ret = PTR_ERR(chap->key);
+		chap->key = NULL;
+		return ret;
+	}
+
+	if (key_hash == 0)
+		return 0;
+
+	hmac_name = nvme_auth_hmac_name(key_hash);
+	if (!hmac_name) {
+		pr_debug("Invalid key hash id %d\n", key_hash);
+		return -EKEYREJECTED;
+	}
+
+	key_tfm = crypto_alloc_shash(hmac_name, 0, 0);
+	if (IS_ERR(key_tfm)) {
+		kfree(chap->key);
+		chap->key = NULL;
+		ret = PTR_ERR(key_tfm);
+	} else {
+		SHASH_DESC_ON_STACK(shash, key_tfm);
+
+		shash->tfm = key_tfm;
+		ret = crypto_shash_setkey(key_tfm, chap->key,
+					  chap->key_len);
+		if (ret < 0) {
+			crypto_free_shash(key_tfm);
+			kfree(chap->key);
+			chap->key = NULL;
+			return ret;
+		}
+		crypto_shash_init(shash);
+		crypto_shash_update(shash, ctrl->opts->host->nqn,
+				    strlen(ctrl->opts->host->nqn));
+		crypto_shash_update(shash, "NVMe-over-Fabrics", 17);
+		crypto_shash_final(shash, chap->key);
+		crypto_free_shash(key_tfm);
+	}
+	return 0;
+}
+
+void nvme_auth_free(struct nvme_dhchap_context *chap)
+{
+	if (chap->shash_tfm)
+		crypto_free_shash(chap->shash_tfm);
+	if (chap->key)
+		kfree(chap->key);
+	if (chap->ctrl_key)
+		kfree(chap->ctrl_key);
+	if (chap->host_key)
+		kfree(chap->host_key);
+	if (chap->sess_key)
+		kfree(chap->sess_key);
+	kfree(chap);
+}
+
+int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid)
+{
+	struct nvme_dhchap_context *chap;
+	void *buf;
+	size_t buf_size, tl;
+	int ret = 0;
+
+	chap = kzalloc(sizeof(*chap), GFP_KERNEL);
+	if (!chap)
+		return -ENOMEM;
+	chap->qid = qid;
+	chap->transaction = ctrl->transaction++;
+
+	ret = nvme_auth_generate_key(ctrl, chap);
+	if (ret) {
+		dev_dbg(ctrl->device, "%s: failed to generate key, error %d\n",
+			__func__, ret);
+		nvme_auth_free(chap);
+		return ret;
+	}
+
+	/*
+	 * Allocate a large enough buffer for the entire negotiation:
+	 * 4k should be enough to ffdhe8192.
+	 */
+	buf_size = 4096;
+	buf = kzalloc(buf_size, GFP_KERNEL);
+	if (!buf) {
+		ret = -ENOMEM;
+		goto out;
+	}
+
+	/* DH-HMAC-CHAP Step 1: send negotiate */
+	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP negotiate\n",
+		__func__, qid);
+	ret = nvme_auth_dhchap_negotiate(ctrl, chap, buf, buf_size);
+	if (ret < 0)
+		goto out;
+	tl = ret;
+	ret = nvme_auth_send(ctrl, qid, buf, tl);
+	if (ret)
+		goto out;
+
+	memset(buf, 0, buf_size);
+	ret = nvme_auth_receive(ctrl, qid, buf, buf_size, chap->transaction,
+				NVME_AUTH_DHCHAP_MESSAGE_CHALLENGE);
+	if (ret < 0) {
+		dev_dbg(ctrl->device,
+			"%s: qid %d DH-HMAC-CHAP failed to receive challenge\n",
+			__func__, qid);
+		goto out;
+	}
+	if (ret > 0) {
+		chap->status = ret;
+		goto fail1;
+	}
+
+	/* DH-HMAC-CHAP Step 2: receive challenge */
+	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP challenge\n",
+		__func__, qid);
+
+	ret = nvme_auth_dhchap_challenge(ctrl, chap, buf, buf_size);
+	if (ret) {
+		/* Invalid parameters for negotiate */
+		goto fail2;
+	}
+
+	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP select hash\n",
+		__func__, qid);
+	ret = nvme_auth_select_hash(ctrl, chap);
+	if (ret)
+		goto fail2;
+
+	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP host response\n",
+		__func__, qid);
+	ret = nvme_auth_dhchap_host_response(ctrl, chap);
+	if (ret)
+		goto fail2;
+
+	/* DH-HMAC-CHAP Step 3: send reply */
+	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP reply\n",
+		__func__, qid);
+	ret = nvme_auth_dhchap_reply(ctrl, chap, buf, buf_size);
+	if (ret < 0)
+		goto fail2;
+
+	tl = ret;
+	ret = nvme_auth_send(ctrl, qid, buf, tl);
+	if (ret)
+		goto fail2;
+
+	memset(buf, 0, buf_size);
+	ret = nvme_auth_receive(ctrl, qid, buf, buf_size, chap->transaction,
+				NVME_AUTH_DHCHAP_MESSAGE_SUCCESS1);
+	if (ret < 0) {
+		dev_dbg(ctrl->device,
+			"%s: qid %d DH-HMAC-CHAP failed to receive success1\n",
+			__func__, qid);
+		goto out;
+	}
+	if (ret > 0) {
+		chap->status = ret;
+		goto fail1;
+	}
+
+	if (ctrl->opts->dhchap_auth) {
+		dev_dbg(ctrl->device,
+			"%s: qid %d DH-HMAC-CHAP controller response\n",
+			__func__, qid);
+		ret = nvme_auth_dhchap_ctrl_response(ctrl, chap);
+		if (ret)
+			goto fail2;
+	}
+
+	/* DH-HMAC-CHAP Step 4: receive success1 */
+	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP success1\n",
+		__func__, qid);
+	ret = nvme_auth_dhchap_success1(ctrl, chap, buf, buf_size);
+	if (ret < 0) {
+		/* Controller authentication failed */
+		goto fail2;
+	}
+	tl = ret;
+	/* DH-HMAC-CHAP Step 5: send success2 */
+	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP success2\n",
+		__func__, qid);
+	tl = nvme_auth_dhchap_success2(ctrl, chap, buf, buf_size);
+	ret = nvme_auth_send(ctrl, qid, buf, tl);
+	if (!ret)
+		goto out;
+
+fail1:
+	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP failure1, status %x\n",
+		__func__, qid, chap->status);
+	goto out;
+
+fail2:
+	dev_dbg(ctrl->device, "%s: qid %d DH-HMAC-CHAP failure2, status %x\n",
+		__func__, qid, chap->status);
+	tl = nvme_auth_dhchap_failure2(ctrl, chap, buf, buf_size);
+	ret = nvme_auth_send(ctrl, qid, buf, tl);
+
+out:
+	if (!ret && chap->status)
+		ret = -EPROTO;
+	if (!ret) {
+		ctrl->dhchap_hash = chap->hash_id;
+	}
+	kfree(buf);
+	nvme_auth_free(chap);
+	return ret;
+}
diff --git a/drivers/nvme/host/auth.h b/drivers/nvme/host/auth.h
new file mode 100644
index 000000000000..4950b1cb9470
--- /dev/null
+++ b/drivers/nvme/host/auth.h
@@ -0,0 +1,23 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2021 Hannes Reinecke, SUSE Software Solutions
+ */
+
+#ifndef _NVME_AUTH_H
+#define _NVME_AUTH_H
+
+const char *nvme_auth_dhgroup_name(int dhgroup_id);
+int nvme_auth_dhgroup_pubkey_size(int dhgroup_id);
+int nvme_auth_dhgroup_privkey_size(int dhgroup_id);
+const char *nvme_auth_dhgroup_kpp(int dhgroup_id);
+int nvme_auth_dhgroup_id(const char *dhgroup_name);
+
+const char *nvme_auth_hmac_name(int hmac_id);
+const char *nvme_auth_digest_name(int hmac_id);
+int nvme_auth_hmac_id(const char *hmac_name);
+int nvme_auth_hmac_len(int hmac_len);
+
+unsigned char *nvme_auth_extract_secret(unsigned char *dhchap_secret,
+					size_t *dhchap_key_len);
+
+#endif /* _NVME_AUTH_H */
diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 11779be42186..7ce9b666dc09 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -708,7 +708,9 @@ bool __nvme_check_ready(struct nvme_ctrl *ctrl, struct request *rq,
 		switch (ctrl->state) {
 		case NVME_CTRL_CONNECTING:
 			if (blk_rq_is_passthrough(rq) && nvme_is_fabrics(req->cmd) &&
-			    req->cmd->fabrics.fctype == nvme_fabrics_type_connect)
+			    (req->cmd->fabrics.fctype == nvme_fabrics_type_connect ||
+			     req->cmd->fabrics.fctype == nvme_fabrics_type_auth_send ||
+			     req->cmd->fabrics.fctype == nvme_fabrics_type_auth_receive))
 				return true;
 			break;
 		default:
@@ -3426,6 +3428,66 @@ static ssize_t nvme_ctrl_fast_io_fail_tmo_store(struct device *dev,
 static DEVICE_ATTR(fast_io_fail_tmo, S_IRUGO | S_IWUSR,
 	nvme_ctrl_fast_io_fail_tmo_show, nvme_ctrl_fast_io_fail_tmo_store);
 
+#ifdef CONFIG_NVME_AUTH
+struct nvmet_dhchap_hash_map {
+	int id;
+	const char name[15];
+} hash_map[] = {
+	{.id = NVME_AUTH_DHCHAP_HASH_SHA256,
+	 .name = "hmac(sha256)", },
+	{.id = NVME_AUTH_DHCHAP_HASH_SHA384,
+	 .name = "hmac(sha384)", },
+	{.id = NVME_AUTH_DHCHAP_HASH_SHA512,
+	 .name = "hmac(sha512)", },
+};
+
+static ssize_t dhchap_hash_show(struct device *dev,
+	struct device_attribute *attr, char *buf)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(hash_map); i++) {
+		if (hash_map[i].id == ctrl->dhchap_hash)
+			return sprintf(buf, "%s\n", hash_map[i].name);
+	}
+	return sprintf(buf, "none\n");
+}
+DEVICE_ATTR_RO(dhchap_hash);
+
+struct nvmet_dhchap_group_map {
+	int id;
+	const char name[15];
+} dhgroup_map[] = {
+	{.id = NVME_AUTH_DHCHAP_DHGROUP_NULL,
+	 .name = "NULL", },
+	{.id = NVME_AUTH_DHCHAP_DHGROUP_2048,
+	 .name = "ffdhe2048", },
+	{.id = NVME_AUTH_DHCHAP_DHGROUP_3072,
+	 .name = "ffdhe3072", },
+	{.id = NVME_AUTH_DHCHAP_DHGROUP_4096,
+	 .name = "ffdhe4096", },
+	{.id = NVME_AUTH_DHCHAP_DHGROUP_6144,
+	 .name = "ffdhe6144", },
+	{.id = NVME_AUTH_DHCHAP_DHGROUP_8192,
+	 .name = "ffdhe8192", },
+};
+
+static ssize_t dhchap_dhgroup_show(struct device *dev,
+	struct device_attribute *attr, char *buf)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(dhgroup_map); i++) {
+		if (hash_map[i].id == ctrl->dhchap_dhgroup)
+			return sprintf(buf, "%s\n", dhgroup_map[i].name);
+	}
+	return sprintf(buf, "none\n");
+}
+DEVICE_ATTR_RO(dhchap_dhgroup);
+#endif
+
 static struct attribute *nvme_dev_attrs[] = {
 	&dev_attr_reset_controller.attr,
 	&dev_attr_rescan_controller.attr,
@@ -3447,6 +3509,10 @@ static struct attribute *nvme_dev_attrs[] = {
 	&dev_attr_reconnect_delay.attr,
 	&dev_attr_fast_io_fail_tmo.attr,
 	&dev_attr_kato.attr,
+#ifdef CONFIG_NVME_AUTH
+	&dev_attr_dhchap_hash.attr,
+	&dev_attr_dhchap_dhgroup.attr,
+#endif
 	NULL
 };
 
@@ -3470,6 +3536,10 @@ static umode_t nvme_dev_attrs_are_visible(struct kobject *kobj,
 		return 0;
 	if (a == &dev_attr_fast_io_fail_tmo.attr && !ctrl->opts)
 		return 0;
+#ifdef CONFIG_NVME_AUTH
+	if (a == &dev_attr_dhchap_hash.attr && !ctrl->opts)
+		return 0;
+#endif
 
 	return a->mode;
 }
@@ -4581,6 +4651,11 @@ static inline void _nvme_check_size(void)
 	BUILD_BUG_ON(sizeof(struct nvme_smart_log) != 512);
 	BUILD_BUG_ON(sizeof(struct nvme_dbbuf) != 64);
 	BUILD_BUG_ON(sizeof(struct nvme_directive_cmd) != 64);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_negotiate_data) != 8);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_challenge_data) != 16);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_reply_data) != 16);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_success1_data) != 16);
+	BUILD_BUG_ON(sizeof(struct nvmf_auth_dhchap_success2_data) != 16);
 }
 
 
diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index a5469fd9d4c3..6404ab9b604b 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -366,6 +366,7 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)
 	union nvme_result res;
 	struct nvmf_connect_data *data;
 	int ret;
+	u32 result;
 
 	cmd.connect.opcode = nvme_fabrics_command;
 	cmd.connect.fctype = nvme_fabrics_type_connect;
@@ -398,8 +399,18 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)
 		goto out_free_data;
 	}
 
-	ctrl->cntlid = le16_to_cpu(res.u16);
-
+	result = le32_to_cpu(res.u32);
+	ctrl->cntlid = result & 0xFFFF;
+	if ((result >> 16) & 2) {
+		/* Authentication required */
+		ret = nvme_auth_negotiate(ctrl, NVME_QID_ANY);
+		if (ret)
+			dev_warn(ctrl->device,
+				 "qid 0: authentication failed\n");
+		else
+			dev_info(ctrl->device,
+				 "qid 0: authenticated\n");
+	}
 out_free_data:
 	kfree(data);
 	return ret;
@@ -432,6 +443,7 @@ int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, u16 qid)
 	struct nvmf_connect_data *data;
 	union nvme_result res;
 	int ret;
+	u32 result;
 
 	cmd.connect.opcode = nvme_fabrics_command;
 	cmd.connect.fctype = nvme_fabrics_type_connect;
@@ -457,6 +469,17 @@ int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, u16 qid)
 		nvmf_log_connect_error(ctrl, ret, le32_to_cpu(res.u32),
 				       &cmd, data);
 	}
+	result = le32_to_cpu(res.u32);
+	if ((result >> 16) & 2) {
+		/* Authentication required */
+		ret = nvme_auth_negotiate(ctrl, qid);
+		if (ret)
+			dev_warn(ctrl->device,
+				 "qid %u: authentication failed\n", qid);
+		else
+			dev_info(ctrl->device,
+				 "qid %u: authenticated\n", qid);
+	}
 	kfree(data);
 	return ret;
 }
@@ -548,6 +571,9 @@ static const match_table_t opt_tokens = {
 	{ NVMF_OPT_NR_POLL_QUEUES,	"nr_poll_queues=%d"	},
 	{ NVMF_OPT_TOS,			"tos=%d"		},
 	{ NVMF_OPT_FAIL_FAST_TMO,	"fast_io_fail_tmo=%d"	},
+	{ NVMF_OPT_DHCHAP_SECRET,	"dhchap_secret=%s"	},
+	{ NVMF_OPT_DHCHAP_AUTH,		"authenticate"		},
+	{ NVMF_OPT_DHCHAP_GROUP,	"dhchap_group=%s"	},
 	{ NVMF_OPT_ERR,			NULL			}
 };
 
@@ -824,6 +850,35 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
 			}
 			opts->tos = token;
 			break;
+		case NVMF_OPT_DHCHAP_SECRET:
+			p = match_strdup(args);
+			if (!p) {
+				ret = -ENOMEM;
+				goto out;
+			}
+			if (strncmp(p, "DHHC-1:00:", 10)) {
+				pr_err("Invalid DH-CHAP secret %s\n", p);
+				ret = -EINVAL;
+				goto out;
+			}
+			kfree(opts->dhchap_secret);
+			opts->dhchap_secret = p;
+			break;
+		case NVMF_OPT_DHCHAP_AUTH:
+			opts->dhchap_auth = true;
+			break;
+		case NVMF_OPT_DHCHAP_GROUP:
+			if (match_int(args, &token)) {
+				ret = -EINVAL;
+				goto out;
+			}
+			if (token <= 0) {
+				pr_err("Invalid dhchap_group %d\n", token);
+				ret = -EINVAL;
+				goto out;
+			}
+			opts->dhchap_group = token;
+			break;
 		default:
 			pr_warn("unknown parameter or missing value '%s' in ctrl creation request\n",
 				p);
@@ -942,6 +997,7 @@ void nvmf_free_options(struct nvmf_ctrl_options *opts)
 	kfree(opts->subsysnqn);
 	kfree(opts->host_traddr);
 	kfree(opts->host_iface);
+	kfree(opts->dhchap_secret);
 	kfree(opts);
 }
 EXPORT_SYMBOL_GPL(nvmf_free_options);
@@ -951,7 +1007,10 @@ EXPORT_SYMBOL_GPL(nvmf_free_options);
 				 NVMF_OPT_KATO | NVMF_OPT_HOSTNQN | \
 				 NVMF_OPT_HOST_ID | NVMF_OPT_DUP_CONNECT |\
 				 NVMF_OPT_DISABLE_SQFLOW |\
-				 NVMF_OPT_FAIL_FAST_TMO)
+				 NVMF_OPT_CTRL_LOSS_TMO |\
+				 NVMF_OPT_FAIL_FAST_TMO |\
+				 NVMF_OPT_DHCHAP_SECRET |\
+				 NVMF_OPT_DHCHAP_AUTH | NVMF_OPT_DHCHAP_GROUP)
 
 static struct nvme_ctrl *
 nvmf_create_ctrl(struct device *dev, const char *buf)
diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h
index a146cb903869..535bc544f0f6 100644
--- a/drivers/nvme/host/fabrics.h
+++ b/drivers/nvme/host/fabrics.h
@@ -67,6 +67,9 @@ enum {
 	NVMF_OPT_TOS		= 1 << 19,
 	NVMF_OPT_FAIL_FAST_TMO	= 1 << 20,
 	NVMF_OPT_HOST_IFACE	= 1 << 21,
+	NVMF_OPT_DHCHAP_SECRET	= 1 << 22,
+	NVMF_OPT_DHCHAP_AUTH	= 1 << 23,
+	NVMF_OPT_DHCHAP_GROUP	= 1 << 24,
 };
 
 /**
@@ -96,6 +99,8 @@ enum {
  * @max_reconnects: maximum number of allowed reconnect attempts before removing
  *              the controller, (-1) means reconnect forever, zero means remove
  *              immediately;
+ * @dhchap_secret: DH-HMAC-CHAP secret
+ * @dhchap_auth: DH-HMAC-CHAP authenticate controller
  * @disable_sqflow: disable controller sq flow control
  * @hdr_digest: generate/verify header digest (TCP)
  * @data_digest: generate/verify data digest (TCP)
@@ -120,6 +125,9 @@ struct nvmf_ctrl_options {
 	unsigned int		kato;
 	struct nvmf_host	*host;
 	int			max_reconnects;
+	char			*dhchap_secret;
+	int			dhchap_group;
+	bool			dhchap_auth;
 	bool			disable_sqflow;
 	bool			hdr_digest;
 	bool			data_digest;
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 18ef8dd03a90..bcd5b8276c26 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -328,6 +328,12 @@ struct nvme_ctrl {
 	struct work_struct ana_work;
 #endif
 
+#ifdef CONFIG_NVME_AUTH
+	u16 transaction;
+	u8 dhchap_hash;
+	u8 dhchap_dhgroup;
+#endif
+
 	/* Power saving configuration */
 	u64 ps_max_latency_us;
 	bool apst_enabled;
@@ -874,6 +880,15 @@ static inline bool nvme_ctrl_sgl_supported(struct nvme_ctrl *ctrl)
 	return ctrl->sgls & ((1 << 0) | (1 << 1));
 }
 
+#ifdef CONFIG_NVME_AUTH
+int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid);
+#else
+static inline int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid)
+{
+	return -EPROTONOSUPPORT;
+}
+#endif
+
 u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 			 u8 opcode);
 int nvme_execute_passthru_rq(struct request *rq);
diff --git a/drivers/nvme/host/trace.c b/drivers/nvme/host/trace.c
index 6543015b6121..66f75d8ea925 100644
--- a/drivers/nvme/host/trace.c
+++ b/drivers/nvme/host/trace.c
@@ -271,6 +271,34 @@ static const char *nvme_trace_fabrics_property_get(struct trace_seq *p, u8 *spc)
 	return ret;
 }
 
+static const char *nvme_trace_fabrics_auth_send(struct trace_seq *p, u8 *spc)
+{
+	const char *ret = trace_seq_buffer_ptr(p);
+	u8 spsp0 = spc[1];
+	u8 spsp1 = spc[2];
+	u8 secp = spc[3];
+	u32 tl = get_unaligned_le32(spc + 4);
+
+	trace_seq_printf(p, "spsp0=%02x, spsp1=%02x, secp=%02x, tl=%u",
+			 spsp0, spsp1, secp, tl);
+	trace_seq_putc(p, 0);
+	return ret;
+}
+
+static const char *nvme_trace_fabrics_auth_receive(struct trace_seq *p, u8 *spc)
+{
+	const char *ret = trace_seq_buffer_ptr(p);
+	u8 spsp0 = spc[1];
+	u8 spsp1 = spc[2];
+	u8 secp = spc[3];
+	u32 al = get_unaligned_le32(spc + 4);
+
+	trace_seq_printf(p, "spsp0=%02x, spsp1=%02x, secp=%02x, al=%u",
+			 spsp0, spsp1, secp, al);
+	trace_seq_putc(p, 0);
+	return ret;
+}
+
 static const char *nvme_trace_fabrics_common(struct trace_seq *p, u8 *spc)
 {
 	const char *ret = trace_seq_buffer_ptr(p);
@@ -290,6 +318,10 @@ const char *nvme_trace_parse_fabrics_cmd(struct trace_seq *p,
 		return nvme_trace_fabrics_connect(p, spc);
 	case nvme_fabrics_type_property_get:
 		return nvme_trace_fabrics_property_get(p, spc);
+	case nvme_fabrics_type_auth_send:
+		return nvme_trace_fabrics_auth_send(p, spc);
+	case nvme_fabrics_type_auth_receive:
+		return nvme_trace_fabrics_auth_receive(p, spc);
 	default:
 		return nvme_trace_fabrics_common(p, spc);
 	}
-- 
2.29.2


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme