[PATCH rdma-next 00/13] Add RDMA inline crypto support

[PATCH rdma-next 00/13] Add RDMA inline crypto support

[Leon Romanovsky]

From Israel,

The purpose of this patchset is to add support for inline
encryption/decryption of the data at storage protocols like nvmf over
RDMA (at a similar way like integrity is used via unique mkey).

This patchset adds support for plaintext keys. The patches were tested
on BF-3 HW with fscrypt tool to test this feature, which showed reduce
in CPU utilization when comparing at 64k or more IO size. The CPU utilization
was improved by more than 50% comparing to the SW only solution at this case.

How to configure fscrypt to enable plaintext keys:
 # mkfs.ext4 -O encrypt /dev/nvme0n1
 # mount /dev/nvme0n1 /mnt/crypto -o inlinecrypt
 # head -c 64 /dev/urandom > /tmp/master_key
 # fscryptctl add_key /mnt/crypto/ < /tmp/master_key
 # mkdir /mnt/crypto/test1
 # fscryptctl set_policy 152c41b2ea39fa3d90ea06448456e7fb /mnt/crypto/test1
   ** “152c41b2ea39fa3d90ea06448456e7fb” is the output of the
      “fscryptctl add_key” command.
 # echo foo > /mnt/crypto/test1/foo

Notes:
 - At plaintext mode only, the user set a master key and the fscrypt
   driver derived from it the DEK and the key identifier.
 - 152c41b2ea39fa3d90ea06448456e7fb is the derived key identifier
 - Only on the first IO, nvme-rdma gets a callback to load the derived DEK. 

There is no special configuration to support crypto at nvme modules.

Thanks

Israel Rukshin (13):
  net/mlx5: Introduce crypto IFC bits and structures
  net/mlx5: Introduce crypto capabilities macro
  RDMA: Split kernel-only create QP flags from uverbs create QP flags
  RDMA/core: Add cryptographic device capabilities
  RDMA/core: Add DEK management API
  RDMA/core: Introduce MR type for crypto operations
  RDMA/core: Add support for creating crypto enabled QPs
  RDMA/mlx5: Add cryptographic device capabilities
  RDMA/mlx5: Add DEK management API
  RDMA/mlx5: Add AES-XTS crypto support
  nvme: Introduce a local variable
  nvme: Add crypto profile at nvme controller
  nvme-rdma: Add inline encryption support

 drivers/infiniband/core/device.c              |   3 +
 drivers/infiniband/core/mr_pool.c             |   2 +
 drivers/infiniband/core/uverbs_std_types_qp.c |  12 +-
 drivers/infiniband/core/verbs.c               |  91 ++++++-
 drivers/infiniband/hw/bnxt_re/ib_verbs.c      |   2 +-
 drivers/infiniband/hw/mlx4/mlx4_ib.h          |   4 +-
 drivers/infiniband/hw/mlx4/qp.c               |   4 +-
 drivers/infiniband/hw/mlx5/Makefile           |   1 +
 drivers/infiniband/hw/mlx5/crypto.c           | 115 +++++++++
 drivers/infiniband/hw/mlx5/crypto.h           |  46 ++++
 drivers/infiniband/hw/mlx5/main.c             |   5 +
 drivers/infiniband/hw/mlx5/mlx5_ib.h          |   5 +-
 drivers/infiniband/hw/mlx5/mr.c               |  33 +++
 drivers/infiniband/hw/mlx5/qp.c               |  15 +-
 drivers/infiniband/hw/mlx5/wr.c               | 164 +++++++++++-
 drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c  |   2 +-
 drivers/net/ethernet/mellanox/mlx5/core/fw.c  |   6 +
 .../net/ethernet/mellanox/mlx5/core/main.c    |   1 +
 drivers/nvme/host/core.c                      |  10 +-
 drivers/nvme/host/nvme.h                      |   4 +
 drivers/nvme/host/rdma.c                      | 236 +++++++++++++++++-
 include/linux/mlx5/device.h                   |   4 +
 include/linux/mlx5/mlx5_ifc.h                 |  36 ++-
 include/rdma/crypto.h                         | 118 +++++++++
 include/rdma/ib_verbs.h                       |  46 +++-
 include/trace/events/rdma_core.h              |  33 +++
 26 files changed, 954 insertions(+), 44 deletions(-)
 create mode 100644 drivers/infiniband/hw/mlx5/crypto.c
 create mode 100644 drivers/infiniband/hw/mlx5/crypto.h
 create mode 100644 include/rdma/crypto.h

-- 
2.39.0

[PATCH mlx5-next 01/13] net/mlx5: Introduce crypto IFC bits and structures

[Leon Romanovsky]

From: Israel Rukshin <israelr@nvidia.com>

Add crypto related IFC structs, layouts and enumerations.

Signed-off-by: Israel Rukshin <israelr@nvidia.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
---
 include/linux/mlx5/mlx5_ifc.h | 36 ++++++++++++++++++++++++++++++++---
 1 file changed, 33 insertions(+), 3 deletions(-)

diff --git a/include/linux/mlx5/mlx5_ifc.h b/include/linux/mlx5/mlx5_ifc.h
index 8bbf15433bb2..170fe1081820 100644
--- a/include/linux/mlx5/mlx5_ifc.h
+++ b/include/linux/mlx5/mlx5_ifc.h
@@ -1331,6 +1331,29 @@ struct mlx5_ifc_macsec_cap_bits {
 	u8    reserved_at_40[0x7c0];
 };
 
+enum {
+	MLX5_CRYPTO_WRAPPED_IMPORT_METHOD_CAP_AES_XTS = 0x4,
+};
+
+struct mlx5_ifc_crypto_cap_bits {
+	u8         wrapped_crypto_operational[0x1];
+	u8         reserved_at_1[0x17];
+	u8         wrapped_import_method[0x8];
+
+	u8         reserved_at_20[0xb];
+	u8         log_max_num_deks[0x5];
+	u8         reserved_at_30[0x3];
+	u8         log_max_num_import_keks[0x5];
+	u8         reserved_at_38[0x3];
+	u8         log_max_num_creds[0x5];
+
+	u8         failed_selftests[0x10];
+	u8         num_nv_import_keks[0x8];
+	u8         num_nv_credentials[0x8];
+
+	u8         reserved_at_60[0x7a0];
+};
+
 enum {
 	MLX5_WQ_TYPE_LINKED_LIST  = 0x0,
 	MLX5_WQ_TYPE_CYCLIC       = 0x1,
@@ -1758,7 +1781,9 @@ struct mlx5_ifc_cmd_hca_cap_bits {
 	u8         reserved_at_3e8[0x2];
 	u8         vhca_state[0x1];
 	u8         log_max_vlan_list[0x5];
-	u8         reserved_at_3f0[0x3];
+	u8         reserved_at_3f0[0x1];
+	u8         aes_xts_single_block_le_tweak[0x1];
+	u8         aes_xts_multi_block_be_tweak[0x1];
 	u8         log_max_current_mc_list[0x5];
 	u8         reserved_at_3f8[0x3];
 	u8         log_max_current_uc_list[0x5];
@@ -1774,7 +1799,8 @@ struct mlx5_ifc_cmd_hca_cap_bits {
 	u8         ats[0x1];
 	u8         reserved_at_462[0x1];
 	u8         log_max_uctx[0x5];
-	u8         reserved_at_468[0x2];
+	u8         aes_xts_multi_block_le_tweak[0x1];
+	u8         crypto[0x1];
 	u8         ipsec_offload[0x1];
 	u8         log_max_umem[0x5];
 	u8         max_num_eqs[0x10];
@@ -3377,6 +3403,7 @@ union mlx5_ifc_hca_cap_union_bits {
 	struct mlx5_ifc_virtio_emulation_cap_bits virtio_emulation_cap;
 	struct mlx5_ifc_shampo_cap_bits shampo_cap;
 	struct mlx5_ifc_macsec_cap_bits macsec_cap;
+	struct mlx5_ifc_crypto_cap_bits crypto_cap;
 	u8         reserved_at_0[0x8000];
 };
 
@@ -3995,7 +4022,9 @@ struct mlx5_ifc_mkc_bits {
 	u8         reserved_at_1d9[0x1];
 	u8         log_page_size[0x5];
 
-	u8         reserved_at_1e0[0x20];
+	u8         reserved_at_1e0[0x3];
+	u8         crypto_en[0x2];
+	u8         reserved_at_1e5[0x1b];
 };
 
 struct mlx5_ifc_pkey_bits {
@@ -11978,6 +12007,7 @@ enum {
 enum {
 	MLX5_GENERAL_OBJECT_TYPE_ENCRYPTION_KEY_TYPE_TLS = 0x1,
 	MLX5_GENERAL_OBJECT_TYPE_ENCRYPTION_KEY_TYPE_IPSEC = 0x2,
+	MLX5_GENERAL_OBJECT_TYPE_ENCRYPTION_KEY_TYPE_AES_XTS = 0x3,
 	MLX5_GENERAL_OBJECT_TYPE_ENCRYPTION_KEY_TYPE_MACSEC = 0x4,
 };
 
-- 
2.39.0

[PATCH mlx5-next 02/13] net/mlx5: Introduce crypto capabilities macro

[Leon Romanovsky]

From: Israel Rukshin <israelr@nvidia.com>

Add MLX5_CAP_CRYPTO() macro to the list of capabilities.

Signed-off-by: Israel Rukshin <israelr@nvidia.com>
Reviewed-by: Max Gurtovoy <mgurtovoy@nvidia.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
---
 drivers/net/ethernet/mellanox/mlx5/core/fw.c   | 6 ++++++
 drivers/net/ethernet/mellanox/mlx5/core/main.c | 1 +
 include/linux/mlx5/device.h                    | 4 ++++
 3 files changed, 11 insertions(+)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fw.c b/drivers/net/ethernet/mellanox/mlx5/core/fw.c
index f34e758a2f1f..4603f7ffd8d6 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/fw.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/fw.c
@@ -147,6 +147,12 @@ int mlx5_query_hca_caps(struct mlx5_core_dev *dev)
 	if (err)
 		return err;
 
+	if (MLX5_CAP_GEN(dev, crypto)) {
+		err = mlx5_core_get_caps(dev, MLX5_CAP_CRYPTO);
+		if (err)
+			return err;
+	}
+
 	if (MLX5_CAP_GEN(dev, port_selection_cap)) {
 		err = mlx5_core_get_caps(dev, MLX5_CAP_PORT_SELECTION);
 		if (err)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/main.c b/drivers/net/ethernet/mellanox/mlx5/core/main.c
index df134f6d32dc..81348a009666 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c
@@ -1555,6 +1555,7 @@ static const int types[] = {
 	MLX5_CAP_DEV_SHAMPO,
 	MLX5_CAP_MACSEC,
 	MLX5_CAP_ADV_VIRTUALIZATION,
+	MLX5_CAP_CRYPTO,
 };
 
 static void mlx5_hca_caps_free(struct mlx5_core_dev *dev)
diff --git a/include/linux/mlx5/device.h b/include/linux/mlx5/device.h
index 29d4b201c7b2..fd095f0ed3ec 100644
--- a/include/linux/mlx5/device.h
+++ b/include/linux/mlx5/device.h
@@ -1204,6 +1204,7 @@ enum mlx5_cap_type {
 	MLX5_CAP_VDPA_EMULATION = 0x13,
 	MLX5_CAP_DEV_EVENT = 0x14,
 	MLX5_CAP_IPSEC,
+	MLX5_CAP_CRYPTO = 0x1a,
 	MLX5_CAP_DEV_SHAMPO = 0x1d,
 	MLX5_CAP_MACSEC = 0x1f,
 	MLX5_CAP_GENERAL_2 = 0x20,
@@ -1466,6 +1467,9 @@ enum mlx5_qcam_feature_groups {
 #define MLX5_CAP_MACSEC(mdev, cap)\
 	MLX5_GET(macsec_cap, (mdev)->caps.hca[MLX5_CAP_MACSEC]->cur, cap)
 
+#define MLX5_CAP_CRYPTO(mdev, cap)\
+	MLX5_GET(crypto_cap, (mdev)->caps.hca[MLX5_CAP_CRYPTO]->cur, cap)
+
 enum {
 	MLX5_CMD_STAT_OK			= 0x0,
 	MLX5_CMD_STAT_INT_ERR			= 0x1,
-- 
2.39.0

[PATCH rdma-next 03/13] RDMA: Split kernel-only create QP flags from uverbs create QP flags

[Leon Romanovsky]

From: Israel Rukshin <israelr@nvidia.com>

The current code limits the create_flags to being the same bitmap that
will be copied to user space. Extend the create QP flags to 64 bits and
move all the flags that are only used internally in the kernel to the
upper 32 bits. This cleanly split out the uverbs flags from the kernel
flags to avoid confusion in the flags bitmap. Also add some short
comments describing which each of the kernel flags is connected to.

Signed-off-by: Israel Rukshin <israelr@nvidia.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
---
 drivers/infiniband/core/uverbs_std_types_qp.c | 12 +++++-----
 drivers/infiniband/hw/bnxt_re/ib_verbs.c      |  2 +-
 drivers/infiniband/hw/mlx4/mlx4_ib.h          |  4 ++--
 drivers/infiniband/hw/mlx4/qp.c               |  4 ++--
 drivers/infiniband/hw/mlx5/mlx5_ib.h          |  2 +-
 drivers/infiniband/hw/mlx5/qp.c               |  9 ++++----
 drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c  |  2 +-
 include/rdma/ib_verbs.h                       | 22 ++++++++++++++-----
 8 files changed, 35 insertions(+), 22 deletions(-)

diff --git a/drivers/infiniband/core/uverbs_std_types_qp.c b/drivers/infiniband/core/uverbs_std_types_qp.c
index 7b4773fa4bc0..fe2c7427eac4 100644
--- a/drivers/infiniband/core/uverbs_std_types_qp.c
+++ b/drivers/infiniband/core/uverbs_std_types_qp.c
@@ -97,6 +97,7 @@ static int UVERBS_HANDLER(UVERBS_METHOD_QP_CREATE)(
 	struct ib_uobject *xrcd_uobj = NULL;
 	struct ib_device *device;
 	u64 user_handle;
+	u32 create_flags;
 	int ret;
 
 	ret = uverbs_copy_from_or_zero(&cap, attrs,
@@ -191,7 +192,7 @@ static int UVERBS_HANDLER(UVERBS_METHOD_QP_CREATE)(
 		return -EINVAL;
 	}
 
-	ret = uverbs_get_flags32(&attr.create_flags, attrs,
+	ret = uverbs_get_flags32(&create_flags, attrs,
 			 UVERBS_ATTR_CREATE_QP_FLAGS,
 			 IB_UVERBS_QP_CREATE_BLOCK_MULTICAST_LOOPBACK |
 			 IB_UVERBS_QP_CREATE_SCATTER_FCS |
@@ -201,7 +202,7 @@ static int UVERBS_HANDLER(UVERBS_METHOD_QP_CREATE)(
 	if (ret)
 		return ret;
 
-	ret = check_creation_flags(attr.qp_type, attr.create_flags);
+	ret = check_creation_flags(attr.qp_type, create_flags);
 	if (ret)
 		return ret;
 
@@ -211,7 +212,7 @@ static int UVERBS_HANDLER(UVERBS_METHOD_QP_CREATE)(
 				       UVERBS_ATTR_CREATE_QP_SOURCE_QPN);
 		if (ret)
 			return ret;
-		attr.create_flags |= IB_QP_CREATE_SOURCE_QPN;
+		create_flags |= IB_QP_CREATE_SOURCE_QPN;
 	}
 
 	srq = uverbs_attr_get_obj(attrs,
@@ -234,16 +235,17 @@ static int UVERBS_HANDLER(UVERBS_METHOD_QP_CREATE)(
 	attr.send_cq = send_cq;
 	attr.recv_cq = recv_cq;
 	attr.xrcd = xrcd;
-	if (attr.create_flags & IB_UVERBS_QP_CREATE_SQ_SIG_ALL) {
+	if (create_flags & IB_UVERBS_QP_CREATE_SQ_SIG_ALL) {
 		/* This creation bit is uverbs one, need to mask before
 		 * calling drivers. It was added to prevent an extra user attr
 		 * only for that when using ioctl.
 		 */
-		attr.create_flags &= ~IB_UVERBS_QP_CREATE_SQ_SIG_ALL;
+		create_flags &= ~IB_UVERBS_QP_CREATE_SQ_SIG_ALL;
 		attr.sq_sig_type = IB_SIGNAL_ALL_WR;
 	} else {
 		attr.sq_sig_type = IB_SIGNAL_REQ_WR;
 	}
+	attr.create_flags = create_flags;
 
 	set_caps(&attr, &cap, true);
 	mutex_init(&obj->mcast_lock);
diff --git a/drivers/infiniband/hw/bnxt_re/ib_verbs.c b/drivers/infiniband/hw/bnxt_re/ib_verbs.c
index 989edc789633..1493ee9ed2b8 100644
--- a/drivers/infiniband/hw/bnxt_re/ib_verbs.c
+++ b/drivers/infiniband/hw/bnxt_re/ib_verbs.c
@@ -1273,7 +1273,7 @@ static int bnxt_re_init_qp_attr(struct bnxt_re_qp *qp, struct bnxt_re_pd *pd,
 	qplqp->dpi = &rdev->dpi_privileged; /* Doorbell page */
 	if (init_attr->create_flags) {
 		ibdev_dbg(&rdev->ibdev,
-			  "QP create flags 0x%x not supported",
+			  "QP create flags 0x%llx not supported",
 			  init_attr->create_flags);
 		return -EOPNOTSUPP;
 	}
diff --git a/drivers/infiniband/hw/mlx4/mlx4_ib.h b/drivers/infiniband/hw/mlx4/mlx4_ib.h
index 17fee1e73a45..c553bf0eb257 100644
--- a/drivers/infiniband/hw/mlx4/mlx4_ib.h
+++ b/drivers/infiniband/hw/mlx4/mlx4_ib.h
@@ -184,7 +184,7 @@ enum mlx4_ib_qp_flags {
 	/* Mellanox specific flags start from IB_QP_CREATE_RESERVED_START */
 	MLX4_IB_ROCE_V2_GSI_QP = MLX4_IB_QP_CREATE_ROCE_V2_GSI,
 	MLX4_IB_SRIOV_TUNNEL_QP = 1 << 30,
-	MLX4_IB_SRIOV_SQP = 1 << 31,
+	MLX4_IB_SRIOV_SQP = 1ULL << 31,
 };
 
 struct mlx4_ib_gid_entry {
@@ -342,7 +342,7 @@ struct mlx4_ib_qp {
 	int			buf_size;
 	struct mutex		mutex;
 	u16			xrcdn;
-	u32			flags;
+	u64			flags;
 	u8			port;
 	u8			alt_port;
 	u8			atomic_rd_en;
diff --git a/drivers/infiniband/hw/mlx4/qp.c b/drivers/infiniband/hw/mlx4/qp.c
index 884825b2e5f7..f3ad436bf5c9 100644
--- a/drivers/infiniband/hw/mlx4/qp.c
+++ b/drivers/infiniband/hw/mlx4/qp.c
@@ -287,7 +287,7 @@ static void mlx4_ib_wq_event(struct mlx4_qp *qp, enum mlx4_event type)
 			    type, qp->qpn);
 }
 
-static int send_wqe_overhead(enum mlx4_ib_qp_type type, u32 flags)
+static int send_wqe_overhead(enum mlx4_ib_qp_type type, u64 flags)
 {
 	/*
 	 * UD WQEs must have a datagram segment.
@@ -1514,7 +1514,7 @@ static int _mlx4_ib_create_qp(struct ib_pd *pd, struct mlx4_ib_qp *qp,
 			      struct ib_udata *udata)
 {
 	int err;
-	int sup_u_create_flags = MLX4_IB_QP_BLOCK_MULTICAST_LOOPBACK;
+	u64 sup_u_create_flags = MLX4_IB_QP_BLOCK_MULTICAST_LOOPBACK;
 	u16 xrcdn = 0;
 
 	if (init_attr->rwq_ind_tbl)
diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h
index ddb36c757074..295502692da2 100644
--- a/drivers/infiniband/hw/mlx5/mlx5_ib.h
+++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h
@@ -500,7 +500,7 @@ struct mlx5_ib_qp {
 	 */
 	struct mutex		mutex;
 	/* cached variant of create_flags from struct ib_qp_init_attr */
-	u32			flags;
+	u64			flags;
 	u32			port;
 	u8			state;
 	int			max_inline_data;
diff --git a/drivers/infiniband/hw/mlx5/qp.c b/drivers/infiniband/hw/mlx5/qp.c
index 80504db92ee9..f04adc18e63b 100644
--- a/drivers/infiniband/hw/mlx5/qp.c
+++ b/drivers/infiniband/hw/mlx5/qp.c
@@ -2938,7 +2938,7 @@ static int process_vendor_flags(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,
 	return (flags) ? -EINVAL : 0;
 	}
 
-static void process_create_flag(struct mlx5_ib_dev *dev, int *flags, int flag,
+static void process_create_flag(struct mlx5_ib_dev *dev, u64 *flags, u64 flag,
 				bool cond, struct mlx5_ib_qp *qp)
 {
 	if (!(*flags & flag))
@@ -2958,7 +2958,8 @@ static void process_create_flag(struct mlx5_ib_dev *dev, int *flags, int flag,
 		*flags &= ~MLX5_IB_QP_CREATE_WC_TEST;
 		return;
 	}
-	mlx5_ib_dbg(dev, "Verbs create QP flag 0x%X is not supported\n", flag);
+	mlx5_ib_dbg(dev, "Verbs create QP flag 0x%llX is not supported\n",
+		    flag);
 }
 
 static int process_create_flags(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,
@@ -2966,7 +2967,7 @@ static int process_create_flags(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,
 {
 	enum ib_qp_type qp_type = qp->type;
 	struct mlx5_core_dev *mdev = dev->mdev;
-	int create_flags = attr->create_flags;
+	u64 create_flags = attr->create_flags;
 	bool cond;
 
 	if (qp_type == MLX5_IB_QPT_DCT)
@@ -3024,7 +3025,7 @@ static int process_create_flags(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,
 			    true, qp);
 
 	if (create_flags) {
-		mlx5_ib_dbg(dev, "Create QP has unsupported flags 0x%X\n",
+		mlx5_ib_dbg(dev, "Create QP has unsupported flags 0x%llX\n",
 			    create_flags);
 		return -EOPNOTSUPP;
 	}
diff --git a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c
index f83cd4a9d992..0dbfc3c9e274 100644
--- a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c
+++ b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c
@@ -206,7 +206,7 @@ int pvrdma_create_qp(struct ib_qp *ibqp, struct ib_qp_init_attr *init_attr,
 
 	if (init_attr->create_flags) {
 		dev_warn(&dev->pdev->dev,
-			 "invalid create queuepair flags %#x\n",
+			 "invalid create queuepair flags 0x%llx\n",
 			 init_attr->create_flags);
 		return -EOPNOTSUPP;
 	}
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 949cf4ffc536..cc2ddd4e6c12 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -1140,16 +1140,15 @@ enum ib_qp_type {
 	IB_QPT_RESERVED10,
 };
 
+/*
+ * bits 0, 5, 6 and 7 may be set by old kernels and should not be used.
+ */
 enum ib_qp_create_flags {
-	IB_QP_CREATE_IPOIB_UD_LSO		= 1 << 0,
 	IB_QP_CREATE_BLOCK_MULTICAST_LOOPBACK	=
 		IB_UVERBS_QP_CREATE_BLOCK_MULTICAST_LOOPBACK,
 	IB_QP_CREATE_CROSS_CHANNEL              = 1 << 2,
 	IB_QP_CREATE_MANAGED_SEND               = 1 << 3,
 	IB_QP_CREATE_MANAGED_RECV               = 1 << 4,
-	IB_QP_CREATE_NETIF_QP			= 1 << 5,
-	IB_QP_CREATE_INTEGRITY_EN		= 1 << 6,
-	IB_QP_CREATE_NETDEV_USE			= 1 << 7,
 	IB_QP_CREATE_SCATTER_FCS		=
 		IB_UVERBS_QP_CREATE_SCATTER_FCS,
 	IB_QP_CREATE_CVLAN_STRIPPING		=
@@ -1159,7 +1158,18 @@ enum ib_qp_create_flags {
 		IB_UVERBS_QP_CREATE_PCI_WRITE_END_PADDING,
 	/* reserve bits 26-31 for low level drivers' internal use */
 	IB_QP_CREATE_RESERVED_START		= 1 << 26,
-	IB_QP_CREATE_RESERVED_END		= 1 << 31,
+	IB_QP_CREATE_RESERVED_END		= 1ULL << 31,
+
+	/* The below flags are used only by the kernel */
+
+	 /* The created QP will be used for IPoIB UD LSO */
+	IB_QP_CREATE_IPOIB_UD_LSO		= 1ULL << 32,
+	/* Create a QP that supports flow-steering */
+	IB_QP_CREATE_NETIF_QP			= 1ULL << 33,
+	/* The created QP can carry out integrity handover operations */
+	IB_QP_CREATE_INTEGRITY_EN		= 1ULL << 34,
+	/* Create an accelerated UD QP */
+	IB_QP_CREATE_NETDEV_USE			= 1ULL << 35,
 };
 
 /*
@@ -1179,7 +1189,7 @@ struct ib_qp_init_attr {
 	struct ib_qp_cap	cap;
 	enum ib_sig_type	sq_sig_type;
 	enum ib_qp_type		qp_type;
-	u32			create_flags;
+	u64			create_flags;
 
 	/*
 	 * Only needed for special QP types, or when using the RW API.
-- 
2.39.0

[PATCH rdma-next 04/13] RDMA/core: Add cryptographic device capabilities

[Leon Romanovsky]

From: Israel Rukshin <israelr@nvidia.com>

Some advanced RDMA devices have HW engines with cryptographic
capabilities. Those devices can encrypt/decrypt data when transmitting
from memory domain to wire domain and when receiving data from wire
domain to memory domain. Expose these capabilities via common RDMA
device attributes. For now, add only AES-XTS cryptographic support.

Signed-off-by: Israel Rukshin <israelr@nvidia.com>
Reviewed-by: Max Gurtovoy <mgurtovoy@nvidia.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
---
 include/rdma/crypto.h   | 37 +++++++++++++++++++++++++++++++++++++
 include/rdma/ib_verbs.h |  2 ++
 2 files changed, 39 insertions(+)
 create mode 100644 include/rdma/crypto.h

diff --git a/include/rdma/crypto.h b/include/rdma/crypto.h
new file mode 100644
index 000000000000..4779eacb000e
--- /dev/null
+++ b/include/rdma/crypto.h
@@ -0,0 +1,37 @@
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/*
+ * Copyright (c) 2022, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+ */
+
+#ifndef _RDMA_CRYPTO_H_
+#define _RDMA_CRYPTO_H_
+
+#include <linux/types.h>
+
+/**
+ * Encryption and decryption operations are done by attaching crypto properties
+ * to a memory region. Once done, every access to the memory via the crypto
+ * memory region will result in inline encryption or decryption of the data
+ * by the RDMA device. The crypto properties contain the Data Encryption Key
+ * (DEK) and the crypto standard that should be used and its attributes.
+ */
+
+/**
+ * Cryptographic engines in clear text mode capabilities.
+ * @IB_CRYPTO_ENGINES_CAP_AES_XTS: Support AES-XTS engine.
+ */
+enum {
+	IB_CRYPTO_ENGINES_CAP_AES_XTS = 1 << 0,
+};
+
+/**
+ * struct ib_crypto_caps - Cryptographic capabilities
+ * @crypto_engines: From enum ib_crypto_engines_cap_bits.
+ * @max_num_deks: Maximum number of Data Encryption Keys.
+ */
+struct ib_crypto_caps {
+	u32 crypto_engines;
+	u32 max_num_deks;
+};
+
+#endif /* _RDMA_CRYPTO_H_ */
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index cc2ddd4e6c12..83be7e49c5f7 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -40,6 +40,7 @@
 #include <rdma/rdma_counter.h>
 #include <rdma/restrack.h>
 #include <rdma/signature.h>
+#include <rdma/crypto.h>
 #include <uapi/rdma/rdma_user_ioctl.h>
 #include <uapi/rdma/ib_user_ioctl_verbs.h>
 
@@ -450,6 +451,7 @@ struct ib_device_attr {
 	u64			max_dm_size;
 	/* Max entries for sgl for optimized performance per READ */
 	u32			max_sgl_rd;
+	struct ib_crypto_caps	crypto_caps;
 };
 
 enum ib_mtu {
-- 
2.39.0

[PATCH rdma-next 05/13] RDMA/core: Add DEK management API

[Leon Romanovsky]

From: Israel Rukshin <israelr@nvidia.com>

Add an API to manage Data Encryption Keys (DEKs). The API allows
creating and destroying a DEK. DEKs allow encryption and decryption
of transmitted data and are used in MKeys for crypto operations.
A crypto setter for the MKey configuration API will be added in the
following commit.

Signed-off-by: Israel Rukshin <israelr@nvidia.com>
Reviewed-by: Max Gurtovoy <mgurtovoy@nvidia.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
---
 drivers/infiniband/core/device.c |  2 ++
 drivers/infiniband/core/verbs.c  | 32 +++++++++++++++++++++++++++
 include/rdma/crypto.h            | 38 ++++++++++++++++++++++++++++++++
 include/rdma/ib_verbs.h          |  3 +++
 4 files changed, 75 insertions(+)

diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c
index a666847bd714..b2016725c3d8 100644
--- a/drivers/infiniband/core/device.c
+++ b/drivers/infiniband/core/device.c
@@ -2615,6 +2615,7 @@ void ib_set_device_ops(struct ib_device *dev, const struct ib_device_ops *ops)
 	SET_DEVICE_OP(dev_ops, create_ah);
 	SET_DEVICE_OP(dev_ops, create_counters);
 	SET_DEVICE_OP(dev_ops, create_cq);
+	SET_DEVICE_OP(dev_ops, create_dek);
 	SET_DEVICE_OP(dev_ops, create_flow);
 	SET_DEVICE_OP(dev_ops, create_qp);
 	SET_DEVICE_OP(dev_ops, create_rwq_ind_table);
@@ -2632,6 +2633,7 @@ void ib_set_device_ops(struct ib_device *dev, const struct ib_device_ops *ops)
 	SET_DEVICE_OP(dev_ops, destroy_ah);
 	SET_DEVICE_OP(dev_ops, destroy_counters);
 	SET_DEVICE_OP(dev_ops, destroy_cq);
+	SET_DEVICE_OP(dev_ops, destroy_dek);
 	SET_DEVICE_OP(dev_ops, destroy_flow);
 	SET_DEVICE_OP(dev_ops, destroy_flow_action);
 	SET_DEVICE_OP(dev_ops, destroy_qp);
diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c
index 26b021f43ba4..03633d706106 100644
--- a/drivers/infiniband/core/verbs.c
+++ b/drivers/infiniband/core/verbs.c
@@ -2306,6 +2306,38 @@ struct ib_mr *ib_alloc_mr_integrity(struct ib_pd *pd,
 }
 EXPORT_SYMBOL(ib_alloc_mr_integrity);
 
+/**
+ * ib_create_dek - Create a DEK (Data Encryption Key) associated with the
+ * specific protection domain.
+ * @pd: The protection domain associated with the DEK.
+ * @attr: The attributes of the DEK.
+ *
+ * Return: Allocated DEK in case of success; IS_ERR() is true in case of an
+ * error, PTR_ERR() returns the error code.
+ */
+struct ib_dek *ib_create_dek(struct ib_pd *pd, struct ib_dek_attr *attr)
+{
+	struct ib_device *device = pd->device;
+
+	if (!device->ops.create_dek || !device->ops.destroy_dek)
+		return ERR_PTR(-EOPNOTSUPP);
+
+	return device->ops.create_dek(pd, attr);
+}
+EXPORT_SYMBOL(ib_create_dek);
+
+/**
+ * ib_destroy_dek - Destroys the specified DEK.
+ * @dek: The DEK to destroy.
+ */
+void ib_destroy_dek(struct ib_dek *dek)
+{
+	struct ib_device *device = dek->pd->device;
+
+	device->ops.destroy_dek(dek);
+}
+EXPORT_SYMBOL(ib_destroy_dek);
+
 /* Multicast groups */
 
 static bool is_valid_mcast_lid(struct ib_qp *qp, u16 lid)
diff --git a/include/rdma/crypto.h b/include/rdma/crypto.h
index 4779eacb000e..cdf287c94737 100644
--- a/include/rdma/crypto.h
+++ b/include/rdma/crypto.h
@@ -34,4 +34,42 @@ struct ib_crypto_caps {
 	u32 max_num_deks;
 };
 
+/**
+ * enum ib_crypto_key_type - Cryptographic key types
+ * @IB_CRYPTO_KEY_TYPE_AES_XTS: Key of type AES-XTS, which can be used when
+ * IB_CRYPTO_AES_XTS is supported.
+ */
+enum ib_crypto_key_type {
+	IB_CRYPTO_KEY_TYPE_AES_XTS,
+};
+
+/**
+ * struct ib_dek_attr - Parameters for DEK (Data Encryption Key)
+ * @key_blob: the key blob that will be used for encryption and decryption of
+ *     transmitted data. Actual size and layout of this field depends on the
+ *     provided key_type and key_blob_size.
+ *     The layout of AES_XTS key is: key1_128b + key2_128b or key1_256b +
+ *     key2_256b.
+ * @key_blob_size: size of the key blob in bytes.
+ * @key_type: specific cryptographic key type.
+ */
+struct ib_dek_attr {
+	const void *key_blob;
+	u32 key_blob_size;
+	enum ib_crypto_key_type key_type;
+};
+
+/**
+ * struct ib_dek - Data Encryption Key
+ * @pd: The protection domain associated with the DEK.
+ * @id: DEK identifier.
+ */
+struct ib_dek {
+	struct ib_pd *pd;
+	u32 id;
+};
+
+struct ib_dek *ib_create_dek(struct ib_pd *pd, struct ib_dek_attr *attr);
+void ib_destroy_dek(struct ib_dek *dek);
+
 #endif /* _RDMA_CRYPTO_H_ */
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 83be7e49c5f7..5fb42d553ca1 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -2512,6 +2512,9 @@ struct ib_device_ops {
 	struct ib_mr *(*alloc_mr_integrity)(struct ib_pd *pd,
 					    u32 max_num_data_sg,
 					    u32 max_num_meta_sg);
+	struct ib_dek *(*create_dek)(struct ib_pd *pd,
+				     struct ib_dek_attr *attr);
+	void (*destroy_dek)(struct ib_dek *dek);
 	int (*advise_mr)(struct ib_pd *pd,
 			 enum ib_uverbs_advise_mr_advice advice, u32 flags,
 			 struct ib_sge *sg_list, u32 num_sge,
-- 
2.39.0

[PATCH rdma-next 06/13] RDMA/core: Introduce MR type for crypto operations

[Leon Romanovsky]

From: Israel Rukshin <israelr@nvidia.com>

Add crypto attributes for MKey. With those attributes, the device
can encrypt/decrypt data when transmitting from memory domain to wire
domain and when receiving data from wire domain to memory domain.

Signed-off-by: Israel Rukshin <israelr@nvidia.com>
Reviewed-by: Max Gurtovoy <mgurtovoy@nvidia.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
---
 drivers/infiniband/core/device.c  |  1 +
 drivers/infiniband/core/mr_pool.c |  2 ++
 drivers/infiniband/core/verbs.c   | 56 ++++++++++++++++++++++++++++++-
 include/rdma/crypto.h             | 43 ++++++++++++++++++++++++
 include/rdma/ib_verbs.h           |  7 ++++
 include/trace/events/rdma_core.h  | 33 ++++++++++++++++++
 6 files changed, 141 insertions(+), 1 deletion(-)

diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c
index b2016725c3d8..d9e98fe92b3c 100644
--- a/drivers/infiniband/core/device.c
+++ b/drivers/infiniband/core/device.c
@@ -2599,6 +2599,7 @@ void ib_set_device_ops(struct ib_device *dev, const struct ib_device_ops *ops)
 	SET_DEVICE_OP(dev_ops, alloc_hw_device_stats);
 	SET_DEVICE_OP(dev_ops, alloc_hw_port_stats);
 	SET_DEVICE_OP(dev_ops, alloc_mr);
+	SET_DEVICE_OP(dev_ops, alloc_mr_crypto);
 	SET_DEVICE_OP(dev_ops, alloc_mr_integrity);
 	SET_DEVICE_OP(dev_ops, alloc_mw);
 	SET_DEVICE_OP(dev_ops, alloc_pd);
diff --git a/drivers/infiniband/core/mr_pool.c b/drivers/infiniband/core/mr_pool.c
index c0e2df128b34..d102cb4caefd 100644
--- a/drivers/infiniband/core/mr_pool.c
+++ b/drivers/infiniband/core/mr_pool.c
@@ -44,6 +44,8 @@ int ib_mr_pool_init(struct ib_qp *qp, struct list_head *list, int nr,
 		if (type == IB_MR_TYPE_INTEGRITY)
 			mr = ib_alloc_mr_integrity(qp->pd, max_num_sg,
 						   max_num_meta_sg);
+		else if (type == IB_MR_TYPE_CRYPTO)
+			mr = ib_alloc_mr_crypto(qp->pd, max_num_sg);
 		else
 			mr = ib_alloc_mr(qp->pd, type, max_num_sg);
 		if (IS_ERR(mr)) {
diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c
index 03633d706106..61473fee4b54 100644
--- a/drivers/infiniband/core/verbs.c
+++ b/drivers/infiniband/core/verbs.c
@@ -2179,6 +2179,7 @@ int ib_dereg_mr_user(struct ib_mr *mr, struct ib_udata *udata)
 	struct ib_pd *pd = mr->pd;
 	struct ib_dm *dm = mr->dm;
 	struct ib_sig_attrs *sig_attrs = mr->sig_attrs;
+	struct ib_crypto_attrs *crypto_attrs = mr->crypto_attrs;
 	int ret;
 
 	trace_mr_dereg(mr);
@@ -2189,6 +2190,7 @@ int ib_dereg_mr_user(struct ib_mr *mr, struct ib_udata *udata)
 		if (dm)
 			atomic_dec(&dm->usecnt);
 		kfree(sig_attrs);
+		kfree(crypto_attrs);
 	}
 
 	return ret;
@@ -2217,7 +2219,8 @@ struct ib_mr *ib_alloc_mr(struct ib_pd *pd, enum ib_mr_type mr_type,
 		goto out;
 	}
 
-	if (mr_type == IB_MR_TYPE_INTEGRITY) {
+	if (mr_type == IB_MR_TYPE_INTEGRITY ||
+	    mr_type == IB_MR_TYPE_CRYPTO) {
 		WARN_ON_ONCE(1);
 		mr = ERR_PTR(-EINVAL);
 		goto out;
@@ -2294,6 +2297,7 @@ struct ib_mr *ib_alloc_mr_integrity(struct ib_pd *pd,
 	mr->uobject = NULL;
 	atomic_inc(&pd->usecnt);
 	mr->need_inval = false;
+	mr->crypto_attrs = NULL;
 	mr->type = IB_MR_TYPE_INTEGRITY;
 	mr->sig_attrs = sig_attrs;
 
@@ -2306,6 +2310,56 @@ struct ib_mr *ib_alloc_mr_integrity(struct ib_pd *pd,
 }
 EXPORT_SYMBOL(ib_alloc_mr_integrity);
 
+/**
+ * ib_alloc_mr_crypto() - Allocates a crypto memory region
+ * @pd:                   protection domain associated with the region
+ * @max_num_sg:           maximum sg entries available for registration
+ *
+ * Notes:
+ * Memory registration page/sg lists must not exceed max_num_sg.
+ *
+ */
+struct ib_mr *ib_alloc_mr_crypto(struct ib_pd *pd, u32 max_num_sg)
+{
+	struct ib_mr *mr;
+	struct ib_crypto_attrs *crypto_attrs;
+
+	if (!pd->device->ops.alloc_mr_crypto) {
+		mr = ERR_PTR(-EOPNOTSUPP);
+		goto out;
+	}
+
+	crypto_attrs = kzalloc(sizeof(*crypto_attrs), GFP_KERNEL);
+	if (!crypto_attrs) {
+		mr = ERR_PTR(-ENOMEM);
+		goto out;
+	}
+
+	mr = pd->device->ops.alloc_mr_crypto(pd, max_num_sg);
+	if (IS_ERR(mr)) {
+		kfree(crypto_attrs);
+		goto out;
+	}
+
+	mr->device = pd->device;
+	mr->pd = pd;
+	mr->dm = NULL;
+	mr->uobject = NULL;
+	atomic_inc(&pd->usecnt);
+	mr->need_inval = false;
+	mr->sig_attrs = NULL;
+	mr->type = IB_MR_TYPE_CRYPTO;
+	mr->crypto_attrs = crypto_attrs;
+
+	rdma_restrack_new(&mr->res, RDMA_RESTRACK_MR);
+	rdma_restrack_parent_name(&mr->res, &pd->res);
+	rdma_restrack_add(&mr->res);
+out:
+	trace_mr_crypto_alloc(pd, max_num_sg, mr);
+	return mr;
+}
+EXPORT_SYMBOL(ib_alloc_mr_crypto);
+
 /**
  * ib_create_dek - Create a DEK (Data Encryption Key) associated with the
  * specific protection domain.
diff --git a/include/rdma/crypto.h b/include/rdma/crypto.h
index cdf287c94737..ba1c6576a8ba 100644
--- a/include/rdma/crypto.h
+++ b/include/rdma/crypto.h
@@ -34,6 +34,49 @@ struct ib_crypto_caps {
 	u32 max_num_deks;
 };
 
+/**
+ * enum ib_crypto_domain - Encryption domain
+ * According to the encryption domain and the data direction, the HW can
+ * conclude if need to encrypt or decrypt the data.
+ * @IB_CRYPTO_ENCRYPTED_WIRE_DOMAIN: encrypted data is in the wire domain.
+ * @IB_CRYPTO_ENCRYPTED_MEM_DOMAIN: encrypted data is in the memory domain.
+ */
+enum ib_crypto_domain {
+	IB_CRYPTO_ENCRYPTED_WIRE_DOMAIN,
+	IB_CRYPTO_ENCRYPTED_MEM_DOMAIN,
+};
+
+/**
+ * enum ib_crypto_standard - Encryption standard
+ * @IB_CRYPTO_AES_XTS: AES-XTS encryption.
+ */
+enum ib_crypto_standard {
+	IB_CRYPTO_AES_XTS,
+};
+
+/* XTS initial tweak size is up to 128 bits, 16 bytes. */
+#define IB_CRYPTO_XTS_TWEAK_MAX_SIZE 16
+
+/**
+ * struct ib_crypto_attrs - Parameters for crypto handover operation
+ * @encrypt_domain: specific encryption domain.
+ * @encrypt_standard: specific encryption standard.
+ * @data_unit_size: data unit size in bytes. It might be e.g., the filesystem
+ *     block size or the disk sector size.
+ * @xts_init_tweak: a value to be used during encryption of each data unit.
+ *     This value is incremented by the device for every data_unit_size in the
+ *     message.
+ * @dek: Data Encryption Key index.
+ */
+struct ib_crypto_attrs {
+	enum ib_crypto_domain encrypt_domain;
+	enum ib_crypto_standard encrypt_standard;
+	int data_unit_size;
+	/* Today we support only AES-XTS */
+	u32 xts_init_tweak[IB_CRYPTO_XTS_TWEAK_MAX_SIZE / sizeof(u32)];
+	u32 dek;
+};
+
 /**
  * enum ib_crypto_key_type - Cryptographic key types
  * @IB_CRYPTO_KEY_TYPE_AES_XTS: Key of type AES-XTS, which can be used when
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 5fb42d553ca1..7507661c78d0 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -876,6 +876,8 @@ __attribute_const__ int ib_rate_to_mbps(enum ib_rate rate);
  *                            without address translations (VA=PA)
  * @IB_MR_TYPE_INTEGRITY:     memory region that is used for
  *                            data integrity operations
+ * @IB_MR_TYPE_CRYPTO:        memory region that is used for cryptographic
+ *                            operations
  */
 enum ib_mr_type {
 	IB_MR_TYPE_MEM_REG,
@@ -884,6 +886,7 @@ enum ib_mr_type {
 	IB_MR_TYPE_USER,
 	IB_MR_TYPE_DMA,
 	IB_MR_TYPE_INTEGRITY,
+	IB_MR_TYPE_CRYPTO,
 };
 
 enum ib_mr_status_check {
@@ -1854,6 +1857,7 @@ struct ib_mr {
 
 	struct ib_dm      *dm;
 	struct ib_sig_attrs *sig_attrs; /* only for IB_MR_TYPE_INTEGRITY MRs */
+	struct ib_crypto_attrs *crypto_attrs; /* only for IB_MR_TYPE_CRYPTO */
 	/*
 	 * Implementation details of the RDMA core, don't use in drivers:
 	 */
@@ -2512,6 +2516,7 @@ struct ib_device_ops {
 	struct ib_mr *(*alloc_mr_integrity)(struct ib_pd *pd,
 					    u32 max_num_data_sg,
 					    u32 max_num_meta_sg);
+	struct ib_mr *(*alloc_mr_crypto)(struct ib_pd *pd, u32 max_num_sg);
 	struct ib_dek *(*create_dek)(struct ib_pd *pd,
 				     struct ib_dek_attr *attr);
 	void (*destroy_dek)(struct ib_dek *dek);
@@ -4295,6 +4300,8 @@ struct ib_mr *ib_alloc_mr_integrity(struct ib_pd *pd,
 				    u32 max_num_data_sg,
 				    u32 max_num_meta_sg);
 
+struct ib_mr *ib_alloc_mr_crypto(struct ib_pd *pd, u32 max_num_sg);
+
 /**
  * ib_update_fast_reg_key - updates the key portion of the fast_reg MR
  *   R_Key and L_Key.
diff --git a/include/trace/events/rdma_core.h b/include/trace/events/rdma_core.h
index 17642aa54437..b6a3d82b89ca 100644
--- a/include/trace/events/rdma_core.h
+++ b/include/trace/events/rdma_core.h
@@ -371,6 +371,39 @@ TRACE_EVENT(mr_integ_alloc,
 		__entry->max_num_meta_sg, __entry->rc)
 );
 
+TRACE_EVENT(mr_crypto_alloc,
+	TP_PROTO(
+		const struct ib_pd *pd,
+		u32 max_num_sg,
+		const struct ib_mr *mr
+	),
+
+	TP_ARGS(pd, max_num_sg, mr),
+
+	TP_STRUCT__entry(
+		__field(u32, pd_id)
+		__field(u32, mr_id)
+		__field(u32, max_num_sg)
+		__field(int, rc)
+	),
+
+	TP_fast_assign(
+		__entry->pd_id = pd->res.id;
+		if (IS_ERR(mr)) {
+			__entry->mr_id = 0;
+			__entry->rc = PTR_ERR(mr);
+		} else {
+			__entry->mr_id = mr->res.id;
+			__entry->rc = 0;
+		}
+		__entry->max_num_sg = max_num_sg;
+	),
+
+	TP_printk("pd.id=%u mr.id=%u max_num_sg=%u rc=%d",
+		__entry->pd_id, __entry->mr_id, __entry->max_num_sg,
+		__entry->rc)
+);
+
 TRACE_EVENT(mr_dereg,
 	TP_PROTO(
 		const struct ib_mr *mr
-- 
2.39.0

[PATCH rdma-next 07/13] RDMA/core: Add support for creating crypto enabled QPs

[Leon Romanovsky]

From: Israel Rukshin <israelr@nvidia.com>

Add a list of crypto MRs and introduce a crypto WR type to post
on those QPs.

Signed-off-by: Israel Rukshin <israelr@nvidia.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
---
 drivers/infiniband/core/verbs.c |  3 +++
 include/rdma/ib_verbs.h         | 12 +++++++++++-
 2 files changed, 14 insertions(+), 1 deletion(-)

diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c
index 61473fee4b54..01aefff6760e 100644
--- a/drivers/infiniband/core/verbs.c
+++ b/drivers/infiniband/core/verbs.c
@@ -1223,6 +1223,7 @@ static struct ib_qp *create_qp(struct ib_device *dev, struct ib_pd *pd,
 	spin_lock_init(&qp->mr_lock);
 	INIT_LIST_HEAD(&qp->rdma_mrs);
 	INIT_LIST_HEAD(&qp->sig_mrs);
+	INIT_LIST_HEAD(&qp->crypto_mrs);
 
 	qp->send_cq = attr->send_cq;
 	qp->recv_cq = attr->recv_cq;
@@ -1363,6 +1364,8 @@ struct ib_qp *ib_create_qp_kernel(struct ib_pd *pd,
 				 device->attrs.max_sge_rd);
 	if (qp_init_attr->create_flags & IB_QP_CREATE_INTEGRITY_EN)
 		qp->integrity_en = true;
+	if (qp_init_attr->create_flags & IB_QP_CREATE_CRYPTO_EN)
+		qp->crypto_en = true;
 
 	return qp;
 
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 7507661c78d0..1770cd30c0f0 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -1175,6 +1175,8 @@ enum ib_qp_create_flags {
 	IB_QP_CREATE_INTEGRITY_EN		= 1ULL << 34,
 	/* Create an accelerated UD QP */
 	IB_QP_CREATE_NETDEV_USE			= 1ULL << 35,
+	/* The created QP can carry out cryptographic handover operations */
+	IB_QP_CREATE_CRYPTO_EN			= 1ULL << 36,
 };
 
 /*
@@ -1352,6 +1354,12 @@ enum ib_wr_opcode {
 	/* These are kernel only and can not be issued by userspace */
 	IB_WR_REG_MR = 0x20,
 	IB_WR_REG_MR_INTEGRITY,
+	/*
+	 * It is used to assign crypto properties to a MKey. Use the MKey in
+	 * any RDMA transaction (SEND/RECV/READ/WRITE) to encrypt/decrypt data
+	 * on-the-fly.
+	 */
+	IB_WR_REG_MR_CRYPTO,
 
 	/* reserve values for low level drivers' internal use.
 	 * These values will not be used at all in the ib core layer.
@@ -1800,6 +1808,7 @@ struct ib_qp {
 	int			mrs_used;
 	struct list_head	rdma_mrs;
 	struct list_head	sig_mrs;
+	struct list_head	crypto_mrs;
 	struct ib_srq	       *srq;
 	struct ib_xrcd	       *xrcd; /* XRC TGT QPs only */
 	struct list_head	xrcd_list;
@@ -1822,7 +1831,8 @@ struct ib_qp {
 	struct ib_qp_security  *qp_sec;
 	u32			port;
 
-	bool			integrity_en;
+	u8			integrity_en:1;
+	u8			crypto_en:1;
 	/*
 	 * Implementation details of the RDMA core, don't use in drivers:
 	 */
-- 
2.39.0

[PATCH rdma-next 08/13] RDMA/mlx5: Add cryptographic device capabilities

[Leon Romanovsky]

From: Israel Rukshin <israelr@nvidia.com>

The capabilities provide information on general cryptographic support,
maximum number of DEKs and status for RDMA devices. Also, they include
the supported cryptographic engines and their import method (wrapped or
plaintext). Wrapped crypto operational flag indicates the import method
mode that can be used. For now, add only AES-XTS cryptographic support.

Signed-off-by: Israel Rukshin <israelr@nvidia.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
---
 drivers/infiniband/hw/mlx5/Makefile  |  1 +
 drivers/infiniband/hw/mlx5/crypto.c  | 31 ++++++++++++++++++++++++++++
 drivers/infiniband/hw/mlx5/crypto.h  | 11 ++++++++++
 drivers/infiniband/hw/mlx5/main.c    |  5 +++++
 drivers/infiniband/hw/mlx5/mlx5_ib.h |  2 ++
 5 files changed, 50 insertions(+)
 create mode 100644 drivers/infiniband/hw/mlx5/crypto.c
 create mode 100644 drivers/infiniband/hw/mlx5/crypto.h

diff --git a/drivers/infiniband/hw/mlx5/Makefile b/drivers/infiniband/hw/mlx5/Makefile
index 612ee8190a2d..d6ae1a08b5b2 100644
--- a/drivers/infiniband/hw/mlx5/Makefile
+++ b/drivers/infiniband/hw/mlx5/Makefile
@@ -6,6 +6,7 @@ mlx5_ib-y := ah.o \
 	     cong.o \
 	     counters.o \
 	     cq.o \
+	     crypto.o \
 	     dm.o \
 	     doorbell.o \
 	     gsi.o \
diff --git a/drivers/infiniband/hw/mlx5/crypto.c b/drivers/infiniband/hw/mlx5/crypto.c
new file mode 100644
index 000000000000..6fad9084877e
--- /dev/null
+++ b/drivers/infiniband/hw/mlx5/crypto.c
@@ -0,0 +1,31 @@
+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
+/* Copyright (c) 2022, NVIDIA CORPORATION & AFFILIATES. */
+
+#include "crypto.h"
+
+void mlx5r_crypto_caps_init(struct mlx5_ib_dev *dev)
+{
+	struct ib_crypto_caps *caps = &dev->crypto_caps;
+	struct mlx5_core_dev *mdev = dev->mdev;
+
+	if (!(MLX5_CAP_GEN_64(dev->mdev, general_obj_types) &
+	      MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_ENCRYPTION_KEY))
+		return;
+
+	if (!MLX5_CAP_GEN(mdev, aes_xts_multi_block_le_tweak) &&
+	    !MLX5_CAP_GEN(mdev, aes_xts_multi_block_be_tweak))
+		return;
+
+	if (MLX5_CAP_CRYPTO(mdev, wrapped_import_method) &
+	    MLX5_CRYPTO_WRAPPED_IMPORT_METHOD_CAP_AES_XTS)
+		return;
+
+	if (MLX5_CAP_CRYPTO(mdev, failed_selftests)) {
+		mlx5_ib_warn(dev, "crypto self-tests failed with error 0x%x\n",
+			     MLX5_CAP_CRYPTO(mdev, failed_selftests));
+		return;
+	}
+
+	caps->crypto_engines |= IB_CRYPTO_ENGINES_CAP_AES_XTS;
+	caps->max_num_deks = 1 << MLX5_CAP_CRYPTO(mdev, log_max_num_deks);
+}
diff --git a/drivers/infiniband/hw/mlx5/crypto.h b/drivers/infiniband/hw/mlx5/crypto.h
new file mode 100644
index 000000000000..8686ac6fb0b0
--- /dev/null
+++ b/drivers/infiniband/hw/mlx5/crypto.h
@@ -0,0 +1,11 @@
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/* Copyright (c) 2022, NVIDIA CORPORATION & AFFILIATES. */
+
+#ifndef _MLX5_IB_CRYPTO_H
+#define _MLX5_IB_CRYPTO_H
+
+#include "mlx5_ib.h"
+
+void mlx5r_crypto_caps_init(struct mlx5_ib_dev *dev);
+
+#endif /* _MLX5_IB_CRYPTO_H */
diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c
index fb0d97bd4074..10f12e9a4dc3 100644
--- a/drivers/infiniband/hw/mlx5/main.c
+++ b/drivers/infiniband/hw/mlx5/main.c
@@ -39,6 +39,7 @@
 #include "srq.h"
 #include "qp.h"
 #include "wr.h"
+#include "crypto.h"
 #include "restrack.h"
 #include "counters.h"
 #include "umr.h"
@@ -989,6 +990,7 @@ static int mlx5_ib_query_device(struct ib_device *ibdev,
 	props->max_ah = INT_MAX;
 	props->hca_core_clock = MLX5_CAP_GEN(mdev, device_frequency_khz);
 	props->timestamp_mask = 0x7FFFFFFFFFFFFFFFULL;
+	props->crypto_caps = dev->crypto_caps;
 
 	if (IS_ENABLED(CONFIG_INFINIBAND_ON_DEMAND_PAGING)) {
 		if (dev->odp_caps.general_caps & IB_ODP_SUPPORT)
@@ -3826,6 +3828,9 @@ static int mlx5_ib_stage_caps_init(struct mlx5_ib_dev *dev)
 	if (MLX5_CAP_GEN(mdev, xrc))
 		ib_set_device_ops(&dev->ib_dev, &mlx5_ib_dev_xrc_ops);
 
+	if (MLX5_CAP_GEN(mdev, crypto))
+		mlx5r_crypto_caps_init(dev);
+
 	if (MLX5_CAP_DEV_MEM(mdev, memic) ||
 	    MLX5_CAP_GEN_64(dev->mdev, general_obj_types) &
 	    MLX5_GENERAL_OBJ_TYPES_CAP_SW_ICM)
diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h
index 295502692da2..8f6850539542 100644
--- a/drivers/infiniband/hw/mlx5/mlx5_ib.h
+++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h
@@ -1100,6 +1100,8 @@ struct mlx5_ib_dev {
 	struct mlx5_ib_delay_drop	delay_drop;
 	const struct mlx5_ib_profile	*profile;
 
+	struct ib_crypto_caps		crypto_caps;
+
 	struct mlx5_ib_lb_state		lb;
 	u8			umr_fence;
 	struct list_head	ib_dev_list;
-- 
2.39.0

[PATCH rdma-next 09/13] RDMA/mlx5: Add DEK management API

[Leon Romanovsky]

From: Israel Rukshin <israelr@nvidia.com>

Add an API to manage Data Encryption Keys (DEKs). The API allows
creating and destroying a DEK. DEKs allow encryption and decryption
of transmitted data and are used in MKeys for crypto operations.

Signed-off-by: Israel Rukshin <israelr@nvidia.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
---
 drivers/infiniband/hw/mlx5/crypto.c | 83 +++++++++++++++++++++++++++++
 drivers/infiniband/hw/mlx5/crypto.h |  8 +++
 2 files changed, 91 insertions(+)

diff --git a/drivers/infiniband/hw/mlx5/crypto.c b/drivers/infiniband/hw/mlx5/crypto.c
index 6fad9084877e..36e978c0fb85 100644
--- a/drivers/infiniband/hw/mlx5/crypto.c
+++ b/drivers/infiniband/hw/mlx5/crypto.c
@@ -3,6 +3,87 @@
 
 #include "crypto.h"
 
+static struct ib_dek *mlx5r_create_dek(struct ib_pd *pd,
+				       struct ib_dek_attr *attr)
+{
+	u32 in[MLX5_ST_SZ_DW(create_encryption_key_in)] = {};
+	u32 out[MLX5_ST_SZ_DW(general_obj_out_cmd_hdr)];
+	struct mlx5_ib_dev *dev = to_mdev(pd->device);
+	u32 key_blob_size = attr->key_blob_size;
+	void *ptr, *key_addr;
+	struct ib_dek *dek;
+	u8 key_size;
+	int err;
+
+	if (attr->key_type != IB_CRYPTO_KEY_TYPE_AES_XTS)
+		return ERR_PTR(-EOPNOTSUPP);
+
+	switch (key_blob_size) {
+	case MLX5_IB_CRYPTO_AES_128_XTS_KEY_SIZE:
+		key_size = MLX5_GENERAL_OBJECT_TYPE_ENCRYPTION_KEY_KEY_SIZE_128;
+		break;
+	case MLX5_IB_CRYPTO_AES_256_XTS_KEY_SIZE:
+		key_size = MLX5_GENERAL_OBJECT_TYPE_ENCRYPTION_KEY_KEY_SIZE_256;
+		break;
+	default:
+		return ERR_PTR(-EOPNOTSUPP);
+	}
+
+	dek = kzalloc(sizeof(*dek), GFP_KERNEL);
+	if (!dek)
+		return ERR_PTR(-ENOMEM);
+
+	ptr = MLX5_ADDR_OF(create_encryption_key_in, in,
+			   general_obj_in_cmd_hdr);
+	MLX5_SET(general_obj_in_cmd_hdr, ptr, opcode,
+		 MLX5_CMD_OP_CREATE_GENERAL_OBJECT);
+	MLX5_SET(general_obj_in_cmd_hdr, ptr, obj_type,
+		 MLX5_GENERAL_OBJECT_TYPES_ENCRYPTION_KEY);
+	ptr = MLX5_ADDR_OF(create_encryption_key_in, in, encryption_key_object);
+	MLX5_SET(encryption_key_obj, ptr, key_size, key_size);
+	MLX5_SET(encryption_key_obj, ptr, key_type,
+		 MLX5_GENERAL_OBJECT_TYPE_ENCRYPTION_KEY_TYPE_AES_XTS);
+	MLX5_SET(encryption_key_obj, ptr, pd, to_mpd(pd)->pdn);
+	key_addr = MLX5_ADDR_OF(encryption_key_obj, ptr, key);
+	memcpy(key_addr, attr->key_blob, key_blob_size);
+
+	err = mlx5_cmd_exec(dev->mdev, in, sizeof(in), out, sizeof(out));
+	/* avoid leaking key on the stack */
+	memzero_explicit(in, sizeof(in));
+	if (err)
+		goto err_free;
+
+	dek->id = MLX5_GET(general_obj_out_cmd_hdr, out, obj_id);
+	dek->pd = pd;
+
+	return dek;
+
+err_free:
+	kfree(dek);
+	return ERR_PTR(err);
+}
+
+static void mlx5r_destroy_dek(struct ib_dek *dek)
+{
+	struct mlx5_ib_dev *dev = to_mdev(dek->pd->device);
+	u32 in[MLX5_ST_SZ_DW(general_obj_in_cmd_hdr)] = {};
+	u32 out[MLX5_ST_SZ_DW(general_obj_out_cmd_hdr)];
+
+	MLX5_SET(general_obj_in_cmd_hdr, in, opcode,
+		 MLX5_CMD_OP_DESTROY_GENERAL_OBJECT);
+	MLX5_SET(general_obj_in_cmd_hdr, in, obj_type,
+		 MLX5_GENERAL_OBJECT_TYPES_ENCRYPTION_KEY);
+	MLX5_SET(general_obj_in_cmd_hdr, in, obj_id, dek->id);
+
+	mlx5_cmd_exec(dev->mdev, in, sizeof(in), out, sizeof(out));
+	kfree(dek);
+}
+
+static const struct ib_device_ops mlx5r_dev_crypto_ops = {
+	.create_dek = mlx5r_create_dek,
+	.destroy_dek = mlx5r_destroy_dek,
+};
+
 void mlx5r_crypto_caps_init(struct mlx5_ib_dev *dev)
 {
 	struct ib_crypto_caps *caps = &dev->crypto_caps;
@@ -28,4 +109,6 @@ void mlx5r_crypto_caps_init(struct mlx5_ib_dev *dev)
 
 	caps->crypto_engines |= IB_CRYPTO_ENGINES_CAP_AES_XTS;
 	caps->max_num_deks = 1 << MLX5_CAP_CRYPTO(mdev, log_max_num_deks);
+
+	ib_set_device_ops(&dev->ib_dev, &mlx5r_dev_crypto_ops);
 }
diff --git a/drivers/infiniband/hw/mlx5/crypto.h b/drivers/infiniband/hw/mlx5/crypto.h
index 8686ac6fb0b0..b132b780030f 100644
--- a/drivers/infiniband/hw/mlx5/crypto.h
+++ b/drivers/infiniband/hw/mlx5/crypto.h
@@ -6,6 +6,14 @@
 
 #include "mlx5_ib.h"
 
+/*
+ * The standard AES-XTS key blob composed of two keys.
+ * AES-128-XTS key blob composed of two 128-bit keys, which is 32 bytes and
+ * AES-256-XTS key blob composed of two 256-bit keys, which is 64 bytes.
+ */
+#define MLX5_IB_CRYPTO_AES_128_XTS_KEY_SIZE	32
+#define MLX5_IB_CRYPTO_AES_256_XTS_KEY_SIZE	64
+
 void mlx5r_crypto_caps_init(struct mlx5_ib_dev *dev);
 
 #endif /* _MLX5_IB_CRYPTO_H */
-- 
2.39.0

[PATCH rdma-next 10/13] RDMA/mlx5: Add AES-XTS crypto support

[Leon Romanovsky]

From: Israel Rukshin <israelr@nvidia.com>

Add crypto attributes for MKey and QP. With this, the device
can encrypt/decrypt data when transmitting data from memory to
network and when receiving data from network to memory.

Signed-off-by: Israel Rukshin <israelr@nvidia.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
---
 drivers/infiniband/hw/mlx5/crypto.c  |   1 +
 drivers/infiniband/hw/mlx5/crypto.h  |  27 +++++
 drivers/infiniband/hw/mlx5/mlx5_ib.h |   1 +
 drivers/infiniband/hw/mlx5/mr.c      |  33 ++++++
 drivers/infiniband/hw/mlx5/qp.c      |   6 +
 drivers/infiniband/hw/mlx5/wr.c      | 164 +++++++++++++++++++++++++--
 6 files changed, 222 insertions(+), 10 deletions(-)

diff --git a/drivers/infiniband/hw/mlx5/crypto.c b/drivers/infiniband/hw/mlx5/crypto.c
index 36e978c0fb85..28bc661bd5bc 100644
--- a/drivers/infiniband/hw/mlx5/crypto.c
+++ b/drivers/infiniband/hw/mlx5/crypto.c
@@ -82,6 +82,7 @@ static void mlx5r_destroy_dek(struct ib_dek *dek)
 static const struct ib_device_ops mlx5r_dev_crypto_ops = {
 	.create_dek = mlx5r_create_dek,
 	.destroy_dek = mlx5r_destroy_dek,
+	.alloc_mr_crypto = mlx5r_alloc_mr_crypto,
 };
 
 void mlx5r_crypto_caps_init(struct mlx5_ib_dev *dev)
diff --git a/drivers/infiniband/hw/mlx5/crypto.h b/drivers/infiniband/hw/mlx5/crypto.h
index b132b780030f..33f7e3b8bcce 100644
--- a/drivers/infiniband/hw/mlx5/crypto.h
+++ b/drivers/infiniband/hw/mlx5/crypto.h
@@ -6,6 +6,33 @@
 
 #include "mlx5_ib.h"
 
+enum {
+	MLX5_CRYPTO_ENCRYPTED_WIRE = 0x0,
+	MLX5_CRYPTO_ENCRYPTED_MEM = 0x1,
+};
+
+enum {
+	MLX5_CRYPTO_AES_XTS = 0x0,
+};
+
+struct mlx5_bsf_crypto {
+	u8 bsf_size_type;
+	u8 encryption_order;
+	u8 rsvd0;
+	u8 encryption_standard;
+	__be32 raw_data_size;
+	u8 block_size_p;
+	u8 rsvd1[7];
+	union {
+		__be32 be_xts_init_tweak[4];
+		__le32 le_xts_init_tweak[4];
+	};
+	__be32 rsvd_dek_pointer;
+	u8 rsvd2[4];
+	u8 keytag[8];
+	u8 rsvd3[16];
+};
+
 /*
  * The standard AES-XTS key blob composed of two keys.
  * AES-128-XTS key blob composed of two 128-bit keys, which is 32 bytes and
diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h
index 8f6850539542..6e5b1a65f91b 100644
--- a/drivers/infiniband/hw/mlx5/mlx5_ib.h
+++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h
@@ -1291,6 +1291,7 @@ struct ib_mr *mlx5_ib_alloc_mr(struct ib_pd *pd, enum ib_mr_type mr_type,
 struct ib_mr *mlx5_ib_alloc_mr_integrity(struct ib_pd *pd,
 					 u32 max_num_sg,
 					 u32 max_num_meta_sg);
+struct ib_mr *mlx5r_alloc_mr_crypto(struct ib_pd *pd, u32 max_num_sg);
 int mlx5_ib_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sg, int sg_nents,
 		      unsigned int *sg_offset);
 int mlx5_ib_map_mr_sg_pi(struct ib_mr *ibmr, struct scatterlist *data_sg,
diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c
index e3b0f41aef0d..8e2f32ae4196 100644
--- a/drivers/infiniband/hw/mlx5/mr.c
+++ b/drivers/infiniband/hw/mlx5/mr.c
@@ -40,6 +40,7 @@
 #include <linux/dma-buf.h>
 #include <linux/dma-resv.h>
 #include <rdma/ib_umem_odp.h>
+#include "crypto.h"
 #include "dm.h"
 #include "mlx5_ib.h"
 #include "umr.h"
@@ -1553,6 +1554,8 @@ mlx5_alloc_priv_descs(struct ib_device *device,
 	int add_size;
 	int ret;
 
+	if (mr->ibmr.type == IB_MR_TYPE_CRYPTO)
+		size += sizeof(struct mlx5_bsf_crypto);
 	add_size = max_t(int, MLX5_UMR_ALIGN - ARCH_KMALLOC_MINALIGN, 0);
 
 	mr->descs_alloc = kzalloc(size + add_size, GFP_KERNEL);
@@ -1582,6 +1585,8 @@ mlx5_free_priv_descs(struct mlx5_ib_mr *mr)
 		int size = mr->max_descs * mr->desc_size;
 		struct mlx5_ib_dev *dev = to_mdev(device);
 
+		if (mr->ibmr.type == IB_MR_TYPE_CRYPTO)
+			size += sizeof(struct mlx5_bsf_crypto);
 		dma_unmap_single(&dev->mdev->pdev->dev, mr->desc_map, size,
 				 DMA_TO_DEVICE);
 		kfree(mr->descs_alloc);
@@ -1862,6 +1867,21 @@ static int mlx5_alloc_integrity_descs(struct ib_pd *pd, struct mlx5_ib_mr *mr,
 	return err;
 }
 
+static int mlx5_alloc_crypto_descs(struct ib_pd *pd, struct mlx5_ib_mr *mr,
+				   int ndescs, u32 *in, int inlen)
+{
+	void *mkc;
+
+	mkc = MLX5_ADDR_OF(create_mkey_in, in, memory_key_mkey_entry);
+	MLX5_SET(mkc, mkc, crypto_en, 1);
+	/* Set bsf descriptors for mkey */
+	MLX5_SET(mkc, mkc, bsf_en, 1);
+	MLX5_SET(mkc, mkc, bsf_octword_size, MLX5_MKEY_BSF_OCTO_SIZE);
+
+	return _mlx5_alloc_mkey_descs(pd, mr, ndescs, sizeof(struct mlx5_klm),
+				      0, MLX5_MKC_ACCESS_MODE_KLMS, in, inlen);
+}
+
 static struct ib_mr *__mlx5_ib_alloc_mr(struct ib_pd *pd,
 					enum ib_mr_type mr_type, u32 max_num_sg,
 					u32 max_num_meta_sg)
@@ -1897,6 +1917,9 @@ static struct ib_mr *__mlx5_ib_alloc_mr(struct ib_pd *pd,
 		err = mlx5_alloc_integrity_descs(pd, mr, max_num_sg,
 						 max_num_meta_sg, in, inlen);
 		break;
+	case IB_MR_TYPE_CRYPTO:
+		err = mlx5_alloc_crypto_descs(pd, mr, ndescs, in, inlen);
+		break;
 	default:
 		mlx5_ib_warn(dev, "Invalid mr type %d\n", mr_type);
 		err = -EINVAL;
@@ -1929,6 +1952,11 @@ struct ib_mr *mlx5_ib_alloc_mr_integrity(struct ib_pd *pd,
 				  max_num_meta_sg);
 }
 
+struct ib_mr *mlx5r_alloc_mr_crypto(struct ib_pd *pd, u32 max_num_sg)
+{
+	return __mlx5_ib_alloc_mr(pd, IB_MR_TYPE_CRYPTO, max_num_sg, 0);
+}
+
 int mlx5_ib_alloc_mw(struct ib_mw *ibmw, struct ib_udata *udata)
 {
 	struct mlx5_ib_dev *dev = to_mdev(ibmw->device);
@@ -2368,6 +2396,11 @@ int mlx5_ib_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sg, int sg_nents,
 				      mr->desc_size * mr->max_descs,
 				      DMA_TO_DEVICE);
 
+	if (ibmr->type == IB_MR_TYPE_CRYPTO) {
+		/* This is zero-based memory region */
+		ibmr->iova = 0;
+	}
+
 	return n;
 }
 
diff --git a/drivers/infiniband/hw/mlx5/qp.c b/drivers/infiniband/hw/mlx5/qp.c
index f04adc18e63b..8eb777d422e4 100644
--- a/drivers/infiniband/hw/mlx5/qp.c
+++ b/drivers/infiniband/hw/mlx5/qp.c
@@ -40,6 +40,7 @@
 #include "ib_rep.h"
 #include "counters.h"
 #include "cmd.h"
+#include "crypto.h"
 #include "umr.h"
 #include "qp.h"
 #include "wr.h"
@@ -554,6 +555,8 @@ static int calc_send_wqe(struct ib_qp_init_attr *attr)
 	}
 
 	size += attr->cap.max_send_sge * sizeof(struct mlx5_wqe_data_seg);
+	if (attr->create_flags & IB_QP_CREATE_CRYPTO_EN)
+		size += sizeof(struct mlx5_bsf_crypto);
 	if (attr->create_flags & IB_QP_CREATE_INTEGRITY_EN &&
 	    ALIGN(max_t(int, inl_size, size), MLX5_SEND_WQE_BB) < MLX5_SIG_WQE_SIZE)
 		return MLX5_SIG_WQE_SIZE;
@@ -3024,6 +3027,9 @@ static int process_create_flags(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,
 	process_create_flag(dev, &create_flags, MLX5_IB_QP_CREATE_SQPN_QP1,
 			    true, qp);
 
+	process_create_flag(dev, &create_flags, IB_QP_CREATE_CRYPTO_EN,
+			    !!dev->crypto_caps.crypto_engines, qp);
+
 	if (create_flags) {
 		mlx5_ib_dbg(dev, "Create QP has unsupported flags 0x%llX\n",
 			    create_flags);
diff --git a/drivers/infiniband/hw/mlx5/wr.c b/drivers/infiniband/hw/mlx5/wr.c
index df1d1b0a3ef7..e047b8aabceb 100644
--- a/drivers/infiniband/hw/mlx5/wr.c
+++ b/drivers/infiniband/hw/mlx5/wr.c
@@ -6,6 +6,7 @@
 #include <linux/gfp.h>
 #include <linux/mlx5/qp.h>
 #include <linux/mlx5/driver.h>
+#include "crypto.h"
 #include "wr.h"
 #include "umr.h"
 
@@ -21,6 +22,7 @@ static const u32 mlx5_ib_opcode[] = {
 	[IB_WR_SEND_WITH_INV]			= MLX5_OPCODE_SEND_INVAL,
 	[IB_WR_LOCAL_INV]			= MLX5_OPCODE_UMR,
 	[IB_WR_REG_MR]				= MLX5_OPCODE_UMR,
+	[IB_WR_REG_MR_CRYPTO]			= MLX5_OPCODE_UMR,
 	[IB_WR_MASKED_ATOMIC_CMP_AND_SWP]	= MLX5_OPCODE_ATOMIC_MASKED_CS,
 	[IB_WR_MASKED_ATOMIC_FETCH_AND_ADD]	= MLX5_OPCODE_ATOMIC_MASKED_FA,
 	[MLX5_IB_WR_UMR]			= MLX5_OPCODE_UMR,
@@ -115,7 +117,7 @@ static void set_data_ptr_seg(struct mlx5_wqe_data_seg *dseg, struct ib_sge *sg)
 	dseg->addr       = cpu_to_be64(sg->addr);
 }
 
-static __be64 frwr_mkey_mask(bool atomic)
+static __be64 frwr_mkey_mask(bool atomic, bool crypto)
 {
 	u64 result;
 
@@ -134,6 +136,9 @@ static __be64 frwr_mkey_mask(bool atomic)
 	if (atomic)
 		result |= MLX5_MKEY_MASK_A;
 
+	if (crypto)
+		result |= MLX5_MKEY_MASK_BSF_EN;
+
 	return cpu_to_be64(result);
 }
 
@@ -159,7 +164,8 @@ static __be64 sig_mkey_mask(void)
 }
 
 static void set_reg_umr_seg(struct mlx5_wqe_umr_ctrl_seg *umr,
-			    struct mlx5_ib_mr *mr, u8 flags, bool atomic)
+			    struct mlx5_ib_mr *mr, u8 flags, bool atomic,
+			    bool crypto)
 {
 	int size = (mr->mmkey.ndescs + mr->meta_ndescs) * mr->desc_size;
 
@@ -167,7 +173,9 @@ static void set_reg_umr_seg(struct mlx5_wqe_umr_ctrl_seg *umr,
 
 	umr->flags = flags;
 	umr->xlt_octowords = cpu_to_be16(mlx5r_umr_get_xlt_octo(size));
-	umr->mkey_mask = frwr_mkey_mask(atomic);
+	umr->mkey_mask = frwr_mkey_mask(atomic, crypto);
+	if (crypto)
+		umr->bsf_octowords = cpu_to_be16(MLX5_MKEY_BSF_OCTO_SIZE);
 }
 
 static void set_linv_umr_seg(struct mlx5_wqe_umr_ctrl_seg *umr)
@@ -188,7 +196,7 @@ static u8 get_umr_flags(int acc)
 
 static void set_reg_mkey_seg(struct mlx5_mkey_seg *seg,
 			     struct mlx5_ib_mr *mr,
-			     u32 key, int access)
+			     u32 key, int access, bool crypto)
 {
 	int ndescs = ALIGN(mr->mmkey.ndescs + mr->meta_ndescs, 8) >> 1;
 
@@ -203,6 +211,8 @@ static void set_reg_mkey_seg(struct mlx5_mkey_seg *seg,
 	seg->flags = get_umr_flags(access) | mr->access_mode;
 	seg->qpn_mkey7_0 = cpu_to_be32((key & 0xff) | 0xffffff00);
 	seg->flags_pd = cpu_to_be32(MLX5_MKEY_REMOTE_INVAL);
+	if (crypto)
+		seg->flags_pd |= cpu_to_be32(MLX5_MKEY_BSF_EN);
 	seg->start_addr = cpu_to_be64(mr->ibmr.iova);
 	seg->len = cpu_to_be64(mr->ibmr.length);
 	seg->xlt_oct_size = cpu_to_be32(ndescs);
@@ -353,6 +363,66 @@ static void mlx5_fill_inl_bsf(struct ib_sig_domain *domain,
 		cpu_to_be16(domain->sig.dif.apptag_check_mask);
 }
 
+static void mlx5_set_xts_tweak(struct ib_crypto_attrs *crypto_attrs,
+			       struct mlx5_bsf_crypto *bsf, bool is_be)
+{
+	int tweak_array_size = sizeof(bsf->be_xts_init_tweak) / sizeof(u32);
+	int i, j;
+
+	/* The endianness of the initial tweak in the kernel is LE */
+	if (is_be) {
+		for (i = 0; i < tweak_array_size; i++) {
+			j = tweak_array_size - i - 1;
+			bsf->be_xts_init_tweak[i] =
+				cpu_to_be32(crypto_attrs->xts_init_tweak[j]);
+		}
+	} else {
+		for (i = 0; i < tweak_array_size; i++)
+			bsf->le_xts_init_tweak[i] =
+				cpu_to_le32(crypto_attrs->xts_init_tweak[i]);
+	}
+}
+
+static int mlx5_set_bsf_crypto(struct ib_mr *ibmr, struct mlx5_bsf_crypto *bsf)
+{
+	struct mlx5_ib_dev *dev = to_mdev(ibmr->device);
+	struct mlx5_core_dev *mdev = dev->mdev;
+	struct ib_crypto_attrs *crypto_attrs = ibmr->crypto_attrs;
+	u64 data_size = ibmr->length;
+
+	if (crypto_attrs->encrypt_standard != IB_CRYPTO_AES_XTS)
+		return -EINVAL;
+
+	memset(bsf, 0, sizeof(*bsf));
+
+	/* Crypto only */
+	bsf->bsf_size_type = 1 << 7;
+	/* Crypto type */
+	bsf->bsf_size_type |= 1;
+
+	switch (crypto_attrs->encrypt_domain) {
+	case IB_CRYPTO_ENCRYPTED_WIRE_DOMAIN:
+		bsf->encryption_order = MLX5_CRYPTO_ENCRYPTED_WIRE;
+		break;
+	case IB_CRYPTO_ENCRYPTED_MEM_DOMAIN:
+		bsf->encryption_order = MLX5_CRYPTO_ENCRYPTED_MEM;
+		break;
+	default:
+		WARN_ONCE(1, "Bad encryption domain (%d) is given.\n",
+			  crypto_attrs->encrypt_domain);
+		return -EINVAL;
+	}
+
+	bsf->encryption_standard = MLX5_CRYPTO_AES_XTS;
+	bsf->raw_data_size = cpu_to_be32(data_size);
+	bsf->block_size_p = bs_selector(crypto_attrs->data_unit_size);
+	mlx5_set_xts_tweak(crypto_attrs, bsf,
+			   MLX5_CAP_GEN(mdev, aes_xts_multi_block_be_tweak));
+	bsf->rsvd_dek_pointer |= cpu_to_be32(crypto_attrs->dek & 0xffffff);
+
+	return 0;
+}
+
 static int mlx5_set_bsf(struct ib_mr *sig_mr,
 			struct ib_sig_attrs *sig_attrs,
 			struct mlx5_bsf *bsf, u32 data_size)
@@ -632,10 +702,53 @@ static int set_psv_wr(struct ib_sig_domain *domain,
 	return 0;
 }
 
+static int set_crypto_data_segment(struct mlx5_ib_qp *qp, struct mlx5_ib_mr *mr,
+				   void **seg, int *size, void **cur_edge)
+{
+	int mr_list_size = (mr->mmkey.ndescs + mr->meta_ndescs) * mr->desc_size;
+	int tmp_size;
+
+	mlx5r_memcpy_send_wqe(&qp->sq, cur_edge, seg, size, mr->descs,
+			      mr_list_size);
+	tmp_size = *size;
+	*size = ALIGN(*size, MLX5_SEND_WQE_BB >> 4);
+	*seg += (*size - tmp_size) * 16;
+	handle_post_send_edge(&qp->sq, seg, *size, cur_edge);
+
+	if (mlx5_set_bsf_crypto(&mr->ibmr, *seg))
+		return -EINVAL;
+
+	*seg += sizeof(struct mlx5_bsf_crypto);
+	*size += sizeof(struct mlx5_bsf_crypto) / 16;
+	handle_post_send_edge(&qp->sq, seg, *size, cur_edge);
+
+	return 0;
+}
+
+static int set_dma_bsf_crypto(struct mlx5_ib_mr *mr)
+{
+	int mr_list_size = (mr->mmkey.ndescs + mr->meta_ndescs) * mr->desc_size;
+	int aligned_size = ALIGN(mr_list_size, MLX5_SEND_WQE_BB);
+
+	ib_dma_sync_single_for_cpu(mr->ibmr.device, mr->desc_map + aligned_size,
+				   sizeof(struct mlx5_bsf_crypto),
+				   DMA_TO_DEVICE);
+
+	if (mlx5_set_bsf_crypto(&mr->ibmr, mr->descs + aligned_size))
+		return -EINVAL;
+
+	ib_dma_sync_single_for_device(mr->ibmr.device,
+				      mr->desc_map + aligned_size,
+				      sizeof(struct mlx5_bsf_crypto),
+				      DMA_TO_DEVICE);
+
+	return 0;
+}
+
 static int set_reg_wr(struct mlx5_ib_qp *qp,
 		      const struct ib_reg_wr *wr,
 		      void **seg, int *size, void **cur_edge,
-		      bool check_not_free)
+		      bool check_not_free, bool crypto)
 {
 	struct mlx5_ib_mr *mr = to_mmr(wr->mr);
 	struct mlx5_ib_pd *pd = to_mpd(qp->ibqp.pd);
@@ -667,17 +780,21 @@ static int set_reg_wr(struct mlx5_ib_qp *qp,
 	if (umr_inline)
 		flags |= MLX5_UMR_INLINE;
 
-	set_reg_umr_seg(*seg, mr, flags, atomic);
+	set_reg_umr_seg(*seg, mr, flags, atomic, crypto);
 	*seg += sizeof(struct mlx5_wqe_umr_ctrl_seg);
 	*size += sizeof(struct mlx5_wqe_umr_ctrl_seg) / 16;
 	handle_post_send_edge(&qp->sq, seg, *size, cur_edge);
 
-	set_reg_mkey_seg(*seg, mr, wr->key, wr->access);
+	set_reg_mkey_seg(*seg, mr, wr->key, wr->access, crypto);
 	*seg += sizeof(struct mlx5_mkey_seg);
 	*size += sizeof(struct mlx5_mkey_seg) / 16;
 	handle_post_send_edge(&qp->sq, seg, *size, cur_edge);
 
 	if (umr_inline) {
+		if (crypto)
+			return set_crypto_data_segment(qp, mr, seg, size,
+						       cur_edge);
+
 		mlx5r_memcpy_send_wqe(&qp->sq, cur_edge, seg, size, mr->descs,
 				      mr_list_size);
 		*size = ALIGN(*size, MLX5_SEND_WQE_BB >> 4);
@@ -685,6 +802,9 @@ static int set_reg_wr(struct mlx5_ib_qp *qp,
 		set_reg_data_seg(*seg, mr, pd);
 		*seg += sizeof(struct mlx5_wqe_data_seg);
 		*size += (sizeof(struct mlx5_wqe_data_seg) / 16);
+
+		if (crypto)
+			return set_dma_bsf_crypto(mr);
 	}
 	return 0;
 }
@@ -806,7 +926,7 @@ static int handle_reg_mr(struct mlx5_ib_qp *qp, const struct ib_send_wr *wr,
 {
 	qp->sq.wr_data[idx] = IB_WR_REG_MR;
 	(*ctrl)->imm = cpu_to_be32(reg_wr(wr)->key);
-	return set_reg_wr(qp, reg_wr(wr), seg, size, cur_edge, true);
+	return set_reg_wr(qp, reg_wr(wr), seg, size, cur_edge, true, false);
 }
 
 static int handle_psv(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,
@@ -870,7 +990,8 @@ static int handle_reg_mr_integrity(struct mlx5_ib_dev *dev,
 
 		(*ctrl)->imm = cpu_to_be32(reg_pi_wr.key);
 		/* UMR for data + prot registration */
-		err = set_reg_wr(qp, &reg_pi_wr, seg, size, cur_edge, false);
+		err = set_reg_wr(qp, &reg_pi_wr, seg, size, cur_edge, false,
+				 false);
 		if (unlikely(err))
 			goto out;
 
@@ -928,6 +1049,20 @@ static int handle_reg_mr_integrity(struct mlx5_ib_dev *dev,
 	return err;
 }
 
+static int handle_reg_mr_crypto(struct mlx5_ib_qp *qp,
+				const struct ib_send_wr *wr,
+				struct mlx5_wqe_ctrl_seg **ctrl, void **seg,
+				int *size, void **cur_edge, unsigned int idx)
+{
+	qp->sq.wr_data[idx] = IB_WR_REG_MR_CRYPTO;
+	(*ctrl)->imm = cpu_to_be32(reg_wr(wr)->key);
+
+	if (unlikely(!qp->ibqp.crypto_en))
+		return -EINVAL;
+
+	return set_reg_wr(qp, reg_wr(wr), seg, size, cur_edge, true, true);
+}
+
 static int handle_qpt_rc(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,
 			 const struct ib_send_wr *wr,
 			 struct mlx5_wqe_ctrl_seg **ctrl, void **seg, int *size,
@@ -971,6 +1106,14 @@ static int handle_qpt_rc(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,
 		*num_sge = 0;
 		break;
 
+	case IB_WR_REG_MR_CRYPTO:
+		err = handle_reg_mr_crypto(qp, wr, ctrl, seg, size, cur_edge,
+					   *idx);
+		if (unlikely(err))
+			goto out;
+		*num_sge = 0;
+		break;
+
 	default:
 		break;
 	}
@@ -1105,7 +1248,8 @@ int mlx5_ib_post_send(struct ib_qp *ibqp, const struct ib_send_wr *wr,
 		}
 
 		if (wr->opcode == IB_WR_REG_MR ||
-		    wr->opcode == IB_WR_REG_MR_INTEGRITY) {
+		    wr->opcode == IB_WR_REG_MR_INTEGRITY ||
+		    wr->opcode == IB_WR_REG_MR_CRYPTO) {
 			fence = dev->umr_fence;
 			next_fence = MLX5_FENCE_MODE_INITIATOR_SMALL;
 		} else  {
-- 
2.39.0

[PATCH rdma-next 11/13] nvme: Introduce a local variable

[Leon Romanovsky]

From: Israel Rukshin <israelr@nvidia.com>

The patch doesn't change any logic.

Signed-off-by: Israel Rukshin <israelr@nvidia.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
---
 drivers/nvme/host/core.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 7be562a4e1aa..51a9880db6ce 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -1870,6 +1870,7 @@ static void nvme_update_disk_info(struct gendisk *disk,
 	sector_t capacity = nvme_lba_to_sect(ns, le64_to_cpu(id->nsze));
 	unsigned short bs = 1 << ns->lba_shift;
 	u32 atomic_bs, phys_bs, io_opt = 0;
+	struct nvme_ctrl *ctrl = ns->ctrl;
 
 	/*
 	 * The block layer can't support LBA sizes larger than the page size
@@ -1892,7 +1893,7 @@ static void nvme_update_disk_info(struct gendisk *disk,
 		if (id->nsfeat & NVME_NS_FEAT_ATOMICS && id->nawupf)
 			atomic_bs = (1 + le16_to_cpu(id->nawupf)) * bs;
 		else
-			atomic_bs = (1 + ns->ctrl->subsys->awupf) * bs;
+			atomic_bs = (1 + ctrl->subsys->awupf) * bs;
 	}
 
 	if (id->nsfeat & NVME_NS_FEAT_IO_OPT) {
@@ -1922,7 +1923,7 @@ static void nvme_update_disk_info(struct gendisk *disk,
 		if (IS_ENABLED(CONFIG_BLK_DEV_INTEGRITY) &&
 		    (ns->features & NVME_NS_METADATA_SUPPORTED))
 			nvme_init_integrity(disk, ns,
-					    ns->ctrl->max_integrity_segments);
+					    ctrl->max_integrity_segments);
 		else if (!nvme_ns_has_pi(ns))
 			capacity = 0;
 	}
@@ -1931,7 +1932,7 @@ static void nvme_update_disk_info(struct gendisk *disk,
 
 	nvme_config_discard(disk, ns);
 	blk_queue_max_write_zeroes_sectors(disk->queue,
-					   ns->ctrl->max_zeroes_sectors);
+					   ctrl->max_zeroes_sectors);
 }
 
 static bool nvme_ns_is_readonly(struct nvme_ns *ns, struct nvme_ns_info *info)
-- 
2.39.0

[PATCH rdma-next 12/13] nvme: Add crypto profile at nvme controller

[Leon Romanovsky]

From: Israel Rukshin <israelr@nvidia.com>

The crypto profile will be filled by the transport drivers. This
is a preparation patch for adding support of inline encryption
at nvme-rdma driver.

Signed-off-by: Israel Rukshin <israelr@nvidia.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
---
 drivers/nvme/host/core.c | 3 +++
 drivers/nvme/host/nvme.h | 4 ++++
 2 files changed, 7 insertions(+)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 51a9880db6ce..f09e4e0216b3 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -1928,6 +1928,9 @@ static void nvme_update_disk_info(struct gendisk *disk,
 			capacity = 0;
 	}
 
+	if (ctrl->crypto_enable)
+		blk_crypto_register(&ctrl->crypto_profile, disk->queue);
+
 	set_capacity_and_notify(disk, capacity);
 
 	nvme_config_discard(disk, ns);
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 424c8a467a0c..591380f53744 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -16,6 +16,7 @@
 #include <linux/rcupdate.h>
 #include <linux/wait.h>
 #include <linux/t10-pi.h>
+#include <linux/blk-crypto-profile.h>
 
 #include <trace/events/block.h>
 
@@ -374,6 +375,9 @@ struct nvme_ctrl {
 
 	enum nvme_ctrl_type cntrltype;
 	enum nvme_dctype dctype;
+
+	bool crypto_enable;
+	struct blk_crypto_profile crypto_profile;
 };
 
 enum nvme_iopolicy {
-- 
2.39.0

[PATCH rdma-next 13/13] nvme-rdma: Add inline encryption support

[Leon Romanovsky]

From: Israel Rukshin <israelr@nvidia.com>

Add support for inline encryption/decryption of the data at a
similar way like integrity is used via a unique Mkey. The feature
is enabled only when CONFIG_BLK_INLINE_ENCRYPTION is configured.
There is no special configuration to enable crypto at nvme modules.

The code was tested with fscrypt and BF-3 HW, which had more than
50% CPU utilization improvement at 64K and bigger I/O sizes when
comparing to the SW only solution at this case.

Signed-off-by: Israel Rukshin <israelr@nvidia.com>
Signed-off-by: Leon Romanovsky <leon@kernel.org>
---
 drivers/nvme/host/rdma.c | 236 ++++++++++++++++++++++++++++++++++++++-
 1 file changed, 232 insertions(+), 4 deletions(-)

diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index bbad26b82b56..8bea38eb976f 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -40,6 +40,10 @@
 #define NVME_RDMA_METADATA_SGL_SIZE \
 	(sizeof(struct scatterlist) * NVME_INLINE_METADATA_SG_CNT)
 
+#define NVME_RDMA_NUM_CRYPTO_KEYSLOTS	(32U)
+/* Bitmask for 512B and 4K data unit sizes */
+#define NVME_RDMA_DATA_UNIT_SIZES	(512U | 4096U)
+
 struct nvme_rdma_device {
 	struct ib_device	*dev;
 	struct ib_pd		*pd;
@@ -75,6 +79,7 @@ struct nvme_rdma_request {
 	struct nvme_rdma_sgl	data_sgl;
 	struct nvme_rdma_sgl	*metadata_sgl;
 	bool			use_sig_mr;
+	bool			use_crypto;
 };
 
 enum nvme_rdma_queue_flags {
@@ -97,6 +102,7 @@ struct nvme_rdma_queue {
 	int			cm_error;
 	struct completion	cm_done;
 	bool			pi_support;
+	bool			crypto_support;
 	int			cq_size;
 	struct mutex		queue_lock;
 };
@@ -126,6 +132,8 @@ struct nvme_rdma_ctrl {
 	struct nvme_ctrl	ctrl;
 	bool			use_inline_data;
 	u32			io_queues[HCTX_MAX_TYPES];
+
+	struct ib_dek		*dek[NVME_RDMA_NUM_CRYPTO_KEYSLOTS];
 };
 
 static inline struct nvme_rdma_ctrl *to_rdma_ctrl(struct nvme_ctrl *ctrl)
@@ -275,6 +283,8 @@ static int nvme_rdma_create_qp(struct nvme_rdma_queue *queue, const int factor)
 	init_attr.recv_cq = queue->ib_cq;
 	if (queue->pi_support)
 		init_attr.create_flags |= IB_QP_CREATE_INTEGRITY_EN;
+	if (queue->crypto_support)
+		init_attr.create_flags |= IB_QP_CREATE_CRYPTO_EN;
 	init_attr.qp_context = queue;
 
 	ret = rdma_create_qp(queue->cm_id, dev->pd, &init_attr);
@@ -364,6 +374,77 @@ static int nvme_rdma_dev_get(struct nvme_rdma_device *dev)
 	return kref_get_unless_zero(&dev->ref);
 }
 
+static int nvme_rdma_crypto_keyslot_program(struct blk_crypto_profile *profile,
+					    const struct blk_crypto_key *key,
+					    unsigned int slot)
+{
+	struct nvme_ctrl *nctrl =
+		container_of(profile, struct nvme_ctrl, crypto_profile);
+	struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl);
+	struct ib_dek_attr dek_attr = { };
+	int err = 0;
+
+	if (slot > NVME_RDMA_NUM_CRYPTO_KEYSLOTS) {
+		dev_err(nctrl->device, "slot index reached the limit %d/%d",
+			slot, NVME_RDMA_NUM_CRYPTO_KEYSLOTS);
+		return -EOPNOTSUPP;
+	}
+
+	if (WARN_ON_ONCE(key->crypto_cfg.crypto_mode !=
+			 BLK_ENCRYPTION_MODE_AES_256_XTS))
+		return -EOPNOTSUPP;
+
+	if (ctrl->dek[slot]) {
+		dev_dbg(nctrl->device, "slot %d is taken, free DEK ID %d\n",
+			slot, ctrl->dek[slot]->id);
+		ib_destroy_dek(ctrl->dek[slot]);
+	}
+
+	dek_attr.key_blob = key->raw;
+	dek_attr.key_blob_size = key->size;
+	dek_attr.key_type = IB_CRYPTO_KEY_TYPE_AES_XTS;
+	ctrl->dek[slot] = ib_create_dek(ctrl->device->pd, &dek_attr);
+	if (IS_ERR(ctrl->dek[slot])) {
+		err = PTR_ERR(ctrl->dek[slot]);
+		dev_err(nctrl->device,
+			"failed to create DEK at slot %d, err %d\n", slot, err);
+		ctrl->dek[slot] = NULL;
+	} else {
+		dev_dbg(nctrl->device, "DEK ID %d was created at slot %d\n",
+			ctrl->dek[slot]->id, slot);
+	}
+
+	return err;
+}
+
+static int nvme_rdma_crypto_keyslot_evict(struct blk_crypto_profile *profile,
+					  const struct blk_crypto_key *key,
+					  unsigned int slot)
+{
+	struct nvme_ctrl *nctrl =
+		container_of(profile, struct nvme_ctrl, crypto_profile);
+	struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl);
+
+	if (slot > NVME_RDMA_NUM_CRYPTO_KEYSLOTS) {
+		dev_err(nctrl->device, "slot index reached the limit %d/%d\n",
+			slot, NVME_RDMA_NUM_CRYPTO_KEYSLOTS);
+		return -EOPNOTSUPP;
+	}
+
+	if (!ctrl->dek[slot]) {
+		dev_err(nctrl->device, "slot %d is empty\n", slot);
+		return -EINVAL;
+	}
+
+	dev_dbg(nctrl->device, "Destroy DEK ID %d slot %d\n",
+		ctrl->dek[slot]->id, slot);
+
+	ib_destroy_dek(ctrl->dek[slot]);
+	ctrl->dek[slot] = NULL;
+
+	return 0;
+}
+
 static struct nvme_rdma_device *
 nvme_rdma_find_get_device(struct rdma_cm_id *cm_id)
 {
@@ -430,6 +511,8 @@ static void nvme_rdma_destroy_queue_ib(struct nvme_rdma_queue *queue)
 	dev = queue->device;
 	ibdev = dev->dev;
 
+	if (queue->crypto_support)
+		ib_mr_pool_destroy(queue->qp, &queue->qp->crypto_mrs);
 	if (queue->pi_support)
 		ib_mr_pool_destroy(queue->qp, &queue->qp->sig_mrs);
 	ib_mr_pool_destroy(queue->qp, &queue->qp->rdma_mrs);
@@ -553,10 +636,24 @@ static int nvme_rdma_create_queue_ib(struct nvme_rdma_queue *queue)
 		}
 	}
 
+	if (queue->crypto_support) {
+		ret = ib_mr_pool_init(queue->qp, &queue->qp->crypto_mrs,
+				      queue->queue_size, IB_MR_TYPE_CRYPTO,
+				      pages_per_mr, 0);
+		if (ret) {
+			dev_err(queue->ctrl->ctrl.device,
+				"failed to initialize crypto MR pool sized %d for QID %d\n",
+				queue->queue_size, nvme_rdma_queue_idx(queue));
+			goto out_destroy_sig_mr_pool;
+		}
+	}
+
 	set_bit(NVME_RDMA_Q_TR_READY, &queue->flags);
 
 	return 0;
 
+out_destroy_sig_mr_pool:
+	ib_mr_pool_destroy(queue->qp, &queue->qp->sig_mrs);
 out_destroy_mr_pool:
 	ib_mr_pool_destroy(queue->qp, &queue->qp->rdma_mrs);
 out_destroy_ring:
@@ -585,6 +682,9 @@ static int nvme_rdma_alloc_queue(struct nvme_rdma_ctrl *ctrl,
 		queue->pi_support = true;
 	else
 		queue->pi_support = false;
+
+	queue->crypto_support = idx && ctrl->ctrl.crypto_enable;
+
 	init_completion(&queue->cm_done);
 
 	if (idx > 0)
@@ -805,15 +905,109 @@ static int nvme_rdma_alloc_tag_set(struct nvme_ctrl *ctrl)
 
 static void nvme_rdma_destroy_admin_queue(struct nvme_rdma_ctrl *ctrl)
 {
+	unsigned int i;
+
 	if (ctrl->async_event_sqe.data) {
 		cancel_work_sync(&ctrl->ctrl.async_event_work);
 		nvme_rdma_free_qe(ctrl->device->dev, &ctrl->async_event_sqe,
 				sizeof(struct nvme_command), DMA_TO_DEVICE);
 		ctrl->async_event_sqe.data = NULL;
 	}
+
+	for (i = 0; i < NVME_RDMA_NUM_CRYPTO_KEYSLOTS; i++) {
+		if (!ctrl->dek[i])
+			continue;
+		ib_destroy_dek(ctrl->dek[i]);
+		ctrl->dek[i] = NULL;
+	}
+
 	nvme_rdma_free_queue(&ctrl->queues[0]);
 }
 
+#ifdef CONFIG_BLK_INLINE_ENCRYPTION
+static const struct blk_crypto_ll_ops nvme_rdma_crypto_ops = {
+	.keyslot_program	= nvme_rdma_crypto_keyslot_program,
+	.keyslot_evict		= nvme_rdma_crypto_keyslot_evict,
+};
+
+static int nvme_rdma_crypto_profile_init(struct nvme_rdma_ctrl *ctrl, bool new)
+{
+	struct blk_crypto_profile *profile = &ctrl->ctrl.crypto_profile;
+	int ret;
+
+	if (!new) {
+		blk_crypto_reprogram_all_keys(profile);
+		return 0;
+	}
+
+	ret = devm_blk_crypto_profile_init(ctrl->ctrl.device, profile,
+					   NVME_RDMA_NUM_CRYPTO_KEYSLOTS);
+	if (ret) {
+		dev_err(ctrl->ctrl.device,
+			"devm_blk_crypto_profile_init failed err %d\n", ret);
+		return ret;
+	}
+
+	profile->ll_ops = nvme_rdma_crypto_ops;
+	profile->dev = ctrl->ctrl.device;
+	profile->max_dun_bytes_supported = IB_CRYPTO_XTS_TWEAK_MAX_SIZE;
+	profile->modes_supported[BLK_ENCRYPTION_MODE_AES_256_XTS] =
+		NVME_RDMA_DATA_UNIT_SIZES;
+
+	return 0;
+}
+
+static void nvme_rdma_set_crypto_attrs(struct ib_crypto_attrs *crypto_attrs,
+		struct nvme_rdma_queue *queue, struct nvme_rdma_request *req)
+{
+	struct request *rq = blk_mq_rq_from_pdu(req);
+	u32 slot_idx = blk_crypto_keyslot_index(rq->crypt_keyslot);
+
+	memset(crypto_attrs, 0, sizeof(*crypto_attrs));
+
+	crypto_attrs->encrypt_domain = IB_CRYPTO_ENCRYPTED_WIRE_DOMAIN;
+	crypto_attrs->encrypt_standard = IB_CRYPTO_AES_XTS;
+	crypto_attrs->data_unit_size =
+		rq->crypt_ctx->bc_key->crypto_cfg.data_unit_size;
+	crypto_attrs->dek = queue->ctrl->dek[slot_idx]->id;
+	memcpy(crypto_attrs->xts_init_tweak, rq->crypt_ctx->bc_dun,
+	       sizeof(crypto_attrs->xts_init_tweak));
+}
+
+static int nvme_rdma_fill_crypto_caps(struct nvme_rdma_ctrl *ctrl, bool new)
+{
+	struct ib_crypto_caps *caps = &ctrl->device->dev->attrs.crypto_caps;
+
+	if (caps->crypto_engines & IB_CRYPTO_ENGINES_CAP_AES_XTS) {
+		ctrl->ctrl.crypto_enable = true;
+		return 0;
+	}
+
+	if (!new && ctrl->ctrl.crypto_enable) {
+		dev_err(ctrl->ctrl.device, "Must support crypto!\n");
+		return -EOPNOTSUPP;
+	}
+	ctrl->ctrl.crypto_enable = false;
+
+	return 0;
+}
+#else
+static int nvme_rdma_crypto_profile_init(struct nvme_rdma_ctrl *ctrl, bool new)
+{
+	return -EOPNOTSUPP;
+}
+
+static void nvme_rdma_set_crypto_attrs(struct ib_crypto_attrs *crypto_attrs,
+		struct nvme_rdma_queue *queue, struct nvme_rdma_request *req)
+{
+}
+
+static int nvme_rdma_fill_crypto_caps(struct nvme_rdma_ctrl *ctrl, bool new)
+{
+	return 0;
+}
+#endif /* CONFIG_BLK_INLINE_ENCRYPTION */
+
 static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl,
 		bool new)
 {
@@ -835,6 +1029,10 @@ static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl,
 	ctrl->max_fr_pages = nvme_rdma_get_max_fr_pages(ctrl->device->dev,
 							pi_capable);
 
+	error = nvme_rdma_fill_crypto_caps(ctrl, new);
+	if (error)
+		goto out_free_queue;
+
 	/*
 	 * Bind the async event SQE DMA mapping to the admin queue lifetime.
 	 * It's safe, since any chage in the underlying RDMA device will issue
@@ -870,6 +1068,12 @@ static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl,
 	else
 		ctrl->ctrl.max_integrity_segments = 0;
 
+	if (ctrl->ctrl.crypto_enable) {
+		error = nvme_rdma_crypto_profile_init(ctrl, new);
+		if (error)
+			goto out_quiesce_queue;
+	}
+
 	nvme_unquiesce_admin_queue(&ctrl->ctrl);
 
 	error = nvme_init_ctrl_finish(&ctrl->ctrl, false);
@@ -1268,6 +1472,8 @@ static void nvme_rdma_unmap_data(struct nvme_rdma_queue *queue,
 
 	if (req->use_sig_mr)
 		pool = &queue->qp->sig_mrs;
+	else if (req->use_crypto)
+		pool = &queue->qp->crypto_mrs;
 
 	if (req->mr) {
 		ib_mr_pool_put(queue->qp, pool, req->mr);
@@ -1331,9 +1537,13 @@ static int nvme_rdma_map_sg_fr(struct nvme_rdma_queue *queue,
 		int count)
 {
 	struct nvme_keyed_sgl_desc *sg = &c->common.dptr.ksgl;
+	struct list_head *pool = &queue->qp->rdma_mrs;
 	int nr;
 
-	req->mr = ib_mr_pool_get(queue->qp, &queue->qp->rdma_mrs);
+	if (req->use_crypto)
+		pool = &queue->qp->crypto_mrs;
+
+	req->mr = ib_mr_pool_get(queue->qp, pool);
 	if (WARN_ON_ONCE(!req->mr))
 		return -EAGAIN;
 
@@ -1344,18 +1554,24 @@ static int nvme_rdma_map_sg_fr(struct nvme_rdma_queue *queue,
 	nr = ib_map_mr_sg(req->mr, req->data_sgl.sg_table.sgl, count, NULL,
 			  SZ_4K);
 	if (unlikely(nr < count)) {
-		ib_mr_pool_put(queue->qp, &queue->qp->rdma_mrs, req->mr);
+		ib_mr_pool_put(queue->qp, pool, req->mr);
 		req->mr = NULL;
 		if (nr < 0)
 			return nr;
 		return -EINVAL;
 	}
 
+	if (req->use_crypto)
+		nvme_rdma_set_crypto_attrs(req->mr->crypto_attrs, queue, req);
+
 	ib_update_fast_reg_key(req->mr, ib_inc_rkey(req->mr->rkey));
 
 	req->reg_cqe.done = nvme_rdma_memreg_done;
 	memset(&req->reg_wr, 0, sizeof(req->reg_wr));
-	req->reg_wr.wr.opcode = IB_WR_REG_MR;
+	if (req->use_crypto)
+		req->reg_wr.wr.opcode = IB_WR_REG_MR_CRYPTO;
+	else
+		req->reg_wr.wr.opcode = IB_WR_REG_MR;
 	req->reg_wr.wr.wr_cqe = &req->reg_cqe;
 	req->reg_wr.wr.num_sge = 0;
 	req->reg_wr.mr = req->mr;
@@ -1571,7 +1787,7 @@ static int nvme_rdma_map_data(struct nvme_rdma_queue *queue,
 		goto out;
 	}
 
-	if (count <= dev->num_inline_segments) {
+	if (count <= dev->num_inline_segments && !req->use_crypto) {
 		if (rq_data_dir(rq) == WRITE && nvme_rdma_queue_idx(queue) &&
 		    queue->ctrl->use_inline_data &&
 		    blk_rq_payload_bytes(rq) <=
@@ -2052,6 +2268,18 @@ static blk_status_t nvme_rdma_queue_rq(struct blk_mq_hw_ctx *hctx,
 	else
 		req->use_sig_mr = false;
 
+#ifdef CONFIG_BLK_INLINE_ENCRYPTION
+	if (rq->crypt_ctx) {
+		if (WARN_ON_ONCE(!queue->crypto_support)) {
+			err = -EIO;
+			goto err;
+		}
+		req->use_crypto = true;
+	} else {
+		req->use_crypto = false;
+	}
+#endif
+
 	err = nvme_rdma_map_data(queue, rq, c);
 	if (unlikely(err < 0)) {
 		dev_err(queue->ctrl->ctrl.device,
-- 
2.39.0

Re: [PATCH rdma-next 03/13] RDMA: Split kernel-only create QP flags from uverbs create QP flags

[Jason Gunthorpe]

On Mon, Jan 16, 2023 at 03:05:50PM +0200, Leon Romanovsky wrote:

> diff --git a/drivers/infiniband/hw/mlx4/mlx4_ib.h b/drivers/infiniband/hw/mlx4/mlx4_ib.h
> index 17fee1e73a45..c553bf0eb257 100644
> --- a/drivers/infiniband/hw/mlx4/mlx4_ib.h
> +++ b/drivers/infiniband/hw/mlx4/mlx4_ib.h
> @@ -184,7 +184,7 @@ enum mlx4_ib_qp_flags {
>  	/* Mellanox specific flags start from IB_QP_CREATE_RESERVED_START */
>  	MLX4_IB_ROCE_V2_GSI_QP = MLX4_IB_QP_CREATE_ROCE_V2_GSI,
>  	MLX4_IB_SRIOV_TUNNEL_QP = 1 << 30,
> -	MLX4_IB_SRIOV_SQP = 1 << 31,
> +	MLX4_IB_SRIOV_SQP = 1ULL << 31,
>  };

These should be moved to a uapi if we are saying they are userspace
available

But I'm not sure they are?


> diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
> index 949cf4ffc536..cc2ddd4e6c12 100644
> --- a/include/rdma/ib_verbs.h
> +++ b/include/rdma/ib_verbs.h
> @@ -1140,16 +1140,15 @@ enum ib_qp_type {
>  	IB_QPT_RESERVED10,
>  };
>  
> +/*
> + * bits 0, 5, 6 and 7 may be set by old kernels and should not be used.
> + */

This is backwards "bits 0 5 6 7 were understood by older kernels and
should not be used"

>  enum ib_qp_create_flags {
> -	IB_QP_CREATE_IPOIB_UD_LSO		= 1 << 0,
>  	IB_QP_CREATE_BLOCK_MULTICAST_LOOPBACK	=
>  		IB_UVERBS_QP_CREATE_BLOCK_MULTICAST_LOOPBACK,
>  	IB_QP_CREATE_CROSS_CHANNEL              = 1 << 2,
>  	IB_QP_CREATE_MANAGED_SEND               = 1 << 3,
>  	IB_QP_CREATE_MANAGED_RECV               = 1 << 4,
> -	IB_QP_CREATE_NETIF_QP			= 1 << 5,
> -	IB_QP_CREATE_INTEGRITY_EN		= 1 << 6,
> -	IB_QP_CREATE_NETDEV_USE			= 1 << 7,
>  	IB_QP_CREATE_SCATTER_FCS		=
>  		IB_UVERBS_QP_CREATE_SCATTER_FCS,
>  	IB_QP_CREATE_CVLAN_STRIPPING		=
> @@ -1159,7 +1158,18 @@ enum ib_qp_create_flags {
>  		IB_UVERBS_QP_CREATE_PCI_WRITE_END_PADDING,
>  	/* reserve bits 26-31 for low level drivers' internal use */
>  	IB_QP_CREATE_RESERVED_START		= 1 << 26,
> -	IB_QP_CREATE_RESERVED_END		= 1 << 31,
> +	IB_QP_CREATE_RESERVED_END		= 1ULL << 31,

And these should be shifted to the uapi header

Jason

Re: [PATCH rdma-next 12/13] nvme: Add crypto profile at nvme controller

[Chaitanya Kulkarni]

> diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
> index 51a9880db6ce..f09e4e0216b3 100644
> --- a/drivers/nvme/host/core.c
> +++ b/drivers/nvme/host/core.c
> @@ -1928,6 +1928,9 @@ static void nvme_update_disk_info(struct gendisk *disk,
>   			capacity = 0;
>   	}
>   
> +	if (ctrl->crypto_enable)
> +		blk_crypto_register(&ctrl->crypto_profile, disk->queue);
> +
>   	set_capacity_and_notify(disk, capacity);
>   
>   	nvme_config_discard(disk, ns);
> diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
> index 424c8a467a0c..591380f53744 100644
> --- a/drivers/nvme/host/nvme.h
> +++ b/drivers/nvme/host/nvme.h
> @@ -16,6 +16,7 @@
>   #include <linux/rcupdate.h>
>   #include <linux/wait.h>
>   #include <linux/t10-pi.h>
> +#include <linux/blk-crypto-profile.h>
>   
>   #include <trace/events/block.h>
>   
> @@ -374,6 +375,9 @@ struct nvme_ctrl {
>   
>   	enum nvme_ctrl_type cntrltype;
>   	enum nvme_dctype dctype;
> +
> +	bool crypto_enable;

why not decalre crypto_profile a pointer, allocate that at init
controller and NULL check against that pointer instead of using
an extra variable crypto_enable ?

e.g. :-

	if (ctrl->crypto_profile)
		blk_crypto_register(ctrl->crypto_profile, disk->queue);

> +	struct blk_crypto_profile crypto_profile;

you are increasing the size of struct nvme_ctrl unconditionally,
why not guard above with CONFIG_BLK_INLINE_ENCRYPTION ?

-ck

Re: [PATCH rdma-next 11/13] nvme: Introduce a local variable

[Chaitanya Kulkarni]

On 1/16/23 05:05, Leon Romanovsky wrote:
> From: Israel Rukshin <israelr@nvidia.com>
> 
> The patch doesn't change any logic.
> 
> Signed-off-by: Israel Rukshin <israelr@nvidia.com>
> Signed-off-by: Leon Romanovsky <leon@kernel.org>
> ---
>   drivers/nvme/host/core.c | 7 ++++---
>   1 file changed, 4 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
> index 7be562a4e1aa..51a9880db6ce 100644
> --- a/drivers/nvme/host/core.c
> +++ b/drivers/nvme/host/core.c
> @@ -1870,6 +1870,7 @@ static void nvme_update_disk_info(struct gendisk *disk,
>   	sector_t capacity = nvme_lba_to_sect(ns, le64_to_cpu(id->nsze));
>   	unsigned short bs = 1 << ns->lba_shift;
>   	u32 atomic_bs, phys_bs, io_opt = 0;
> +	struct nvme_ctrl *ctrl = ns->ctrl;
>   

I don't think this patch is needed, existing code is more readable and
gives much clarity that we are accessing ctrl from namespace since we
are in nvme_update_disk_info() which is namespace based.

-ck

Re: [PATCH rdma-next 12/13] nvme: Add crypto profile at nvme controller

[Leon Romanovsky]

On Tue, Jan 17, 2023 at 12:31:22AM +0000, Chaitanya Kulkarni wrote:
> > diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
> > index 51a9880db6ce..f09e4e0216b3 100644
> > --- a/drivers/nvme/host/core.c
> > +++ b/drivers/nvme/host/core.c
> > @@ -1928,6 +1928,9 @@ static void nvme_update_disk_info(struct gendisk *disk,
> >   			capacity = 0;
> >   	}
> >   
> > +	if (ctrl->crypto_enable)
> > +		blk_crypto_register(&ctrl->crypto_profile, disk->queue);
> > +
> >   	set_capacity_and_notify(disk, capacity);
> >   
> >   	nvme_config_discard(disk, ns);
> > diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
> > index 424c8a467a0c..591380f53744 100644
> > --- a/drivers/nvme/host/nvme.h
> > +++ b/drivers/nvme/host/nvme.h
> > @@ -16,6 +16,7 @@
> >   #include <linux/rcupdate.h>
> >   #include <linux/wait.h>
> >   #include <linux/t10-pi.h>
> > +#include <linux/blk-crypto-profile.h>
> >   
> >   #include <trace/events/block.h>
> >   
> > @@ -374,6 +375,9 @@ struct nvme_ctrl {
> >   
> >   	enum nvme_ctrl_type cntrltype;
> >   	enum nvme_dctype dctype;
> > +
> > +	bool crypto_enable;
> 
> why not decalre crypto_profile a pointer, allocate that at init
> controller and NULL check against that pointer instead of using
> an extra variable crypto_enable ?

The embedded struct allows us to rely on container_of() macro.

   377 static int nvme_rdma_crypto_keyslot_program(struct blk_crypto_profile *profile,
   378                                             const struct blk_crypto_key *key,
   379                                             unsigned int slot)
   380 {
   381         struct nvme_ctrl *nctrl =
   382                 container_of(profile, struct nvme_ctrl, crypto_profile);

> 
> e.g. :-
> 
> 	if (ctrl->crypto_profile)
> 		blk_crypto_register(ctrl->crypto_profile, disk->queue);
> 
> > +	struct blk_crypto_profile crypto_profile;
> 
> you are increasing the size of struct nvme_ctrl unconditionally,
> why not guard above with CONFIG_BLK_INLINE_ENCRYPTION ?

We can do it.

Thanks

> 
> -ck
> 

Re: [PATCH rdma-next 11/13] nvme: Introduce a local variable

[Leon Romanovsky]

On Tue, Jan 17, 2023 at 12:32:59AM +0000, Chaitanya Kulkarni wrote:
> On 1/16/23 05:05, Leon Romanovsky wrote:
> > From: Israel Rukshin <israelr@nvidia.com>
> > 
> > The patch doesn't change any logic.
> > 
> > Signed-off-by: Israel Rukshin <israelr@nvidia.com>
> > Signed-off-by: Leon Romanovsky <leon@kernel.org>
> > ---
> >   drivers/nvme/host/core.c | 7 ++++---
> >   1 file changed, 4 insertions(+), 3 deletions(-)
> > 
> > diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
> > index 7be562a4e1aa..51a9880db6ce 100644
> > --- a/drivers/nvme/host/core.c
> > +++ b/drivers/nvme/host/core.c
> > @@ -1870,6 +1870,7 @@ static void nvme_update_disk_info(struct gendisk *disk,
> >   	sector_t capacity = nvme_lba_to_sect(ns, le64_to_cpu(id->nsze));
> >   	unsigned short bs = 1 << ns->lba_shift;
> >   	u32 atomic_bs, phys_bs, io_opt = 0;
> > +	struct nvme_ctrl *ctrl = ns->ctrl;
> >   
> 
> I don't think this patch is needed, existing code is more readable and
> gives much clarity that we are accessing ctrl from namespace since we
> are in nvme_update_disk_info() which is namespace based.

I don't have strong opinion here.

Thanks

> 
> -ck
> 

Re: [PATCH rdma-next 03/13] RDMA: Split kernel-only create QP flags from uverbs create QP flags

[Leon Romanovsky]

On Mon, Jan 16, 2023 at 01:39:38PM -0400, Jason Gunthorpe wrote:
> On Mon, Jan 16, 2023 at 03:05:50PM +0200, Leon Romanovsky wrote:
> 
> > diff --git a/drivers/infiniband/hw/mlx4/mlx4_ib.h b/drivers/infiniband/hw/mlx4/mlx4_ib.h
> > index 17fee1e73a45..c553bf0eb257 100644
> > --- a/drivers/infiniband/hw/mlx4/mlx4_ib.h
> > +++ b/drivers/infiniband/hw/mlx4/mlx4_ib.h
> > @@ -184,7 +184,7 @@ enum mlx4_ib_qp_flags {
> >  	/* Mellanox specific flags start from IB_QP_CREATE_RESERVED_START */
> >  	MLX4_IB_ROCE_V2_GSI_QP = MLX4_IB_QP_CREATE_ROCE_V2_GSI,
> >  	MLX4_IB_SRIOV_TUNNEL_QP = 1 << 30,
> > -	MLX4_IB_SRIOV_SQP = 1 << 31,
> > +	MLX4_IB_SRIOV_SQP = 1ULL << 31,
> >  };
> 
> These should be moved to a uapi if we are saying they are userspace
> available
> 
> But I'm not sure they are?

I don't think so.

> 
> 
> > diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
> > index 949cf4ffc536..cc2ddd4e6c12 100644
> > --- a/include/rdma/ib_verbs.h
> > +++ b/include/rdma/ib_verbs.h
> > @@ -1140,16 +1140,15 @@ enum ib_qp_type {
> >  	IB_QPT_RESERVED10,
> >  };
> >  
> > +/*
> > + * bits 0, 5, 6 and 7 may be set by old kernels and should not be used.
> > + */
> 
> This is backwards "bits 0 5 6 7 were understood by older kernels and
> should not be used"

will change

> 
> >  enum ib_qp_create_flags {
> > -	IB_QP_CREATE_IPOIB_UD_LSO		= 1 << 0,
> >  	IB_QP_CREATE_BLOCK_MULTICAST_LOOPBACK	=
> >  		IB_UVERBS_QP_CREATE_BLOCK_MULTICAST_LOOPBACK,
> >  	IB_QP_CREATE_CROSS_CHANNEL              = 1 << 2,
> >  	IB_QP_CREATE_MANAGED_SEND               = 1 << 3,
> >  	IB_QP_CREATE_MANAGED_RECV               = 1 << 4,
> > -	IB_QP_CREATE_NETIF_QP			= 1 << 5,
> > -	IB_QP_CREATE_INTEGRITY_EN		= 1 << 6,
> > -	IB_QP_CREATE_NETDEV_USE			= 1 << 7,
> >  	IB_QP_CREATE_SCATTER_FCS		=
> >  		IB_UVERBS_QP_CREATE_SCATTER_FCS,
> >  	IB_QP_CREATE_CVLAN_STRIPPING		=
> > @@ -1159,7 +1158,18 @@ enum ib_qp_create_flags {
> >  		IB_UVERBS_QP_CREATE_PCI_WRITE_END_PADDING,
> >  	/* reserve bits 26-31 for low level drivers' internal use */
> >  	IB_QP_CREATE_RESERVED_START		= 1 << 26,
> > -	IB_QP_CREATE_RESERVED_END		= 1 << 31,
> > +	IB_QP_CREATE_RESERVED_END		= 1ULL << 31,
> 
> And these should be shifted to the uapi header

No problem.

> 
> Jason

Re: [PATCH rdma-next 03/13] RDMA: Split kernel-only create QP flags from uverbs create QP flags

[Jason Gunthorpe]

On Tue, Jan 17, 2023 at 02:03:07PM +0200, Leon Romanovsky wrote:
> On Mon, Jan 16, 2023 at 01:39:38PM -0400, Jason Gunthorpe wrote:
> > On Mon, Jan 16, 2023 at 03:05:50PM +0200, Leon Romanovsky wrote:
> > 
> > > diff --git a/drivers/infiniband/hw/mlx4/mlx4_ib.h b/drivers/infiniband/hw/mlx4/mlx4_ib.h
> > > index 17fee1e73a45..c553bf0eb257 100644
> > > --- a/drivers/infiniband/hw/mlx4/mlx4_ib.h
> > > +++ b/drivers/infiniband/hw/mlx4/mlx4_ib.h
> > > @@ -184,7 +184,7 @@ enum mlx4_ib_qp_flags {
> > >  	/* Mellanox specific flags start from IB_QP_CREATE_RESERVED_START */
> > >  	MLX4_IB_ROCE_V2_GSI_QP = MLX4_IB_QP_CREATE_ROCE_V2_GSI,
> > >  	MLX4_IB_SRIOV_TUNNEL_QP = 1 << 30,
> > > -	MLX4_IB_SRIOV_SQP = 1 << 31,
> > > +	MLX4_IB_SRIOV_SQP = 1ULL << 31,
> > >  };
> > 
> > These should be moved to a uapi if we are saying they are userspace
> > available
> > 
> > But I'm not sure they are?
> 
> I don't think so.

Then they should be > 32 bits right?

Jason

Re: [PATCH rdma-next 03/13] RDMA: Split kernel-only create QP flags from uverbs create QP flags

[Leon Romanovsky]

On Tue, Jan 17, 2023 at 09:49:28AM -0400, Jason Gunthorpe wrote:
> On Tue, Jan 17, 2023 at 02:03:07PM +0200, Leon Romanovsky wrote:
> > On Mon, Jan 16, 2023 at 01:39:38PM -0400, Jason Gunthorpe wrote:
> > > On Mon, Jan 16, 2023 at 03:05:50PM +0200, Leon Romanovsky wrote:
> > > 
> > > > diff --git a/drivers/infiniband/hw/mlx4/mlx4_ib.h b/drivers/infiniband/hw/mlx4/mlx4_ib.h
> > > > index 17fee1e73a45..c553bf0eb257 100644
> > > > --- a/drivers/infiniband/hw/mlx4/mlx4_ib.h
> > > > +++ b/drivers/infiniband/hw/mlx4/mlx4_ib.h
> > > > @@ -184,7 +184,7 @@ enum mlx4_ib_qp_flags {
> > > >  	/* Mellanox specific flags start from IB_QP_CREATE_RESERVED_START */
> > > >  	MLX4_IB_ROCE_V2_GSI_QP = MLX4_IB_QP_CREATE_ROCE_V2_GSI,
> > > >  	MLX4_IB_SRIOV_TUNNEL_QP = 1 << 30,
> > > > -	MLX4_IB_SRIOV_SQP = 1 << 31,
> > > > +	MLX4_IB_SRIOV_SQP = 1ULL << 31,
> > > >  };
> > > 
> > > These should be moved to a uapi if we are saying they are userspace
> > > available
> > > 
> > > But I'm not sure they are?
> > 
> > I don't think so.
> 
> Then they should be > 32 bits right?

Right now, they are in reserved range:
        /* reserve bits 26-31 for low level drivers' internal use */
        IB_QP_CREATE_RESERVED_START             = 1 << 26,
        IB_QP_CREATE_RESERVED_END               = 1ULL << 31,

If we move them to kernel part, we will need to define reserved range
there too. So we just "burn" extra bits just for mlx4, also I don't see
any reason to promote mlx4 bits to be general ones.

Thanks

> 
> Jason

Re: [PATCH rdma-next 03/13] RDMA: Split kernel-only create QP flags from uverbs create QP flags

[Jason Gunthorpe]

On Tue, Jan 17, 2023 at 04:14:22PM +0200, Leon Romanovsky wrote:
> On Tue, Jan 17, 2023 at 09:49:28AM -0400, Jason Gunthorpe wrote:
> > On Tue, Jan 17, 2023 at 02:03:07PM +0200, Leon Romanovsky wrote:
> > > On Mon, Jan 16, 2023 at 01:39:38PM -0400, Jason Gunthorpe wrote:
> > > > On Mon, Jan 16, 2023 at 03:05:50PM +0200, Leon Romanovsky wrote:
> > > > 
> > > > > diff --git a/drivers/infiniband/hw/mlx4/mlx4_ib.h b/drivers/infiniband/hw/mlx4/mlx4_ib.h
> > > > > index 17fee1e73a45..c553bf0eb257 100644
> > > > > --- a/drivers/infiniband/hw/mlx4/mlx4_ib.h
> > > > > +++ b/drivers/infiniband/hw/mlx4/mlx4_ib.h
> > > > > @@ -184,7 +184,7 @@ enum mlx4_ib_qp_flags {
> > > > >  	/* Mellanox specific flags start from IB_QP_CREATE_RESERVED_START */
> > > > >  	MLX4_IB_ROCE_V2_GSI_QP = MLX4_IB_QP_CREATE_ROCE_V2_GSI,
> > > > >  	MLX4_IB_SRIOV_TUNNEL_QP = 1 << 30,
> > > > > -	MLX4_IB_SRIOV_SQP = 1 << 31,
> > > > > +	MLX4_IB_SRIOV_SQP = 1ULL << 31,
> > > > >  };
> > > > 
> > > > These should be moved to a uapi if we are saying they are userspace
> > > > available
> > > > 
> > > > But I'm not sure they are?
> > > 
> > > I don't think so.
> > 
> > Then they should be > 32 bits right?
> 
> Right now, they are in reserved range:
>         /* reserve bits 26-31 for low level drivers' internal use */
>         IB_QP_CREATE_RESERVED_START             = 1 << 26,
>         IB_QP_CREATE_RESERVED_END               = 1ULL << 31,
> 
> If we move them to kernel part, we will need to define reserved range
> there too. So we just "burn" extra bits just for mlx4, also I don't see
> any reason to promote mlx4 bits to be general ones.

Is the reserved range kernel only? It would be nice to clarify that
detail

If yes we should move it so that userspace cannot set it. Do we have a
bug here already?

Jason

Re: [PATCH rdma-next 03/13] RDMA: Split kernel-only create QP flags from uverbs create QP flags

[Leon Romanovsky]

On Tue, Jan 17, 2023 at 10:21:18AM -0400, Jason Gunthorpe wrote:
> On Tue, Jan 17, 2023 at 04:14:22PM +0200, Leon Romanovsky wrote:
> > On Tue, Jan 17, 2023 at 09:49:28AM -0400, Jason Gunthorpe wrote:
> > > On Tue, Jan 17, 2023 at 02:03:07PM +0200, Leon Romanovsky wrote:
> > > > On Mon, Jan 16, 2023 at 01:39:38PM -0400, Jason Gunthorpe wrote:
> > > > > On Mon, Jan 16, 2023 at 03:05:50PM +0200, Leon Romanovsky wrote:
> > > > > 
> > > > > > diff --git a/drivers/infiniband/hw/mlx4/mlx4_ib.h b/drivers/infiniband/hw/mlx4/mlx4_ib.h
> > > > > > index 17fee1e73a45..c553bf0eb257 100644
> > > > > > --- a/drivers/infiniband/hw/mlx4/mlx4_ib.h
> > > > > > +++ b/drivers/infiniband/hw/mlx4/mlx4_ib.h
> > > > > > @@ -184,7 +184,7 @@ enum mlx4_ib_qp_flags {
> > > > > >  	/* Mellanox specific flags start from IB_QP_CREATE_RESERVED_START */
> > > > > >  	MLX4_IB_ROCE_V2_GSI_QP = MLX4_IB_QP_CREATE_ROCE_V2_GSI,
> > > > > >  	MLX4_IB_SRIOV_TUNNEL_QP = 1 << 30,
> > > > > > -	MLX4_IB_SRIOV_SQP = 1 << 31,
> > > > > > +	MLX4_IB_SRIOV_SQP = 1ULL << 31,
> > > > > >  };
> > > > > 
> > > > > These should be moved to a uapi if we are saying they are userspace
> > > > > available
> > > > > 
> > > > > But I'm not sure they are?
> > > > 
> > > > I don't think so.
> > > 
> > > Then they should be > 32 bits right?
> > 
> > Right now, they are in reserved range:
> >         /* reserve bits 26-31 for low level drivers' internal use */
> >         IB_QP_CREATE_RESERVED_START             = 1 << 26,
> >         IB_QP_CREATE_RESERVED_END               = 1ULL << 31,
> > 
> > If we move them to kernel part, we will need to define reserved range
> > there too. So we just "burn" extra bits just for mlx4, also I don't see
> > any reason to promote mlx4 bits to be general ones.
> 
> Is the reserved range kernel only? It would be nice to clarify that
> detail
> 
> If yes we should move it so that userspace cannot set it. Do we have a
> bug here already?

No, we always checked that users can't provide these bits and fail create QP.

It means that we can safely move that range too.

Thanks

> 
> Jason

Re: [PATCH rdma-next 06/13] RDMA/core: Introduce MR type for crypto operations

[Steven Rostedt]

On Mon, 16 Jan 2023 15:05:53 +0200
Leon Romanovsky <leon@kernel.org> wrote:

> index 17642aa54437..b6a3d82b89ca 100644
> --- a/include/trace/events/rdma_core.h
> +++ b/include/trace/events/rdma_core.h
> @@ -371,6 +371,39 @@ TRACE_EVENT(mr_integ_alloc,
>  		__entry->max_num_meta_sg, __entry->rc)
>  );
>  
> +TRACE_EVENT(mr_crypto_alloc,
> +	TP_PROTO(
> +		const struct ib_pd *pd,
> +		u32 max_num_sg,
> +		const struct ib_mr *mr
> +	),
> +
> +	TP_ARGS(pd, max_num_sg, mr),
> +
> +	TP_STRUCT__entry(
> +		__field(u32, pd_id)
> +		__field(u32, mr_id)
> +		__field(u32, max_num_sg)
> +		__field(int, rc)
> +	),
> +
> +	TP_fast_assign(
> +		__entry->pd_id = pd->res.id;
> +		if (IS_ERR(mr)) {
> +			__entry->mr_id = 0;
> +			__entry->rc = PTR_ERR(mr);
> +		} else {
> +			__entry->mr_id = mr->res.id;
> +			__entry->rc = 0;
> +		}
> +		__entry->max_num_sg = max_num_sg;
> +	),
> +
> +	TP_printk("pd.id=%u mr.id=%u max_num_sg=%u rc=%d",
> +		__entry->pd_id, __entry->mr_id, __entry->max_num_sg,
> +		__entry->rc)
> +);
> +

Reviewed-by: Steven Rostedt (Google) <rostedt@goodmis.org>

-- Steve

>  TRACE_EVENT(mr_dereg,
>  	TP_PROTO(
>  		const struct ib_mr *mr

Re: [PATCH rdma-next 00/13] Add RDMA inline crypto support

[Eric Biggers]

Hi Leon,

On Mon, Jan 16, 2023 at 03:05:47PM +0200, Leon Romanovsky wrote:
> >From Israel,
> 
> The purpose of this patchset is to add support for inline
> encryption/decryption of the data at storage protocols like nvmf over
> RDMA (at a similar way like integrity is used via unique mkey).
> 
> This patchset adds support for plaintext keys. The patches were tested
> on BF-3 HW with fscrypt tool to test this feature, which showed reduce
> in CPU utilization when comparing at 64k or more IO size. The CPU utilization
> was improved by more than 50% comparing to the SW only solution at this case.
> 
> How to configure fscrypt to enable plaintext keys:
>  # mkfs.ext4 -O encrypt /dev/nvme0n1
>  # mount /dev/nvme0n1 /mnt/crypto -o inlinecrypt
>  # head -c 64 /dev/urandom > /tmp/master_key
>  # fscryptctl add_key /mnt/crypto/ < /tmp/master_key
>  # mkdir /mnt/crypto/test1
>  # fscryptctl set_policy 152c41b2ea39fa3d90ea06448456e7fb /mnt/crypto/test1
>    ** “152c41b2ea39fa3d90ea06448456e7fb” is the output of the
>       “fscryptctl add_key” command.
>  # echo foo > /mnt/crypto/test1/foo
> 
> Notes:
>  - At plaintext mode only, the user set a master key and the fscrypt
>    driver derived from it the DEK and the key identifier.
>  - 152c41b2ea39fa3d90ea06448456e7fb is the derived key identifier
>  - Only on the first IO, nvme-rdma gets a callback to load the derived DEK. 
> 
> There is no special configuration to support crypto at nvme modules.
> 
> Thanks

Very interesting work!  Can you Cc me on future versions?

I'm glad to see that this hardware allows all 16 IV bytes to be specified.

Does it also handle programming and evicting keys efficiently?

Also, just checking: have you tested that the ciphertext that this inline
encryption hardware produces is correct?  That's always super important to test.
There are xfstests that test for it, e.g. generic/582.  Another way to test it
is to just manually test whether encrypted files that were created when the
filesystem was mounted with '-o inlinecrypt' show the same contents when the
filesystem is *not* mounted with '-o inlinecrypt' (or vice versa).

- Eric

Re: [PATCH rdma-next 00/13] Add RDMA inline crypto support

[Chaitanya Kulkarni]

Eric,

>> Notes:
>>   - At plaintext mode only, the user set a master key and the fscrypt
>>     driver derived from it the DEK and the key identifier.
>>   - 152c41b2ea39fa3d90ea06448456e7fb is the derived key identifier
>>   - Only on the first IO, nvme-rdma gets a callback to load the derived DEK.
>>
>> There is no special configuration to support crypto at nvme modules.
>>
>> Thanks
> 
> Very interesting work!  Can you Cc me on future versions?
> 
> I'm glad to see that this hardware allows all 16 IV bytes to be specified.
> 
> Does it also handle programming and evicting keys efficiently?
> 
> Also, just checking: have you tested that the ciphertext that this inline
> encryption hardware produces is correct?  That's always super important to test.
> There are xfstests that test for it, e.g. generic/582.  Another way to test it
> is to just manually test whether encrypted files that were created when the
> filesystem was mounted with '-o inlinecrypt' show the same contents when the
> filesystem is *not* mounted with '-o inlinecrypt' (or vice versa).
> 
> - Eric
> 

I'm wondering which are the xfstests that needs to run in order
to establish the correctness/stability apart from generic/582
this work ?

-ck

Re: [PATCH rdma-next 00/13] Add RDMA inline crypto support

[Eric Biggers]

On Wed, Jan 18, 2023 at 07:14:30AM +0000, Chaitanya Kulkarni wrote:
> Eric,
> 
> >> Notes:
> >>   - At plaintext mode only, the user set a master key and the fscrypt
> >>     driver derived from it the DEK and the key identifier.
> >>   - 152c41b2ea39fa3d90ea06448456e7fb is the derived key identifier
> >>   - Only on the first IO, nvme-rdma gets a callback to load the derived DEK.
> >>
> >> There is no special configuration to support crypto at nvme modules.
> >>
> >> Thanks
> > 
> > Very interesting work!  Can you Cc me on future versions?
> > 
> > I'm glad to see that this hardware allows all 16 IV bytes to be specified.
> > 
> > Does it also handle programming and evicting keys efficiently?
> > 
> > Also, just checking: have you tested that the ciphertext that this inline
> > encryption hardware produces is correct?  That's always super important to test.
> > There are xfstests that test for it, e.g. generic/582.  Another way to test it
> > is to just manually test whether encrypted files that were created when the
> > filesystem was mounted with '-o inlinecrypt' show the same contents when the
> > filesystem is *not* mounted with '-o inlinecrypt' (or vice versa).
> > 
> > - Eric
> > 
> 
> I'm wondering which are the xfstests that needs to run in order
> to establish the correctness/stability apart from generic/582
> this work ?
> 

See https://docs.kernel.org/filesystems/fscrypt.html#tests.

- Eric

Re: [PATCH rdma-next 00/13] Add RDMA inline crypto support

[Christoph Hellwig]

On Mon, Jan 16, 2023 at 03:05:47PM +0200, Leon Romanovsky wrote:
> >From Israel,
> 
> The purpose of this patchset is to add support for inline
> encryption/decryption of the data at storage protocols like nvmf over
> RDMA (at a similar way like integrity is used via unique mkey).
> 
> This patchset adds support for plaintext keys. The patches were tested
> on BF-3 HW with fscrypt tool to test this feature, which showed reduce
> in CPU utilization when comparing at 64k or more IO size. The CPU utilization
> was improved by more than 50% comparing to the SW only solution at this case.

One thing that needs to be solved before we can look into this is the
interaction with protection information (or integrity data in Linux
terms).  Currently inline encryption and protection information are
mutally incompatible.

Re: [PATCH rdma-next 00/13] Add RDMA inline crypto support

[Leon Romanovsky]

On Tue, Jan 17, 2023 at 10:47:44PM -0800, Eric Biggers wrote:
> Hi Leon,
> 
> On Mon, Jan 16, 2023 at 03:05:47PM +0200, Leon Romanovsky wrote:
> > >From Israel,
> > 
> > The purpose of this patchset is to add support for inline
> > encryption/decryption of the data at storage protocols like nvmf over
> > RDMA (at a similar way like integrity is used via unique mkey).
> > 
> > This patchset adds support for plaintext keys. The patches were tested
> > on BF-3 HW with fscrypt tool to test this feature, which showed reduce
> > in CPU utilization when comparing at 64k or more IO size. The CPU utilization
> > was improved by more than 50% comparing to the SW only solution at this case.
> > 
> > How to configure fscrypt to enable plaintext keys:
> >  # mkfs.ext4 -O encrypt /dev/nvme0n1
> >  # mount /dev/nvme0n1 /mnt/crypto -o inlinecrypt
> >  # head -c 64 /dev/urandom > /tmp/master_key
> >  # fscryptctl add_key /mnt/crypto/ < /tmp/master_key
> >  # mkdir /mnt/crypto/test1
> >  # fscryptctl set_policy 152c41b2ea39fa3d90ea06448456e7fb /mnt/crypto/test1
> >    ** “152c41b2ea39fa3d90ea06448456e7fb” is the output of the
> >       “fscryptctl add_key” command.
> >  # echo foo > /mnt/crypto/test1/foo
> > 
> > Notes:
> >  - At plaintext mode only, the user set a master key and the fscrypt
> >    driver derived from it the DEK and the key identifier.
> >  - 152c41b2ea39fa3d90ea06448456e7fb is the derived key identifier
> >  - Only on the first IO, nvme-rdma gets a callback to load the derived DEK. 
> > 
> > There is no special configuration to support crypto at nvme modules.
> > 
> > Thanks
> 
> Very interesting work!  Can you Cc me on future versions?

Sure

> 
> I'm glad to see that this hardware allows all 16 IV bytes to be specified.
> 
> Does it also handle programming and evicting keys efficiently?

"efficiently" is a very subjective term. We are using FW command
interface to program keys and this interface can do hundreds/thousands
commands per-second.

Thanks

Re: [PATCH rdma-next 00/13] Add RDMA inline crypto support

[Israel Rukshin]

Hi Eric,

On 1/18/2023 8:47 AM, Eric Biggers wrote:
> Hi Leon,
>
> On Mon, Jan 16, 2023 at 03:05:47PM +0200, Leon Romanovsky wrote:
>> >From Israel,
>>
>> The purpose of this patchset is to add support for inline
>> encryption/decryption of the data at storage protocols like nvmf over
>> RDMA (at a similar way like integrity is used via unique mkey).
>>
>> This patchset adds support for plaintext keys. The patches were tested
>> on BF-3 HW with fscrypt tool to test this feature, which showed reduce
>> in CPU utilization when comparing at 64k or more IO size. The CPU utilization
>> was improved by more than 50% comparing to the SW only solution at this case.
>>
>> How to configure fscrypt to enable plaintext keys:
>>   # mkfs.ext4 -O encrypt /dev/nvme0n1
>>   # mount /dev/nvme0n1 /mnt/crypto -o inlinecrypt
>>   # head -c 64 /dev/urandom > /tmp/master_key
>>   # fscryptctl add_key /mnt/crypto/ < /tmp/master_key
>>   # mkdir /mnt/crypto/test1
>>   # fscryptctl set_policy 152c41b2ea39fa3d90ea06448456e7fb /mnt/crypto/test1
>>     ** “152c41b2ea39fa3d90ea06448456e7fb” is the output of the
>>        “fscryptctl add_key” command.
>>   # echo foo > /mnt/crypto/test1/foo
>>
>> Notes:
>>   - At plaintext mode only, the user set a master key and the fscrypt
>>     driver derived from it the DEK and the key identifier.
>>   - 152c41b2ea39fa3d90ea06448456e7fb is the derived key identifier
>>   - Only on the first IO, nvme-rdma gets a callback to load the derived DEK.
>>
>> There is no special configuration to support crypto at nvme modules.
>>
>> Thanks
> Very interesting work!  Can you Cc me on future versions?
>
> I'm glad to see that this hardware allows all 16 IV bytes to be specified.
>
> Does it also handle programming and evicting keys efficiently?
>
> Also, just checking: have you tested that the ciphertext that this inline
> encryption hardware produces is correct?  That's always super important to test.
> There are xfstests that test for it, e.g. generic/582.  Another way to test it
> is to just manually test whether encrypted files that were created when the
> filesystem was mounted with '-o inlinecrypt' show the same contents when the
> filesystem is *not* mounted with '-o inlinecrypt' (or vice versa).
sure, I ran the manual test of comparing the encrypted files content 
with and without the '-o inlinecrypt' at the mount command.
>
> - Eric
  - Israel

Re: [PATCH rdma-next 00/13] Add RDMA inline crypto support

[Max Gurtovoy]

On 18/01/2023 9:36, Christoph Hellwig wrote:
> On Mon, Jan 16, 2023 at 03:05:47PM +0200, Leon Romanovsky wrote:
>> >From Israel,
>>
>> The purpose of this patchset is to add support for inline
>> encryption/decryption of the data at storage protocols like nvmf over
>> RDMA (at a similar way like integrity is used via unique mkey).
>>
>> This patchset adds support for plaintext keys. The patches were tested
>> on BF-3 HW with fscrypt tool to test this feature, which showed reduce
>> in CPU utilization when comparing at 64k or more IO size. The CPU utilization
>> was improved by more than 50% comparing to the SW only solution at this case.
> One thing that needs to be solved before we can look into this is the
> interaction with protection information (or integrity data in Linux
> terms).  Currently inline encryption and protection information are
> mutally incompatible.

Well, for sure this should be solved. Not sure that it should be done 
before this series.

This patch set doesn't break/change existing behavior of PI and 
Encryption offloads so the above can be done incrementally.
It's already big enough series and 2 subsystems are involved.

Today, if one would like to run both features using a capable device 
then the PI will be offloaded (no SW fallback) and the Encryption will 
be using SW fallback.
There should be a serious instrumentation in the block layer to make 
both operations offloaded.
We'll start looking into it. Any suggestions and designs are welcomed.

We also prepared patches to extend the blk lavel encryption (in addition 
to fscrypt exist today).

Re: [PATCH rdma-next 00/13] Add RDMA inline crypto support

[Sagi Grimberg]

>  From Israel,
> 
> The purpose of this patchset is to add support for inline
> encryption/decryption of the data at storage protocols like nvmf over
> RDMA (at a similar way like integrity is used via unique mkey).
> 
> This patchset adds support for plaintext keys. The patches were tested
> on BF-3 HW with fscrypt tool to test this feature, which showed reduce
> in CPU utilization when comparing at 64k or more IO size. The CPU utilization
> was improved by more than 50% comparing to the SW only solution at this case.
> 
> How to configure fscrypt to enable plaintext keys:
>   # mkfs.ext4 -O encrypt /dev/nvme0n1
>   # mount /dev/nvme0n1 /mnt/crypto -o inlinecrypt
>   # head -c 64 /dev/urandom > /tmp/master_key
>   # fscryptctl add_key /mnt/crypto/ < /tmp/master_key
>   # mkdir /mnt/crypto/test1
>   # fscryptctl set_policy 152c41b2ea39fa3d90ea06448456e7fb /mnt/crypto/test1
>     ** “152c41b2ea39fa3d90ea06448456e7fb” is the output of the
>        “fscryptctl add_key” command.
>   # echo foo > /mnt/crypto/test1/foo
> 
> Notes:
>   - At plaintext mode only, the user set a master key and the fscrypt
>     driver derived from it the DEK and the key identifier.
>   - 152c41b2ea39fa3d90ea06448456e7fb is the derived key identifier
>   - Only on the first IO, nvme-rdma gets a callback to load the derived DEK.
> 
> There is no special configuration to support crypto at nvme modules.

Hey, this looks sane to me in a very first glance.

Few high level questions:
- what happens with multipathing? when if not all devices are
capable. SW fallback?
- Does the crypt stuff stay intact when bio is requeued?

I'm assuming you tested this with multipathing? This is not very
useful if it is incompatible with it.

Re: [PATCH rdma-next 00/13] Add RDMA inline crypto support

[Israel Rukshin]

Hi Sagi,

On 1/23/2023 1:27 PM, Sagi Grimberg wrote:
>
>>  From Israel,
>>
>> The purpose of this patchset is to add support for inline
>> encryption/decryption of the data at storage protocols like nvmf over
>> RDMA (at a similar way like integrity is used via unique mkey).
>>
>> This patchset adds support for plaintext keys. The patches were tested
>> on BF-3 HW with fscrypt tool to test this feature, which showed reduce
>> in CPU utilization when comparing at 64k or more IO size. The CPU 
>> utilization
>> was improved by more than 50% comparing to the SW only solution at 
>> this case.
>>
>> How to configure fscrypt to enable plaintext keys:
>>   # mkfs.ext4 -O encrypt /dev/nvme0n1
>>   # mount /dev/nvme0n1 /mnt/crypto -o inlinecrypt
>>   # head -c 64 /dev/urandom > /tmp/master_key
>>   # fscryptctl add_key /mnt/crypto/ < /tmp/master_key
>>   # mkdir /mnt/crypto/test1
>>   # fscryptctl set_policy 152c41b2ea39fa3d90ea06448456e7fb 
>> /mnt/crypto/test1
>>     ** “152c41b2ea39fa3d90ea06448456e7fb” is the output of the
>>        “fscryptctl add_key” command.
>>   # echo foo > /mnt/crypto/test1/foo
>>
>> Notes:
>>   - At plaintext mode only, the user set a master key and the fscrypt
>>     driver derived from it the DEK and the key identifier.
>>   - 152c41b2ea39fa3d90ea06448456e7fb is the derived key identifier
>>   - Only on the first IO, nvme-rdma gets a callback to load the 
>> derived DEK.
>>
>> There is no special configuration to support crypto at nvme modules.
>
> Hey, this looks sane to me in a very first glance.
>
> Few high level questions:
> - what happens with multipathing? when if not all devices are
> capable. SW fallback?
SW fallback happens every time the device doesn't support the specific 
crypto request (which include data-unit-size, mode and dun_bytes).
So with multipathing, one path uses the HW crypto offload and the other 
one uses the SW fallback.
> - Does the crypt stuff stay intact when bio is requeued?
Yes, the crypto ctx is copied when cloning the bio.
>
> I'm assuming you tested this with multipathing? This is not very
> useful if it is incompatible with it.
Yes, sure.
You can see the call to  blk_crypto_reprogram_all_keys(), which is 
called when the controller was reconnected after port toggling.

- Israel

Re: [PATCH rdma-next 00/13] Add RDMA inline crypto support

[Christoph Hellwig]

On Wed, Jan 18, 2023 at 04:20:27PM +0200, Max Gurtovoy wrote:
> Today, if one would like to run both features using a capable device then 
> the PI will be offloaded (no SW fallback) and the Encryption will be using 
> SW fallback.

It's not just running both - it's offering both as currently registering
these fatures is mutally incompatible.

Re: [PATCH rdma-next 00/13] Add RDMA inline crypto support

[Christoph Hellwig]

On Mon, Jan 23, 2023 at 02:57:18PM +0200, Israel Rukshin wrote:
>> - what happens with multipathing? when if not all devices are
>> capable. SW fallback?
> SW fallback happens every time the device doesn't support the specific 
> crypto request (which include data-unit-size, mode and dun_bytes).
> So with multipathing, one path uses the HW crypto offload and the other one 
> uses the SW fallback.

That's a big no-go.  The blk-crypto-fallback code is just a toy example
and not actually safe to use in prodution.  Most importantly it just
kmallocs a bio clone and pages for it without any mempool that guarantees
forward progress.

Re: [PATCH rdma-next 00/13] Add RDMA inline crypto support

[Max Gurtovoy]

On 30/01/2023 14:35, Christoph Hellwig wrote:
> On Wed, Jan 18, 2023 at 04:20:27PM +0200, Max Gurtovoy wrote:
>> Today, if one would like to run both features using a capable device then
>> the PI will be offloaded (no SW fallback) and the Encryption will be using
>> SW fallback.
> It's not just running both - it's offering both as currently registering
> these fatures is mutally incompatible.

For sure we would like to offer both, but as mentioned before, this will 
require blk layer instrumentation and ib_core instrumentation to some 
pipeline_mr or something like that.
We are interested in doing this IO path but doing all in 1 series 
doesn't sounds possible.

First we would like to have at least the mutual exclusive solution since 
there are users that may want only one type of acceleration.

re-designing, both blk layer and ib_core for the PI + CRYPTO case in 
this series is not feasible IMO.

Re: [PATCH rdma-next 00/13] Add RDMA inline crypto support

[Sagi Grimberg]

>>> Today, if one would like to run both features using a capable device 
>>> then
>>> the PI will be offloaded (no SW fallback) and the Encryption will be 
>>> using
>>> SW fallback.
>> It's not just running both - it's offering both as currently registering
>> these fatures is mutally incompatible.
> 
> For sure we would like to offer both, but as mentioned before, this will 
> require blk layer instrumentation and ib_core instrumentation to some 
> pipeline_mr or something like that.

Can you explain what is needed in the block layer that prevents queueing
a request that has PI and crypto ctx?

As for the rdma portion, I don't know enough if this is something
that is supported by the HW and lacks a SW interface, or inherently
unsupported in HW...