[dpdk-dev] [RFC] drivers: introduce mlx5 crypto PMD

[dpdk-dev] [RFC] drivers: introduce mlx5 crypto PMD

[Matan Azrad]

From: Shiri Kuzin <shirik@nvidia.com>

Add a new PMD for Mellanox devices- crypto PMD.

The crypto PMD will be supported starting Mellanox ConnectX6 and
BlueField2.

The crypto PMD will add the support of encryption and decryption using
the AES-XTS symmetric algorithm.

The crypto PMD requires rdma-core and uses mlx5 DevX.

This patch adds the PCI probing, basic functions, build files and
log utility.

Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 MAINTAINERS                             |   4 +
 drivers/common/mlx5/mlx5_common.h       |   1 +
 drivers/common/mlx5/mlx5_common_pci.c   |  14 ++
 drivers/common/mlx5/mlx5_common_pci.h   |  21 +--
 drivers/crypto/meson.build              |   1 +
 drivers/crypto/mlx5/meson.build         |  26 +++
 drivers/crypto/mlx5/mlx5_crypto.c       | 288 ++++++++++++++++++++++++++++++++
 drivers/crypto/mlx5/mlx5_crypto_utils.h |  19 +++
 drivers/crypto/mlx5/version.map         |   3 +
 9 files changed, 367 insertions(+), 10 deletions(-)
 create mode 100644 drivers/crypto/mlx5/meson.build
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto.c
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto_utils.h
 create mode 100644 drivers/crypto/mlx5/version.map

diff --git a/MAINTAINERS b/MAINTAINERS
index 1962284..a449180 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1064,6 +1064,10 @@ F: drivers/crypto/octeontx2/
 F: doc/guides/cryptodevs/octeontx2.rst
 F: doc/guides/cryptodevs/features/octeontx2.ini
 
+Mellanox mlx5
+M: Matan Azrad <matan@nvidia.com>
+F: drivers/crypto/mlx5/
+
 Null Crypto
 M: Declan Doherty <declan.doherty@intel.com>
 F: drivers/crypto/null/
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index 99efa22..5b4d8d1 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -219,6 +219,7 @@ enum mlx5_class {
 	MLX5_CLASS_VDPA = RTE_BIT64(1),
 	MLX5_CLASS_REGEX = RTE_BIT64(2),
 	MLX5_CLASS_COMPRESS = RTE_BIT64(3),
+	MLX5_CLASS_CRYPTO = RTE_BIT64(4),
 };
 
 #define MLX5_DBR_SIZE RTE_CACHE_LINE_SIZE
diff --git a/drivers/common/mlx5/mlx5_common_pci.c b/drivers/common/mlx5/mlx5_common_pci.c
index 4ecc4d6..167d55b 100644
--- a/drivers/common/mlx5/mlx5_common_pci.c
+++ b/drivers/common/mlx5/mlx5_common_pci.c
@@ -30,6 +30,7 @@ static TAILQ_HEAD(mlx5_pci_devices_head, mlx5_pci_device) devices_list =
 	{ .name = "net", .driver_class = MLX5_CLASS_NET },
 	{ .name = "regex", .driver_class = MLX5_CLASS_REGEX },
 	{ .name = "compress", .driver_class = MLX5_CLASS_COMPRESS },
+	{ .name = "crypto", .driver_class = MLX5_CLASS_CRYPTO },
 };
 
 static const unsigned int mlx5_class_combinations[] = {
@@ -37,13 +38,26 @@ static TAILQ_HEAD(mlx5_pci_devices_head, mlx5_pci_device) devices_list =
 	MLX5_CLASS_VDPA,
 	MLX5_CLASS_REGEX,
 	MLX5_CLASS_COMPRESS,
+	MLX5_CLASS_CRYPTO,
 	MLX5_CLASS_NET | MLX5_CLASS_REGEX,
 	MLX5_CLASS_VDPA | MLX5_CLASS_REGEX,
 	MLX5_CLASS_NET | MLX5_CLASS_COMPRESS,
 	MLX5_CLASS_VDPA | MLX5_CLASS_COMPRESS,
 	MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS,
+	MLX5_CLASS_NET | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_VDPA | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_REGEX | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_COMPRESS | MLX5_CLASS_CRYPTO,
 	MLX5_CLASS_NET | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS,
 	MLX5_CLASS_VDPA | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS,
+	MLX5_CLASS_NET | MLX5_CLASS_REGEX | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_VDPA | MLX5_CLASS_REGEX | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_NET | MLX5_CLASS_COMPRESS | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_VDPA | MLX5_CLASS_COMPRESS | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_NET | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS |
+	MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_VDPA | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS |
+	MLX5_CLASS_CRYPTO,
 	/* New class combination should be added here. */
 };
 
diff --git a/drivers/common/mlx5/mlx5_common_pci.h b/drivers/common/mlx5/mlx5_common_pci.h
index de89bb9..cb8d2f5 100644
--- a/drivers/common/mlx5/mlx5_common_pci.h
+++ b/drivers/common/mlx5/mlx5_common_pci.h
@@ -9,17 +9,18 @@
  * @file
  *
  * RTE Mellanox PCI Driver Interface
- * Mellanox ConnectX PCI device supports multiple class: net,vdpa,regex and
- * compress devices. This layer enables creating such multiple class of devices
- * on a single PCI device by allowing to bind multiple class specific device
- * driver to attach to mlx5_pci driver.
+ * Mellanox ConnectX PCI device supports multiple class: net,vdpa,regex,compress
+ * and crypto devices. This layer enables creating such multiple class of
+ * devices on a single PCI device by allowing to bind multiple class specific
+ * device driver to attach to mlx5_pci driver.
  *
- * -----------    ------------    -------------    ----------------
- * |   mlx5  |    |   mlx5   |    |   mlx5    |    |     mlx5     |
- * | net pmd |    | vdpa pmd |    | regex pmd |    | compress pmd |
- * -----------    ------------    -------------    ----------------
- *      \              \                    /              /
- *       \              \                  /              /
+ * --------    --------    ---------    ------------    ----------
+ * | mlx5 |    | mlx5 |    | mlx5  |    |   mlx5   |    |  mlx5  |
+ * | net  |    | vdpa |    | regex |    | compress |    | crypto |
+ * | pmd  |    | pmd  |    |  pmd  |    |   pmd    |    |  pmd   |
+ * --------    --------    ---------    ------------    ----------
+ *      \              \         |          /              /
+ *       \              \        |         /              /
  *        \              \_--------------_/              /
  *         \_______________|   mlx5     |_______________/
  *                         | pci common |
diff --git a/drivers/crypto/meson.build b/drivers/crypto/meson.build
index c927c5f..90b2149 100644
--- a/drivers/crypto/meson.build
+++ b/drivers/crypto/meson.build
@@ -14,6 +14,7 @@ drivers = ['aesni_gcm',
 	   'dpaa_sec',
 	   'dpaa2_sec',
 	   'kasumi',
+	   'mlx5',
 	   'mvsam',
 	   'nitrox',
 	   'null',
diff --git a/drivers/crypto/mlx5/meson.build b/drivers/crypto/mlx5/meson.build
new file mode 100644
index 0000000..5bf0912
--- /dev/null
+++ b/drivers/crypto/mlx5/meson.build
@@ -0,0 +1,26 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright 2021 Mellanox Technologies, Ltd
+
+if not is_linux
+	build = false
+	reason = 'only supported on Linux'
+	subdir_done()
+endif
+
+fmt_name = 'mlx5_crypto'
+deps += ['common_mlx5', 'eal', 'cryptodev']
+sources = files(
+	'mlx5_crypto.c',
+)
+cflags_options = [
+	'-std=c11',
+	'-Wno-strict-prototypes',
+	'-D_BSD_SOURCE',
+	'-D_DEFAULT_SOURCE',
+	'-D_XOPEN_SOURCE=600'
+]
+foreach option:cflags_options
+	if cc.has_argument(option)
+		cflags += option
+	endif
+endforeach
\ No newline at end of file
diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
new file mode 100644
index 0000000..6e40176
--- /dev/null
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -0,0 +1,288 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2021 Mellanox Technologies, Ltd
+ */
+
+#include <rte_malloc.h>
+#include <rte_log.h>
+#include <rte_errno.h>
+#include <rte_pci.h>
+#include <rte_crypto.h>
+#include <rte_cryptodev.h>
+#include <rte_cryptodev_pmd.h>
+
+#include <mlx5_glue.h>
+#include <mlx5_common.h>
+#include <mlx5_common_pci.h>
+#include <mlx5_devx_cmds.h>
+#include <mlx5_common_os.h>
+
+#include "mlx5_crypto_utils.h"
+
+#define MLX5_CRYPTO_DRIVER_NAME mlx5_crypto
+#define MLX5_CRYPTO_LOG_NAME    pmd.crypto.mlx5
+
+struct mlx5_crypto_priv {
+	TAILQ_ENTRY(mlx5_crypto_priv) next;
+	struct ibv_context *ctx; /* Device context. */
+	struct rte_pci_device *pci_dev;
+	struct rte_cryptodev *crypto_dev;
+	void *uar; /* User Access Region. */
+	uint32_t pdn; /* Protection Domain number. */
+	struct ibv_pd *pd;
+};
+
+TAILQ_HEAD(mlx5_crypto_privs, mlx5_crypto_priv) mlx5_crypto_priv_list =
+				TAILQ_HEAD_INITIALIZER(mlx5_crypto_priv_list);
+static pthread_mutex_t priv_list_lock = PTHREAD_MUTEX_INITIALIZER;
+
+int mlx5_crypto_logtype;
+
+uint8_t mlx5_crypto_driver_id;
+
+static const char mlx5_crypto_drv_name[] = RTE_STR(MLX5_CRYPTO_DRIVER_NAME);
+
+static const struct rte_driver mlx5_drv = {
+	.name = mlx5_crypto_drv_name,
+	.alias = mlx5_crypto_drv_name
+};
+
+static struct cryptodev_driver mlx5_cryptodev_driver;
+
+static struct rte_cryptodev_ops mlx5_crypto_ops = {
+	.dev_configure			= NULL,
+	.dev_start			= NULL,
+	.dev_stop			= NULL,
+	.dev_close			= NULL,
+	.dev_infos_get			= NULL,
+	.stats_get			= NULL,
+	.stats_reset			= NULL,
+	.queue_pair_setup		= NULL,
+	.queue_pair_release		= NULL,
+	.sym_session_get_size		= NULL,
+	.sym_session_configure		= NULL,
+	.sym_session_clear		= NULL,
+	.sym_get_raw_dp_ctx_size	= NULL,
+	.sym_configure_raw_dp_ctx	= NULL,
+};
+
+static void
+mlx5_crypto_hw_global_release(struct mlx5_crypto_priv *priv)
+{
+	if (priv->pd != NULL) {
+		claim_zero(mlx5_glue->dealloc_pd(priv->pd));
+		priv->pd = NULL;
+	}
+	if (priv->uar != NULL) {
+		mlx5_glue->devx_free_uar(priv->uar);
+		priv->uar = NULL;
+	}
+}
+
+static int
+mlx5_crypto_pd_create(struct mlx5_crypto_priv *priv)
+{
+#ifdef HAVE_IBV_FLOW_DV_SUPPORT
+	struct mlx5dv_obj obj;
+	struct mlx5dv_pd pd_info;
+	int ret;
+
+	priv->pd = mlx5_glue->alloc_pd(priv->ctx);
+	if (priv->pd == NULL) {
+		DRV_LOG(ERR, "Failed to allocate PD.");
+		return errno ? -errno : -ENOMEM;
+	}
+	obj.pd.in = priv->pd;
+	obj.pd.out = &pd_info;
+	ret = mlx5_glue->dv_init_obj(&obj, MLX5DV_OBJ_PD);
+	if (ret != 0) {
+		DRV_LOG(ERR, "Fail to get PD object info.");
+		mlx5_glue->dealloc_pd(priv->pd);
+		priv->pd = NULL;
+		return -errno;
+	}
+	priv->pdn = pd_info.pdn;
+	return 0;
+#else
+	(void)priv;
+	DRV_LOG(ERR, "Cannot get pdn - no DV support.");
+	return -ENOTSUP;
+#endif /* HAVE_IBV_FLOW_DV_SUPPORT */
+}
+
+static int
+mlx5_crypto_hw_global_prepare(struct mlx5_crypto_priv *priv)
+{
+	if (mlx5_crypto_pd_create(priv) != 0)
+		return -1;
+	priv->uar = mlx5_devx_alloc_uar(priv->ctx, -1);
+	if (priv->uar == NULL || mlx5_os_get_devx_uar_reg_addr(priv->uar) ==
+	    NULL) {
+		rte_errno = errno;
+		claim_zero(mlx5_glue->dealloc_pd(priv->pd));
+		DRV_LOG(ERR, "Failed to allocate UAR.");
+		return -1;
+	}
+	return 0;
+}
+
+/**
+ * DPDK callback to register a PCI device.
+ *
+ * This function spawns crypto device out of a given PCI device.
+ *
+ * @param[in] pci_drv
+ *   PCI driver structure (mlx5_crypto_driver).
+ * @param[in] pci_dev
+ *   PCI device information.
+ *
+ * @return
+ *   0 on success, 1 to skip this driver, a negative errno value otherwise
+ *   and rte_errno is set.
+ */
+static int
+mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
+			struct rte_pci_device *pci_dev)
+{
+	struct ibv_device *ibv;
+	struct rte_cryptodev *crypto_dev;
+	struct ibv_context *ctx;
+	struct mlx5_crypto_priv *priv;
+	struct mlx5_hca_attr attr = { 0 };
+	struct rte_cryptodev_pmd_init_params init_params = {
+		.name = "",
+		.private_data_size = sizeof(struct mlx5_crypto_priv),
+		.socket_id = pci_dev->device.numa_node,
+		.max_nb_queue_pairs =
+				RTE_CRYPTODEV_PMD_DEFAULT_MAX_NB_QUEUE_PAIRS,
+	};
+	RTE_SET_USED(pci_drv);
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		DRV_LOG(ERR, "Non-primary process type is not supported.");
+		rte_errno = ENOTSUP;
+		return -rte_errno;
+	}
+	ibv = mlx5_os_get_ib_device_match(&pci_dev->addr);
+	if (ibv == NULL) {
+		DRV_LOG(ERR, "No matching IB device for PCI slot "
+			PCI_PRI_FMT ".", pci_dev->addr.domain,
+			pci_dev->addr.bus, pci_dev->addr.devid,
+			pci_dev->addr.function);
+		return -rte_errno;
+	}
+	DRV_LOG(INFO, "PCI information matches for device \"%s\".", ibv->name);
+	ctx = mlx5_glue->dv_open_device(ibv);
+	if (ctx == NULL) {
+		DRV_LOG(ERR, "Failed to open IB device \"%s\".", ibv->name);
+		rte_errno = ENODEV;
+		return -rte_errno;
+	}
+	if (mlx5_devx_cmd_query_hca_attr(ctx, &attr) != 0 ||
+	    attr.crypto == 0 || attr.aes_xts == 0) {
+		DRV_LOG(ERR, "Not enough capabilities to support crypto "
+			"operations, maybe old FW/OFED version?");
+		claim_zero(mlx5_glue->close_device(ctx));
+		rte_errno = ENOTSUP;
+		return -ENOTSUP;
+	}
+	crypto_dev = rte_cryptodev_pmd_create(ibv->name, &pci_dev->device,
+					&init_params);
+	if (crypto_dev == NULL) {
+		DRV_LOG(ERR, "Failed to create device \"%s\".", ibv->name);
+		claim_zero(mlx5_glue->close_device(ctx));
+		return -ENODEV;
+	}
+	DRV_LOG(INFO,
+		"Crypto device %s was created successfully.", ibv->name);
+	crypto_dev->dev_ops = &mlx5_crypto_ops;
+	crypto_dev->dequeue_burst = NULL;
+	crypto_dev->enqueue_burst = NULL;
+	crypto_dev->feature_flags = RTE_CRYPTODEV_FF_HW_ACCELERATED;
+	crypto_dev->driver_id = mlx5_crypto_driver_id;
+	priv = crypto_dev->data->dev_private;
+	priv->ctx = ctx;
+	priv->pci_dev = pci_dev;
+	priv->crypto_dev = crypto_dev;
+	if (mlx5_crypto_hw_global_prepare(priv) != 0) {
+		rte_cryptodev_pmd_destroy(priv->crypto_dev);
+		claim_zero(mlx5_glue->close_device(priv->ctx));
+		return -1;
+	}
+	pthread_mutex_lock(&priv_list_lock);
+	TAILQ_INSERT_TAIL(&mlx5_crypto_priv_list, priv, next);
+	pthread_mutex_unlock(&priv_list_lock);
+	return 0;
+}
+
+static int
+mlx5_crypto_pci_remove(struct rte_pci_device *pdev)
+{
+	struct mlx5_crypto_priv *priv = NULL;
+
+	pthread_mutex_lock(&priv_list_lock);
+	TAILQ_FOREACH(priv, &mlx5_crypto_priv_list, next)
+		if (rte_pci_addr_cmp(&priv->pci_dev->addr, &pdev->addr) != 0)
+			break;
+	if (priv)
+		TAILQ_REMOVE(&mlx5_crypto_priv_list, priv, next);
+	pthread_mutex_unlock(&priv_list_lock);
+	if (priv) {
+		mlx5_crypto_hw_global_release(priv);
+		rte_cryptodev_pmd_destroy(priv->crypto_dev);
+		claim_zero(mlx5_glue->close_device(priv->ctx));
+	}
+	return 0;
+}
+
+static const struct rte_pci_id mlx5_crypto_pci_id_map[] = {
+		{
+			RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
+					PCI_DEVICE_ID_MELLANOX_CONNECTX6)
+		},
+		{
+			RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
+					PCI_DEVICE_ID_MELLANOX_CONNECTX6DX)
+		},
+		{
+			RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
+					PCI_DEVICE_ID_MELLANOX_CONNECTX6DXBF)
+		},
+		{
+			RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
+					PCI_DEVICE_ID_MELLANOX_CONNECTX7)
+		},
+		{
+			RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
+					PCI_DEVICE_ID_MELLANOX_CONNECTX7BF)
+		},
+		{
+			.vendor_id = 0
+		}
+	};
+
+static struct mlx5_pci_driver mlx5_crypto_driver = {
+	.driver_class = MLX5_CLASS_CRYPTO,
+	.pci_driver = {
+		.driver = {
+			.name = RTE_STR(MLX5_CRYPTO_DRIVER_NAME),
+		},
+		.id_table = mlx5_crypto_pci_id_map,
+		.probe = mlx5_crypto_pci_probe,
+		.remove = mlx5_crypto_pci_remove,
+		.drv_flags = 0,
+	},
+};
+
+RTE_INIT(rte_mlx5_crypto_init)
+{
+	mlx5_common_init();
+	if (mlx5_glue != NULL)
+		mlx5_pci_driver_register(&mlx5_crypto_driver);
+}
+
+RTE_PMD_REGISTER_CRYPTO_DRIVER(mlx5_cryptodev_driver, mlx5_drv,
+			       mlx5_crypto_driver_id);
+
+RTE_LOG_REGISTER(mlx5_crypto_logtype, MLX5_CRYPTO_LOG_NAME, NOTICE)
+RTE_PMD_EXPORT_NAME(MLX5_CRYPTO_DRIVER_NAME, __COUNTER__);
+RTE_PMD_REGISTER_PCI_TABLE(MLX5_CRYPTO_DRIVER_NAME, mlx5_crypto_pci_id_map);
+RTE_PMD_REGISTER_KMOD_DEP(MLX5_CRYPTO_DRIVER_NAME, "* ib_uverbs & mlx5_core & mlx5_ib");
diff --git a/drivers/crypto/mlx5/mlx5_crypto_utils.h b/drivers/crypto/mlx5/mlx5_crypto_utils.h
new file mode 100644
index 0000000..cef4b07
--- /dev/null
+++ b/drivers/crypto/mlx5/mlx5_crypto_utils.h
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2021 Mellanox Technologies, Ltd
+ */
+
+#ifndef RTE_PMD_MLX5_CRYPTO_UTILS_H_
+#define RTE_PMD_MLX5_CRYPTO_UTILS_H_
+
+#include <mlx5_common.h>
+
+extern int mlx5_crypto_logtype;
+
+#define MLX5_CRYPTO_LOG_PREFIX "mlx5_crypto"
+/* Generic printf()-like logging macro with automatic line feed. */
+#define DRV_LOG(level, ...) \
+	PMD_DRV_LOG_(level, mlx5_crypto_logtype, MLX5_CRYPTO_LOG_PREFIX, \
+		__VA_ARGS__ PMD_DRV_LOG_STRIP PMD_DRV_LOG_OPAREN, \
+		PMD_DRV_LOG_CPAREN)
+
+#endif /* RTE_PMD_MLX5_CRYPTO_UTILS_H_ */
diff --git a/drivers/crypto/mlx5/version.map b/drivers/crypto/mlx5/version.map
new file mode 100644
index 0000000..4a76d1d
--- /dev/null
+++ b/drivers/crypto/mlx5/version.map
@@ -0,0 +1,3 @@
+DPDK_21 {
+	local: *;
+};
-- 
1.8.3.1

[dpdk-dev] [PATCH 00/24] drivers: introduce mlx5 crypto PMD

[Shiri Kuzin]

Add a new PMD for Nvidia devices- crypto PMD.

The crypto PMD will be supported on Nvidia ConnectX6-Dx and
Bluefield 2.

The crypto PMD will add the support of encryption and decryption using
the AES-XTS symmetric algorithm.

The crypto PMD requires rdma-core and uses mlx5 DevX.

This series includes control path PMD implementation.
We are currently almost done working on the data path implementation
and doing our best effort to send the data path for rc1, but we might
be done for rc2.
If there is a suggestion or need for the series earlier, we will make
the effort to meet the requirement.

Dekel Peled (13):
  common/mlx5: remove redundant spaces in header file
  common/mlx5: update GENEVE TLV OPT obj name
  common/mlx5: optimize read of general obj type caps
  common/mlx5: add HCA cap for AES-XTS crypto
  common/mlx5: support general object DEK create op
  common/mlx5: add crypto en field to MKEY context
  common/mlx5: support umr en field in MKEY context
  common/mlx5: support general obj IMPORT KEK create
  common/mlx5: support general obj CRYPTO LOGIN create
  common/mlx5: add crypto BSF struct and defines
  common/mlx5: support general obj CREDENTIAL create
  common/mlx5: add crypto register structs and defs
  common/mlx5: support register write access

Shiri Kuzin (11):
  common/mlx5: share hash list tool
  common/mlx5: share get ib device match function
  drivers: introduce mlx5 crypto PMD
  crypto/mlx5: add DEK object management
  crypto/mlx5: support session operations
  crypto/mlx5: add basic operations
  crypto/mlx5: support queue pairs operations
  crypto/mlx5: add dev stop and start operations
  crypto/mlx5: add memory region management
  crypto/mlx5: create login object using DevX
  crypto/mlx5: adjust to the multiple data unit API

 MAINTAINERS                                   |   4 +
 drivers/common/mlx5/linux/mlx5_common_os.c    |  30 +-
 drivers/common/mlx5/linux/mlx5_common_os.h    |   5 +
 drivers/common/mlx5/linux/mlx5_nl.c           |   2 +-
 drivers/common/mlx5/meson.build               |   1 +
 drivers/common/mlx5/mlx5_common.c             |   2 +-
 drivers/common/mlx5/mlx5_common.h             |   2 +
 drivers/common/mlx5/mlx5_common_devx.c        |   2 +-
 drivers/common/mlx5/mlx5_common_log.h         |  21 +
 drivers/common/mlx5/mlx5_common_mp.c          |   2 +-
 drivers/common/mlx5/mlx5_common_mr.c          |   2 +-
 drivers/common/mlx5/mlx5_common_pci.c         |  17 +-
 drivers/common/mlx5/mlx5_common_pci.h         |  21 +-
 drivers/common/mlx5/mlx5_common_utils.c       | 221 ++++++
 drivers/common/mlx5/mlx5_common_utils.h       | 202 ++++-
 drivers/common/mlx5/mlx5_devx_cmds.c          | 314 +++++++-
 drivers/common/mlx5/mlx5_devx_cmds.h          |  66 ++
 drivers/common/mlx5/mlx5_malloc.c             |   2 +-
 drivers/common/mlx5/mlx5_prm.h                | 545 +++++++++-----
 .../common/mlx5/rte_common_mlx5_exports.def   |   7 +
 drivers/common/mlx5/version.map               |  24 +-
 drivers/common/mlx5/windows/mlx5_common_os.c  |   2 +-
 drivers/common/mlx5/windows/mlx5_glue.c       |   2 +-
 drivers/compress/mlx5/mlx5_compress.c         |  30 +-
 drivers/crypto/meson.build                    |   1 +
 drivers/crypto/mlx5/meson.build               |  27 +
 drivers/crypto/mlx5/mlx5_crypto.c             | 687 ++++++++++++++++++
 drivers/crypto/mlx5/mlx5_crypto.h             |  67 ++
 drivers/crypto/mlx5/mlx5_crypto_dek.c         | 135 ++++
 drivers/crypto/mlx5/mlx5_crypto_utils.h       |  19 +
 drivers/crypto/mlx5/version.map               |   3 +
 drivers/net/mlx5/mlx5_utils.c                 | 209 ------
 drivers/net/mlx5/mlx5_utils.h                 | 194 +----
 drivers/regex/mlx5/mlx5_regex.c               |  30 +-
 drivers/vdpa/mlx5/mlx5_vdpa.c                 |  32 +-
 35 files changed, 2216 insertions(+), 714 deletions(-)
 create mode 100644 drivers/common/mlx5/mlx5_common_log.h
 create mode 100644 drivers/common/mlx5/mlx5_common_utils.c
 create mode 100644 drivers/crypto/mlx5/meson.build
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto.c
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto.h
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto_dek.c
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto_utils.h
 create mode 100644 drivers/crypto/mlx5/version.map

-- 
2.21.0

[dpdk-dev] [PATCH 01/24] common/mlx5: remove redundant spaces in header file

[Shiri Kuzin]

From: Dekel Peled <dekelp@nvidia.com>

File drivers/common/mlx5/mlx5_prm.h includes structs representing
data items as defined in PRM document.
Some of these structs were copied as-is from kernel file mlx5_ifc.h.
As result the structs are not all aligned with the same spacing.

This patch removes redundant spaces and new lines from several structs,
to align all structs in mlx5_prm.h to the same format.

Signed-off-by: Dekel Peled <dekelp@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/mlx5_prm.h | 334 +++++++++++++++------------------
 1 file changed, 155 insertions(+), 179 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h
index 0ef0574f92..f0a4f70d7d 100644
--- a/drivers/common/mlx5/mlx5_prm.h
+++ b/drivers/common/mlx5/mlx5_prm.h
@@ -953,163 +953,139 @@ enum {
 
 /* Flow counters. */
 struct mlx5_ifc_alloc_flow_counter_out_bits {
-	u8         status[0x8];
-	u8         reserved_at_8[0x18];
-	u8         syndrome[0x20];
-	u8         flow_counter_id[0x20];
-	u8         reserved_at_60[0x20];
+	u8 status[0x8];
+	u8 reserved_at_8[0x18];
+	u8 syndrome[0x20];
+	u8 flow_counter_id[0x20];
+	u8 reserved_at_60[0x20];
 };
 
 struct mlx5_ifc_alloc_flow_counter_in_bits {
-	u8         opcode[0x10];
-	u8         reserved_at_10[0x10];
-	u8         reserved_at_20[0x10];
-	u8         op_mod[0x10];
-	u8         flow_counter_id[0x20];
-	u8         reserved_at_40[0x18];
-	u8         flow_counter_bulk[0x8];
+	u8 opcode[0x10];
+	u8 reserved_at_10[0x10];
+	u8 reserved_at_20[0x10];
+	u8 op_mod[0x10];
+	u8 flow_counter_id[0x20];
+	u8 reserved_at_40[0x18];
+	u8 flow_counter_bulk[0x8];
 };
 
 struct mlx5_ifc_dealloc_flow_counter_out_bits {
-	u8         status[0x8];
-	u8         reserved_at_8[0x18];
-	u8         syndrome[0x20];
-	u8         reserved_at_40[0x40];
+	u8 status[0x8];
+	u8 reserved_at_8[0x18];
+	u8 syndrome[0x20];
+	u8 reserved_at_40[0x40];
 };
 
 struct mlx5_ifc_dealloc_flow_counter_in_bits {
-	u8         opcode[0x10];
-	u8         reserved_at_10[0x10];
-	u8         reserved_at_20[0x10];
-	u8         op_mod[0x10];
-	u8         flow_counter_id[0x20];
-	u8         reserved_at_60[0x20];
+	u8 opcode[0x10];
+	u8 reserved_at_10[0x10];
+	u8 reserved_at_20[0x10];
+	u8 op_mod[0x10];
+	u8 flow_counter_id[0x20];
+	u8 reserved_at_60[0x20];
 };
 
 struct mlx5_ifc_traffic_counter_bits {
-	u8         packets[0x40];
-	u8         octets[0x40];
+	u8 packets[0x40];
+	u8 octets[0x40];
 };
 
 struct mlx5_ifc_query_flow_counter_out_bits {
-	u8         status[0x8];
-	u8         reserved_at_8[0x18];
-	u8         syndrome[0x20];
-	u8         reserved_at_40[0x40];
+	u8 status[0x8];
+	u8 reserved_at_8[0x18];
+	u8 syndrome[0x20];
+	u8 reserved_at_40[0x40];
 	struct mlx5_ifc_traffic_counter_bits flow_statistics[];
 };
 
 struct mlx5_ifc_query_flow_counter_in_bits {
-	u8         opcode[0x10];
-	u8         reserved_at_10[0x10];
-	u8         reserved_at_20[0x10];
-	u8         op_mod[0x10];
-	u8         reserved_at_40[0x20];
-	u8         mkey[0x20];
-	u8         address[0x40];
-	u8         clear[0x1];
-	u8         dump_to_memory[0x1];
-	u8         num_of_counters[0x1e];
-	u8         flow_counter_id[0x20];
+	u8 opcode[0x10];
+	u8 reserved_at_10[0x10];
+	u8 reserved_at_20[0x10];
+	u8 op_mod[0x10];
+	u8 reserved_at_40[0x20];
+	u8 mkey[0x20];
+	u8 address[0x40];
+	u8 clear[0x1];
+	u8 dump_to_memory[0x1];
+	u8 num_of_counters[0x1e];
+	u8 flow_counter_id[0x20];
 };
 
 #define MLX5_MAX_KLM_BYTE_COUNT 0x80000000u
 #define MLX5_MIN_KLM_FIXED_BUFFER_SIZE 0x1000u
 
-
 struct mlx5_ifc_klm_bits {
-	u8         byte_count[0x20];
-	u8         mkey[0x20];
-	u8         address[0x40];
+	u8 byte_count[0x20];
+	u8 mkey[0x20];
+	u8 address[0x40];
 };
 
 struct mlx5_ifc_mkc_bits {
-	u8         reserved_at_0[0x1];
-	u8         free[0x1];
-	u8         reserved_at_2[0x1];
-	u8         access_mode_4_2[0x3];
-	u8         reserved_at_6[0x7];
-	u8         relaxed_ordering_write[0x1];
-	u8         reserved_at_e[0x1];
-	u8         small_fence_on_rdma_read_response[0x1];
-	u8         umr_en[0x1];
-	u8         a[0x1];
-	u8         rw[0x1];
-	u8         rr[0x1];
-	u8         lw[0x1];
-	u8         lr[0x1];
-	u8         access_mode_1_0[0x2];
-	u8         reserved_at_18[0x8];
-
-	u8         qpn[0x18];
-	u8         mkey_7_0[0x8];
-
-	u8         reserved_at_40[0x20];
-
-	u8         length64[0x1];
-	u8         bsf_en[0x1];
-	u8         sync_umr[0x1];
-	u8         reserved_at_63[0x2];
-	u8         expected_sigerr_count[0x1];
-	u8         reserved_at_66[0x1];
-	u8         en_rinval[0x1];
-	u8         pd[0x18];
-
-	u8         start_addr[0x40];
-
-	u8         len[0x40];
-
-	u8         bsf_octword_size[0x20];
-
-	u8         reserved_at_120[0x80];
-
-	u8         translations_octword_size[0x20];
-
-	u8         reserved_at_1c0[0x19];
-	u8		   relaxed_ordering_read[0x1];
-	u8		   reserved_at_1da[0x1];
-	u8         log_page_size[0x5];
-
-	u8         reserved_at_1e0[0x20];
+	u8 reserved_at_0[0x1];
+	u8 free[0x1];
+	u8 reserved_at_2[0x1];
+	u8 access_mode_4_2[0x3];
+	u8 reserved_at_6[0x7];
+	u8 relaxed_ordering_write[0x1];
+	u8 reserved_at_e[0x1];
+	u8 small_fence_on_rdma_read_response[0x1];
+	u8 umr_en[0x1];
+	u8 a[0x1];
+	u8 rw[0x1];
+	u8 rr[0x1];
+	u8 lw[0x1];
+	u8 lr[0x1];
+	u8 access_mode_1_0[0x2];
+	u8 reserved_at_18[0x8];
+	u8 qpn[0x18];
+	u8 mkey_7_0[0x8];
+	u8 reserved_at_40[0x20];
+	u8 length64[0x1];
+	u8 bsf_en[0x1];
+	u8 sync_umr[0x1];
+	u8 reserved_at_63[0x2];
+	u8 expected_sigerr_count[0x1];
+	u8 reserved_at_66[0x1];
+	u8 en_rinval[0x1];
+	u8 pd[0x18];
+	u8 start_addr[0x40];
+	u8 len[0x40];
+	u8 bsf_octword_size[0x20];
+	u8 reserved_at_120[0x80];
+	u8 translations_octword_size[0x20];
+	u8 reserved_at_1c0[0x19];
+	u8 relaxed_ordering_read[0x1];
+	u8 reserved_at_1da[0x1];
+	u8 log_page_size[0x5];
+	u8 reserved_at_1e0[0x20];
 };
 
 struct mlx5_ifc_create_mkey_out_bits {
-	u8         status[0x8];
-	u8         reserved_at_8[0x18];
-
-	u8         syndrome[0x20];
-
-	u8         reserved_at_40[0x8];
-	u8         mkey_index[0x18];
-
-	u8         reserved_at_60[0x20];
+	u8 status[0x8];
+	u8 reserved_at_8[0x18];
+	u8 syndrome[0x20];
+	u8 reserved_at_40[0x8];
+	u8 mkey_index[0x18];
+	u8 reserved_at_60[0x20];
 };
 
 struct mlx5_ifc_create_mkey_in_bits {
-	u8         opcode[0x10];
-	u8         reserved_at_10[0x10];
-
-	u8         reserved_at_20[0x10];
-	u8         op_mod[0x10];
-
-	u8         reserved_at_40[0x20];
-
-	u8         pg_access[0x1];
-	u8         reserved_at_61[0x1f];
-
+	u8 opcode[0x10];
+	u8 reserved_at_10[0x10];
+	u8 reserved_at_20[0x10];
+	u8 op_mod[0x10];
+	u8 reserved_at_40[0x20];
+	u8 pg_access[0x1];
+	u8 reserved_at_61[0x1f];
 	struct mlx5_ifc_mkc_bits memory_key_mkey_entry;
-
-	u8         reserved_at_280[0x80];
-
-	u8         translations_octword_actual_size[0x20];
-
-	u8         mkey_umem_id[0x20];
-
-	u8         mkey_umem_offset[0x40];
-
-	u8         reserved_at_380[0x500];
-
-	u8         klm_pas_mtt[][0x20];
+	u8 reserved_at_280[0x80];
+	u8 translations_octword_actual_size[0x20];
+	u8 mkey_umem_id[0x20];
+	u8 mkey_umem_offset[0x40];
+	u8 reserved_at_380[0x500];
+	u8 klm_pas_mtt[][0x20];
 };
 
 enum {
@@ -2262,27 +2238,27 @@ enum {
 };
 
 struct mlx5_ifc_flow_meter_parameters_bits {
-	u8         valid[0x1];			// 00h
-	u8         bucket_overflow[0x1];
-	u8         start_color[0x2];
-	u8         both_buckets_on_green[0x1];
-	u8         meter_mode[0x2];
-	u8         reserved_at_1[0x19];
-	u8         reserved_at_2[0x20]; //04h
-	u8         reserved_at_3[0x3];
-	u8         cbs_exponent[0x5];		// 08h
-	u8         cbs_mantissa[0x8];
-	u8         reserved_at_4[0x3];
-	u8         cir_exponent[0x5];
-	u8         cir_mantissa[0x8];
-	u8         reserved_at_5[0x20];		// 0Ch
-	u8         reserved_at_6[0x3];
-	u8         ebs_exponent[0x5];		// 10h
-	u8         ebs_mantissa[0x8];
-	u8         reserved_at_7[0x3];
-	u8         eir_exponent[0x5];
-	u8         eir_mantissa[0x8];
-	u8         reserved_at_8[0x60];		// 14h-1Ch
+	u8 valid[0x1];
+	u8 bucket_overflow[0x1];
+	u8 start_color[0x2];
+	u8 both_buckets_on_green[0x1];
+	u8 meter_mode[0x2];
+	u8 reserved_at_1[0x19];
+	u8 reserved_at_2[0x20];
+	u8 reserved_at_3[0x3];
+	u8 cbs_exponent[0x5];
+	u8 cbs_mantissa[0x8];
+	u8 reserved_at_4[0x3];
+	u8 cir_exponent[0x5];
+	u8 cir_mantissa[0x8];
+	u8 reserved_at_5[0x20];
+	u8 reserved_at_6[0x3];
+	u8 ebs_exponent[0x5];
+	u8 ebs_mantissa[0x8];
+	u8 reserved_at_7[0x3];
+	u8 eir_exponent[0x5];
+	u8 eir_mantissa[0x8];
+	u8 reserved_at_8[0x60];
 };
 
 enum {
@@ -2340,46 +2316,46 @@ struct mlx5_ifc_cqc_bits {
 };
 
 struct mlx5_ifc_health_buffer_bits {
-	u8         reserved_0[0x100];
-	u8         assert_existptr[0x20];
-	u8         assert_callra[0x20];
-	u8         reserved_1[0x40];
-	u8         fw_version[0x20];
-	u8         hw_id[0x20];
-	u8         reserved_2[0x20];
-	u8         irisc_index[0x8];
-	u8         synd[0x8];
-	u8         ext_synd[0x10];
+	u8 reserved_0[0x100];
+	u8 assert_existptr[0x20];
+	u8 assert_callra[0x20];
+	u8 reserved_1[0x40];
+	u8 fw_version[0x20];
+	u8 hw_id[0x20];
+	u8 reserved_2[0x20];
+	u8 irisc_index[0x8];
+	u8 synd[0x8];
+	u8 ext_synd[0x10];
 };
 
 struct mlx5_ifc_initial_seg_bits {
-	u8         fw_rev_minor[0x10];
-	u8         fw_rev_major[0x10];
-	u8         cmd_interface_rev[0x10];
-	u8         fw_rev_subminor[0x10];
-	u8         reserved_0[0x40];
-	u8         cmdq_phy_addr_63_32[0x20];
-	u8         cmdq_phy_addr_31_12[0x14];
-	u8         reserved_1[0x2];
-	u8         nic_interface[0x2];
-	u8         log_cmdq_size[0x4];
-	u8         log_cmdq_stride[0x4];
-	u8         command_doorbell_vector[0x20];
-	u8         reserved_2[0xf00];
-	u8         initializing[0x1];
-	u8         nic_interface_supported[0x7];
-	u8         reserved_4[0x18];
+	u8 fw_rev_minor[0x10];
+	u8 fw_rev_major[0x10];
+	u8 cmd_interface_rev[0x10];
+	u8 fw_rev_subminor[0x10];
+	u8 reserved_0[0x40];
+	u8 cmdq_phy_addr_63_32[0x20];
+	u8 cmdq_phy_addr_31_12[0x14];
+	u8 reserved_1[0x2];
+	u8 nic_interface[0x2];
+	u8 log_cmdq_size[0x4];
+	u8 log_cmdq_stride[0x4];
+	u8 command_doorbell_vector[0x20];
+	u8 reserved_2[0xf00];
+	u8 initializing[0x1];
+	u8 nic_interface_supported[0x7];
+	u8 reserved_4[0x18];
 	struct mlx5_ifc_health_buffer_bits health_buffer;
-	u8         no_dram_nic_offset[0x20];
-	u8         reserved_5[0x6de0];
-	u8         internal_timer_h[0x20];
-	u8         internal_timer_l[0x20];
-	u8         reserved_6[0x20];
-	u8         reserved_7[0x1f];
-	u8         clear_int[0x1];
-	u8         health_syndrome[0x8];
-	u8         health_counter[0x18];
-	u8         reserved_8[0x17fc0];
+	u8 no_dram_nic_offset[0x20];
+	u8 reserved_5[0x6de0];
+	u8 internal_timer_h[0x20];
+	u8 internal_timer_l[0x20];
+	u8 reserved_6[0x20];
+	u8 reserved_7[0x1f];
+	u8 clear_int[0x1];
+	u8 health_syndrome[0x8];
+	u8 health_counter[0x18];
+	u8 reserved_8[0x17fc0];
 };
 
 struct mlx5_ifc_create_cq_out_bits {
-- 
2.21.0

[dpdk-dev] [PATCH 02/24] common/mlx5: update GENEVE TLV OPT obj name

[Shiri Kuzin]

From: Dekel Peled <dekelp@nvidia.com>

Rename MLX5_OBJ_TYPE_GENEVE_TLV_OPT as
MLX5_GENERAL_OBJ_TYPE_GENEVE_TLV_OPT, to align with other general
objects names.

Signed-off-by: Dekel Peled <dekelp@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/mlx5_devx_cmds.c | 2 +-
 drivers/common/mlx5/mlx5_prm.h       | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c
index c90e020643..dfd2cb6af4 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.c
+++ b/drivers/common/mlx5/mlx5_devx_cmds.c
@@ -2176,7 +2176,7 @@ mlx5_devx_cmd_create_geneve_tlv_option(void *ctx,
 	MLX5_SET(general_obj_in_cmd_hdr, hdr, opcode,
 			MLX5_CMD_OP_CREATE_GENERAL_OBJECT);
 	MLX5_SET(general_obj_in_cmd_hdr, hdr, obj_type,
-			MLX5_OBJ_TYPE_GENEVE_TLV_OPT);
+		 MLX5_GENERAL_OBJ_TYPE_GENEVE_TLV_OPT);
 	MLX5_SET(geneve_tlv_option, opt, option_class,
 			rte_be_to_cpu_16(class));
 	MLX5_SET(geneve_tlv_option, opt, option_type, type);
diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h
index f0a4f70d7d..e3ec4201c3 100644
--- a/drivers/common/mlx5/mlx5_prm.h
+++ b/drivers/common/mlx5/mlx5_prm.h
@@ -1106,7 +1106,7 @@ enum {
 #define MLX5_GENERAL_OBJ_TYPES_CAP_FLOW_HIT_ASO \
 			(1ULL << MLX5_GENERAL_OBJ_TYPE_FLOW_HIT_ASO)
 #define MLX5_GENERAL_OBJ_TYPES_CAP_GENEVE_TLV_OPT \
-			(1ULL << MLX5_OBJ_TYPE_GENEVE_TLV_OPT)
+			(1ULL << MLX5_GENERAL_OBJ_TYPE_GENEVE_TLV_OPT)
 
 enum {
 	MLX5_HCA_CAP_OPMOD_GET_MAX   = 0,
@@ -2383,7 +2383,7 @@ struct mlx5_ifc_create_cq_in_bits {
 };
 
 enum {
-	MLX5_OBJ_TYPE_GENEVE_TLV_OPT = 0x000b,
+	MLX5_GENERAL_OBJ_TYPE_GENEVE_TLV_OPT = 0x000b,
 	MLX5_GENERAL_OBJ_TYPE_VIRTQ = 0x000d,
 	MLX5_GENERAL_OBJ_TYPE_VIRTIO_Q_COUNTERS = 0x001c,
 	MLX5_GENERAL_OBJ_TYPE_FLEX_PARSE_GRAPH = 0x0022,
-- 
2.21.0

[dpdk-dev] [PATCH 03/24] common/mlx5: optimize read of general obj type caps

[Shiri Kuzin]

From: Dekel Peled <dekelp@nvidia.com>

General object types support is indicated in bitmap general_obj_types,
which is part of HCA capabilities list.
Currently this bitmap is read multiple times, and each time a different
bit is extracted.

This patch optimizes the code, reading the bitmap once into a local
variable, and then extracting the required bits.

Signed-off-by: Dekel Peled <dekelp@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/mlx5_devx_cmds.c | 30 +++++++++++++++-------------
 1 file changed, 16 insertions(+), 14 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c
index dfd2cb6af4..8e4a5d71b6 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.c
+++ b/drivers/common/mlx5/mlx5_devx_cmds.c
@@ -646,6 +646,7 @@ mlx5_devx_cmd_query_hca_attr(void *ctx,
 	uint32_t out[MLX5_ST_SZ_DW(query_hca_cap_out)] = {0};
 	void *hcattr;
 	int status, syndrome, rc, i;
+	uint64_t general_obj_types_supported = 0;
 
 	MLX5_SET(query_hca_cap_in, in, opcode, MLX5_CMD_OP_QUERY_HCA_CAP);
 	MLX5_SET(query_hca_cap_in, in, op_mod,
@@ -695,15 +696,6 @@ mlx5_devx_cmd_query_hca_attr(void *ctx,
 	attr->max_geneve_tlv_option_data_len = MLX5_GET(cmd_hca_cap, hcattr,
 			max_geneve_tlv_option_data_len);
 	attr->qos.sup = MLX5_GET(cmd_hca_cap, hcattr, qos);
-	attr->vdpa.valid = !!(MLX5_GET64(cmd_hca_cap, hcattr,
-					 general_obj_types) &
-			      MLX5_GENERAL_OBJ_TYPES_CAP_VIRTQ_NET_Q);
-	attr->vdpa.queue_counters_valid = !!(MLX5_GET64(cmd_hca_cap, hcattr,
-							general_obj_types) &
-				  MLX5_GENERAL_OBJ_TYPES_CAP_VIRTIO_Q_COUNTERS);
-	attr->parse_graph_flex_node = !!(MLX5_GET64(cmd_hca_cap, hcattr,
-					 general_obj_types) &
-			      MLX5_GENERAL_OBJ_TYPES_CAP_PARSE_GRAPH_FLEX_NODE);
 	attr->wqe_index_ignore = MLX5_GET(cmd_hca_cap, hcattr,
 					  wqe_index_ignore_cap);
 	attr->cross_channel = MLX5_GET(cmd_hca_cap, hcattr, cd);
@@ -721,12 +713,22 @@ mlx5_devx_cmd_query_hca_attr(void *ctx,
 	attr->regex = MLX5_GET(cmd_hca_cap, hcattr, regexp);
 	attr->regexp_num_of_engines = MLX5_GET(cmd_hca_cap, hcattr,
 					       regexp_num_of_engines);
-	attr->flow_hit_aso = !!(MLX5_GET64(cmd_hca_cap, hcattr,
-					   general_obj_types) &
+	/* Read the general_obj_types bitmap and extract the relevant bits. */
+	general_obj_types_supported = MLX5_GET64(cmd_hca_cap, hcattr,
+						 general_obj_types);
+	attr->vdpa.valid = !!(general_obj_types_supported &
+			      MLX5_GENERAL_OBJ_TYPES_CAP_VIRTQ_NET_Q);
+	attr->vdpa.queue_counters_valid =
+			!!(general_obj_types_supported &
+			   MLX5_GENERAL_OBJ_TYPES_CAP_VIRTIO_Q_COUNTERS);
+	attr->parse_graph_flex_node =
+			!!(general_obj_types_supported &
+			   MLX5_GENERAL_OBJ_TYPES_CAP_PARSE_GRAPH_FLEX_NODE);
+	attr->flow_hit_aso = !!(general_obj_types_supported &
 				MLX5_GENERAL_OBJ_TYPES_CAP_FLOW_HIT_ASO);
-	attr->geneve_tlv_opt = !!(MLX5_GET64(cmd_hca_cap, hcattr,
-					   general_obj_types) &
-				MLX5_GENERAL_OBJ_TYPES_CAP_GENEVE_TLV_OPT);
+	attr->geneve_tlv_opt = !!(general_obj_types_supported &
+				  MLX5_GENERAL_OBJ_TYPES_CAP_GENEVE_TLV_OPT);
+	/* Add reading of other GENERAL_OBJ_TYPES_CAP bits above this line. */
 	attr->log_max_cq = MLX5_GET(cmd_hca_cap, hcattr, log_max_cq);
 	attr->log_max_qp = MLX5_GET(cmd_hca_cap, hcattr, log_max_qp);
 	attr->log_max_cq_sz = MLX5_GET(cmd_hca_cap, hcattr, log_max_cq_sz);
-- 
2.21.0

[dpdk-dev] [PATCH 04/24] common/mlx5: add HCA cap for AES-XTS crypto

[Shiri Kuzin]

From: Dekel Peled <dekelp@nvidia.com>

Update the PRM structure and HCA capabilities reading, to include
relevant capabilities for AES-XTS crypto.

Signed-off-by: Dekel Peled <dekelp@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/mlx5_devx_cmds.c | 3 +++
 drivers/common/mlx5/mlx5_devx_cmds.h | 2 ++
 drivers/common/mlx5/mlx5_prm.h       | 5 ++++-
 3 files changed, 9 insertions(+), 1 deletion(-)

diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c
index 8e4a5d71b6..f96f706ccd 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.c
+++ b/drivers/common/mlx5/mlx5_devx_cmds.c
@@ -754,6 +754,9 @@ mlx5_devx_cmd_query_hca_attr(void *ctx,
 						mini_cqe_resp_flow_tag);
 	attr->mini_cqe_resp_l3_l4_tag = MLX5_GET(cmd_hca_cap, hcattr,
 						 mini_cqe_resp_l3_l4_tag);
+	attr->crypto = MLX5_GET(cmd_hca_cap, hcattr, crypto);
+	if (attr->crypto)
+		attr->aes_xts = MLX5_GET(cmd_hca_cap, hcattr, aes_xts);
 	if (attr->qos.sup) {
 		MLX5_SET(query_hca_cap_in, in, op_mod,
 			 MLX5_GET_HCA_CAP_OP_MOD_QOS_CAP |
diff --git a/drivers/common/mlx5/mlx5_devx_cmds.h b/drivers/common/mlx5/mlx5_devx_cmds.h
index 2826c0b2c6..0c7a9ed76e 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.h
+++ b/drivers/common/mlx5/mlx5_devx_cmds.h
@@ -127,6 +127,8 @@ struct mlx5_hca_attr {
 	uint32_t qp_ts_format:2;
 	uint32_t regex:1;
 	uint32_t reg_c_preserve:1;
+	uint32_t crypto:1; /* Crypto engine is supported. */
+	uint32_t aes_xts:1; /* AES-XTS crypto is supported. */
 	uint32_t regexp_num_of_engines;
 	uint32_t log_max_ft_sampler_num:8;
 	uint32_t geneve_tlv_opt;
diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h
index e3ec4201c3..c7f973953e 100644
--- a/drivers/common/mlx5/mlx5_prm.h
+++ b/drivers/common/mlx5/mlx5_prm.h
@@ -1425,7 +1425,10 @@ struct mlx5_ifc_cmd_hca_cap_bits {
 	u8 sq_ts_format[0x2];
 	u8 rq_ts_format[0x2];
 	u8 reserved_at_444[0x1C];
-	u8 reserved_at_460[0x10];
+	u8 reserved_at_460[0x8];
+	u8 aes_xts[0x1];
+	u8 crypto[0x1];
+	u8 reserved_at_46a[0x6];
 	u8 max_num_eqs[0x10];
 	u8 reserved_at_480[0x3];
 	u8 log_max_l2_table[0x5];
-- 
2.21.0

[dpdk-dev] [PATCH 05/24] common/mlx5: support general object DEK create op

[Shiri Kuzin]

From: Dekel Peled <dekelp@nvidia.com>

Data Encryption Keys (DEKs) are the keys used for data
encryption/decryption operations.

Add reading of DEK support capability.
Add function to create general object type DEK, using DevX API.

Signed-off-by: Dekel Peled <dekelp@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/mlx5_devx_cmds.c | 53 ++++++++++++++++++++++++++++
 drivers/common/mlx5/mlx5_devx_cmds.h | 17 +++++++++
 drivers/common/mlx5/mlx5_prm.h       | 39 ++++++++++++++++++++
 drivers/common/mlx5/version.map      |  1 +
 4 files changed, 110 insertions(+)

diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c
index f96f706ccd..bad474aeae 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.c
+++ b/drivers/common/mlx5/mlx5_devx_cmds.c
@@ -728,6 +728,8 @@ mlx5_devx_cmd_query_hca_attr(void *ctx,
 				MLX5_GENERAL_OBJ_TYPES_CAP_FLOW_HIT_ASO);
 	attr->geneve_tlv_opt = !!(general_obj_types_supported &
 				  MLX5_GENERAL_OBJ_TYPES_CAP_GENEVE_TLV_OPT);
+	attr->dek = !!(general_obj_types_supported &
+		       MLX5_GENERAL_OBJ_TYPES_CAP_DEK);
 	/* Add reading of other GENERAL_OBJ_TYPES_CAP bits above this line. */
 	attr->log_max_cq = MLX5_GET(cmd_hca_cap, hcattr, log_max_cq);
 	attr->log_max_qp = MLX5_GET(cmd_hca_cap, hcattr, log_max_qp);
@@ -2299,3 +2301,54 @@ mlx5_devx_cmd_queue_counter_query(struct mlx5_devx_obj *dcs, int clear,
 	*out_of_buffers = MLX5_GET(query_q_counter_out, out, out_of_buffer);
 	return 0;
 }
+
+/**
+ * Create general object of type DEK using DevX API.
+ *
+ * @param[in] ctx
+ *   Context returned from mlx5 open_device() glue function.
+ * @param [in] attr
+ *   Pointer to DEK attributes structure.
+ *
+ * @return
+ *   The DevX object created, NULL otherwise and rte_errno is set.
+ */
+struct mlx5_devx_obj *
+mlx5_devx_cmd_create_dek_obj(void *ctx, struct mlx5_devx_dek_attr *attr)
+{
+	uint32_t in[MLX5_ST_SZ_DW(create_dek_in)] = {0};
+	uint32_t out[MLX5_ST_SZ_DW(general_obj_out_cmd_hdr)] = {0};
+	struct mlx5_devx_obj *dek_obj = NULL;
+	void *ptr = NULL, *key_addr = NULL;
+
+	dek_obj = mlx5_malloc(MLX5_MEM_ZERO, sizeof(*dek_obj),
+			      0, SOCKET_ID_ANY);
+	if (dek_obj == NULL) {
+		DRV_LOG(ERR, "Failed to allocate DEK object data");
+		rte_errno = ENOMEM;
+		return NULL;
+	}
+	ptr = MLX5_ADDR_OF(create_dek_in, in, hdr);
+	MLX5_SET(general_obj_in_cmd_hdr, ptr, opcode,
+		 MLX5_CMD_OP_CREATE_GENERAL_OBJECT);
+	MLX5_SET(general_obj_in_cmd_hdr, ptr, obj_type,
+		 MLX5_GENERAL_OBJ_TYPE_DEK);
+	ptr = MLX5_ADDR_OF(create_dek_in, in, dek);
+	MLX5_SET(dek, ptr, key_size, attr->key_size);
+	MLX5_SET(dek, ptr, has_keytag, attr->has_keytag);
+	MLX5_SET(dek, ptr, key_purpose, attr->key_purpose);
+	MLX5_SET(dek, ptr, pd, attr->pd);
+	MLX5_SET64(dek, ptr, opaque, attr->opaque);
+	key_addr = MLX5_ADDR_OF(dek, ptr, key);
+	memcpy(key_addr, (void *)(attr->key), MLX5_CRYPTO_KEY_MAX_SIZE);
+	dek_obj->obj = mlx5_glue->devx_obj_create(ctx, in, sizeof(in),
+						  out, sizeof(out));
+	if (dek_obj->obj == NULL) {
+		rte_errno = errno;
+		DRV_LOG(ERR, "Failed to create DEK obj using DevX.");
+		mlx5_free(dek_obj);
+		return NULL;
+	}
+	dek_obj->id = MLX5_GET(general_obj_out_cmd_hdr, out, obj_id);
+	return dek_obj;
+}
diff --git a/drivers/common/mlx5/mlx5_devx_cmds.h b/drivers/common/mlx5/mlx5_devx_cmds.h
index 0c7a9ed76e..e17bad6bce 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.h
+++ b/drivers/common/mlx5/mlx5_devx_cmds.h
@@ -129,6 +129,7 @@ struct mlx5_hca_attr {
 	uint32_t reg_c_preserve:1;
 	uint32_t crypto:1; /* Crypto engine is supported. */
 	uint32_t aes_xts:1; /* AES-XTS crypto is supported. */
+	uint32_t dek:1; /* General obj type DEK is supported. */
 	uint32_t regexp_num_of_engines;
 	uint32_t log_max_ft_sampler_num:8;
 	uint32_t geneve_tlv_opt;
@@ -423,6 +424,18 @@ struct mlx5_devx_graph_node_attr {
 	struct mlx5_devx_graph_arc_attr out[MLX5_GRAPH_NODE_ARC_NUM];
 };
 
+/* Encryption key size is up to 1024 bit, 128 bytes. */
+#define MLX5_CRYPTO_KEY_MAX_SIZE	128
+
+struct mlx5_devx_dek_attr {
+	uint32_t key_size:4;
+	uint32_t has_keytag:1;
+	uint32_t key_purpose:4;
+	uint32_t pd:24;
+	uint64_t opaque;
+	uint8_t key[MLX5_CRYPTO_KEY_MAX_SIZE];
+};
+
 /* mlx5_devx_cmds.c */
 
 __rte_internal
@@ -557,4 +570,8 @@ struct mlx5_devx_obj *mlx5_devx_cmd_queue_counter_alloc(void *ctx);
 __rte_internal
 int mlx5_devx_cmd_queue_counter_query(struct mlx5_devx_obj *dcs, int clear,
 				      uint32_t *out_of_buffers);
+__rte_internal
+struct mlx5_devx_obj *
+mlx5_devx_cmd_create_dek_obj(void *ctx, struct mlx5_devx_dek_attr *attr);
+
 #endif /* RTE_PMD_MLX5_DEVX_CMDS_H_ */
diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h
index c7f973953e..fbe191cc94 100644
--- a/drivers/common/mlx5/mlx5_prm.h
+++ b/drivers/common/mlx5/mlx5_prm.h
@@ -1107,6 +1107,8 @@ enum {
 			(1ULL << MLX5_GENERAL_OBJ_TYPE_FLOW_HIT_ASO)
 #define MLX5_GENERAL_OBJ_TYPES_CAP_GENEVE_TLV_OPT \
 			(1ULL << MLX5_GENERAL_OBJ_TYPE_GENEVE_TLV_OPT)
+#define MLX5_GENERAL_OBJ_TYPES_CAP_DEK \
+			(1ULL << MLX5_GENERAL_OBJ_TYPE_DEK)
 
 enum {
 	MLX5_HCA_CAP_OPMOD_GET_MAX   = 0,
@@ -2387,6 +2389,7 @@ struct mlx5_ifc_create_cq_in_bits {
 
 enum {
 	MLX5_GENERAL_OBJ_TYPE_GENEVE_TLV_OPT = 0x000b,
+	MLX5_GENERAL_OBJ_TYPE_DEK = 0x000c,
 	MLX5_GENERAL_OBJ_TYPE_VIRTQ = 0x000d,
 	MLX5_GENERAL_OBJ_TYPE_VIRTIO_Q_COUNTERS = 0x001c,
 	MLX5_GENERAL_OBJ_TYPE_FLEX_PARSE_GRAPH = 0x0022,
@@ -2447,6 +2450,42 @@ struct mlx5_ifc_create_geneve_tlv_option_in_bits {
 	struct mlx5_ifc_geneve_tlv_option_bits geneve_tlv_opt;
 };
 
+enum {
+	MLX5_CRYPTO_KEY_SIZE_128b = 0x0,
+	MLX5_CRYPTO_KEY_SIZE_256b = 0x1,
+};
+
+enum {
+	MLX5_CRYPTO_KEY_PURPOSE_TLS	= 0x1,
+	MLX5_CRYPTO_KEY_PURPOSE_IPSEC	= 0x2,
+	MLX5_CRYPTO_KEY_PURPOSE_AES_XTS	= 0x3,
+	MLX5_CRYPTO_KEY_PURPOSE_MACSEC	= 0x4,
+	MLX5_CRYPTO_KEY_PURPOSE_GCM	= 0x5,
+	MLX5_CRYPTO_KEY_PURPOSE_PSP	= 0x6,
+};
+
+struct mlx5_ifc_dek_bits {
+	u8 modify_field_select[0x40];
+	u8 state[0x8];
+	u8 reserved_at_48[0xc];
+	u8 key_size[0x4];
+	u8 has_keytag[0x1];
+	u8 reserved_at_59[0x3];
+	u8 key_purpose[0x4];
+	u8 reserved_at_60[0x8];
+	u8 pd[0x18];
+	u8 reserved_at_80[0x100];
+	u8 opaque[0x40];
+	u8 reserved_at_1c0[0x40];
+	u8 key[0x400];
+	u8 reserved_at_600[0x200];
+};
+
+struct mlx5_ifc_create_dek_in_bits {
+	struct mlx5_ifc_general_obj_in_cmd_hdr_bits hdr;
+	struct mlx5_ifc_dek_bits dek;
+};
+
 enum {
 	MLX5_VIRTQ_STATE_INIT = 0,
 	MLX5_VIRTQ_STATE_RDY = 1,
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index 91f3fa5779..50017f9005 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -24,6 +24,7 @@ INTERNAL {
 	mlx5_devx_cmd_create_virtq;
 	mlx5_devx_cmd_create_flow_hit_aso_obj;
 	mlx5_devx_cmd_create_geneve_tlv_option;
+        mlx5_devx_cmd_create_dek_obj;
 	mlx5_devx_cmd_destroy;
 	mlx5_devx_cmd_flow_counter_alloc;
 	mlx5_devx_cmd_flow_counter_query;
-- 
2.21.0

[dpdk-dev] [PATCH 06/24] common/mlx5: add crypto en field to MKEY context

[Shiri Kuzin]

From: Dekel Peled <dekelp@nvidia.com>

MKEY that will be used for crypto purposes must be created with
crypto_en attribute.
This patch adds support for crypto_en attribute in MKEY context.

Signed-off-by: Dekel Peled <dekelp@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/mlx5_devx_cmds.c |  1 +
 drivers/common/mlx5/mlx5_devx_cmds.h |  1 +
 drivers/common/mlx5/mlx5_prm.h       | 10 +++++++++-
 3 files changed, 11 insertions(+), 1 deletion(-)

diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c
index bad474aeae..262930675a 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.c
+++ b/drivers/common/mlx5/mlx5_devx_cmds.c
@@ -272,6 +272,7 @@ mlx5_devx_cmd_mkey_create(void *ctx,
 	MLX5_SET(mkc, mkc, relaxed_ordering_read, attr->relaxed_ordering_read);
 	MLX5_SET64(mkc, mkc, start_addr, attr->addr);
 	MLX5_SET64(mkc, mkc, len, attr->size);
+	MLX5_SET(mkc, mkc, crypto_en, attr->crypto_en);
 	mkey->obj = mlx5_glue->devx_obj_create(ctx, in, in_size_dw * 4, out,
 					       sizeof(out));
 	if (!mkey->obj) {
diff --git a/drivers/common/mlx5/mlx5_devx_cmds.h b/drivers/common/mlx5/mlx5_devx_cmds.h
index e17bad6bce..94bd8f292f 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.h
+++ b/drivers/common/mlx5/mlx5_devx_cmds.h
@@ -31,6 +31,7 @@ struct mlx5_devx_mkey_attr {
 	uint32_t pg_access:1;
 	uint32_t relaxed_ordering_write:1;
 	uint32_t relaxed_ordering_read:1;
+	uint32_t crypto_en:2;
 	struct mlx5_klm *klm_array;
 	int klm_num;
 };
diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h
index fbe191cc94..4a06e656fd 100644
--- a/drivers/common/mlx5/mlx5_prm.h
+++ b/drivers/common/mlx5/mlx5_prm.h
@@ -1059,7 +1059,15 @@ struct mlx5_ifc_mkc_bits {
 	u8 relaxed_ordering_read[0x1];
 	u8 reserved_at_1da[0x1];
 	u8 log_page_size[0x5];
-	u8 reserved_at_1e0[0x20];
+	u8 reserved_at_1e0[0x3];
+	u8 crypto_en[0x2];
+	u8 reserved_at_1e5[0x1b];
+};
+
+/* Range of values for MKEY context crypto_en field. */
+enum {
+	MLX5_MKEY_CRYPTO_DISABLED = 0x0,
+	MLX5_MKEY_CRYPTO_ENABLED = 0x1,
 };
 
 struct mlx5_ifc_create_mkey_out_bits {
-- 
2.21.0

[dpdk-dev] [PATCH 07/24] common/mlx5: support umr en field in MKEY context

[Shiri Kuzin]

From: Dekel Peled <dekelp@nvidia.com>

MKEY that will be used in UMR operations must be created with
umr_en attribute.
This patch adds support for setting umr_en attribute in MKEY context.

Signed-off-by: Dekel Peled <dekelp@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/mlx5_devx_cmds.c | 1 +
 drivers/common/mlx5/mlx5_devx_cmds.h | 1 +
 2 files changed, 2 insertions(+)

diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c
index 262930675a..7208ae1f11 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.c
+++ b/drivers/common/mlx5/mlx5_devx_cmds.c
@@ -273,6 +273,7 @@ mlx5_devx_cmd_mkey_create(void *ctx,
 	MLX5_SET64(mkc, mkc, start_addr, attr->addr);
 	MLX5_SET64(mkc, mkc, len, attr->size);
 	MLX5_SET(mkc, mkc, crypto_en, attr->crypto_en);
+	MLX5_SET(mkc, mkc, umr_en, attr->umr_en);
 	mkey->obj = mlx5_glue->devx_obj_create(ctx, in, in_size_dw * 4, out,
 					       sizeof(out));
 	if (!mkey->obj) {
diff --git a/drivers/common/mlx5/mlx5_devx_cmds.h b/drivers/common/mlx5/mlx5_devx_cmds.h
index 94bd8f292f..1571542dbc 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.h
+++ b/drivers/common/mlx5/mlx5_devx_cmds.h
@@ -32,6 +32,7 @@ struct mlx5_devx_mkey_attr {
 	uint32_t relaxed_ordering_write:1;
 	uint32_t relaxed_ordering_read:1;
 	uint32_t crypto_en:2;
+	uint32_t umr_en:1;
 	struct mlx5_klm *klm_array;
 	int klm_num;
 };
-- 
2.21.0

[dpdk-dev] [PATCH 08/24] common/mlx5: support general obj IMPORT KEK create

[Shiri Kuzin]

From: Dekel Peled <dekelp@nvidia.com>

IMPORT_KEK object is used to wrap (encrypt) critical security
parameters, such as other keys and credentials, when those need
to be passed between the device and the software.

This patch add support of IMPORT_KEK object create operation.
Add reading of IMPORT_KEK support capability.
Add function to create general object type IMPORT_KEK, using DevX API.

Signed-off-by: Dekel Peled <dekelp@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/mlx5_devx_cmds.c | 50 ++++++++++++++++++++++++++++
 drivers/common/mlx5/mlx5_devx_cmds.h | 13 ++++++++
 drivers/common/mlx5/mlx5_prm.h       | 18 ++++++++++
 drivers/common/mlx5/version.map      |  1 +
 4 files changed, 82 insertions(+)

diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c
index 7208ae1f11..5855b5ead5 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.c
+++ b/drivers/common/mlx5/mlx5_devx_cmds.c
@@ -732,6 +732,8 @@ mlx5_devx_cmd_query_hca_attr(void *ctx,
 				  MLX5_GENERAL_OBJ_TYPES_CAP_GENEVE_TLV_OPT);
 	attr->dek = !!(general_obj_types_supported &
 		       MLX5_GENERAL_OBJ_TYPES_CAP_DEK);
+	attr->import_kek = !!(general_obj_types_supported &
+			      MLX5_GENERAL_OBJ_TYPES_CAP_IMPORT_KEK);
 	/* Add reading of other GENERAL_OBJ_TYPES_CAP bits above this line. */
 	attr->log_max_cq = MLX5_GET(cmd_hca_cap, hcattr, log_max_cq);
 	attr->log_max_qp = MLX5_GET(cmd_hca_cap, hcattr, log_max_qp);
@@ -2354,3 +2356,51 @@ mlx5_devx_cmd_create_dek_obj(void *ctx, struct mlx5_devx_dek_attr *attr)
 	dek_obj->id = MLX5_GET(general_obj_out_cmd_hdr, out, obj_id);
 	return dek_obj;
 }
+
+/**
+ * Create general object of type IMPORT_KEK using DevX API.
+ *
+ * @param[in] ctx
+ *   Context returned from mlx5 open_device() glue function.
+ * @param [in] attr
+ *   Pointer to IMPORT_KEK attributes structure.
+ *
+ * @return
+ *   The DevX object created, NULL otherwise and rte_errno is set.
+ */
+struct mlx5_devx_obj *
+mlx5_devx_cmd_create_import_kek_obj(void *ctx,
+				    struct mlx5_devx_import_kek_attr *attr)
+{
+	uint32_t in[MLX5_ST_SZ_DW(create_import_kek_in)] = {0};
+	uint32_t out[MLX5_ST_SZ_DW(general_obj_out_cmd_hdr)] = {0};
+	struct mlx5_devx_obj *import_kek_obj = NULL;
+	void *ptr = NULL, *key_addr = NULL;
+
+	import_kek_obj = mlx5_malloc(MLX5_MEM_ZERO, sizeof(*import_kek_obj),
+				     0, SOCKET_ID_ANY);
+	if (import_kek_obj == NULL) {
+		DRV_LOG(ERR, "Failed to allocate IMPORT_KEK object data");
+		rte_errno = ENOMEM;
+		return NULL;
+	}
+	ptr = MLX5_ADDR_OF(create_import_kek_in, in, hdr);
+	MLX5_SET(general_obj_in_cmd_hdr, ptr, opcode,
+		 MLX5_CMD_OP_CREATE_GENERAL_OBJECT);
+	MLX5_SET(general_obj_in_cmd_hdr, ptr, obj_type,
+		 MLX5_GENERAL_OBJ_TYPE_IMPORT_KEK);
+	ptr = MLX5_ADDR_OF(create_import_kek_in, in, import_kek);
+	MLX5_SET(import_kek, ptr, key_size, attr->key_size);
+	key_addr = MLX5_ADDR_OF(import_kek, ptr, key);
+	memcpy(key_addr, (void *)(attr->key), MLX5_CRYPTO_KEY_MAX_SIZE);
+	import_kek_obj->obj = mlx5_glue->devx_obj_create(ctx, in, sizeof(in),
+							 out, sizeof(out));
+	if (import_kek_obj->obj == NULL) {
+		rte_errno = errno;
+		DRV_LOG(ERR, "Failed to create IMPORT_KEK object using DevX.");
+		mlx5_free(import_kek_obj);
+		return NULL;
+	}
+	import_kek_obj->id = MLX5_GET(general_obj_out_cmd_hdr, out, obj_id);
+	return import_kek_obj;
+}
diff --git a/drivers/common/mlx5/mlx5_devx_cmds.h b/drivers/common/mlx5/mlx5_devx_cmds.h
index 1571542dbc..eebe05911f 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.h
+++ b/drivers/common/mlx5/mlx5_devx_cmds.h
@@ -132,6 +132,7 @@ struct mlx5_hca_attr {
 	uint32_t crypto:1; /* Crypto engine is supported. */
 	uint32_t aes_xts:1; /* AES-XTS crypto is supported. */
 	uint32_t dek:1; /* General obj type DEK is supported. */
+	uint32_t import_kek:1; /* General obj type IMPORT_KEK supported. */
 	uint32_t regexp_num_of_engines;
 	uint32_t log_max_ft_sampler_num:8;
 	uint32_t geneve_tlv_opt;
@@ -438,6 +439,13 @@ struct mlx5_devx_dek_attr {
 	uint8_t key[MLX5_CRYPTO_KEY_MAX_SIZE];
 };
 
+struct mlx5_devx_import_kek_attr {
+	uint64_t modify_field_select;
+	uint32_t state:8;
+	uint32_t key_size:4;
+	uint8_t key[MLX5_CRYPTO_KEY_MAX_SIZE];
+};
+
 /* mlx5_devx_cmds.c */
 
 __rte_internal
@@ -576,4 +584,9 @@ __rte_internal
 struct mlx5_devx_obj *
 mlx5_devx_cmd_create_dek_obj(void *ctx, struct mlx5_devx_dek_attr *attr);
 
+__rte_internal
+struct mlx5_devx_obj *
+mlx5_devx_cmd_create_import_kek_obj(void *ctx,
+				    struct mlx5_devx_import_kek_attr *attr);
+
 #endif /* RTE_PMD_MLX5_DEVX_CMDS_H_ */
diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h
index 4a06e656fd..1569dae9a4 100644
--- a/drivers/common/mlx5/mlx5_prm.h
+++ b/drivers/common/mlx5/mlx5_prm.h
@@ -1117,6 +1117,8 @@ enum {
 			(1ULL << MLX5_GENERAL_OBJ_TYPE_GENEVE_TLV_OPT)
 #define MLX5_GENERAL_OBJ_TYPES_CAP_DEK \
 			(1ULL << MLX5_GENERAL_OBJ_TYPE_DEK)
+#define MLX5_GENERAL_OBJ_TYPES_CAP_IMPORT_KEK \
+			(1ULL << MLX5_GENERAL_OBJ_TYPE_IMPORT_KEK)
 
 enum {
 	MLX5_HCA_CAP_OPMOD_GET_MAX   = 0,
@@ -2400,6 +2402,7 @@ enum {
 	MLX5_GENERAL_OBJ_TYPE_DEK = 0x000c,
 	MLX5_GENERAL_OBJ_TYPE_VIRTQ = 0x000d,
 	MLX5_GENERAL_OBJ_TYPE_VIRTIO_Q_COUNTERS = 0x001c,
+	MLX5_GENERAL_OBJ_TYPE_IMPORT_KEK = 0x001d,
 	MLX5_GENERAL_OBJ_TYPE_FLEX_PARSE_GRAPH = 0x0022,
 	MLX5_GENERAL_OBJ_TYPE_FLOW_HIT_ASO = 0x0025,
 };
@@ -2494,6 +2497,21 @@ struct mlx5_ifc_create_dek_in_bits {
 	struct mlx5_ifc_dek_bits dek;
 };
 
+struct mlx5_ifc_import_kek_bits {
+	u8 modify_field_select[0x40];
+	u8 state[0x8];
+	u8 reserved_at_48[0xc];
+	u8 key_size[0x4];
+	u8 reserved_at_58[0x1a8];
+	u8 key[0x400];
+	u8 reserved_at_600[0x200];
+};
+
+struct mlx5_ifc_create_import_kek_in_bits {
+	struct mlx5_ifc_general_obj_in_cmd_hdr_bits hdr;
+	struct mlx5_ifc_import_kek_bits import_kek;
+};
+
 enum {
 	MLX5_VIRTQ_STATE_INIT = 0,
 	MLX5_VIRTQ_STATE_RDY = 1,
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index 50017f9005..1407e52d02 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -25,6 +25,7 @@ INTERNAL {
 	mlx5_devx_cmd_create_flow_hit_aso_obj;
 	mlx5_devx_cmd_create_geneve_tlv_option;
         mlx5_devx_cmd_create_dek_obj;
+        mlx5_devx_cmd_create_import_kek_obj;
 	mlx5_devx_cmd_destroy;
 	mlx5_devx_cmd_flow_counter_alloc;
 	mlx5_devx_cmd_flow_counter_query;
-- 
2.21.0

[dpdk-dev] [PATCH 09/24] common/mlx5: support general obj CRYPTO LOGIN create

[Shiri Kuzin]

From: Dekel Peled <dekelp@nvidia.com>

CRYPTO_LOGIN Object is used to login to the device as crypto user
or crypto officer.
Required in order to perform any crypto related control operations.

This patch adds support of CRYPTO_LOGIN object create operation.
Add reading of CRYPTO_LOGIN support capability.
Add function to create general object type CRYPTO_LOGIN, using DevX API.

Signed-off-by: Dekel Peled <dekelp@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/mlx5_devx_cmds.c | 54 ++++++++++++++++++++++++++++
 drivers/common/mlx5/mlx5_devx_cmds.h | 15 ++++++++
 drivers/common/mlx5/mlx5_prm.h       | 19 ++++++++++
 drivers/common/mlx5/version.map      |  1 +
 4 files changed, 89 insertions(+)

diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c
index 5855b5ead5..c3b3b50b51 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.c
+++ b/drivers/common/mlx5/mlx5_devx_cmds.c
@@ -734,6 +734,8 @@ mlx5_devx_cmd_query_hca_attr(void *ctx,
 		       MLX5_GENERAL_OBJ_TYPES_CAP_DEK);
 	attr->import_kek = !!(general_obj_types_supported &
 			      MLX5_GENERAL_OBJ_TYPES_CAP_IMPORT_KEK);
+	attr->crypto_login = !!(general_obj_types_supported &
+				MLX5_GENERAL_OBJ_TYPES_CAP_CRYPTO_LOGIN);
 	/* Add reading of other GENERAL_OBJ_TYPES_CAP bits above this line. */
 	attr->log_max_cq = MLX5_GET(cmd_hca_cap, hcattr, log_max_cq);
 	attr->log_max_qp = MLX5_GET(cmd_hca_cap, hcattr, log_max_qp);
@@ -2404,3 +2406,55 @@ mlx5_devx_cmd_create_import_kek_obj(void *ctx,
 	import_kek_obj->id = MLX5_GET(general_obj_out_cmd_hdr, out, obj_id);
 	return import_kek_obj;
 }
+
+/**
+ * Create general object of type CRYPTO_LOGIN using DevX API.
+ *
+ * @param[in] ctx
+ *   Context returned from mlx5 open_device() glue function.
+ * @param [in] attr
+ *   Pointer to CRYPTO_LOGIN attributes structure.
+ *
+ * @return
+ *   The DevX object created, NULL otherwise and rte_errno is set.
+ */
+struct mlx5_devx_obj *
+mlx5_devx_cmd_create_crypto_login_obj(void *ctx,
+				      struct mlx5_devx_crypto_login_attr *attr)
+{
+	uint32_t in[MLX5_ST_SZ_DW(create_crypto_login_in)] = {0};
+	uint32_t out[MLX5_ST_SZ_DW(general_obj_out_cmd_hdr)] = {0};
+	struct mlx5_devx_obj *crypto_login_obj = NULL;
+	void *ptr = NULL, *credential_addr = NULL;
+
+	crypto_login_obj = mlx5_malloc(MLX5_MEM_ZERO, sizeof(*crypto_login_obj),
+				       0, SOCKET_ID_ANY);
+	if (crypto_login_obj == NULL) {
+		DRV_LOG(ERR, "Failed to allocate CRYPTO_LOGIN object data");
+		rte_errno = ENOMEM;
+		return NULL;
+	}
+	ptr = MLX5_ADDR_OF(create_crypto_login_in, in, hdr);
+	MLX5_SET(general_obj_in_cmd_hdr, ptr, opcode,
+		 MLX5_CMD_OP_CREATE_GENERAL_OBJECT);
+	MLX5_SET(general_obj_in_cmd_hdr, ptr, obj_type,
+		 MLX5_GENERAL_OBJ_TYPE_CRYPTO_LOGIN);
+	ptr = MLX5_ADDR_OF(create_crypto_login_in, in, crypto_login);
+	MLX5_SET(crypto_login, ptr, credential_pointer,
+		 attr->credential_pointer);
+	MLX5_SET(crypto_login, ptr, session_import_kek_ptr,
+		 attr->session_import_kek_ptr);
+	credential_addr = MLX5_ADDR_OF(crypto_login, ptr, credential);
+	memcpy(credential_addr, (void *)(attr->credential),
+	       MLX5_CRYPTO_LOGIN_CREDENTIAL_SIZE);
+	crypto_login_obj->obj = mlx5_glue->devx_obj_create(ctx, in, sizeof(in),
+							   out, sizeof(out));
+	if (crypto_login_obj->obj == NULL) {
+		rte_errno = errno;
+		DRV_LOG(ERR, "Failed to create CRYPTO_LOGIN obj using DevX.");
+		mlx5_free(crypto_login_obj);
+		return NULL;
+	}
+	crypto_login_obj->id = MLX5_GET(general_obj_out_cmd_hdr, out, obj_id);
+	return crypto_login_obj;
+}
diff --git a/drivers/common/mlx5/mlx5_devx_cmds.h b/drivers/common/mlx5/mlx5_devx_cmds.h
index eebe05911f..b723aee126 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.h
+++ b/drivers/common/mlx5/mlx5_devx_cmds.h
@@ -133,6 +133,7 @@ struct mlx5_hca_attr {
 	uint32_t aes_xts:1; /* AES-XTS crypto is supported. */
 	uint32_t dek:1; /* General obj type DEK is supported. */
 	uint32_t import_kek:1; /* General obj type IMPORT_KEK supported. */
+	uint32_t crypto_login:1; /* General obj type CRYPTO_LOGIN supported. */
 	uint32_t regexp_num_of_engines;
 	uint32_t log_max_ft_sampler_num:8;
 	uint32_t geneve_tlv_opt;
@@ -446,6 +447,15 @@ struct mlx5_devx_import_kek_attr {
 	uint8_t key[MLX5_CRYPTO_KEY_MAX_SIZE];
 };
 
+#define MLX5_CRYPTO_LOGIN_CREDENTIAL_SIZE	48
+
+struct mlx5_devx_crypto_login_attr {
+	uint64_t modify_field_select;
+	uint32_t credential_pointer:24;
+	uint32_t session_import_kek_ptr:24;
+	uint8_t credential[MLX5_CRYPTO_LOGIN_CREDENTIAL_SIZE];
+};
+
 /* mlx5_devx_cmds.c */
 
 __rte_internal
@@ -589,4 +599,9 @@ struct mlx5_devx_obj *
 mlx5_devx_cmd_create_import_kek_obj(void *ctx,
 				    struct mlx5_devx_import_kek_attr *attr);
 
+__rte_internal
+struct mlx5_devx_obj *
+mlx5_devx_cmd_create_crypto_login_obj(void *ctx,
+				      struct mlx5_devx_crypto_login_attr *attr);
+
 #endif /* RTE_PMD_MLX5_DEVX_CMDS_H_ */
diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h
index 1569dae9a4..ca2e70881d 100644
--- a/drivers/common/mlx5/mlx5_prm.h
+++ b/drivers/common/mlx5/mlx5_prm.h
@@ -1119,6 +1119,8 @@ enum {
 			(1ULL << MLX5_GENERAL_OBJ_TYPE_DEK)
 #define MLX5_GENERAL_OBJ_TYPES_CAP_IMPORT_KEK \
 			(1ULL << MLX5_GENERAL_OBJ_TYPE_IMPORT_KEK)
+#define MLX5_GENERAL_OBJ_TYPES_CAP_CRYPTO_LOGIN \
+			(1ULL << MLX5_GENERAL_OBJ_TYPE_CRYPTO_LOGIN)
 
 enum {
 	MLX5_HCA_CAP_OPMOD_GET_MAX   = 0,
@@ -2403,6 +2405,7 @@ enum {
 	MLX5_GENERAL_OBJ_TYPE_VIRTQ = 0x000d,
 	MLX5_GENERAL_OBJ_TYPE_VIRTIO_Q_COUNTERS = 0x001c,
 	MLX5_GENERAL_OBJ_TYPE_IMPORT_KEK = 0x001d,
+	MLX5_GENERAL_OBJ_TYPE_CRYPTO_LOGIN = 0x001f,
 	MLX5_GENERAL_OBJ_TYPE_FLEX_PARSE_GRAPH = 0x0022,
 	MLX5_GENERAL_OBJ_TYPE_FLOW_HIT_ASO = 0x0025,
 };
@@ -2512,6 +2515,22 @@ struct mlx5_ifc_create_import_kek_in_bits {
 	struct mlx5_ifc_import_kek_bits import_kek;
 };
 
+struct mlx5_ifc_crypto_login_bits {
+	u8 modify_field_select[0x40];
+	u8 reserved_at_40[0x48];
+	u8 credential_pointer[0x18];
+	u8 reserved_at_a0[0x8];
+	u8 session_import_kek_ptr[0x18];
+	u8 reserved_at_c0[0x140];
+	u8 credential[0x180];
+	u8 reserved_at_380[0x480];
+};
+
+struct mlx5_ifc_create_crypto_login_in_bits {
+	struct mlx5_ifc_general_obj_in_cmd_hdr_bits hdr;
+	struct mlx5_ifc_crypto_login_bits crypto_login;
+};
+
 enum {
 	MLX5_VIRTQ_STATE_INIT = 0,
 	MLX5_VIRTQ_STATE_RDY = 1,
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index 1407e52d02..120be7a99d 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -26,6 +26,7 @@ INTERNAL {
 	mlx5_devx_cmd_create_geneve_tlv_option;
         mlx5_devx_cmd_create_dek_obj;
         mlx5_devx_cmd_create_import_kek_obj;
+        mlx5_devx_cmd_create_crypto_login_obj;
 	mlx5_devx_cmd_destroy;
 	mlx5_devx_cmd_flow_counter_alloc;
 	mlx5_devx_cmd_flow_counter_query;
-- 
2.21.0

[dpdk-dev] [PATCH 10/24] common/mlx5: add crypto BSF struct and defines

[Shiri Kuzin]

From: Dekel Peled <dekelp@nvidia.com>

This patch adds the struct defining crypto BSF segment of UMR WQE,
and the related value definitions and offsets.

Signed-off-by: Dekel Peled <dekelp@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/mlx5_prm.h | 66 ++++++++++++++++++++++++++++++++++
 1 file changed, 66 insertions(+)

diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h
index ca2e70881d..736badb6df 100644
--- a/drivers/common/mlx5/mlx5_prm.h
+++ b/drivers/common/mlx5/mlx5_prm.h
@@ -1096,6 +1096,72 @@ struct mlx5_ifc_create_mkey_in_bits {
 	u8 klm_pas_mtt[][0x20];
 };
 
+enum {
+	MLX5_BSF_SIZE_16B = 0x0,
+	MLX5_BSF_SIZE_32B = 0x1,
+	MLX5_BSF_SIZE_64B = 0x2,
+	MLX5_BSF_SIZE_128B = 0x3,
+};
+
+enum {
+	MLX5_BSF_P_TYPE_SIGNATURE = 0x0,
+	MLX5_BSF_P_TYPE_CRYPTO = 0x1,
+};
+
+enum {
+	MLX5_ENCRYPTION_ORDER_ENCRYPTED_WIRE_SIGNATURE = 0x0,
+	MLX5_ENCRYPTION_ORDER_ENCRYPTED_MEMORY_SIGNATURE = 0x1,
+	MLX5_ENCRYPTION_ORDER_ENCRYPTED_RAW_WIRE = 0x2,
+	MLX5_ENCRYPTION_ORDER_ENCRYPTED_RAW_MEMORY = 0x3,
+};
+
+enum {
+	MLX5_ENCRYPTION_STANDARD_AES_XTS = 0x0,
+};
+
+enum {
+	MLX5_BLOCK_SIZE_512B	= 0x1,
+	MLX5_BLOCK_SIZE_520B	= 0x2,
+	MLX5_BLOCK_SIZE_4096B	= 0x3,
+	MLX5_BLOCK_SIZE_4160B	= 0x4,
+	MLX5_BLOCK_SIZE_1MB	= 0x5,
+	MLX5_BLOCK_SIZE_4048B	= 0x6,
+};
+
+#define MLX5_BSF_SIZE_OFFSET		30
+#define MLX5_BSF_P_TYPE_OFFSET		24
+#define MLX5_ENCRYPTION_ORDER_OFFSET	16
+#define MLX5_BLOCK_SIZE_OFFSET		24
+
+struct mlx5_wqe_umr_bsf_seg {
+	/*
+	 * bs_bpt_eo_es contains:
+	 * bs	bsf_size		2 bits at MLX5_BSF_SIZE_OFFSET
+	 * bpt	bsf_p_type		2 bits at MLX5_BSF_P_TYPE_OFFSET
+	 * eo	encryption_order	4 bits at MLX5_ENCRYPTION_ORDER_OFFSET
+	 * es	encryption_standard	4 bits at offset 0
+	 */
+	uint32_t bs_bpt_eo_es;
+	uint32_t raw_data_size;
+	/*
+	 * bsp_res contains:
+	 * bsp	crypto_block_size_pointer	8 bits at MLX5_BLOCK_SIZE_OFFSET
+	 * res	reserved 24 bits
+	 */
+	uint32_t bsp_res;
+	uint32_t reserved0;
+	uint8_t xts_initial_tweak[16];
+	/*
+	 * res_dp contains:
+	 * res	reserved 8 bits
+	 * dp	dek_pointer		24 bits at offset 0
+	 */
+	uint32_t res_dp;
+	uint32_t reserved1;
+	uint64_t keytag;
+	uint32_t reserved2[4];
+} __rte_packed;
+
 enum {
 	MLX5_GET_HCA_CAP_OP_MOD_GENERAL_DEVICE = 0x0 << 1,
 	MLX5_GET_HCA_CAP_OP_MOD_ETHERNET_OFFLOAD_CAPS = 0x1 << 1,
-- 
2.21.0

[dpdk-dev] [PATCH 11/24] common/mlx5: share hash list tool

[Shiri Kuzin]

In order to use the hash list defined in net in other drivers, the
hash list is moved to common utilities.

In addition, the log definition was moved from the common utilities to
a dedicated new log file in common in order to prevent a conflict.

Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/linux/mlx5_common_os.c    |   2 +-
 drivers/common/mlx5/linux/mlx5_nl.c           |   2 +-
 drivers/common/mlx5/meson.build               |   1 +
 drivers/common/mlx5/mlx5_common.c             |   2 +-
 drivers/common/mlx5/mlx5_common_devx.c        |   2 +-
 drivers/common/mlx5/mlx5_common_log.h         |  21 ++
 drivers/common/mlx5/mlx5_common_mp.c          |   2 +-
 drivers/common/mlx5/mlx5_common_mr.c          |   2 +-
 drivers/common/mlx5/mlx5_common_pci.c         |   3 +-
 drivers/common/mlx5/mlx5_common_utils.c       | 221 ++++++++++++++++++
 drivers/common/mlx5/mlx5_common_utils.h       | 202 +++++++++++++++-
 drivers/common/mlx5/mlx5_devx_cmds.c          |   2 +-
 drivers/common/mlx5/mlx5_malloc.c             |   2 +-
 .../common/mlx5/rte_common_mlx5_exports.def   |   7 +
 drivers/common/mlx5/version.map               |   6 +
 drivers/common/mlx5/windows/mlx5_common_os.c  |   2 +-
 drivers/common/mlx5/windows/mlx5_glue.c       |   2 +-
 drivers/net/mlx5/mlx5_utils.c                 | 209 -----------------
 drivers/net/mlx5/mlx5_utils.h                 | 194 +--------------
 19 files changed, 464 insertions(+), 420 deletions(-)
 create mode 100644 drivers/common/mlx5/mlx5_common_log.h
 create mode 100644 drivers/common/mlx5/mlx5_common_utils.c

diff --git a/drivers/common/mlx5/linux/mlx5_common_os.c b/drivers/common/mlx5/linux/mlx5_common_os.c
index 5cf9576921..fba8245b8b 100644
--- a/drivers/common/mlx5/linux/mlx5_common_os.c
+++ b/drivers/common/mlx5/linux/mlx5_common_os.c
@@ -15,7 +15,7 @@
 #include <rte_string_fns.h>
 
 #include "mlx5_common.h"
-#include "mlx5_common_utils.h"
+#include "mlx5_common_log.h"
 #include "mlx5_glue.h"
 
 #ifdef MLX5_GLUE
diff --git a/drivers/common/mlx5/linux/mlx5_nl.c b/drivers/common/mlx5/linux/mlx5_nl.c
index 752c57b33d..f0d04f9473 100644
--- a/drivers/common/mlx5/linux/mlx5_nl.c
+++ b/drivers/common/mlx5/linux/mlx5_nl.c
@@ -20,7 +20,7 @@
 #include <rte_errno.h>
 
 #include "mlx5_nl.h"
-#include "mlx5_common_utils.h"
+#include "../mlx5_common_log.h"
 #include "mlx5_malloc.h"
 #ifdef HAVE_DEVLINK
 #include <linux/devlink.h>
diff --git a/drivers/common/mlx5/meson.build b/drivers/common/mlx5/meson.build
index b2efea4432..1459b9bd89 100644
--- a/drivers/common/mlx5/meson.build
+++ b/drivers/common/mlx5/meson.build
@@ -16,6 +16,7 @@ sources += files(
 	'mlx5_malloc.c',
 	'mlx5_common_pci.c',
 	'mlx5_common_devx.c',
+	'mlx5_common_utils.c',
 )
 
 cflags_options = [
diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index c26a2cfa30..acb785f9ce 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -11,7 +11,7 @@
 
 #include "mlx5_common.h"
 #include "mlx5_common_os.h"
-#include "mlx5_common_utils.h"
+#include "mlx5_common_log.h"
 #include "mlx5_common_pci.h"
 
 int mlx5_common_logtype;
diff --git a/drivers/common/mlx5/mlx5_common_devx.c b/drivers/common/mlx5/mlx5_common_devx.c
index d19be122bd..22c8d356c4 100644
--- a/drivers/common/mlx5/mlx5_common_devx.c
+++ b/drivers/common/mlx5/mlx5_common_devx.c
@@ -12,7 +12,7 @@
 
 #include "mlx5_prm.h"
 #include "mlx5_devx_cmds.h"
-#include "mlx5_common_utils.h"
+#include "mlx5_common_log.h"
 #include "mlx5_malloc.h"
 #include "mlx5_common.h"
 #include "mlx5_common_devx.h"
diff --git a/drivers/common/mlx5/mlx5_common_log.h b/drivers/common/mlx5/mlx5_common_log.h
new file mode 100644
index 0000000000..26b13fedaf
--- /dev/null
+++ b/drivers/common/mlx5/mlx5_common_log.h
@@ -0,0 +1,21 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2019 Mellanox Technologies, Ltd
+ */
+
+#ifndef RTE_PMD_MLX5_COMMON_LOG_H_
+#define RTE_PMD_MLX5_COMMON_LOG_H_
+
+#include "mlx5_common.h"
+
+
+extern int mlx5_common_logtype;
+
+#define MLX5_COMMON_LOG_PREFIX "mlx5_common"
+/* Generic printf()-like logging macro with automatic line feed. */
+#define DRV_LOG(level, ...) \
+	PMD_DRV_LOG_(level, mlx5_common_logtype, MLX5_COMMON_LOG_PREFIX, \
+		__VA_ARGS__ PMD_DRV_LOG_STRIP PMD_DRV_LOG_OPAREN, \
+		PMD_DRV_LOG_CPAREN)
+
+#endif /* RTE_PMD_MLX5_COMMON_LOG_H_ */
+
diff --git a/drivers/common/mlx5/mlx5_common_mp.c b/drivers/common/mlx5/mlx5_common_mp.c
index 40e3956e45..673a7c31de 100644
--- a/drivers/common/mlx5/mlx5_common_mp.c
+++ b/drivers/common/mlx5/mlx5_common_mp.c
@@ -10,7 +10,7 @@
 #include <rte_errno.h>
 
 #include "mlx5_common_mp.h"
-#include "mlx5_common_utils.h"
+#include "mlx5_common_log.h"
 #include "mlx5_malloc.h"
 
 /**
diff --git a/drivers/common/mlx5/mlx5_common_mr.c b/drivers/common/mlx5/mlx5_common_mr.c
index e1ed0caf3a..afb5b3d0a7 100644
--- a/drivers/common/mlx5/mlx5_common_mr.c
+++ b/drivers/common/mlx5/mlx5_common_mr.c
@@ -11,7 +11,7 @@
 #include "mlx5_glue.h"
 #include "mlx5_common_mp.h"
 #include "mlx5_common_mr.h"
-#include "mlx5_common_utils.h"
+#include "mlx5_common_log.h"
 #include "mlx5_malloc.h"
 
 struct mr_find_contig_memsegs_data {
diff --git a/drivers/common/mlx5/mlx5_common_pci.c b/drivers/common/mlx5/mlx5_common_pci.c
index 21b1acdf87..177ccf62d9 100644
--- a/drivers/common/mlx5/mlx5_common_pci.c
+++ b/drivers/common/mlx5/mlx5_common_pci.c
@@ -5,7 +5,8 @@
 #include <stdlib.h>
 #include <rte_malloc.h>
 #include <rte_class.h>
-#include "mlx5_common_utils.h"
+
+#include "mlx5_common_log.h"
 #include "mlx5_common_pci.h"
 
 struct mlx5_pci_device {
diff --git a/drivers/common/mlx5/mlx5_common_utils.c b/drivers/common/mlx5/mlx5_common_utils.c
new file mode 100644
index 0000000000..ad2011e858
--- /dev/null
+++ b/drivers/common/mlx5/mlx5_common_utils.c
@@ -0,0 +1,221 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2019 Mellanox Technologies, Ltd
+ */
+
+#include <rte_malloc.h>
+#include <rte_hash_crc.h>
+#include <rte_errno.h>
+
+#include <mlx5_malloc.h>
+
+#include "mlx5_common_utils.h"
+#include "mlx5_common_log.h"
+
+/********************* Hash List **********************/
+
+static struct mlx5_hlist_entry *
+mlx5_hlist_default_create_cb(struct mlx5_hlist *h, uint64_t key __rte_unused,
+			     void *ctx __rte_unused)
+{
+	return mlx5_malloc(MLX5_MEM_ZERO, h->entry_sz, 0, SOCKET_ID_ANY);
+}
+
+static void
+mlx5_hlist_default_remove_cb(struct mlx5_hlist *h __rte_unused,
+			     struct mlx5_hlist_entry *entry)
+{
+	mlx5_free(entry);
+}
+
+struct mlx5_hlist *
+mlx5_hlist_create(const char *name, uint32_t size, uint32_t entry_size,
+		  uint32_t flags, mlx5_hlist_create_cb cb_create,
+		  mlx5_hlist_match_cb cb_match, mlx5_hlist_remove_cb cb_remove)
+{
+	struct mlx5_hlist *h;
+	uint32_t act_size;
+	uint32_t alloc_size;
+	uint32_t i;
+
+	if (!size || !cb_match || (!cb_create ^ !cb_remove))
+		return NULL;
+	/* Align to the next power of 2, 32bits integer is enough now. */
+	if (!rte_is_power_of_2(size)) {
+		act_size = rte_align32pow2(size);
+		DRV_LOG(DEBUG, "Size 0x%" PRIX32 " is not power of 2, "
+			"will be aligned to 0x%" PRIX32 ".", size, act_size);
+	} else {
+		act_size = size;
+	}
+	alloc_size = sizeof(struct mlx5_hlist) +
+		     sizeof(struct mlx5_hlist_bucket) * act_size;
+	/* Using zmalloc, then no need to initialize the heads. */
+	h = mlx5_malloc(MLX5_MEM_ZERO, alloc_size, RTE_CACHE_LINE_SIZE,
+			SOCKET_ID_ANY);
+	if (!h) {
+		DRV_LOG(ERR, "No memory for hash list %s creation",
+			name ? name : "None");
+		return NULL;
+	}
+	if (name)
+		snprintf(h->name, MLX5_HLIST_NAMESIZE, "%s", name);
+	h->table_sz = act_size;
+	h->mask = act_size - 1;
+	h->entry_sz = entry_size;
+	h->direct_key = !!(flags & MLX5_HLIST_DIRECT_KEY);
+	h->write_most = !!(flags & MLX5_HLIST_WRITE_MOST);
+	h->cb_create = cb_create ? cb_create : mlx5_hlist_default_create_cb;
+	h->cb_match = cb_match;
+	h->cb_remove = cb_remove ? cb_remove : mlx5_hlist_default_remove_cb;
+	for (i = 0; i < act_size; i++)
+		rte_rwlock_init(&h->buckets[i].lock);
+	DRV_LOG(DEBUG, "Hash list with %s size 0x%" PRIX32 " is created.",
+		h->name, act_size);
+	return h;
+}
+
+static struct mlx5_hlist_entry *
+__hlist_lookup(struct mlx5_hlist *h, uint64_t key, uint32_t idx,
+	       void *ctx, bool reuse)
+{
+	struct mlx5_hlist_head *first;
+	struct mlx5_hlist_entry *node;
+
+	MLX5_ASSERT(h);
+	first = &h->buckets[idx].head;
+	LIST_FOREACH(node, first, next) {
+		if (!h->cb_match(h, node, key, ctx)) {
+			if (reuse) {
+				__atomic_add_fetch(&node->ref_cnt, 1,
+						   __ATOMIC_RELAXED);
+				DRV_LOG(DEBUG, "Hash list %s entry %p "
+					"reuse: %u.",
+					h->name, (void *)node, node->ref_cnt);
+			}
+			break;
+		}
+	}
+	return node;
+}
+
+static struct mlx5_hlist_entry *
+hlist_lookup(struct mlx5_hlist *h, uint64_t key, uint32_t idx,
+	     void *ctx, bool reuse)
+{
+	struct mlx5_hlist_entry *node;
+
+	MLX5_ASSERT(h);
+	rte_rwlock_read_lock(&h->buckets[idx].lock);
+	node = __hlist_lookup(h, key, idx, ctx, reuse);
+	rte_rwlock_read_unlock(&h->buckets[idx].lock);
+	return node;
+}
+
+struct mlx5_hlist_entry *
+mlx5_hlist_lookup(struct mlx5_hlist *h, uint64_t key, void *ctx)
+{
+	uint32_t idx;
+
+	if (h->direct_key)
+		idx = (uint32_t)(key & h->mask);
+	else
+		idx = rte_hash_crc_8byte(key, 0) & h->mask;
+	return hlist_lookup(h, key, idx, ctx, false);
+}
+
+struct mlx5_hlist_entry*
+mlx5_hlist_register(struct mlx5_hlist *h, uint64_t key, void *ctx)
+{
+	uint32_t idx;
+	struct mlx5_hlist_head *first;
+	struct mlx5_hlist_bucket *b;
+	struct mlx5_hlist_entry *entry;
+	uint32_t prev_gen_cnt = 0;
+
+	if (h->direct_key)
+		idx = (uint32_t)(key & h->mask);
+	else
+		idx = rte_hash_crc_8byte(key, 0) & h->mask;
+	MLX5_ASSERT(h);
+	b = &h->buckets[idx];
+	/* Use write lock directly for write-most list. */
+	if (!h->write_most) {
+		prev_gen_cnt = __atomic_load_n(&b->gen_cnt, __ATOMIC_ACQUIRE);
+		entry = hlist_lookup(h, key, idx, ctx, true);
+		if (entry)
+			return entry;
+	}
+	rte_rwlock_write_lock(&b->lock);
+	/* Check if the list changed by other threads. */
+	if (h->write_most ||
+	    prev_gen_cnt != __atomic_load_n(&b->gen_cnt, __ATOMIC_ACQUIRE)) {
+		entry = __hlist_lookup(h, key, idx, ctx, true);
+		if (entry)
+			goto done;
+	}
+	first = &b->head;
+	entry = h->cb_create(h, key, ctx);
+	if (!entry) {
+		rte_errno = ENOMEM;
+		DRV_LOG(DEBUG, "Can't allocate hash list %s entry.", h->name);
+		goto done;
+	}
+	entry->idx = idx;
+	entry->ref_cnt = 1;
+	LIST_INSERT_HEAD(first, entry, next);
+	__atomic_add_fetch(&b->gen_cnt, 1, __ATOMIC_ACQ_REL);
+	DRV_LOG(DEBUG, "Hash list %s entry %p new: %u.",
+		h->name, (void *)entry, entry->ref_cnt);
+done:
+	rte_rwlock_write_unlock(&b->lock);
+	return entry;
+}
+
+int
+mlx5_hlist_unregister(struct mlx5_hlist *h, struct mlx5_hlist_entry *entry)
+{
+	uint32_t idx = entry->idx;
+
+	rte_rwlock_write_lock(&h->buckets[idx].lock);
+	MLX5_ASSERT(entry && entry->ref_cnt && entry->next.le_prev);
+	DRV_LOG(DEBUG, "Hash list %s entry %p deref: %u.",
+		h->name, (void *)entry, entry->ref_cnt);
+	if (--entry->ref_cnt) {
+		rte_rwlock_write_unlock(&h->buckets[idx].lock);
+		return 1;
+	}
+	LIST_REMOVE(entry, next);
+	/* Set to NULL to get rid of removing action for more than once. */
+	entry->next.le_prev = NULL;
+	h->cb_remove(h, entry);
+	rte_rwlock_write_unlock(&h->buckets[idx].lock);
+	DRV_LOG(DEBUG, "Hash list %s entry %p removed.",
+		h->name, (void *)entry);
+	return 0;
+}
+
+void
+mlx5_hlist_destroy(struct mlx5_hlist *h)
+{
+	uint32_t idx;
+	struct mlx5_hlist_entry *entry;
+
+	MLX5_ASSERT(h);
+	for (idx = 0; idx < h->table_sz; ++idx) {
+		/* No LIST_FOREACH_SAFE, using while instead. */
+		while (!LIST_EMPTY(&h->buckets[idx].head)) {
+			entry = LIST_FIRST(&h->buckets[idx].head);
+			LIST_REMOVE(entry, next);
+			/*
+			 * The owner of whole element which contains data entry
+			 * is the user, so it's the user's duty to do the clean
+			 * up and the free work because someone may not put the
+			 * hlist entry at the beginning(suggested to locate at
+			 * the beginning). Or else the default free function
+			 * will be used.
+			 */
+			h->cb_remove(h, entry);
+		}
+	}
+	mlx5_free(h);
+}
diff --git a/drivers/common/mlx5/mlx5_common_utils.h b/drivers/common/mlx5/mlx5_common_utils.h
index 6cba39c8cc..ed378ce9bd 100644
--- a/drivers/common/mlx5/mlx5_common_utils.h
+++ b/drivers/common/mlx5/mlx5_common_utils.h
@@ -7,14 +7,202 @@
 
 #include "mlx5_common.h"
 
+#define MLX5_HLIST_DIRECT_KEY 0x0001 /* Use the key directly as hash index. */
+#define MLX5_HLIST_WRITE_MOST 0x0002 /* List mostly used for append new. */
 
-extern int mlx5_common_logtype;
+/** Maximum size of string for naming the hlist table. */
+#define MLX5_HLIST_NAMESIZE			32
 
-#define MLX5_COMMON_LOG_PREFIX "mlx5_common"
-/* Generic printf()-like logging macro with automatic line feed. */
-#define DRV_LOG(level, ...) \
-	PMD_DRV_LOG_(level, mlx5_common_logtype, MLX5_COMMON_LOG_PREFIX, \
-		__VA_ARGS__ PMD_DRV_LOG_STRIP PMD_DRV_LOG_OPAREN, \
-		PMD_DRV_LOG_CPAREN)
+struct mlx5_hlist;
+
+/**
+ * Structure of the entry in the hash list, user should define its own struct
+ * that contains this in order to store the data. The 'key' is 64-bits right
+ * now and its user's responsibility to guarantee there is no collision.
+ */
+struct mlx5_hlist_entry {
+	LIST_ENTRY(mlx5_hlist_entry) next; /* entry pointers in the list. */
+	uint32_t idx; /* Bucket index the entry belongs to. */
+	uint32_t ref_cnt; /* Reference count. */
+};
+
+/** Structure for hash head. */
+LIST_HEAD(mlx5_hlist_head, mlx5_hlist_entry);
+
+/**
+ * Type of callback function for entry removal.
+ *
+ * @param list
+ *   The hash list.
+ * @param entry
+ *   The entry in the list.
+ */
+typedef void (*mlx5_hlist_remove_cb)(struct mlx5_hlist *list,
+				     struct mlx5_hlist_entry *entry);
+
+/**
+ * Type of function for user defined matching.
+ *
+ * @param list
+ *   The hash list.
+ * @param entry
+ *   The entry in the list.
+ * @param key
+ *   The new entry key.
+ * @param ctx
+ *   The pointer to new entry context.
+ *
+ * @return
+ *   0 if matching, non-zero number otherwise.
+ */
+typedef int (*mlx5_hlist_match_cb)(struct mlx5_hlist *list,
+				   struct mlx5_hlist_entry *entry,
+				   uint64_t key, void *ctx);
+
+/**
+ * Type of function for user defined hash list entry creation.
+ *
+ * @param list
+ *   The hash list.
+ * @param key
+ *   The key of the new entry.
+ * @param ctx
+ *   The pointer to new entry context.
+ *
+ * @return
+ *   Pointer to allocated entry on success, NULL otherwise.
+ */
+typedef struct mlx5_hlist_entry *(*mlx5_hlist_create_cb)
+				  (struct mlx5_hlist *list,
+				   uint64_t key, void *ctx);
+
+/* Hash list bucket head. */
+struct mlx5_hlist_bucket {
+	struct mlx5_hlist_head head; /* List head. */
+	rte_rwlock_t lock; /* Bucket lock. */
+	uint32_t gen_cnt; /* List modification will update generation count. */
+} __rte_cache_aligned;
+
+/**
+ * Hash list table structure
+ *
+ * Entry in hash list could be reused if entry already exists, reference
+ * count will increase and the existing entry returns.
+ *
+ * When destroy an entry from list, decrease reference count and only
+ * destroy when no further reference.
+ */
+struct mlx5_hlist {
+	char name[MLX5_HLIST_NAMESIZE]; /**< Name of the hash list. */
+	/**< number of heads, need to be power of 2. */
+	uint32_t table_sz;
+	uint32_t entry_sz; /**< Size of entry, used to allocate entry. */
+	/**< mask to get the index of the list heads. */
+	uint32_t mask;
+	bool direct_key; /* Use the new entry key directly as hash index. */
+	bool write_most; /* List mostly used for append new or destroy. */
+	void *ctx;
+	mlx5_hlist_create_cb cb_create; /**< entry create callback. */
+	mlx5_hlist_match_cb cb_match; /**< entry match callback. */
+	mlx5_hlist_remove_cb cb_remove; /**< entry remove callback. */
+	struct mlx5_hlist_bucket buckets[] __rte_cache_aligned;
+	/**< list bucket arrays. */
+};
+
+/**
+ * Create a hash list table, the user can specify the list heads array size
+ * of the table, now the size should be a power of 2 in order to get better
+ * distribution for the entries. Each entry is a part of the whole data element
+ * and the caller should be responsible for the data element's allocation and
+ * cleanup / free. Key of each entry will be calculated with CRC in order to
+ * generate a little fairer distribution.
+ *
+ * @param name
+ *   Name of the hash list(optional).
+ * @param size
+ *   Heads array size of the hash list.
+ * @param entry_size
+ *   Entry size to allocate if cb_create not specified.
+ * @param flags
+ *   The hash list attribute flags.
+ * @param cb_create
+ *   Callback function for entry create.
+ * @param cb_match
+ *   Callback function for entry match.
+ * @param cb_destroy
+ *   Callback function for entry destroy.
+ * @return
+ *   Pointer of the hash list table created, NULL on failure.
+ */
+__rte_internal
+struct mlx5_hlist *mlx5_hlist_create(const char *name, uint32_t size,
+				     uint32_t entry_size, uint32_t flags,
+				     mlx5_hlist_create_cb cb_create,
+				     mlx5_hlist_match_cb cb_match,
+				     mlx5_hlist_remove_cb cb_destroy);
+
+/**
+ * Search an entry matching the key.
+ *
+ * Result returned might be destroyed by other thread, must use
+ * this function only in main thread.
+ *
+ * @param h
+ *   Pointer to the hast list table.
+ * @param key
+ *   Key for the searching entry.
+ * @param ctx
+ *   Common context parameter used by entry callback function.
+ *
+ * @return
+ *   Pointer of the hlist entry if found, NULL otherwise.
+ */
+__rte_internal
+struct mlx5_hlist_entry *mlx5_hlist_lookup(struct mlx5_hlist *h, uint64_t key,
+					   void *ctx);
+
+/**
+ * Insert an entry to the hash list table, the entry is only part of whole data
+ * element and a 64B key is used for matching. User should construct the key or
+ * give a calculated hash signature and guarantee there is no collision.
+ *
+ * @param h
+ *   Pointer to the hast list table.
+ * @param entry
+ *   Entry to be inserted into the hash list table.
+ * @param ctx
+ *   Common context parameter used by callback function.
+ *
+ * @return
+ *   registered entry on success, NULL otherwise
+ */
+__rte_internal
+struct mlx5_hlist_entry *mlx5_hlist_register(struct mlx5_hlist *h, uint64_t key,
+					     void *ctx);
+
+/**
+ * Remove an entry from the hash list table. User should guarantee the validity
+ * of the entry.
+ *
+ * @param h
+ *   Pointer to the hast list table. (not used)
+ * @param entry
+ *   Entry to be removed from the hash list table.
+ * @return
+ *   0 on entry removed, 1 on entry still referenced.
+ */
+__rte_internal
+int mlx5_hlist_unregister(struct mlx5_hlist *h, struct mlx5_hlist_entry *entry);
+
+/**
+ * Destroy the hash list table, all the entries already inserted into the lists
+ * will be handled by the callback function provided by the user (including
+ * free if needed) before the table is freed.
+ *
+ * @param h
+ *   Pointer to the hast list table.
+ */
+__rte_internal
+void mlx5_hlist_destroy(struct mlx5_hlist *h);
 
 #endif /* RTE_PMD_MLX5_COMMON_UTILS_H_ */
diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c
index c3b3b50b51..1b1227eb16 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.c
+++ b/drivers/common/mlx5/mlx5_devx_cmds.c
@@ -9,7 +9,7 @@
 
 #include "mlx5_prm.h"
 #include "mlx5_devx_cmds.h"
-#include "mlx5_common_utils.h"
+#include "mlx5_common_log.h"
 #include "mlx5_malloc.h"
 
 
diff --git a/drivers/common/mlx5/mlx5_malloc.c b/drivers/common/mlx5/mlx5_malloc.c
index 9d30cedbaa..b19501e1bc 100644
--- a/drivers/common/mlx5/mlx5_malloc.c
+++ b/drivers/common/mlx5/mlx5_malloc.c
@@ -8,7 +8,7 @@
 #include <stdbool.h>
 #include <string.h>
 
-#include "mlx5_common_utils.h"
+#include "mlx5_common_log.h"
 #include "mlx5_common_os.h"
 #include "mlx5_malloc.h"
 
diff --git a/drivers/common/mlx5/rte_common_mlx5_exports.def b/drivers/common/mlx5/rte_common_mlx5_exports.def
index fd62b806ca..f44ae61d52 100644
--- a/drivers/common/mlx5/rte_common_mlx5_exports.def
+++ b/drivers/common/mlx5/rte_common_mlx5_exports.def
@@ -73,3 +73,10 @@ EXPORTS
 	mlx5_os_reg_mr
 	mlx5_os_umem_reg
 	mlx5_os_umem_dereg
+
+	mlx5_hlist_create
+	mlx5_hlist_lookup
+	mlx5_hlist_register
+	mlx5_hlist_unregister
+	mlx5_hlist_destroy
+
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index 120be7a99d..5c2cc4dca9 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -108,4 +108,10 @@ INTERNAL {
 	mlx5_free;
 
 	mlx5_pci_driver_register;
+
+	mlx5_hlist_create;
+	mlx5_hlist_lookup;
+	mlx5_hlist_register;
+	mlx5_hlist_unregister;
+	mlx5_hlist_destroy;
 };
diff --git a/drivers/common/mlx5/windows/mlx5_common_os.c b/drivers/common/mlx5/windows/mlx5_common_os.c
index f2d781a965..2e6e172a96 100644
--- a/drivers/common/mlx5/windows/mlx5_common_os.c
+++ b/drivers/common/mlx5/windows/mlx5_common_os.c
@@ -11,7 +11,7 @@
 #include <rte_errno.h>
 
 #include "mlx5_devx_cmds.h"
-#include "mlx5_common_utils.h"
+#include "../mlx5_common_log.h"
 #include "mlx5_common.h"
 #include "mlx5_common_os.h"
 #include "mlx5_malloc.h"
diff --git a/drivers/common/mlx5/windows/mlx5_glue.c b/drivers/common/mlx5/windows/mlx5_glue.c
index aef6d3b5f4..535487a8d4 100644
--- a/drivers/common/mlx5/windows/mlx5_glue.c
+++ b/drivers/common/mlx5/windows/mlx5_glue.c
@@ -12,7 +12,7 @@
 #include <rte_malloc.h>
 
 #include "mlx5_glue.h"
-#include "mlx5_common_utils.h"
+#include "../mlx5_common_log.h"
 #include "mlx5_win_ext.h"
 
 /*
diff --git a/drivers/net/mlx5/mlx5_utils.c b/drivers/net/mlx5/mlx5_utils.c
index a39b5edddc..18fe23e4fb 100644
--- a/drivers/net/mlx5/mlx5_utils.c
+++ b/drivers/net/mlx5/mlx5_utils.c
@@ -3,220 +3,11 @@
  */
 
 #include <rte_malloc.h>
-#include <rte_hash_crc.h>
 
 #include <mlx5_malloc.h>
 
 #include "mlx5_utils.h"
 
-/********************* Hash List **********************/
-
-static struct mlx5_hlist_entry *
-mlx5_hlist_default_create_cb(struct mlx5_hlist *h, uint64_t key __rte_unused,
-			     void *ctx __rte_unused)
-{
-	return mlx5_malloc(MLX5_MEM_ZERO, h->entry_sz, 0, SOCKET_ID_ANY);
-}
-
-static void
-mlx5_hlist_default_remove_cb(struct mlx5_hlist *h __rte_unused,
-			     struct mlx5_hlist_entry *entry)
-{
-	mlx5_free(entry);
-}
-
-struct mlx5_hlist *
-mlx5_hlist_create(const char *name, uint32_t size, uint32_t entry_size,
-		  uint32_t flags, mlx5_hlist_create_cb cb_create,
-		  mlx5_hlist_match_cb cb_match, mlx5_hlist_remove_cb cb_remove)
-{
-	struct mlx5_hlist *h;
-	uint32_t act_size;
-	uint32_t alloc_size;
-	uint32_t i;
-
-	if (!size || !cb_match || (!cb_create ^ !cb_remove))
-		return NULL;
-	/* Align to the next power of 2, 32bits integer is enough now. */
-	if (!rte_is_power_of_2(size)) {
-		act_size = rte_align32pow2(size);
-		DRV_LOG(DEBUG, "Size 0x%" PRIX32 " is not power of 2, "
-			"will be aligned to 0x%" PRIX32 ".", size, act_size);
-	} else {
-		act_size = size;
-	}
-	alloc_size = sizeof(struct mlx5_hlist) +
-		     sizeof(struct mlx5_hlist_bucket) * act_size;
-	/* Using zmalloc, then no need to initialize the heads. */
-	h = mlx5_malloc(MLX5_MEM_ZERO, alloc_size, RTE_CACHE_LINE_SIZE,
-			SOCKET_ID_ANY);
-	if (!h) {
-		DRV_LOG(ERR, "No memory for hash list %s creation",
-			name ? name : "None");
-		return NULL;
-	}
-	if (name)
-		snprintf(h->name, MLX5_HLIST_NAMESIZE, "%s", name);
-	h->table_sz = act_size;
-	h->mask = act_size - 1;
-	h->entry_sz = entry_size;
-	h->direct_key = !!(flags & MLX5_HLIST_DIRECT_KEY);
-	h->write_most = !!(flags & MLX5_HLIST_WRITE_MOST);
-	h->cb_create = cb_create ? cb_create : mlx5_hlist_default_create_cb;
-	h->cb_match = cb_match;
-	h->cb_remove = cb_remove ? cb_remove : mlx5_hlist_default_remove_cb;
-	for (i = 0; i < act_size; i++)
-		rte_rwlock_init(&h->buckets[i].lock);
-	DRV_LOG(DEBUG, "Hash list with %s size 0x%" PRIX32 " is created.",
-		h->name, act_size);
-	return h;
-}
-
-static struct mlx5_hlist_entry *
-__hlist_lookup(struct mlx5_hlist *h, uint64_t key, uint32_t idx,
-	       void *ctx, bool reuse)
-{
-	struct mlx5_hlist_head *first;
-	struct mlx5_hlist_entry *node;
-
-	MLX5_ASSERT(h);
-	first = &h->buckets[idx].head;
-	LIST_FOREACH(node, first, next) {
-		if (!h->cb_match(h, node, key, ctx)) {
-			if (reuse) {
-				__atomic_add_fetch(&node->ref_cnt, 1,
-						   __ATOMIC_RELAXED);
-				DRV_LOG(DEBUG, "Hash list %s entry %p "
-					"reuse: %u.",
-					h->name, (void *)node, node->ref_cnt);
-			}
-			break;
-		}
-	}
-	return node;
-}
-
-static struct mlx5_hlist_entry *
-hlist_lookup(struct mlx5_hlist *h, uint64_t key, uint32_t idx,
-	     void *ctx, bool reuse)
-{
-	struct mlx5_hlist_entry *node;
-
-	MLX5_ASSERT(h);
-	rte_rwlock_read_lock(&h->buckets[idx].lock);
-	node = __hlist_lookup(h, key, idx, ctx, reuse);
-	rte_rwlock_read_unlock(&h->buckets[idx].lock);
-	return node;
-}
-
-struct mlx5_hlist_entry *
-mlx5_hlist_lookup(struct mlx5_hlist *h, uint64_t key, void *ctx)
-{
-	uint32_t idx;
-
-	if (h->direct_key)
-		idx = (uint32_t)(key & h->mask);
-	else
-		idx = rte_hash_crc_8byte(key, 0) & h->mask;
-	return hlist_lookup(h, key, idx, ctx, false);
-}
-
-struct mlx5_hlist_entry*
-mlx5_hlist_register(struct mlx5_hlist *h, uint64_t key, void *ctx)
-{
-	uint32_t idx;
-	struct mlx5_hlist_head *first;
-	struct mlx5_hlist_bucket *b;
-	struct mlx5_hlist_entry *entry;
-	uint32_t prev_gen_cnt = 0;
-
-	if (h->direct_key)
-		idx = (uint32_t)(key & h->mask);
-	else
-		idx = rte_hash_crc_8byte(key, 0) & h->mask;
-	MLX5_ASSERT(h);
-	b = &h->buckets[idx];
-	/* Use write lock directly for write-most list. */
-	if (!h->write_most) {
-		prev_gen_cnt = __atomic_load_n(&b->gen_cnt, __ATOMIC_ACQUIRE);
-		entry = hlist_lookup(h, key, idx, ctx, true);
-		if (entry)
-			return entry;
-	}
-	rte_rwlock_write_lock(&b->lock);
-	/* Check if the list changed by other threads. */
-	if (h->write_most ||
-	    prev_gen_cnt != __atomic_load_n(&b->gen_cnt, __ATOMIC_ACQUIRE)) {
-		entry = __hlist_lookup(h, key, idx, ctx, true);
-		if (entry)
-			goto done;
-	}
-	first = &b->head;
-	entry = h->cb_create(h, key, ctx);
-	if (!entry) {
-		rte_errno = ENOMEM;
-		DRV_LOG(DEBUG, "Can't allocate hash list %s entry.", h->name);
-		goto done;
-	}
-	entry->idx = idx;
-	entry->ref_cnt = 1;
-	LIST_INSERT_HEAD(first, entry, next);
-	__atomic_add_fetch(&b->gen_cnt, 1, __ATOMIC_ACQ_REL);
-	DRV_LOG(DEBUG, "Hash list %s entry %p new: %u.",
-		h->name, (void *)entry, entry->ref_cnt);
-done:
-	rte_rwlock_write_unlock(&b->lock);
-	return entry;
-}
-
-int
-mlx5_hlist_unregister(struct mlx5_hlist *h, struct mlx5_hlist_entry *entry)
-{
-	uint32_t idx = entry->idx;
-
-	rte_rwlock_write_lock(&h->buckets[idx].lock);
-	MLX5_ASSERT(entry && entry->ref_cnt && entry->next.le_prev);
-	DRV_LOG(DEBUG, "Hash list %s entry %p deref: %u.",
-		h->name, (void *)entry, entry->ref_cnt);
-	if (--entry->ref_cnt) {
-		rte_rwlock_write_unlock(&h->buckets[idx].lock);
-		return 1;
-	}
-	LIST_REMOVE(entry, next);
-	/* Set to NULL to get rid of removing action for more than once. */
-	entry->next.le_prev = NULL;
-	h->cb_remove(h, entry);
-	rte_rwlock_write_unlock(&h->buckets[idx].lock);
-	DRV_LOG(DEBUG, "Hash list %s entry %p removed.",
-		h->name, (void *)entry);
-	return 0;
-}
-
-void
-mlx5_hlist_destroy(struct mlx5_hlist *h)
-{
-	uint32_t idx;
-	struct mlx5_hlist_entry *entry;
-
-	MLX5_ASSERT(h);
-	for (idx = 0; idx < h->table_sz; ++idx) {
-		/* No LIST_FOREACH_SAFE, using while instead. */
-		while (!LIST_EMPTY(&h->buckets[idx].head)) {
-			entry = LIST_FIRST(&h->buckets[idx].head);
-			LIST_REMOVE(entry, next);
-			/*
-			 * The owner of whole element which contains data entry
-			 * is the user, so it's the user's duty to do the clean
-			 * up and the free work because someone may not put the
-			 * hlist entry at the beginning(suggested to locate at
-			 * the beginning). Or else the default free function
-			 * will be used.
-			 */
-			h->cb_remove(h, entry);
-		}
-	}
-	mlx5_free(h);
-}
 
 /********************* Cache list ************************/
 
diff --git a/drivers/net/mlx5/mlx5_utils.h b/drivers/net/mlx5/mlx5_utils.h
index 5088c95e86..008a017402 100644
--- a/drivers/net/mlx5/mlx5_utils.h
+++ b/drivers/net/mlx5/mlx5_utils.h
@@ -18,6 +18,7 @@
 #include <rte_bitmap.h>
 
 #include <mlx5_common.h>
+#include <mlx5_common_utils.h>
 
 #include "mlx5_defs.h"
 
@@ -261,199 +262,6 @@ log2above(unsigned int v)
 	return l + r;
 }
 
-#define MLX5_HLIST_DIRECT_KEY 0x0001 /* Use the key directly as hash index. */
-#define MLX5_HLIST_WRITE_MOST 0x0002 /* List mostly used for append new. */
-
-/** Maximum size of string for naming the hlist table. */
-#define MLX5_HLIST_NAMESIZE			32
-
-struct mlx5_hlist;
-
-/**
- * Structure of the entry in the hash list, user should define its own struct
- * that contains this in order to store the data. The 'key' is 64-bits right
- * now and its user's responsibility to guarantee there is no collision.
- */
-struct mlx5_hlist_entry {
-	LIST_ENTRY(mlx5_hlist_entry) next; /* entry pointers in the list. */
-	uint32_t idx; /* Bucket index the entry belongs to. */
-	uint32_t ref_cnt; /* Reference count. */
-};
-
-/** Structure for hash head. */
-LIST_HEAD(mlx5_hlist_head, mlx5_hlist_entry);
-
-/**
- * Type of callback function for entry removal.
- *
- * @param list
- *   The hash list.
- * @param entry
- *   The entry in the list.
- */
-typedef void (*mlx5_hlist_remove_cb)(struct mlx5_hlist *list,
-				     struct mlx5_hlist_entry *entry);
-
-/**
- * Type of function for user defined matching.
- *
- * @param list
- *   The hash list.
- * @param entry
- *   The entry in the list.
- * @param key
- *   The new entry key.
- * @param ctx
- *   The pointer to new entry context.
- *
- * @return
- *   0 if matching, non-zero number otherwise.
- */
-typedef int (*mlx5_hlist_match_cb)(struct mlx5_hlist *list,
-				   struct mlx5_hlist_entry *entry,
-				   uint64_t key, void *ctx);
-
-/**
- * Type of function for user defined hash list entry creation.
- *
- * @param list
- *   The hash list.
- * @param key
- *   The key of the new entry.
- * @param ctx
- *   The pointer to new entry context.
- *
- * @return
- *   Pointer to allocated entry on success, NULL otherwise.
- */
-typedef struct mlx5_hlist_entry *(*mlx5_hlist_create_cb)
-				  (struct mlx5_hlist *list,
-				   uint64_t key, void *ctx);
-
-/* Hash list bucket head. */
-struct mlx5_hlist_bucket {
-	struct mlx5_hlist_head head; /* List head. */
-	rte_rwlock_t lock; /* Bucket lock. */
-	uint32_t gen_cnt; /* List modification will update generation count. */
-} __rte_cache_aligned;
-
-/**
- * Hash list table structure
- *
- * Entry in hash list could be reused if entry already exists, reference
- * count will increase and the existing entry returns.
- *
- * When destroy an entry from list, decrease reference count and only
- * destroy when no further reference.
- */
-struct mlx5_hlist {
-	char name[MLX5_HLIST_NAMESIZE]; /**< Name of the hash list. */
-	/**< number of heads, need to be power of 2. */
-	uint32_t table_sz;
-	uint32_t entry_sz; /**< Size of entry, used to allocate entry. */
-	/**< mask to get the index of the list heads. */
-	uint32_t mask;
-	bool direct_key; /* Use the new entry key directly as hash index. */
-	bool write_most; /* List mostly used for append new or destroy. */
-	void *ctx;
-	mlx5_hlist_create_cb cb_create; /**< entry create callback. */
-	mlx5_hlist_match_cb cb_match; /**< entry match callback. */
-	mlx5_hlist_remove_cb cb_remove; /**< entry remove callback. */
-	struct mlx5_hlist_bucket buckets[] __rte_cache_aligned;
-	/**< list bucket arrays. */
-};
-
-/**
- * Create a hash list table, the user can specify the list heads array size
- * of the table, now the size should be a power of 2 in order to get better
- * distribution for the entries. Each entry is a part of the whole data element
- * and the caller should be responsible for the data element's allocation and
- * cleanup / free. Key of each entry will be calculated with CRC in order to
- * generate a little fairer distribution.
- *
- * @param name
- *   Name of the hash list(optional).
- * @param size
- *   Heads array size of the hash list.
- * @param entry_size
- *   Entry size to allocate if cb_create not specified.
- * @param flags
- *   The hash list attribute flags.
- * @param cb_create
- *   Callback function for entry create.
- * @param cb_match
- *   Callback function for entry match.
- * @param cb_destroy
- *   Callback function for entry destroy.
- * @return
- *   Pointer of the hash list table created, NULL on failure.
- */
-struct mlx5_hlist *mlx5_hlist_create(const char *name, uint32_t size,
-				     uint32_t entry_size, uint32_t flags,
-				     mlx5_hlist_create_cb cb_create,
-				     mlx5_hlist_match_cb cb_match,
-				     mlx5_hlist_remove_cb cb_destroy);
-
-/**
- * Search an entry matching the key.
- *
- * Result returned might be destroyed by other thread, must use
- * this function only in main thread.
- *
- * @param h
- *   Pointer to the hast list table.
- * @param key
- *   Key for the searching entry.
- * @param ctx
- *   Common context parameter used by entry callback function.
- *
- * @return
- *   Pointer of the hlist entry if found, NULL otherwise.
- */
-struct mlx5_hlist_entry *mlx5_hlist_lookup(struct mlx5_hlist *h, uint64_t key,
-					   void *ctx);
-
-/**
- * Insert an entry to the hash list table, the entry is only part of whole data
- * element and a 64B key is used for matching. User should construct the key or
- * give a calculated hash signature and guarantee there is no collision.
- *
- * @param h
- *   Pointer to the hast list table.
- * @param entry
- *   Entry to be inserted into the hash list table.
- * @param ctx
- *   Common context parameter used by callback function.
- *
- * @return
- *   registered entry on success, NULL otherwise
- */
-struct mlx5_hlist_entry *mlx5_hlist_register(struct mlx5_hlist *h, uint64_t key,
-					     void *ctx);
-
-/**
- * Remove an entry from the hash list table. User should guarantee the validity
- * of the entry.
- *
- * @param h
- *   Pointer to the hast list table. (not used)
- * @param entry
- *   Entry to be removed from the hash list table.
- * @return
- *   0 on entry removed, 1 on entry still referenced.
- */
-int mlx5_hlist_unregister(struct mlx5_hlist *h, struct mlx5_hlist_entry *entry);
-
-/**
- * Destroy the hash list table, all the entries already inserted into the lists
- * will be handled by the callback function provided by the user (including
- * free if needed) before the table is freed.
- *
- * @param h
- *   Pointer to the hast list table.
- */
-void mlx5_hlist_destroy(struct mlx5_hlist *h);
-
 /************************ cache list *****************************/
 
 /** Maximum size of string for naming. */
-- 
2.21.0

[dpdk-dev] [PATCH 12/24] common/mlx5: share get ib device match function

[Shiri Kuzin]

The get_ib_device_match function iterates over the list of ib devices
returned by the get_device_list glue function and returns the ib device
matching the provided address.

Since this function is in use by several drivers, in this patch we
share the function in common part.

Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/linux/mlx5_common_os.c | 28 +++++++++++++++++++
 drivers/common/mlx5/linux/mlx5_common_os.h |  5 ++++
 drivers/common/mlx5/mlx5_common.h          |  1 +
 drivers/common/mlx5/version.map            |  2 ++
 drivers/compress/mlx5/mlx5_compress.c      | 30 +-------------------
 drivers/regex/mlx5/mlx5_regex.c            | 30 ++------------------
 drivers/vdpa/mlx5/mlx5_vdpa.c              | 32 ++--------------------
 7 files changed, 41 insertions(+), 87 deletions(-)

diff --git a/drivers/common/mlx5/linux/mlx5_common_os.c b/drivers/common/mlx5/linux/mlx5_common_os.c
index fba8245b8b..037147fe31 100644
--- a/drivers/common/mlx5/linux/mlx5_common_os.c
+++ b/drivers/common/mlx5/linux/mlx5_common_os.c
@@ -16,6 +16,7 @@
 
 #include "mlx5_common.h"
 #include "mlx5_common_log.h"
+#include "mlx5_common_os.h"
 #include "mlx5_glue.h"
 
 #ifdef MLX5_GLUE
@@ -423,3 +424,30 @@ mlx5_glue_constructor(void)
 	mlx5_glue = NULL;
 }
 
+struct ibv_device *
+mlx5_os_get_ib_device_match(struct rte_pci_addr *addr)
+{
+	int n;
+	struct ibv_device **ibv_list = mlx5_glue->get_device_list(&n);
+	struct ibv_device *ibv_match = NULL;
+
+	if (ibv_list == NULL) {
+		rte_errno = ENOSYS;
+		return NULL;
+	}
+	while (n-- > 0) {
+		struct rte_pci_addr paddr;
+
+		DRV_LOG(DEBUG, "Checking device \"%s\"..", ibv_list[n]->name);
+		if (mlx5_dev_to_pci_addr(ibv_list[n]->ibdev_path, &paddr) != 0)
+			continue;
+		if (rte_pci_addr_cmp(addr, &paddr) != 0)
+			continue;
+		ibv_match = ibv_list[n];
+		break;
+	}
+	if (ibv_match == NULL)
+		rte_errno = ENOENT;
+	mlx5_glue->free_device_list(ibv_list);
+	return ibv_match;
+}
diff --git a/drivers/common/mlx5/linux/mlx5_common_os.h b/drivers/common/mlx5/linux/mlx5_common_os.h
index 63f070d9c4..0eb304e0e3 100644
--- a/drivers/common/mlx5/linux/mlx5_common_os.h
+++ b/drivers/common/mlx5/linux/mlx5_common_os.h
@@ -284,4 +284,9 @@ mlx5_os_free(void *addr)
 {
 	free(addr);
 }
+
+__rte_internal
+struct ibv_device *
+mlx5_os_get_ib_device_match(struct rte_pci_addr *addr);
+
 #endif /* RTE_PMD_MLX5_COMMON_OS_H_ */
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index 327374fdcf..f1ff16eb3e 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -17,6 +17,7 @@
 
 #include "mlx5_prm.h"
 #include "mlx5_devx_cmds.h"
+#include "mlx5_common_os.h"
 
 /* Reported driver name. */
 #define MLX5_PCI_DRIVER_NAME "mlx5_pci"
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index 5c2cc4dca9..045c0fc6fe 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -99,6 +99,8 @@ INTERNAL {
 	mlx5_nl_vlan_vmwa_create;
 	mlx5_nl_vlan_vmwa_delete;
 
+	mlx5_os_get_ib_device_match;
+
 	mlx5_translate_port_name;
 
 	mlx5_malloc_mem_select;
diff --git a/drivers/compress/mlx5/mlx5_compress.c b/drivers/compress/mlx5/mlx5_compress.c
index ec3c237512..ff988d26ef 100644
--- a/drivers/compress/mlx5/mlx5_compress.c
+++ b/drivers/compress/mlx5/mlx5_compress.c
@@ -647,34 +647,6 @@ mlx5_compress_dequeue_burst(void *queue_pair, struct rte_comp_op **ops,
 	return i;
 }
 
-static struct ibv_device *
-mlx5_compress_get_ib_device_match(struct rte_pci_addr *addr)
-{
-	int n;
-	struct ibv_device **ibv_list = mlx5_glue->get_device_list(&n);
-	struct ibv_device *ibv_match = NULL;
-
-	if (ibv_list == NULL) {
-		rte_errno = ENOSYS;
-		return NULL;
-	}
-	while (n-- > 0) {
-		struct rte_pci_addr paddr;
-
-		DRV_LOG(DEBUG, "Checking device \"%s\"..", ibv_list[n]->name);
-		if (mlx5_dev_to_pci_addr(ibv_list[n]->ibdev_path, &paddr) != 0)
-			continue;
-		if (rte_pci_addr_cmp(addr, &paddr) != 0)
-			continue;
-		ibv_match = ibv_list[n];
-		break;
-	}
-	if (ibv_match == NULL)
-		rte_errno = ENOENT;
-	mlx5_glue->free_device_list(ibv_list);
-	return ibv_match;
-}
-
 static void
 mlx5_compress_hw_global_release(struct mlx5_compress_priv *priv)
 {
@@ -774,7 +746,7 @@ mlx5_compress_pci_probe(struct rte_pci_driver *pci_drv,
 		rte_errno = ENOTSUP;
 		return -rte_errno;
 	}
-	ibv = mlx5_compress_get_ib_device_match(&pci_dev->addr);
+	ibv = mlx5_os_get_ib_device_match(&pci_dev->addr);
 	if (ibv == NULL) {
 		DRV_LOG(ERR, "No matching IB device for PCI slot "
 			PCI_PRI_FMT ".", pci_dev->addr.domain,
diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
index ac5b205fa9..88221186e8 100644
--- a/drivers/regex/mlx5/mlx5_regex.c
+++ b/drivers/regex/mlx5/mlx5_regex.c
@@ -11,6 +11,7 @@
 #include <rte_regexdev_driver.h>
 
 #include <mlx5_common_pci.h>
+#include <mlx5_common.h>
 #include <mlx5_glue.h>
 #include <mlx5_devx_cmds.h>
 #include <mlx5_prm.h>
@@ -52,33 +53,6 @@ mlx5_regex_close(struct rte_regexdev *dev __rte_unused)
 	return 0;
 }
 
-static struct ibv_device *
-mlx5_regex_get_ib_device_match(struct rte_pci_addr *addr)
-{
-	int n;
-	struct ibv_device **ibv_list = mlx5_glue->get_device_list(&n);
-	struct ibv_device *ibv_match = NULL;
-
-	if (!ibv_list) {
-		rte_errno = ENOSYS;
-		return NULL;
-	}
-	while (n-- > 0) {
-		struct rte_pci_addr pci_addr;
-
-		DRV_LOG(DEBUG, "Checking device \"%s\"..", ibv_list[n]->name);
-		if (mlx5_dev_to_pci_addr(ibv_list[n]->ibdev_path, &pci_addr))
-			continue;
-		if (rte_pci_addr_cmp(addr, &pci_addr))
-			continue;
-		ibv_match = ibv_list[n];
-		break;
-	}
-	if (!ibv_match)
-		rte_errno = ENOENT;
-	mlx5_glue->free_device_list(ibv_list);
-	return ibv_match;
-}
 static int
 mlx5_regex_engines_status(struct ibv_context *ctx, int num_engines)
 {
@@ -121,7 +95,7 @@ mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	int ret;
 	uint32_t val;
 
-	ibv = mlx5_regex_get_ib_device_match(&pci_dev->addr);
+	ibv = mlx5_os_get_ib_device_match(&pci_dev->addr);
 	if (!ibv) {
 		DRV_LOG(ERR, "No matching IB device for PCI slot "
 			PCI_PRI_FMT ".", pci_dev->addr.domain,
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 619e31d61c..9e5857205d 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -472,34 +472,6 @@ static struct rte_vdpa_dev_ops mlx5_vdpa_ops = {
 	.reset_stats = mlx5_vdpa_reset_stats,
 };
 
-static struct ibv_device *
-mlx5_vdpa_get_ib_device_match(struct rte_pci_addr *addr)
-{
-	int n;
-	struct ibv_device **ibv_list = mlx5_glue->get_device_list(&n);
-	struct ibv_device *ibv_match = NULL;
-
-	if (!ibv_list) {
-		rte_errno = ENOSYS;
-		return NULL;
-	}
-	while (n-- > 0) {
-		struct rte_pci_addr pci_addr;
-
-		DRV_LOG(DEBUG, "Checking device \"%s\"..", ibv_list[n]->name);
-		if (mlx5_dev_to_pci_addr(ibv_list[n]->ibdev_path, &pci_addr))
-			continue;
-		if (rte_pci_addr_cmp(addr, &pci_addr))
-			continue;
-		ibv_match = ibv_list[n];
-		break;
-	}
-	if (!ibv_match)
-		rte_errno = ENOENT;
-	mlx5_glue->free_device_list(ibv_list);
-	return ibv_match;
-}
-
 /* Try to disable ROCE by Netlink\Devlink. */
 static int
 mlx5_vdpa_nl_roce_disable(const char *addr)
@@ -595,7 +567,7 @@ mlx5_vdpa_roce_disable(struct rte_pci_addr *addr, struct ibv_device **ibv)
 		struct ibv_device *ibv_new;
 
 		for (r = MLX5_VDPA_MAX_RETRIES; r; r--) {
-			ibv_new = mlx5_vdpa_get_ib_device_match(addr);
+			ibv_new = mlx5_os_get_ib_device_match(addr);
 			if (ibv_new) {
 				*ibv = ibv_new;
 				return 0;
@@ -698,7 +670,7 @@ mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	struct mlx5_hca_attr attr;
 	int ret;
 
-	ibv = mlx5_vdpa_get_ib_device_match(&pci_dev->addr);
+	ibv = mlx5_os_get_ib_device_match(&pci_dev->addr);
 	if (!ibv) {
 		DRV_LOG(ERR, "No matching IB device for PCI slot "
 			PCI_PRI_FMT ".", pci_dev->addr.domain,
-- 
2.21.0

[dpdk-dev] [PATCH 13/24] common/mlx5: support general obj CREDENTIAL create

[Shiri Kuzin]

From: Dekel Peled <dekelp@nvidia.com>

CREDENTIAL object is used for any crypto operation in wrapped mode.

This patch add support of CREDENTIAL object create operation.
Add reading of CREDENTIAL support capability.
Add function to create general object type CREDENTIAL, using DevX API.

Signed-off-by: Dekel Peled <dekelp@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/mlx5_devx_cmds.c | 53 +++++++++++++++++++++++++++-
 drivers/common/mlx5/mlx5_devx_cmds.h | 17 +++++++--
 drivers/common/mlx5/mlx5_prm.h       | 23 ++++++++++++
 drivers/common/mlx5/version.map      |  1 +
 4 files changed, 91 insertions(+), 3 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c
index 1b1227eb16..c43e320f87 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.c
+++ b/drivers/common/mlx5/mlx5_devx_cmds.c
@@ -734,6 +734,8 @@ mlx5_devx_cmd_query_hca_attr(void *ctx,
 		       MLX5_GENERAL_OBJ_TYPES_CAP_DEK);
 	attr->import_kek = !!(general_obj_types_supported &
 			      MLX5_GENERAL_OBJ_TYPES_CAP_IMPORT_KEK);
+	attr->credential = !!(general_obj_types_supported &
+			      MLX5_GENERAL_OBJ_TYPES_CAP_CREDENTIAL);
 	attr->crypto_login = !!(general_obj_types_supported &
 				MLX5_GENERAL_OBJ_TYPES_CAP_CRYPTO_LOGIN);
 	/* Add reading of other GENERAL_OBJ_TYPES_CAP bits above this line. */
@@ -2407,6 +2409,55 @@ mlx5_devx_cmd_create_import_kek_obj(void *ctx,
 	return import_kek_obj;
 }
 
+/**
+ * Create general object of type CREDENTIAL using DevX API.
+ *
+ * @param[in] ctx
+ *   Context returned from mlx5 open_device() glue function.
+ * @param [in] attr
+ *   Pointer to CREDENTIAL attributes structure.
+ *
+ * @return
+ *   The DevX object created, NULL otherwise and rte_errno is set.
+ */
+struct mlx5_devx_obj *
+mlx5_devx_cmd_create_credential_obj(void *ctx,
+				    struct mlx5_devx_credential_attr *attr)
+{
+	uint32_t in[MLX5_ST_SZ_DW(create_credential_in)] = {0};
+	uint32_t out[MLX5_ST_SZ_DW(general_obj_out_cmd_hdr)] = {0};
+	struct mlx5_devx_obj *credential_obj = NULL;
+	void *ptr = NULL, *credential_addr = NULL;
+
+	credential_obj = mlx5_malloc(MLX5_MEM_ZERO, sizeof(*credential_obj),
+				     0, SOCKET_ID_ANY);
+	if (credential_obj == NULL) {
+		DRV_LOG(ERR, "Failed to allocate CREDENTIAL object data");
+		rte_errno = ENOMEM;
+		return NULL;
+	}
+	ptr = MLX5_ADDR_OF(create_credential_in, in, hdr);
+	MLX5_SET(general_obj_in_cmd_hdr, ptr, opcode,
+		 MLX5_CMD_OP_CREATE_GENERAL_OBJECT);
+	MLX5_SET(general_obj_in_cmd_hdr, ptr, obj_type,
+		 MLX5_GENERAL_OBJ_TYPE_CREDENTIAL);
+	ptr = MLX5_ADDR_OF(create_credential_in, in, credential);
+	MLX5_SET(credential, ptr, credential_role, attr->credential_role);
+	credential_addr = MLX5_ADDR_OF(credential, ptr, credential);
+	memcpy(credential_addr, (void *)(attr->credential),
+	       MLX5_CRYPTO_CREDENTIAL_SIZE);
+	credential_obj->obj = mlx5_glue->devx_obj_create(ctx, in, sizeof(in),
+							 out, sizeof(out));
+	if (credential_obj->obj == NULL) {
+		rte_errno = errno;
+		DRV_LOG(ERR, "Failed to create CREDENTIAL object using DevX.");
+		mlx5_free(credential_obj);
+		return NULL;
+	}
+	credential_obj->id = MLX5_GET(general_obj_out_cmd_hdr, out, obj_id);
+	return credential_obj;
+}
+
 /**
  * Create general object of type CRYPTO_LOGIN using DevX API.
  *
@@ -2446,7 +2497,7 @@ mlx5_devx_cmd_create_crypto_login_obj(void *ctx,
 		 attr->session_import_kek_ptr);
 	credential_addr = MLX5_ADDR_OF(crypto_login, ptr, credential);
 	memcpy(credential_addr, (void *)(attr->credential),
-	       MLX5_CRYPTO_LOGIN_CREDENTIAL_SIZE);
+	       MLX5_CRYPTO_CREDENTIAL_SIZE);
 	crypto_login_obj->obj = mlx5_glue->devx_obj_create(ctx, in, sizeof(in),
 							   out, sizeof(out));
 	if (crypto_login_obj->obj == NULL) {
diff --git a/drivers/common/mlx5/mlx5_devx_cmds.h b/drivers/common/mlx5/mlx5_devx_cmds.h
index b723aee126..1c952495bf 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.h
+++ b/drivers/common/mlx5/mlx5_devx_cmds.h
@@ -133,6 +133,7 @@ struct mlx5_hca_attr {
 	uint32_t aes_xts:1; /* AES-XTS crypto is supported. */
 	uint32_t dek:1; /* General obj type DEK is supported. */
 	uint32_t import_kek:1; /* General obj type IMPORT_KEK supported. */
+	uint32_t credential:1; /* General obj type CREDENTIAL supported. */
 	uint32_t crypto_login:1; /* General obj type CRYPTO_LOGIN supported. */
 	uint32_t regexp_num_of_engines;
 	uint32_t log_max_ft_sampler_num:8;
@@ -447,13 +448,20 @@ struct mlx5_devx_import_kek_attr {
 	uint8_t key[MLX5_CRYPTO_KEY_MAX_SIZE];
 };
 
-#define MLX5_CRYPTO_LOGIN_CREDENTIAL_SIZE	48
+#define MLX5_CRYPTO_CREDENTIAL_SIZE	48
+
+struct mlx5_devx_credential_attr {
+	uint64_t modify_field_select;
+	uint32_t state:8;
+	uint32_t credential_role:8;
+	uint8_t credential[MLX5_CRYPTO_CREDENTIAL_SIZE];
+};
 
 struct mlx5_devx_crypto_login_attr {
 	uint64_t modify_field_select;
 	uint32_t credential_pointer:24;
 	uint32_t session_import_kek_ptr:24;
-	uint8_t credential[MLX5_CRYPTO_LOGIN_CREDENTIAL_SIZE];
+	uint8_t credential[MLX5_CRYPTO_CREDENTIAL_SIZE];
 };
 
 /* mlx5_devx_cmds.c */
@@ -599,6 +607,11 @@ struct mlx5_devx_obj *
 mlx5_devx_cmd_create_import_kek_obj(void *ctx,
 				    struct mlx5_devx_import_kek_attr *attr);
 
+__rte_internal
+struct mlx5_devx_obj *
+mlx5_devx_cmd_create_credential_obj(void *ctx,
+				    struct mlx5_devx_credential_attr *attr);
+
 __rte_internal
 struct mlx5_devx_obj *
 mlx5_devx_cmd_create_crypto_login_obj(void *ctx,
diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h
index 736badb6df..69d6de8cbd 100644
--- a/drivers/common/mlx5/mlx5_prm.h
+++ b/drivers/common/mlx5/mlx5_prm.h
@@ -1185,6 +1185,8 @@ enum {
 			(1ULL << MLX5_GENERAL_OBJ_TYPE_DEK)
 #define MLX5_GENERAL_OBJ_TYPES_CAP_IMPORT_KEK \
 			(1ULL << MLX5_GENERAL_OBJ_TYPE_IMPORT_KEK)
+#define MLX5_GENERAL_OBJ_TYPES_CAP_CREDENTIAL \
+			(1ULL << MLX5_GENERAL_OBJ_TYPE_CREDENTIAL)
 #define MLX5_GENERAL_OBJ_TYPES_CAP_CRYPTO_LOGIN \
 			(1ULL << MLX5_GENERAL_OBJ_TYPE_CRYPTO_LOGIN)
 
@@ -2471,6 +2473,7 @@ enum {
 	MLX5_GENERAL_OBJ_TYPE_VIRTQ = 0x000d,
 	MLX5_GENERAL_OBJ_TYPE_VIRTIO_Q_COUNTERS = 0x001c,
 	MLX5_GENERAL_OBJ_TYPE_IMPORT_KEK = 0x001d,
+	MLX5_GENERAL_OBJ_TYPE_CREDENTIAL = 0x001e,
 	MLX5_GENERAL_OBJ_TYPE_CRYPTO_LOGIN = 0x001f,
 	MLX5_GENERAL_OBJ_TYPE_FLEX_PARSE_GRAPH = 0x0022,
 	MLX5_GENERAL_OBJ_TYPE_FLOW_HIT_ASO = 0x0025,
@@ -2581,6 +2584,26 @@ struct mlx5_ifc_create_import_kek_in_bits {
 	struct mlx5_ifc_import_kek_bits import_kek;
 };
 
+enum {
+	MLX5_CREDENTIAL_ROLE_OFFICER = 0x0,
+	MLX5_CREDENTIAL_ROLE_USER = 0x1,
+};
+
+struct mlx5_ifc_credential_bits {
+	u8 modify_field_select[0x40];
+	u8 state[0x8];
+	u8 reserved_at_48[0x10];
+	u8 credential_role[0x8];
+	u8 reserved_at_60[0x1a0];
+	u8 credential[0x180];
+	u8 reserved_at_380[0x480];
+};
+
+struct mlx5_ifc_create_credential_in_bits {
+	struct mlx5_ifc_general_obj_in_cmd_hdr_bits hdr;
+	struct mlx5_ifc_credential_bits credential;
+};
+
 struct mlx5_ifc_crypto_login_bits {
 	u8 modify_field_select[0x40];
 	u8 reserved_at_40[0x48];
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index 045c0fc6fe..e1f0b324a0 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -26,6 +26,7 @@ INTERNAL {
 	mlx5_devx_cmd_create_geneve_tlv_option;
         mlx5_devx_cmd_create_dek_obj;
         mlx5_devx_cmd_create_import_kek_obj;
+        mlx5_devx_cmd_create_credential_obj;
         mlx5_devx_cmd_create_crypto_login_obj;
 	mlx5_devx_cmd_destroy;
 	mlx5_devx_cmd_flow_counter_alloc;
-- 
2.21.0

[dpdk-dev] [PATCH 14/24] common/mlx5: add crypto register structs and defs

[Shiri Kuzin]

From: Dekel Peled <dekelp@nvidia.com>

Encryption key management requires use of several related registers.
This patch adds the relevant structs and values, according to PRM
definitions.

Signed-off-by: Dekel Peled <dekelp@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/mlx5_prm.h | 41 ++++++++++++++++++++++++++++++++++
 1 file changed, 41 insertions(+)

diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h
index 69d6de8cbd..4518304889 100644
--- a/drivers/common/mlx5/mlx5_prm.h
+++ b/drivers/common/mlx5/mlx5_prm.h
@@ -3232,6 +3232,10 @@ enum {
 
 enum {
 	MLX5_REGISTER_ID_MTUTC  = 0x9055,
+	MLX5_CRYPTO_OPERATIONAL_REGISTER_ID = 0xC002,
+	MLX5_CRYPTO_COMMISSIONING_REGISTER_ID = 0xC003,
+	MLX5_IMPORT_KEK_HANDLE_REGISTER_ID = 0xC004,
+	MLX5_CREDENTIAL_HANDLE_REGISTER_ID = 0xC005,
 };
 
 struct mlx5_ifc_register_mtutc_bits {
@@ -3249,6 +3253,43 @@ struct mlx5_ifc_register_mtutc_bits {
 #define MLX5_MTUTC_TIMESTAMP_MODE_INTERNAL_TIMER 0
 #define MLX5_MTUTC_TIMESTAMP_MODE_REAL_TIME 1
 
+struct mlx5_ifc_crypto_operational_register_bits {
+	u8 wrapped_crypto_operational[0x1];
+	u8 reserved_at_1[0x1b];
+	u8 kek_size[0x4];
+	u8 reserved_at_20[0x20];
+	u8 credential[0x140];
+	u8 kek[0x100];
+	u8 reserved_at_280[0x180];
+};
+
+struct mlx5_ifc_crypto_commissioning_register_bits {
+	u8 token[0x1]; /* TODO: add size after PRM update */
+};
+
+struct mlx5_ifc_import_kek_handle_register_bits {
+	struct mlx5_ifc_crypto_login_bits crypto_login_object;
+	struct mlx5_ifc_import_kek_bits import_kek_object;
+	u8 reserved_at_200[0x4];
+	u8 write_operation[0x4];
+	u8 import_kek_id[0x18];
+	u8 reserved_at_220[0xe0];
+};
+
+struct mlx5_ifc_credential_handle_register_bits {
+	struct mlx5_ifc_crypto_login_bits crypto_login_object;
+	struct mlx5_ifc_credential_bits credential_object;
+	u8 reserved_at_200[0x4];
+	u8 write_operation[0x4];
+	u8 credential_id[0x18];
+	u8 reserved_at_220[0xe0];
+};
+
+enum {
+	MLX5_REGISTER_ADD_OPERATION = 0x1,
+	MLX5_REGISTER_DELETE_OPERATION = 0x2,
+};
+
 struct mlx5_ifc_parse_graph_arc_bits {
 	u8 start_inner_tunnel[0x1];
 	u8 reserved_at_1[0x7];
-- 
2.21.0

[dpdk-dev] [PATCH 15/24] common/mlx5: support register write access

[Shiri Kuzin]

From: Dekel Peled <dekelp@nvidia.com>

This patch adds support of write operation to NIC registers.

Signed-off-by: Dekel Peled <dekelp@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/mlx5_devx_cmds.c | 67 +++++++++++++++++++++++++++-
 drivers/common/mlx5/mlx5_devx_cmds.h |  4 ++
 drivers/common/mlx5/version.map      | 20 +++++----
 3 files changed, 81 insertions(+), 10 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c
index c43e320f87..49042d99ee 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.c
+++ b/drivers/common/mlx5/mlx5_devx_cmds.c
@@ -12,7 +12,6 @@
 #include "mlx5_common_log.h"
 #include "mlx5_malloc.h"
 
-
 /**
  * Perform read access to the registers. Reads data from register
  * and writes ones to the specified buffer.
@@ -61,7 +60,7 @@ mlx5_devx_cmd_register_read(void *ctx, uint16_t reg_id, uint32_t arg,
 	if (status) {
 		int syndrome = MLX5_GET(access_register_out, out, syndrome);
 
-		DRV_LOG(DEBUG, "Failed to access NIC register 0x%X, "
+		DRV_LOG(DEBUG, "Failed to read access NIC register 0x%X, "
 			       "status %x, syndrome = %x",
 			       reg_id, status, syndrome);
 		return -1;
@@ -74,6 +73,70 @@ mlx5_devx_cmd_register_read(void *ctx, uint16_t reg_id, uint32_t arg,
 	return rc;
 }
 
+/**
+ * Perform write access to the registers.
+ *
+ * @param[in] ctx
+ *   Context returned from mlx5 open_device() glue function.
+ * @param[in] reg_id
+ *   Register identifier according to the PRM.
+ * @param[in] arg
+ *   Register access auxiliary parameter according to the PRM.
+ * @param[out] data
+ *   Pointer to the buffer containing data to write.
+ * @param[in] dw_cnt
+ *   Buffer size in double words (32bit units).
+ *
+ * @return
+ *   0 on success, a negative value otherwise.
+ */
+int
+mlx5_devx_cmd_register_write(void *ctx, uint16_t reg_id, uint32_t arg,
+			     uint32_t *data, uint32_t dw_cnt)
+{
+	uint32_t in[MLX5_ST_SZ_DW(access_register_in) +
+		    MLX5_ACCESS_REGISTER_DATA_DWORD_MAX] = {0};
+	uint32_t out[MLX5_ST_SZ_DW(access_register_out)] = {0};
+	int status, rc;
+	void *ptr;
+
+	MLX5_ASSERT(data && dw_cnt);
+	MLX5_ASSERT(dw_cnt <= MLX5_ACCESS_REGISTER_DATA_DWORD_MAX);
+	if (dw_cnt > MLX5_ACCESS_REGISTER_DATA_DWORD_MAX) {
+		DRV_LOG(ERR, "Data to write exceeds max size");
+		return -1;
+	}
+	MLX5_SET(access_register_in, in, opcode,
+		 MLX5_CMD_OP_ACCESS_REGISTER_USER);
+	MLX5_SET(access_register_in, in, op_mod,
+		 MLX5_ACCESS_REGISTER_IN_OP_MOD_WRITE);
+	MLX5_SET(access_register_in, in, register_id, reg_id);
+	MLX5_SET(access_register_in, in, argument, arg);
+	ptr = MLX5_ADDR_OF(access_register_in, in, register_data);
+	memcpy(ptr, data, dw_cnt * sizeof(uint32_t));
+	rc = mlx5_glue->devx_general_cmd(ctx, in, sizeof(in), out, sizeof(out));
+
+	rc = mlx5_glue->devx_general_cmd(ctx, in,
+					 MLX5_ST_SZ_BYTES(access_register_in) +
+					 dw_cnt * sizeof(uint32_t),
+					 out, sizeof(out));
+	if (rc)
+		goto error;
+	status = MLX5_GET(access_register_out, out, status);
+	if (status) {
+		int syndrome = MLX5_GET(access_register_out, out, syndrome);
+
+		DRV_LOG(DEBUG, "Failed to write access NIC register 0x%X, "
+			       "status %x, syndrome = %x",
+			       reg_id, status, syndrome);
+		return -1;
+	}
+	return 0;
+error:
+	rc = (rc > 0) ? -rc : rc;
+	return rc;
+}
+
 /**
  * Allocate flow counters via devx interface.
  *
diff --git a/drivers/common/mlx5/mlx5_devx_cmds.h b/drivers/common/mlx5/mlx5_devx_cmds.h
index 1c952495bf..6e961f77db 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.h
+++ b/drivers/common/mlx5/mlx5_devx_cmds.h
@@ -552,6 +552,10 @@ __rte_internal
 int mlx5_devx_cmd_register_read(void *ctx, uint16_t reg_id,
 				uint32_t arg, uint32_t *data, uint32_t dw_cnt);
 
+__rte_internal
+int mlx5_devx_cmd_register_write(void *ctx, uint16_t reg_id,
+				 uint32_t arg, uint32_t *data, uint32_t dw_cnt);
+
 __rte_internal
 struct mlx5_devx_obj *
 mlx5_devx_cmd_create_geneve_tlv_option(void *ctx,
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index e1f0b324a0..2e213bd3b9 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -11,8 +11,16 @@ INTERNAL {
 	mlx5_dev_to_pci_addr;
 
 	mlx5_devx_cmd_alloc_pd;
+	mlx5_devx_alloc_uar;
+
 	mlx5_devx_cmd_create_cq;
+	mlx5_devx_cmd_create_credential_obj;
+	mlx5_devx_cmd_create_crypto_login_obj;
+	mlx5_devx_cmd_create_dek_obj;
 	mlx5_devx_cmd_create_flex_parser;
+	mlx5_devx_cmd_create_flow_hit_aso_obj;
+	mlx5_devx_cmd_create_geneve_tlv_option;
+	mlx5_devx_cmd_create_import_kek_obj;
 	mlx5_devx_cmd_create_qp;
 	mlx5_devx_cmd_create_rq;
 	mlx5_devx_cmd_create_rqt;
@@ -22,12 +30,6 @@ INTERNAL {
 	mlx5_devx_cmd_create_tis;
 	mlx5_devx_cmd_create_virtio_q_counters;
 	mlx5_devx_cmd_create_virtq;
-	mlx5_devx_cmd_create_flow_hit_aso_obj;
-	mlx5_devx_cmd_create_geneve_tlv_option;
-        mlx5_devx_cmd_create_dek_obj;
-        mlx5_devx_cmd_create_import_kek_obj;
-        mlx5_devx_cmd_create_credential_obj;
-        mlx5_devx_cmd_create_crypto_login_obj;
 	mlx5_devx_cmd_destroy;
 	mlx5_devx_cmd_flow_counter_alloc;
 	mlx5_devx_cmd_flow_counter_query;
@@ -47,12 +49,14 @@ INTERNAL {
 	mlx5_devx_cmd_queue_counter_alloc;
 	mlx5_devx_cmd_queue_counter_query;
 	mlx5_devx_cmd_register_read;
+	mlx5_devx_cmd_register_write;
 	mlx5_devx_cmd_wq_query;
-	mlx5_devx_get_out_command_status;
-	mlx5_devx_alloc_uar;
 
 	mlx5_devx_cq_create;
 	mlx5_devx_cq_destroy;
+
+	mlx5_devx_get_out_command_status;
+
 	mlx5_devx_rq_create;
 	mlx5_devx_rq_destroy;
 	mlx5_devx_sq_create;
-- 
2.21.0

[dpdk-dev] [PATCH 16/24] drivers: introduce mlx5 crypto PMD

[Shiri Kuzin]

Add a new PMD for Nvidia devices- crypto PMD.

The crypto PMD will be supported starting Nvidia ConnectX6 and
BlueField2.

The crypto PMD will add the support of encryption and decryption using
the AES-XTS symmetric algorithm.

The crypto PMD requires rdma-core and uses mlx5 DevX.

This patch adds the PCI probing, basic functions, build files and
log utility.

Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 MAINTAINERS                             |   4 +
 drivers/common/mlx5/mlx5_common.h       |   1 +
 drivers/common/mlx5/mlx5_common_pci.c   |  14 ++
 drivers/common/mlx5/mlx5_common_pci.h   |  21 +-
 drivers/crypto/meson.build              |   1 +
 drivers/crypto/mlx5/meson.build         |  26 +++
 drivers/crypto/mlx5/mlx5_crypto.c       | 288 ++++++++++++++++++++++++
 drivers/crypto/mlx5/mlx5_crypto_utils.h |  19 ++
 drivers/crypto/mlx5/version.map         |   3 +
 9 files changed, 367 insertions(+), 10 deletions(-)
 create mode 100644 drivers/crypto/mlx5/meson.build
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto.c
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto_utils.h
 create mode 100644 drivers/crypto/mlx5/version.map

diff --git a/MAINTAINERS b/MAINTAINERS
index 0ec5588540..448fd32186 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1064,6 +1064,10 @@ F: drivers/crypto/octeontx2/
 F: doc/guides/cryptodevs/octeontx2.rst
 F: doc/guides/cryptodevs/features/octeontx2.ini
 
+Mellanox mlx5
+M: Matan Azrad <matan@nvidia.com>
+F: drivers/crypto/mlx5/
+
 Null Crypto
 M: Declan Doherty <declan.doherty@intel.com>
 F: drivers/crypto/null/
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index f1ff16eb3e..7858cdb7dc 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -215,6 +215,7 @@ enum mlx5_class {
 	MLX5_CLASS_VDPA = RTE_BIT64(1),
 	MLX5_CLASS_REGEX = RTE_BIT64(2),
 	MLX5_CLASS_COMPRESS = RTE_BIT64(3),
+	MLX5_CLASS_CRYPTO = RTE_BIT64(4),
 };
 
 #define MLX5_DBR_SIZE RTE_CACHE_LINE_SIZE
diff --git a/drivers/common/mlx5/mlx5_common_pci.c b/drivers/common/mlx5/mlx5_common_pci.c
index 177ccf62d9..c27d8dd492 100644
--- a/drivers/common/mlx5/mlx5_common_pci.c
+++ b/drivers/common/mlx5/mlx5_common_pci.c
@@ -31,6 +31,7 @@ static const struct {
 	{ .name = "eth", .driver_class = MLX5_CLASS_NET },
 	{ .name = "regex", .driver_class = MLX5_CLASS_REGEX },
 	{ .name = "compress", .driver_class = MLX5_CLASS_COMPRESS },
+	{ .name = "crypto", .driver_class = MLX5_CLASS_CRYPTO },
 };
 
 static const unsigned int mlx5_class_combinations[] = {
@@ -38,13 +39,26 @@ static const unsigned int mlx5_class_combinations[] = {
 	MLX5_CLASS_VDPA,
 	MLX5_CLASS_REGEX,
 	MLX5_CLASS_COMPRESS,
+	MLX5_CLASS_CRYPTO,
 	MLX5_CLASS_NET | MLX5_CLASS_REGEX,
 	MLX5_CLASS_VDPA | MLX5_CLASS_REGEX,
 	MLX5_CLASS_NET | MLX5_CLASS_COMPRESS,
 	MLX5_CLASS_VDPA | MLX5_CLASS_COMPRESS,
 	MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS,
+	MLX5_CLASS_NET | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_VDPA | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_REGEX | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_COMPRESS | MLX5_CLASS_CRYPTO,
 	MLX5_CLASS_NET | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS,
 	MLX5_CLASS_VDPA | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS,
+	MLX5_CLASS_NET | MLX5_CLASS_REGEX | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_VDPA | MLX5_CLASS_REGEX | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_NET | MLX5_CLASS_COMPRESS | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_VDPA | MLX5_CLASS_COMPRESS | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_NET | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS |
+	MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_VDPA | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS |
+	MLX5_CLASS_CRYPTO,
 	/* New class combination should be added here. */
 };
 
diff --git a/drivers/common/mlx5/mlx5_common_pci.h b/drivers/common/mlx5/mlx5_common_pci.h
index de89bb98bc..cb8d2f5f87 100644
--- a/drivers/common/mlx5/mlx5_common_pci.h
+++ b/drivers/common/mlx5/mlx5_common_pci.h
@@ -9,17 +9,18 @@
  * @file
  *
  * RTE Mellanox PCI Driver Interface
- * Mellanox ConnectX PCI device supports multiple class: net,vdpa,regex and
- * compress devices. This layer enables creating such multiple class of devices
- * on a single PCI device by allowing to bind multiple class specific device
- * driver to attach to mlx5_pci driver.
+ * Mellanox ConnectX PCI device supports multiple class: net,vdpa,regex,compress
+ * and crypto devices. This layer enables creating such multiple class of
+ * devices on a single PCI device by allowing to bind multiple class specific
+ * device driver to attach to mlx5_pci driver.
  *
- * -----------    ------------    -------------    ----------------
- * |   mlx5  |    |   mlx5   |    |   mlx5    |    |     mlx5     |
- * | net pmd |    | vdpa pmd |    | regex pmd |    | compress pmd |
- * -----------    ------------    -------------    ----------------
- *      \              \                    /              /
- *       \              \                  /              /
+ * --------    --------    ---------    ------------    ----------
+ * | mlx5 |    | mlx5 |    | mlx5  |    |   mlx5   |    |  mlx5  |
+ * | net  |    | vdpa |    | regex |    | compress |    | crypto |
+ * | pmd  |    | pmd  |    |  pmd  |    |   pmd    |    |  pmd   |
+ * --------    --------    ---------    ------------    ----------
+ *      \              \         |          /              /
+ *       \              \        |         /              /
  *        \              \_--------------_/              /
  *         \_______________|   mlx5     |_______________/
  *                         | pci common |
diff --git a/drivers/crypto/meson.build b/drivers/crypto/meson.build
index c927c5f8d3..90b21494e1 100644
--- a/drivers/crypto/meson.build
+++ b/drivers/crypto/meson.build
@@ -14,6 +14,7 @@ drivers = ['aesni_gcm',
 	   'dpaa_sec',
 	   'dpaa2_sec',
 	   'kasumi',
+	   'mlx5',
 	   'mvsam',
 	   'nitrox',
 	   'null',
diff --git a/drivers/crypto/mlx5/meson.build b/drivers/crypto/mlx5/meson.build
new file mode 100644
index 0000000000..5bf0912766
--- /dev/null
+++ b/drivers/crypto/mlx5/meson.build
@@ -0,0 +1,26 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright 2021 Mellanox Technologies, Ltd
+
+if not is_linux
+	build = false
+	reason = 'only supported on Linux'
+	subdir_done()
+endif
+
+fmt_name = 'mlx5_crypto'
+deps += ['common_mlx5', 'eal', 'cryptodev']
+sources = files(
+	'mlx5_crypto.c',
+)
+cflags_options = [
+	'-std=c11',
+	'-Wno-strict-prototypes',
+	'-D_BSD_SOURCE',
+	'-D_DEFAULT_SOURCE',
+	'-D_XOPEN_SOURCE=600'
+]
+foreach option:cflags_options
+	if cc.has_argument(option)
+		cflags += option
+	endif
+endforeach
\ No newline at end of file
diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
new file mode 100644
index 0000000000..6e40176087
--- /dev/null
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -0,0 +1,288 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2021 Mellanox Technologies, Ltd
+ */
+
+#include <rte_malloc.h>
+#include <rte_log.h>
+#include <rte_errno.h>
+#include <rte_pci.h>
+#include <rte_crypto.h>
+#include <rte_cryptodev.h>
+#include <rte_cryptodev_pmd.h>
+
+#include <mlx5_glue.h>
+#include <mlx5_common.h>
+#include <mlx5_common_pci.h>
+#include <mlx5_devx_cmds.h>
+#include <mlx5_common_os.h>
+
+#include "mlx5_crypto_utils.h"
+
+#define MLX5_CRYPTO_DRIVER_NAME mlx5_crypto
+#define MLX5_CRYPTO_LOG_NAME    pmd.crypto.mlx5
+
+struct mlx5_crypto_priv {
+	TAILQ_ENTRY(mlx5_crypto_priv) next;
+	struct ibv_context *ctx; /* Device context. */
+	struct rte_pci_device *pci_dev;
+	struct rte_cryptodev *crypto_dev;
+	void *uar; /* User Access Region. */
+	uint32_t pdn; /* Protection Domain number. */
+	struct ibv_pd *pd;
+};
+
+TAILQ_HEAD(mlx5_crypto_privs, mlx5_crypto_priv) mlx5_crypto_priv_list =
+				TAILQ_HEAD_INITIALIZER(mlx5_crypto_priv_list);
+static pthread_mutex_t priv_list_lock = PTHREAD_MUTEX_INITIALIZER;
+
+int mlx5_crypto_logtype;
+
+uint8_t mlx5_crypto_driver_id;
+
+static const char mlx5_crypto_drv_name[] = RTE_STR(MLX5_CRYPTO_DRIVER_NAME);
+
+static const struct rte_driver mlx5_drv = {
+	.name = mlx5_crypto_drv_name,
+	.alias = mlx5_crypto_drv_name
+};
+
+static struct cryptodev_driver mlx5_cryptodev_driver;
+
+static struct rte_cryptodev_ops mlx5_crypto_ops = {
+	.dev_configure			= NULL,
+	.dev_start			= NULL,
+	.dev_stop			= NULL,
+	.dev_close			= NULL,
+	.dev_infos_get			= NULL,
+	.stats_get			= NULL,
+	.stats_reset			= NULL,
+	.queue_pair_setup		= NULL,
+	.queue_pair_release		= NULL,
+	.sym_session_get_size		= NULL,
+	.sym_session_configure		= NULL,
+	.sym_session_clear		= NULL,
+	.sym_get_raw_dp_ctx_size	= NULL,
+	.sym_configure_raw_dp_ctx	= NULL,
+};
+
+static void
+mlx5_crypto_hw_global_release(struct mlx5_crypto_priv *priv)
+{
+	if (priv->pd != NULL) {
+		claim_zero(mlx5_glue->dealloc_pd(priv->pd));
+		priv->pd = NULL;
+	}
+	if (priv->uar != NULL) {
+		mlx5_glue->devx_free_uar(priv->uar);
+		priv->uar = NULL;
+	}
+}
+
+static int
+mlx5_crypto_pd_create(struct mlx5_crypto_priv *priv)
+{
+#ifdef HAVE_IBV_FLOW_DV_SUPPORT
+	struct mlx5dv_obj obj;
+	struct mlx5dv_pd pd_info;
+	int ret;
+
+	priv->pd = mlx5_glue->alloc_pd(priv->ctx);
+	if (priv->pd == NULL) {
+		DRV_LOG(ERR, "Failed to allocate PD.");
+		return errno ? -errno : -ENOMEM;
+	}
+	obj.pd.in = priv->pd;
+	obj.pd.out = &pd_info;
+	ret = mlx5_glue->dv_init_obj(&obj, MLX5DV_OBJ_PD);
+	if (ret != 0) {
+		DRV_LOG(ERR, "Fail to get PD object info.");
+		mlx5_glue->dealloc_pd(priv->pd);
+		priv->pd = NULL;
+		return -errno;
+	}
+	priv->pdn = pd_info.pdn;
+	return 0;
+#else
+	(void)priv;
+	DRV_LOG(ERR, "Cannot get pdn - no DV support.");
+	return -ENOTSUP;
+#endif /* HAVE_IBV_FLOW_DV_SUPPORT */
+}
+
+static int
+mlx5_crypto_hw_global_prepare(struct mlx5_crypto_priv *priv)
+{
+	if (mlx5_crypto_pd_create(priv) != 0)
+		return -1;
+	priv->uar = mlx5_devx_alloc_uar(priv->ctx, -1);
+	if (priv->uar == NULL || mlx5_os_get_devx_uar_reg_addr(priv->uar) ==
+	    NULL) {
+		rte_errno = errno;
+		claim_zero(mlx5_glue->dealloc_pd(priv->pd));
+		DRV_LOG(ERR, "Failed to allocate UAR.");
+		return -1;
+	}
+	return 0;
+}
+
+/**
+ * DPDK callback to register a PCI device.
+ *
+ * This function spawns crypto device out of a given PCI device.
+ *
+ * @param[in] pci_drv
+ *   PCI driver structure (mlx5_crypto_driver).
+ * @param[in] pci_dev
+ *   PCI device information.
+ *
+ * @return
+ *   0 on success, 1 to skip this driver, a negative errno value otherwise
+ *   and rte_errno is set.
+ */
+static int
+mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
+			struct rte_pci_device *pci_dev)
+{
+	struct ibv_device *ibv;
+	struct rte_cryptodev *crypto_dev;
+	struct ibv_context *ctx;
+	struct mlx5_crypto_priv *priv;
+	struct mlx5_hca_attr attr = { 0 };
+	struct rte_cryptodev_pmd_init_params init_params = {
+		.name = "",
+		.private_data_size = sizeof(struct mlx5_crypto_priv),
+		.socket_id = pci_dev->device.numa_node,
+		.max_nb_queue_pairs =
+				RTE_CRYPTODEV_PMD_DEFAULT_MAX_NB_QUEUE_PAIRS,
+	};
+	RTE_SET_USED(pci_drv);
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		DRV_LOG(ERR, "Non-primary process type is not supported.");
+		rte_errno = ENOTSUP;
+		return -rte_errno;
+	}
+	ibv = mlx5_os_get_ib_device_match(&pci_dev->addr);
+	if (ibv == NULL) {
+		DRV_LOG(ERR, "No matching IB device for PCI slot "
+			PCI_PRI_FMT ".", pci_dev->addr.domain,
+			pci_dev->addr.bus, pci_dev->addr.devid,
+			pci_dev->addr.function);
+		return -rte_errno;
+	}
+	DRV_LOG(INFO, "PCI information matches for device \"%s\".", ibv->name);
+	ctx = mlx5_glue->dv_open_device(ibv);
+	if (ctx == NULL) {
+		DRV_LOG(ERR, "Failed to open IB device \"%s\".", ibv->name);
+		rte_errno = ENODEV;
+		return -rte_errno;
+	}
+	if (mlx5_devx_cmd_query_hca_attr(ctx, &attr) != 0 ||
+	    attr.crypto == 0 || attr.aes_xts == 0) {
+		DRV_LOG(ERR, "Not enough capabilities to support crypto "
+			"operations, maybe old FW/OFED version?");
+		claim_zero(mlx5_glue->close_device(ctx));
+		rte_errno = ENOTSUP;
+		return -ENOTSUP;
+	}
+	crypto_dev = rte_cryptodev_pmd_create(ibv->name, &pci_dev->device,
+					&init_params);
+	if (crypto_dev == NULL) {
+		DRV_LOG(ERR, "Failed to create device \"%s\".", ibv->name);
+		claim_zero(mlx5_glue->close_device(ctx));
+		return -ENODEV;
+	}
+	DRV_LOG(INFO,
+		"Crypto device %s was created successfully.", ibv->name);
+	crypto_dev->dev_ops = &mlx5_crypto_ops;
+	crypto_dev->dequeue_burst = NULL;
+	crypto_dev->enqueue_burst = NULL;
+	crypto_dev->feature_flags = RTE_CRYPTODEV_FF_HW_ACCELERATED;
+	crypto_dev->driver_id = mlx5_crypto_driver_id;
+	priv = crypto_dev->data->dev_private;
+	priv->ctx = ctx;
+	priv->pci_dev = pci_dev;
+	priv->crypto_dev = crypto_dev;
+	if (mlx5_crypto_hw_global_prepare(priv) != 0) {
+		rte_cryptodev_pmd_destroy(priv->crypto_dev);
+		claim_zero(mlx5_glue->close_device(priv->ctx));
+		return -1;
+	}
+	pthread_mutex_lock(&priv_list_lock);
+	TAILQ_INSERT_TAIL(&mlx5_crypto_priv_list, priv, next);
+	pthread_mutex_unlock(&priv_list_lock);
+	return 0;
+}
+
+static int
+mlx5_crypto_pci_remove(struct rte_pci_device *pdev)
+{
+	struct mlx5_crypto_priv *priv = NULL;
+
+	pthread_mutex_lock(&priv_list_lock);
+	TAILQ_FOREACH(priv, &mlx5_crypto_priv_list, next)
+		if (rte_pci_addr_cmp(&priv->pci_dev->addr, &pdev->addr) != 0)
+			break;
+	if (priv)
+		TAILQ_REMOVE(&mlx5_crypto_priv_list, priv, next);
+	pthread_mutex_unlock(&priv_list_lock);
+	if (priv) {
+		mlx5_crypto_hw_global_release(priv);
+		rte_cryptodev_pmd_destroy(priv->crypto_dev);
+		claim_zero(mlx5_glue->close_device(priv->ctx));
+	}
+	return 0;
+}
+
+static const struct rte_pci_id mlx5_crypto_pci_id_map[] = {
+		{
+			RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
+					PCI_DEVICE_ID_MELLANOX_CONNECTX6)
+		},
+		{
+			RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
+					PCI_DEVICE_ID_MELLANOX_CONNECTX6DX)
+		},
+		{
+			RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
+					PCI_DEVICE_ID_MELLANOX_CONNECTX6DXBF)
+		},
+		{
+			RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
+					PCI_DEVICE_ID_MELLANOX_CONNECTX7)
+		},
+		{
+			RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
+					PCI_DEVICE_ID_MELLANOX_CONNECTX7BF)
+		},
+		{
+			.vendor_id = 0
+		}
+	};
+
+static struct mlx5_pci_driver mlx5_crypto_driver = {
+	.driver_class = MLX5_CLASS_CRYPTO,
+	.pci_driver = {
+		.driver = {
+			.name = RTE_STR(MLX5_CRYPTO_DRIVER_NAME),
+		},
+		.id_table = mlx5_crypto_pci_id_map,
+		.probe = mlx5_crypto_pci_probe,
+		.remove = mlx5_crypto_pci_remove,
+		.drv_flags = 0,
+	},
+};
+
+RTE_INIT(rte_mlx5_crypto_init)
+{
+	mlx5_common_init();
+	if (mlx5_glue != NULL)
+		mlx5_pci_driver_register(&mlx5_crypto_driver);
+}
+
+RTE_PMD_REGISTER_CRYPTO_DRIVER(mlx5_cryptodev_driver, mlx5_drv,
+			       mlx5_crypto_driver_id);
+
+RTE_LOG_REGISTER(mlx5_crypto_logtype, MLX5_CRYPTO_LOG_NAME, NOTICE)
+RTE_PMD_EXPORT_NAME(MLX5_CRYPTO_DRIVER_NAME, __COUNTER__);
+RTE_PMD_REGISTER_PCI_TABLE(MLX5_CRYPTO_DRIVER_NAME, mlx5_crypto_pci_id_map);
+RTE_PMD_REGISTER_KMOD_DEP(MLX5_CRYPTO_DRIVER_NAME, "* ib_uverbs & mlx5_core & mlx5_ib");
diff --git a/drivers/crypto/mlx5/mlx5_crypto_utils.h b/drivers/crypto/mlx5/mlx5_crypto_utils.h
new file mode 100644
index 0000000000..cef4b07a36
--- /dev/null
+++ b/drivers/crypto/mlx5/mlx5_crypto_utils.h
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2021 Mellanox Technologies, Ltd
+ */
+
+#ifndef RTE_PMD_MLX5_CRYPTO_UTILS_H_
+#define RTE_PMD_MLX5_CRYPTO_UTILS_H_
+
+#include <mlx5_common.h>
+
+extern int mlx5_crypto_logtype;
+
+#define MLX5_CRYPTO_LOG_PREFIX "mlx5_crypto"
+/* Generic printf()-like logging macro with automatic line feed. */
+#define DRV_LOG(level, ...) \
+	PMD_DRV_LOG_(level, mlx5_crypto_logtype, MLX5_CRYPTO_LOG_PREFIX, \
+		__VA_ARGS__ PMD_DRV_LOG_STRIP PMD_DRV_LOG_OPAREN, \
+		PMD_DRV_LOG_CPAREN)
+
+#endif /* RTE_PMD_MLX5_CRYPTO_UTILS_H_ */
diff --git a/drivers/crypto/mlx5/version.map b/drivers/crypto/mlx5/version.map
new file mode 100644
index 0000000000..4a76d1d52d
--- /dev/null
+++ b/drivers/crypto/mlx5/version.map
@@ -0,0 +1,3 @@
+DPDK_21 {
+	local: *;
+};
-- 
2.21.0

[dpdk-dev] [PATCH 17/24] crypto/mlx5: add DEK object management

[Shiri Kuzin]

A DEK(Data encryption Key) is an mlx5 HW object which represents the
cipher algorithm key.
The DEKs are used during data encryption/decryption operations.

In symmetric algorithms like AES-STS, we use the same DEK for both
encryption and decryption.

Use the mlx5 hash-list tool to manage the DEK objects in the PMD.

Provide the compare, create and destroy functions to manage DEKs in
hash-list and introduce an internal API to setup and unset the DEK
management and to prepare and destroy specific DEK object.

The DEK hash-list will be created in dev_configure routine and
destroyed in dev_close routine.

Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/crypto/mlx5/meson.build       |   1 +
 drivers/crypto/mlx5/mlx5_crypto.c     |  44 +++++----
 drivers/crypto/mlx5/mlx5_crypto.h     |  51 ++++++++++
 drivers/crypto/mlx5/mlx5_crypto_dek.c | 135 ++++++++++++++++++++++++++
 4 files changed, 214 insertions(+), 17 deletions(-)
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto.h
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto_dek.c

diff --git a/drivers/crypto/mlx5/meson.build b/drivers/crypto/mlx5/meson.build
index 5bf0912766..0666c35094 100644
--- a/drivers/crypto/mlx5/meson.build
+++ b/drivers/crypto/mlx5/meson.build
@@ -11,6 +11,7 @@ fmt_name = 'mlx5_crypto'
 deps += ['common_mlx5', 'eal', 'cryptodev']
 sources = files(
 	'mlx5_crypto.c',
+	'mlx5_crypto_dek.c',
 )
 cflags_options = [
 	'-std=c11',
diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index 6e40176087..17aaaaa53d 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -3,12 +3,9 @@
  */
 
 #include <rte_malloc.h>
-#include <rte_log.h>
 #include <rte_errno.h>
+#include <rte_log.h>
 #include <rte_pci.h>
-#include <rte_crypto.h>
-#include <rte_cryptodev.h>
-#include <rte_cryptodev_pmd.h>
 
 #include <mlx5_glue.h>
 #include <mlx5_common.h>
@@ -17,19 +14,10 @@
 #include <mlx5_common_os.h>
 
 #include "mlx5_crypto_utils.h"
+#include "mlx5_crypto.h"
 
 #define MLX5_CRYPTO_DRIVER_NAME mlx5_crypto
-#define MLX5_CRYPTO_LOG_NAME    pmd.crypto.mlx5
-
-struct mlx5_crypto_priv {
-	TAILQ_ENTRY(mlx5_crypto_priv) next;
-	struct ibv_context *ctx; /* Device context. */
-	struct rte_pci_device *pci_dev;
-	struct rte_cryptodev *crypto_dev;
-	void *uar; /* User Access Region. */
-	uint32_t pdn; /* Protection Domain number. */
-	struct ibv_pd *pd;
-};
+#define MLX5_CRYPTO_LOG_NAME pmd.crypto.mlx5
 
 TAILQ_HEAD(mlx5_crypto_privs, mlx5_crypto_priv) mlx5_crypto_priv_list =
 				TAILQ_HEAD_INITIALIZER(mlx5_crypto_priv_list);
@@ -48,11 +36,33 @@ static const struct rte_driver mlx5_drv = {
 
 static struct cryptodev_driver mlx5_cryptodev_driver;
 
+static int
+mlx5_crypto_dev_configure(struct rte_cryptodev *dev,
+		struct rte_cryptodev_config *config __rte_unused)
+{
+	struct mlx5_crypto_priv *priv = dev->data->dev_private;
+
+	if (mlx5_crypto_dek_setup(priv) != 0) {
+		DRV_LOG(ERR, "Dek hash list creation has failed.");
+		return -ENOMEM;
+	}
+	return 0;
+}
+
+static int
+mlx5_crypto_dev_close(struct rte_cryptodev *dev)
+{
+	struct mlx5_crypto_priv *priv = dev->data->dev_private;
+
+	mlx5_crypto_dek_unset(priv);
+	return 0;
+}
+
 static struct rte_cryptodev_ops mlx5_crypto_ops = {
-	.dev_configure			= NULL,
+	.dev_configure			= mlx5_crypto_dev_configure,
 	.dev_start			= NULL,
 	.dev_stop			= NULL,
-	.dev_close			= NULL,
+	.dev_close			= mlx5_crypto_dev_close,
 	.dev_infos_get			= NULL,
 	.stats_get			= NULL,
 	.stats_reset			= NULL,
diff --git a/drivers/crypto/mlx5/mlx5_crypto.h b/drivers/crypto/mlx5/mlx5_crypto.h
new file mode 100644
index 0000000000..4ec67a7e0f
--- /dev/null
+++ b/drivers/crypto/mlx5/mlx5_crypto.h
@@ -0,0 +1,51 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2021 Mellanox Technologies, Ltd
+ */
+
+#ifndef MLX5_CRYPTO_H_
+#define MLX5_CRYPTO_H_
+
+#include <stdbool.h>
+
+#include <rte_cryptodev.h>
+#include <rte_cryptodev_pmd.h>
+
+#include <mlx5_common_utils.h>
+
+#define MLX5_CRYPTO_DEK_HTABLE_SZ (1 << 11)
+#define MLX5_CRYPTO_KEY_LENGTH 80
+
+struct mlx5_crypto_priv {
+	TAILQ_ENTRY(mlx5_crypto_priv) next;
+	struct ibv_context *ctx; /* Device context. */
+	struct rte_pci_device *pci_dev;
+	struct rte_cryptodev *crypto_dev;
+	void *uar; /* User Access Region. */
+	uint32_t pdn; /* Protection Domain number. */
+	struct ibv_pd *pd;
+	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
+};
+
+struct mlx5_crypto_dek {
+	struct mlx5_hlist_entry entry; /* Pointer to DEK hash list entry. */
+	struct mlx5_devx_obj *obj; /* Pointer to DEK DevX object. */
+	uint8_t data[MLX5_CRYPTO_KEY_LENGTH]; /* DEK key data. */
+	bool size_is_48; /* Whether the key\data size is 48 bytes or not. */
+};
+
+int
+mlx5_crypto_dek_destroy(struct mlx5_crypto_priv *priv,
+			struct mlx5_crypto_dek *dek);
+
+struct mlx5_crypto_dek *
+mlx5_crypto_dek_prepare(struct mlx5_crypto_priv *priv,
+			struct rte_crypto_cipher_xform *cipher);
+
+int
+mlx5_crypto_dek_setup(struct mlx5_crypto_priv *priv);
+
+void
+mlx5_crypto_dek_unset(struct mlx5_crypto_priv *priv);
+
+#endif /* MLX5_CRYPTO_H_ */
+
diff --git a/drivers/crypto/mlx5/mlx5_crypto_dek.c b/drivers/crypto/mlx5/mlx5_crypto_dek.c
new file mode 100644
index 0000000000..70410fed57
--- /dev/null
+++ b/drivers/crypto/mlx5/mlx5_crypto_dek.c
@@ -0,0 +1,135 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2018 Mellanox Technologies, Ltd
+ */
+
+#include <rte_ip.h>
+#include <rte_common.h>
+#include <rte_errno.h>
+#include <rte_log.h>
+
+#include <mlx5_prm.h>
+#include <mlx5_devx_cmds.h>
+
+#include "mlx5_crypto_utils.h"
+#include "mlx5_crypto.h"
+
+struct mlx5_crypto_dek_ctx {
+	struct rte_crypto_cipher_xform *cipher;
+	struct mlx5_crypto_priv *priv;
+};
+
+int
+mlx5_crypto_dek_destroy(struct mlx5_crypto_priv *priv,
+			struct mlx5_crypto_dek *dek)
+{
+	return mlx5_hlist_unregister(priv->dek_hlist, &dek->entry);
+}
+
+struct mlx5_crypto_dek *
+mlx5_crypto_dek_prepare(struct mlx5_crypto_priv *priv,
+			struct rte_crypto_cipher_xform *cipher)
+{
+	struct mlx5_hlist *dek_hlist = priv->dek_hlist;
+	struct mlx5_crypto_dek_ctx dek_ctx = {
+		.cipher = cipher,
+		.priv = priv,
+	};
+	struct rte_crypto_cipher_xform *cipher_ctx = cipher;
+	uint64_t key64 = __rte_raw_cksum(cipher_ctx->key.data,
+					 cipher_ctx->key.length, 0);
+	struct mlx5_hlist_entry *entry = mlx5_hlist_register(dek_hlist,
+							     key64, &dek_ctx);
+
+	return entry == NULL ? NULL :
+			     container_of(entry, struct mlx5_crypto_dek, entry);
+}
+
+static int
+mlx5_crypto_dek_match_cb(struct mlx5_hlist *list __rte_unused,
+			 struct mlx5_hlist_entry *entry,
+			 uint64_t key __rte_unused, void *cb_ctx)
+{
+	struct mlx5_crypto_dek_ctx *ctx = cb_ctx;
+	struct rte_crypto_cipher_xform *cipher_ctx = ctx->cipher;
+	struct mlx5_crypto_dek *dek =
+			container_of(entry, typeof(*dek), entry);
+	uint32_t key_len = dek->size_is_48 ? 48 : 80;
+
+	if (key_len != cipher_ctx->key.length)
+		return -1;
+	return memcmp(cipher_ctx->key.data, dek->data, key_len);
+}
+
+static struct mlx5_hlist_entry *
+mlx5_crypto_dek_create_cb(struct mlx5_hlist *list __rte_unused,
+			  uint64_t key __rte_unused, void *cb_ctx)
+{
+	struct mlx5_crypto_dek_ctx *ctx = cb_ctx;
+	struct rte_crypto_cipher_xform *cipher_ctx = ctx->cipher;
+	struct mlx5_crypto_dek *dek = rte_zmalloc(__func__, sizeof(*dek),
+						  RTE_CACHE_LINE_SIZE);
+	struct mlx5_devx_dek_attr dek_attr = {
+		.pd = ctx->priv->pdn,
+		.key_purpose = MLX5_CRYPTO_KEY_PURPOSE_AES_XTS,
+	};
+
+	if (dek == NULL) {
+		DRV_LOG(ERR, "Failed to allocate dek memory.");
+		return NULL;
+	}
+	switch (cipher_ctx->key.length) {
+	case 48:
+		dek->size_is_48 = true;
+		dek_attr.key_size = MLX5_CRYPTO_KEY_SIZE_128b;
+		break;
+	case 80:
+		dek->size_is_48 = false;
+		dek_attr.key_size = MLX5_CRYPTO_KEY_SIZE_256b;
+		break;
+	default:
+		DRV_LOG(ERR, "Key size not supported.");
+		return NULL;
+	}
+	rte_memcpy(&dek_attr.key, cipher_ctx->key.data, cipher_ctx->key.length);
+	dek->obj = mlx5_devx_cmd_create_dek_obj(ctx->priv->ctx, &dek_attr);
+	if (dek->obj == NULL) {
+		rte_free(dek);
+		return NULL;
+	}
+	rte_memcpy(&dek->data, cipher_ctx->key.data, cipher_ctx->key.length);
+	return &dek->entry;
+}
+
+static void
+mlx5_crypto_dek_remove_cb(struct mlx5_hlist *list __rte_unused,
+			  struct mlx5_hlist_entry *entry)
+{
+	struct mlx5_crypto_dek *dek =
+		container_of(entry, typeof(*dek), entry);
+
+	claim_zero(mlx5_devx_cmd_destroy(dek->obj));
+	rte_free(dek);
+}
+
+
+int
+mlx5_crypto_dek_setup(struct mlx5_crypto_priv *priv)
+{
+	priv->dek_hlist = mlx5_hlist_create("dek_hlist",
+				 MLX5_CRYPTO_DEK_HTABLE_SZ,
+				 0, MLX5_HLIST_WRITE_MOST |
+				 MLX5_HLIST_DIRECT_KEY,
+				 mlx5_crypto_dek_create_cb,
+				 mlx5_crypto_dek_match_cb,
+				 mlx5_crypto_dek_remove_cb);
+	if (priv->dek_hlist == NULL)
+		return -1;
+	return 0;
+}
+
+void
+mlx5_crypto_dek_unset(struct mlx5_crypto_priv *priv)
+{
+	mlx5_hlist_destroy(priv->dek_hlist);
+	priv->dek_hlist = NULL;
+}
-- 
2.21.0

[dpdk-dev] [PATCH 18/24] crypto/mlx5: support session operations

[Shiri Kuzin]

Sessions are used in symmetric transformations in order to prepare
objects and data for packet processing stage.

A mlx5 session includes iv_offset, pointer to mlx5_crypto_dek struct,
bsf_size, bsf_p_type, encryption_order and encryption standard.

Implement the next session operations:
        mlx5_crypto_sym_session_get_size- returns the size of the mlx5
	session struct.
	mlx5_crypto_sym_session_configure- prepares the DEK hash-list
	and saves all the session data.
	mlx5_crypto_sym_session_clear - destroys the DEK hash-list.

Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/crypto/mlx5/mlx5_crypto.c | 92 ++++++++++++++++++++++++++++++-
 1 file changed, 89 insertions(+), 3 deletions(-)

diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index 17aaaaa53d..b0242afec4 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -3,6 +3,7 @@
  */
 
 #include <rte_malloc.h>
+#include <rte_mempool.h>
 #include <rte_errno.h>
 #include <rte_log.h>
 #include <rte_pci.h>
@@ -36,6 +37,23 @@ static const struct rte_driver mlx5_drv = {
 
 static struct cryptodev_driver mlx5_cryptodev_driver;
 
+struct mlx5_crypto_session {
+	uint32_t bs_bpt_eo_es;
+	/*
+	 * bsf_size, bsf_p_type, encryption_order and encryption standard,
+	 * saved in big endian format.
+	 */
+	uint32_t iv_offset:16;
+	/* Starting point for Initialisation Vector. */
+	struct mlx5_crypto_dek *dek; /* Pointer to dek struct. */
+} __rte_packed;
+
+static unsigned int
+mlx5_crypto_sym_session_get_size(struct rte_cryptodev *dev __rte_unused)
+{
+	return sizeof(struct mlx5_crypto_session);
+}
+
 static int
 mlx5_crypto_dev_configure(struct rte_cryptodev *dev,
 		struct rte_cryptodev_config *config __rte_unused)
@@ -58,6 +76,74 @@ mlx5_crypto_dev_close(struct rte_cryptodev *dev)
 	return 0;
 }
 
+static int
+mlx5_crypto_sym_session_configure(struct rte_cryptodev *dev,
+				  struct rte_crypto_sym_xform *xform,
+				  struct rte_cryptodev_sym_session *session,
+				  struct rte_mempool *mp)
+{
+	struct mlx5_crypto_priv *priv = dev->data->dev_private;
+	struct mlx5_crypto_session *sess_private_data;
+	struct rte_crypto_cipher_xform *cipher;
+	uint8_t encryption_order;
+	int ret;
+
+	if (unlikely(xform->next != NULL)) {
+		DRV_LOG(ERR, "Xform next is not supported.");
+		return -ENOTSUP;
+	}
+	if (unlikely((xform->type != RTE_CRYPTO_SYM_XFORM_CIPHER) ||
+		     (xform->cipher.algo != RTE_CRYPTO_CIPHER_AES_XTS))) {
+		DRV_LOG(ERR, "Only AES-XTS algorithm is supported.");
+		return -ENOTSUP;
+	}
+	ret = rte_mempool_get(mp, (void *)&sess_private_data);
+	if (ret != 0) {
+		DRV_LOG(ERR,
+			"Failed to get session %p private data from mempool.",
+			sess_private_data);
+		return -ENOMEM;
+	}
+	cipher = &xform->cipher;
+	sess_private_data->dek = mlx5_crypto_dek_prepare(priv, cipher);
+	if (sess_private_data->dek == NULL) {
+		rte_mempool_put(mp, sess_private_data);
+		DRV_LOG(ERR, "Failed to prepare dek.");
+		return -ENOMEM;
+	}
+	if (cipher->op == RTE_CRYPTO_CIPHER_OP_ENCRYPT)
+		encryption_order = MLX5_ENCRYPTION_ORDER_ENCRYPTED_RAW_MEMORY;
+	else
+		encryption_order = MLX5_ENCRYPTION_ORDER_ENCRYPTED_RAW_WIRE;
+	sess_private_data->bs_bpt_eo_es = rte_cpu_to_be_32
+			(MLX5_BSF_SIZE_64B << MLX5_BSF_SIZE_OFFSET |
+			 MLX5_BSF_P_TYPE_CRYPTO << MLX5_BSF_P_TYPE_OFFSET |
+			 encryption_order << MLX5_ENCRYPTION_ORDER_OFFSET |
+			 MLX5_ENCRYPTION_STANDARD_AES_XTS);
+	sess_private_data->iv_offset = cipher->iv.offset;
+	set_sym_session_private_data(session, dev->driver_id,
+				     sess_private_data);
+	DRV_LOG(DEBUG, "Session %p was configured.", sess_private_data);
+	return 0;
+}
+
+static void
+mlx5_crypto_sym_session_clear(struct rte_cryptodev *dev,
+			      struct rte_cryptodev_sym_session *sess)
+{
+	struct mlx5_crypto_priv *priv = dev->data->dev_private;
+	struct mlx5_crypto_session *sess_private_data =
+			get_sym_session_private_data(sess, dev->driver_id);
+
+	if (unlikely(sess_private_data == NULL)) {
+		DRV_LOG(ERR, "Failed to get session %p private data.",
+				sess_private_data);
+		return;
+	}
+	mlx5_crypto_dek_destroy(priv, sess_private_data->dek);
+	DRV_LOG(DEBUG, "Session %p was cleared.", sess_private_data);
+}
+
 static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_configure			= mlx5_crypto_dev_configure,
 	.dev_start			= NULL,
@@ -68,9 +154,9 @@ static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.stats_reset			= NULL,
 	.queue_pair_setup		= NULL,
 	.queue_pair_release		= NULL,
-	.sym_session_get_size		= NULL,
-	.sym_session_configure		= NULL,
-	.sym_session_clear		= NULL,
+	.sym_session_get_size		= mlx5_crypto_sym_session_get_size,
+	.sym_session_configure		= mlx5_crypto_sym_session_configure,
+	.sym_session_clear		= mlx5_crypto_sym_session_clear,
 	.sym_get_raw_dp_ctx_size	= NULL,
 	.sym_configure_raw_dp_ctx	= NULL,
 };
-- 
2.21.0

[dpdk-dev] [PATCH 19/24] crypto/mlx5: add basic operations

[Shiri Kuzin]

The basic dev control operations are configure, close and get info.

Extended the existing support of configure and close:
	-mlx5_crypto_dev_configure- function used to configure device.
	-mlx5_crypto_dev_close-  function used to close a configured
	 device.

Added support of get info function:
	-mlx5_crypto_dev_infos_get- function used to get specific
	 information of a device.

Added config struct to user private data with the fields socket id,
number of queue pairs and feature flags to be disabled.

Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/crypto/mlx5/mlx5_crypto.c | 46 +++++++++++++++++++++++++++----
 drivers/crypto/mlx5/mlx5_crypto.h |  1 +
 2 files changed, 42 insertions(+), 5 deletions(-)

diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index b0242afec4..3807a01357 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -19,6 +19,7 @@
 
 #define MLX5_CRYPTO_DRIVER_NAME mlx5_crypto
 #define MLX5_CRYPTO_LOG_NAME pmd.crypto.mlx5
+#define MLX5_CRYPTO_MAX_QPS 1024
 
 TAILQ_HEAD(mlx5_crypto_privs, mlx5_crypto_priv) mlx5_crypto_priv_list =
 				TAILQ_HEAD_INITIALIZER(mlx5_crypto_priv_list);
@@ -28,6 +29,9 @@ int mlx5_crypto_logtype;
 
 uint8_t mlx5_crypto_driver_id;
 
+const struct rte_cryptodev_capabilities
+		mlx5_crypto_caps[RTE_CRYPTO_OP_TYPE_UNDEFINED];
+
 static const char mlx5_crypto_drv_name[] = RTE_STR(MLX5_CRYPTO_DRIVER_NAME);
 
 static const struct rte_driver mlx5_drv = {
@@ -48,22 +52,47 @@ struct mlx5_crypto_session {
 	struct mlx5_crypto_dek *dek; /* Pointer to dek struct. */
 } __rte_packed;
 
-static unsigned int
-mlx5_crypto_sym_session_get_size(struct rte_cryptodev *dev __rte_unused)
+static void
+mlx5_crypto_dev_infos_get(struct rte_cryptodev *dev,
+			  struct rte_cryptodev_info *dev_info)
 {
-	return sizeof(struct mlx5_crypto_session);
+	RTE_SET_USED(dev);
+	if (dev_info != NULL) {
+		dev_info->driver_id = mlx5_crypto_driver_id;
+		dev_info->feature_flags = 0;
+		dev_info->capabilities = mlx5_crypto_caps;
+		dev_info->max_nb_queue_pairs = MLX5_CRYPTO_MAX_QPS;
+		dev_info->min_mbuf_headroom_req = 0;
+		dev_info->min_mbuf_tailroom_req = 0;
+		dev_info->sym.max_nb_sessions = 0;
+		/*
+		 * If 0, the device does not have any limitation in number of
+		 * sessions that can be used.
+		 */
+	}
 }
 
 static int
 mlx5_crypto_dev_configure(struct rte_cryptodev *dev,
-		struct rte_cryptodev_config *config __rte_unused)
+			  struct rte_cryptodev_config *config)
 {
 	struct mlx5_crypto_priv *priv = dev->data->dev_private;
 
+	if (config == NULL) {
+		DRV_LOG(ERR, "Invalid crypto dev configure parameters.");
+		return -EINVAL;
+	}
+	if ((config->ff_disable & RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO) != 0) {
+		DRV_LOG(ERR,
+			"Disabled symmetric crypto feature is not supported.");
+		return -ENOTSUP;
+	}
 	if (mlx5_crypto_dek_setup(priv) != 0) {
 		DRV_LOG(ERR, "Dek hash list creation has failed.");
 		return -ENOMEM;
 	}
+	priv->dev_config = *config;
+	DRV_LOG(DEBUG, "Device %u was configured.", dev->driver_id);
 	return 0;
 }
 
@@ -73,9 +102,16 @@ mlx5_crypto_dev_close(struct rte_cryptodev *dev)
 	struct mlx5_crypto_priv *priv = dev->data->dev_private;
 
 	mlx5_crypto_dek_unset(priv);
+	DRV_LOG(DEBUG, "Device %u was closed.", dev->driver_id);
 	return 0;
 }
 
+static unsigned int
+mlx5_crypto_sym_session_get_size(struct rte_cryptodev *dev __rte_unused)
+{
+	return sizeof(struct mlx5_crypto_session);
+}
+
 static int
 mlx5_crypto_sym_session_configure(struct rte_cryptodev *dev,
 				  struct rte_crypto_sym_xform *xform,
@@ -149,7 +185,7 @@ static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_start			= NULL,
 	.dev_stop			= NULL,
 	.dev_close			= mlx5_crypto_dev_close,
-	.dev_infos_get			= NULL,
+	.dev_infos_get			= mlx5_crypto_dev_infos_get,
 	.stats_get			= NULL,
 	.stats_reset			= NULL,
 	.queue_pair_setup		= NULL,
diff --git a/drivers/crypto/mlx5/mlx5_crypto.h b/drivers/crypto/mlx5/mlx5_crypto.h
index 4ec67a7e0f..5e270d3d5a 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.h
+++ b/drivers/crypto/mlx5/mlx5_crypto.h
@@ -24,6 +24,7 @@ struct mlx5_crypto_priv {
 	uint32_t pdn; /* Protection Domain number. */
 	struct ibv_pd *pd;
 	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
+	struct rte_cryptodev_config dev_config;
 };
 
 struct mlx5_crypto_dek {
-- 
2.21.0

[dpdk-dev] [PATCH 20/24] crypto/mlx5: support queue pairs operations

[Shiri Kuzin]

The HW queue pairs are a pair of send queue and receive queue of
independent work queues packed together in one object for the purpose
of transferring data between nodes of a network.

Completion Queue is a FIFO queue of completed work requests.

In crypto driver we use one QP in loopback in order to encrypt and
decrypt data locally without sending it to the wire.
In the configured QP we only use the SQ to preform the encryption and
decryption operations.

Added implementation for the QP setup function which creates the CQ,
creates the QP and changes its state to RTS (ready to send).

Added implementation for the release QP function to release all the QP
resources.

Added the ops structure that contains any operation which is supported
by the cryptodev.

Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/crypto/mlx5/mlx5_crypto.c | 124 +++++++++++++++++++++++++++++-
 drivers/crypto/mlx5/mlx5_crypto.h |  11 +++
 2 files changed, 133 insertions(+), 2 deletions(-)

diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index 3807a01357..07bc6e3a1c 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -7,6 +7,7 @@
 #include <rte_errno.h>
 #include <rte_log.h>
 #include <rte_pci.h>
+#include <rte_memory.h>
 
 #include <mlx5_glue.h>
 #include <mlx5_common.h>
@@ -180,6 +181,125 @@ mlx5_crypto_sym_session_clear(struct rte_cryptodev *dev,
 	DRV_LOG(DEBUG, "Session %p was cleared.", sess_private_data);
 }
 
+static int
+mlx5_crypto_queue_pair_release(struct rte_cryptodev *dev, uint16_t qp_id)
+{
+	struct mlx5_crypto_qp *qp = dev->data->queue_pairs[qp_id];
+
+	if (qp->qp_obj != NULL)
+		claim_zero(mlx5_devx_cmd_destroy(qp->qp_obj));
+	if (qp->umem_obj != NULL)
+		claim_zero(mlx5_glue->devx_umem_dereg(qp->umem_obj));
+	if (qp->umem_buf != NULL)
+		rte_free(qp->umem_buf);
+	mlx5_devx_cq_destroy(&qp->cq_obj);
+	rte_free(qp);
+	dev->data->queue_pairs[qp_id] = NULL;
+	return 0;
+}
+
+static int
+mlx5_crypto_qp2rts(struct mlx5_crypto_qp *qp)
+{
+	/*
+	 * In Order to configure self loopback, when calling these functions the
+	 * remote QP id that is used is the id of the same QP.
+	 */
+	if (mlx5_devx_cmd_modify_qp_state(qp->qp_obj, MLX5_CMD_OP_RST2INIT_QP,
+					  qp->qp_obj->id)) {
+		DRV_LOG(ERR, "Failed to modify QP to INIT state(%u).",
+			rte_errno);
+		return -1;
+	}
+	if (mlx5_devx_cmd_modify_qp_state(qp->qp_obj, MLX5_CMD_OP_INIT2RTR_QP,
+					  qp->qp_obj->id)) {
+		DRV_LOG(ERR, "Failed to modify QP to RTR state(%u).",
+			rte_errno);
+		return -1;
+	}
+	if (mlx5_devx_cmd_modify_qp_state(qp->qp_obj, MLX5_CMD_OP_RTR2RTS_QP,
+					  qp->qp_obj->id)) {
+		DRV_LOG(ERR, "Failed to modify QP to RTS state(%u).",
+			rte_errno);
+		return -1;
+	}
+	return 0;
+}
+
+static int
+mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
+			     const struct rte_cryptodev_qp_conf *qp_conf,
+			     int socket_id)
+{
+	struct mlx5_crypto_priv *priv = dev->data->dev_private;
+	struct mlx5_devx_qp_attr attr = {0};
+	struct mlx5_crypto_qp *qp;
+	uint16_t log_nb_desc = rte_log2_u32(qp_conf->nb_descriptors);
+	uint32_t umem_size = RTE_BIT32(log_nb_desc) *
+			      MLX5_CRYPTO_WQE_SET_SIZE +
+			      sizeof(*qp->db_rec) * 2;
+	uint32_t alloc_size = sizeof(*qp);
+	struct mlx5_devx_cq_attr cq_attr = {
+		.uar_page_id = mlx5_os_get_devx_uar_page_id(priv->uar),
+	};
+
+	alloc_size = RTE_ALIGN(alloc_size, RTE_CACHE_LINE_SIZE);
+	alloc_size += sizeof(struct rte_crypto_op *) * RTE_BIT32(log_nb_desc);
+	qp = rte_zmalloc_socket(__func__, alloc_size, RTE_CACHE_LINE_SIZE,
+				socket_id);
+	if (qp == NULL) {
+		DRV_LOG(ERR, "Failed to allocate QP memory.");
+		rte_errno = ENOMEM;
+		return -rte_errno;
+	}
+	if (mlx5_devx_cq_create(priv->ctx, &qp->cq_obj, log_nb_desc,
+				&cq_attr, socket_id) != 0) {
+		DRV_LOG(ERR, "Failed to create CQ.");
+		goto error;
+	}
+	qp->umem_buf = rte_zmalloc_socket(__func__, umem_size, 4096, socket_id);
+	if (qp->umem_buf == NULL) {
+		DRV_LOG(ERR, "Failed to allocate QP umem.");
+		rte_errno = ENOMEM;
+		goto error;
+	}
+	qp->umem_obj = mlx5_glue->devx_umem_reg(priv->ctx,
+					       (void *)(uintptr_t)qp->umem_buf,
+					       umem_size,
+					       IBV_ACCESS_LOCAL_WRITE);
+	if (qp->umem_obj == NULL) {
+		DRV_LOG(ERR, "Failed to register QP umem.");
+		goto error;
+	}
+	attr.pd = priv->pdn;
+	attr.uar_index = mlx5_os_get_devx_uar_page_id(priv->uar);
+	attr.cqn = qp->cq_obj.cq->id;
+	attr.log_page_size = rte_log2_u32(sysconf(_SC_PAGESIZE));
+	attr.rq_size =  0;
+	attr.sq_size = RTE_BIT32(log_nb_desc);
+	attr.dbr_umem_valid = 1;
+	attr.wq_umem_id = qp->umem_obj->umem_id;
+	attr.wq_umem_offset = 0;
+	attr.dbr_umem_id = qp->umem_obj->umem_id;
+	attr.dbr_address = RTE_BIT64(log_nb_desc) *
+			   MLX5_CRYPTO_WQE_SET_SIZE;
+	qp->qp_obj = mlx5_devx_cmd_create_qp(priv->ctx, &attr);
+	if (qp->qp_obj == NULL) {
+		DRV_LOG(ERR, "Failed to create QP(%u).", rte_errno);
+		goto error;
+	}
+	qp->db_rec = RTE_PTR_ADD(qp->umem_buf, (uintptr_t)attr.dbr_address);
+	if (mlx5_crypto_qp2rts(qp))
+		goto error;
+	qp->ops = (struct rte_crypto_op **)RTE_ALIGN((uintptr_t)(qp + 1),
+							   RTE_CACHE_LINE_SIZE);
+	dev->data->queue_pairs[qp_id] = qp;
+	return 0;
+error:
+	mlx5_crypto_queue_pair_release(dev, qp_id);
+	return -1;
+}
+
 static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_configure			= mlx5_crypto_dev_configure,
 	.dev_start			= NULL,
@@ -188,8 +308,8 @@ static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_infos_get			= mlx5_crypto_dev_infos_get,
 	.stats_get			= NULL,
 	.stats_reset			= NULL,
-	.queue_pair_setup		= NULL,
-	.queue_pair_release		= NULL,
+	.queue_pair_setup		= mlx5_crypto_queue_pair_setup,
+	.queue_pair_release		= mlx5_crypto_queue_pair_release,
 	.sym_session_get_size		= mlx5_crypto_sym_session_get_size,
 	.sym_session_configure		= mlx5_crypto_sym_session_configure,
 	.sym_session_clear		= mlx5_crypto_sym_session_clear,
diff --git a/drivers/crypto/mlx5/mlx5_crypto.h b/drivers/crypto/mlx5/mlx5_crypto.h
index 5e270d3d5a..f5313b89f2 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.h
+++ b/drivers/crypto/mlx5/mlx5_crypto.h
@@ -11,9 +11,11 @@
 #include <rte_cryptodev_pmd.h>
 
 #include <mlx5_common_utils.h>
+#include <mlx5_common_devx.h>
 
 #define MLX5_CRYPTO_DEK_HTABLE_SZ (1 << 11)
 #define MLX5_CRYPTO_KEY_LENGTH 80
+#define MLX5_CRYPTO_WQE_SET_SIZE 1024
 
 struct mlx5_crypto_priv {
 	TAILQ_ENTRY(mlx5_crypto_priv) next;
@@ -27,6 +29,15 @@ struct mlx5_crypto_priv {
 	struct rte_cryptodev_config dev_config;
 };
 
+struct mlx5_crypto_qp {
+	struct mlx5_devx_cq cq_obj;
+	struct mlx5_devx_obj *qp_obj;
+	struct mlx5dv_devx_umem *umem_obj;
+	void *umem_buf;
+	volatile uint32_t *db_rec;
+	struct rte_crypto_op **ops;
+};
+
 struct mlx5_crypto_dek {
 	struct mlx5_hlist_entry entry; /* Pointer to DEK hash list entry. */
 	struct mlx5_devx_obj *obj; /* Pointer to DEK DevX object. */
-- 
2.21.0

[dpdk-dev] [PATCH 21/24] crypto/mlx5: add dev stop and start operations

[Shiri Kuzin]

Add the dev_start function that is used to start a configured device.
Add the dev_stop function that is used to stop a configured device.

Both functions set the dev parameter as used and return 0.

Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/crypto/mlx5/mlx5_crypto.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index 07bc6e3a1c..cb5716ba2a 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -97,6 +97,19 @@ mlx5_crypto_dev_configure(struct rte_cryptodev *dev,
 	return 0;
 }
 
+static void
+mlx5_crypto_dev_stop(struct rte_cryptodev *dev)
+{
+	RTE_SET_USED(dev);
+}
+
+static int
+mlx5_crypto_dev_start(struct rte_cryptodev *dev)
+{
+	RTE_SET_USED(dev);
+	return 0;
+}
+
 static int
 mlx5_crypto_dev_close(struct rte_cryptodev *dev)
 {
@@ -302,8 +315,8 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 
 static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_configure			= mlx5_crypto_dev_configure,
-	.dev_start			= NULL,
-	.dev_stop			= NULL,
+	.dev_start			= mlx5_crypto_dev_start,
+	.dev_stop			= mlx5_crypto_dev_stop,
 	.dev_close			= mlx5_crypto_dev_close,
 	.dev_infos_get			= mlx5_crypto_dev_infos_get,
 	.stats_get			= NULL,
-- 
2.21.0

[dpdk-dev] [PATCH 22/24] crypto/mlx5: add memory region management

[Shiri Kuzin]

Mellanox user space drivers don't deal with physical addresses as part
of a memory protection mechanism.
The device translates the given virtual address to a physical address
using the given memory key as an address space identifier.
That's why any mbuf virtual address is moved directly to the HW
descriptor(WQE).

The mapping between the virtual address to the physical address is saved
in MR configured by the kernel to the HW.

Each MR has a key that should also be moved to the WQE by the SW.

When the SW sees an unmapped address, it extends the address range and
creates a MR using a system call.

Add memory region cache management:
	- 2 level cache per queue-pair - no locks.
	- 1 shared cache between all the queues using a lock.

Using this way, the MR key search per data-path address is optimized.

Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/crypto/mlx5/mlx5_crypto.c | 20 ++++++++++++++++++++
 drivers/crypto/mlx5/mlx5_crypto.h |  3 +++
 2 files changed, 23 insertions(+)

diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index cb5716ba2a..f71de5a724 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -205,6 +205,7 @@ mlx5_crypto_queue_pair_release(struct rte_cryptodev *dev, uint16_t qp_id)
 		claim_zero(mlx5_glue->devx_umem_dereg(qp->umem_obj));
 	if (qp->umem_buf != NULL)
 		rte_free(qp->umem_buf);
+	mlx5_mr_btree_free(&qp->mr_ctrl.cache_bh);
 	mlx5_devx_cq_destroy(&qp->cq_obj);
 	rte_free(qp);
 	dev->data->queue_pairs[qp_id] = NULL;
@@ -284,6 +285,13 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 		DRV_LOG(ERR, "Failed to register QP umem.");
 		goto error;
 	}
+	if (mlx5_mr_btree_init(&qp->mr_ctrl.cache_bh, MLX5_MR_BTREE_CACHE_N,
+			       priv->dev_config.socket_id) != 0) {
+		DRV_LOG(ERR, "Cannot allocate MR Btree for qp %u.",
+			(uint32_t)qp_id);
+		rte_errno = ENOMEM;
+		goto error;
+	}
 	attr.pd = priv->pdn;
 	attr.uar_index = mlx5_os_get_devx_uar_page_id(priv->uar);
 	attr.cqn = qp->cq_obj.cq->id;
@@ -472,6 +480,17 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 		claim_zero(mlx5_glue->close_device(priv->ctx));
 		return -1;
 	}
+	if (mlx5_mr_btree_init(&priv->mr_scache.cache,
+			     MLX5_MR_BTREE_CACHE_N * 2, rte_socket_id()) != 0) {
+		DRV_LOG(ERR, "Failed to allocate shared cache MR memory.");
+		mlx5_crypto_hw_global_release(priv);
+		rte_cryptodev_pmd_destroy(priv->crypto_dev);
+		claim_zero(mlx5_glue->close_device(priv->ctx));
+		rte_errno = ENOMEM;
+		return -rte_errno;
+	}
+	priv->mr_scache.reg_mr_cb = mlx5_common_verbs_reg_mr;
+	priv->mr_scache.dereg_mr_cb = mlx5_common_verbs_dereg_mr;
 	pthread_mutex_lock(&priv_list_lock);
 	TAILQ_INSERT_TAIL(&mlx5_crypto_priv_list, priv, next);
 	pthread_mutex_unlock(&priv_list_lock);
@@ -491,6 +510,7 @@ mlx5_crypto_pci_remove(struct rte_pci_device *pdev)
 		TAILQ_REMOVE(&mlx5_crypto_priv_list, priv, next);
 	pthread_mutex_unlock(&priv_list_lock);
 	if (priv) {
+		mlx5_mr_release_cache(&priv->mr_scache);
 		mlx5_crypto_hw_global_release(priv);
 		rte_cryptodev_pmd_destroy(priv->crypto_dev);
 		claim_zero(mlx5_glue->close_device(priv->ctx));
diff --git a/drivers/crypto/mlx5/mlx5_crypto.h b/drivers/crypto/mlx5/mlx5_crypto.h
index f5313b89f2..397267d249 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.h
+++ b/drivers/crypto/mlx5/mlx5_crypto.h
@@ -12,6 +12,7 @@
 
 #include <mlx5_common_utils.h>
 #include <mlx5_common_devx.h>
+#include <mlx5_common_mr.h>
 
 #define MLX5_CRYPTO_DEK_HTABLE_SZ (1 << 11)
 #define MLX5_CRYPTO_KEY_LENGTH 80
@@ -27,6 +28,7 @@ struct mlx5_crypto_priv {
 	struct ibv_pd *pd;
 	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
 	struct rte_cryptodev_config dev_config;
+	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
 };
 
 struct mlx5_crypto_qp {
@@ -36,6 +38,7 @@ struct mlx5_crypto_qp {
 	void *umem_buf;
 	volatile uint32_t *db_rec;
 	struct rte_crypto_op **ops;
+	struct mlx5_mr_ctrl mr_ctrl;
 };
 
 struct mlx5_crypto_dek {
-- 
2.21.0

[dpdk-dev] [PATCH 23/24] crypto/mlx5: create login object using DevX

[Shiri Kuzin]

To work with crypto engines that are marked with wrapped_import_method,
a login session is required.
A crypto login object needs to be created using DevX.

The crypto login object contains:
	- The credential pointer.
	- The import_KEK pointer to be used for all secured information
	  communicated in crypto commands (key fields), including the
	  provided credential in this command.
	- The credential secret, wrapped by the import_KEK indicated in
	  this command. Size includes 8 bytes IV for wrapping.

Added devargs for the required login values:
	- wcs_file - path to the file containing the credential.
	- import_kek_id - the import KEK pointer.
	- credential_id - the credential pointer.

Create the login DevX object in pci_probe function and destroy it in
pci_remove.
Destroying the crypto login object means logout.

Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/crypto/mlx5/mlx5_crypto.c | 89 +++++++++++++++++++++++++++++++
 drivers/crypto/mlx5/mlx5_crypto.h |  1 +
 2 files changed, 90 insertions(+)

diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index f71de5a724..25a435a999 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -398,6 +398,87 @@ mlx5_crypto_hw_global_prepare(struct mlx5_crypto_priv *priv)
 	return 0;
 }
 
+
+static int
+mlx5_crypto_args_check_handler(const char *key, const char *val, void *opaque)
+{
+	struct mlx5_devx_crypto_login_attr *attr = opaque;
+	unsigned long tmp;
+	FILE *file;
+	int ret;
+
+	if (strcmp(key, "class") == 0)
+		return 0;
+	if (strcmp(key, "wcs_file") == 0) {
+		file = fopen(val, "rb");
+		if (file == NULL) {
+			rte_errno = ENOTSUP;
+			return -rte_errno;
+		}
+		ret = fscanf(file, "%" RTE_STR(MLX5_CRYPTO_CREDENTIAL_SIZE) "s",
+			     &attr->credential[0]);
+		if (ret <= 0) {
+			fclose(file);
+			DRV_LOG(ERR, "Failed to read credential from file.");
+			rte_errno = EINVAL;
+			return -rte_errno;
+		}
+		fclose(file);
+		return 0;
+	}
+	errno = 0;
+	tmp = strtoul(val, NULL, 0);
+	if (errno) {
+		DRV_LOG(WARNING, "%s: \"%s\" is an invalid integer.", key, val);
+		return -errno;
+	}
+	if (strcmp(key, "import_kek_id") == 0)
+		attr->session_import_kek_ptr = (uint32_t)tmp;
+	else if (strcmp(key, "credential_id") == 0)
+		attr->credential_pointer = (uint32_t)tmp;
+	else
+		DRV_LOG(WARNING, "Invalid key %s.", key);
+	return 0;
+}
+
+static struct mlx5_devx_obj *
+mlx5_crypto_config_login(struct rte_devargs *devargs,
+			 struct ibv_context *ctx)
+{
+	/*
+	 * Set credential pointer and session import KEK pointer to a default
+	 * value of 0.
+	 */
+	struct mlx5_devx_crypto_login_attr attr = {
+			.credential_pointer = 0,
+			.session_import_kek_ptr = 0,
+	};
+	struct rte_kvargs *kvlist;
+
+	if (devargs == NULL) {
+		DRV_LOG(ERR,
+	"No login devargs in order to enable crypto operations in the device.");
+		rte_errno = EINVAL;
+		return NULL;
+	}
+	kvlist = rte_kvargs_parse(devargs->args, NULL);
+	if (kvlist == NULL) {
+		DRV_LOG(ERR, "Failed to parse devargs.");
+		rte_errno = EINVAL;
+		return NULL;
+	}
+	rte_kvargs_process(kvlist, NULL, mlx5_crypto_args_check_handler, &attr);
+	rte_kvargs_free(kvlist);
+	if (attr.credential == NULL) {
+		DRV_LOG(ERR,
+	"No login credential devarg in order to enable crypto operations "
+	"in the device.");
+		rte_errno = EINVAL;
+		return NULL;
+	}
+	return mlx5_devx_cmd_create_crypto_login_obj(ctx, &attr);
+}
+
 /**
  * DPDK callback to register a PCI device.
  *
@@ -419,6 +500,7 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 	struct ibv_device *ibv;
 	struct rte_cryptodev *crypto_dev;
 	struct ibv_context *ctx;
+	struct mlx5_devx_obj *login;
 	struct mlx5_crypto_priv *priv;
 	struct mlx5_hca_attr attr = { 0 };
 	struct rte_cryptodev_pmd_init_params init_params = {
@@ -457,6 +539,11 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 		rte_errno = ENOTSUP;
 		return -ENOTSUP;
 	}
+	login = mlx5_crypto_config_login(pci_dev->device.devargs, ctx);
+	if (login == NULL) {
+		DRV_LOG(ERR, "Failed to configure login.");
+		return -rte_errno;
+	}
 	crypto_dev = rte_cryptodev_pmd_create(ibv->name, &pci_dev->device,
 					&init_params);
 	if (crypto_dev == NULL) {
@@ -473,6 +560,7 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 	crypto_dev->driver_id = mlx5_crypto_driver_id;
 	priv = crypto_dev->data->dev_private;
 	priv->ctx = ctx;
+	priv->login_obj = login;
 	priv->pci_dev = pci_dev;
 	priv->crypto_dev = crypto_dev;
 	if (mlx5_crypto_hw_global_prepare(priv) != 0) {
@@ -513,6 +601,7 @@ mlx5_crypto_pci_remove(struct rte_pci_device *pdev)
 		mlx5_mr_release_cache(&priv->mr_scache);
 		mlx5_crypto_hw_global_release(priv);
 		rte_cryptodev_pmd_destroy(priv->crypto_dev);
+		claim_zero(mlx5_devx_cmd_destroy(priv->login_obj));
 		claim_zero(mlx5_glue->close_device(priv->ctx));
 	}
 	return 0;
diff --git a/drivers/crypto/mlx5/mlx5_crypto.h b/drivers/crypto/mlx5/mlx5_crypto.h
index 397267d249..0056d9e3e8 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.h
+++ b/drivers/crypto/mlx5/mlx5_crypto.h
@@ -29,6 +29,7 @@ struct mlx5_crypto_priv {
 	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
 	struct rte_cryptodev_config dev_config;
 	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
+	struct mlx5_devx_obj *login_obj;
 };
 
 struct mlx5_crypto_qp {
-- 
2.21.0

[dpdk-dev] [PATCH 24/24] crypto/mlx5: adjust to the multiple data unit API

[Shiri Kuzin]

In AES-XTS the data to be encrypted\decrypted does not have to be
in multiples of 16B size, the unit of data is called data-unit.

As a result of patch [1] a new field is added to the cipher capability,
called dataunit_set, where the devices can report the range of
supported data-unit sizes.

The new field enables saving the data-unit size in the session
structure to the block size pointer variable in order to support
several data-unit sizes.

[1] https://www.mail-archive.com/dev@dpdk.org/msg203590.html

Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
---
 drivers/crypto/mlx5/mlx5_crypto.c | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index 25a435a999..8810872aff 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -48,6 +48,11 @@ struct mlx5_crypto_session {
 	 * bsf_size, bsf_p_type, encryption_order and encryption standard,
 	 * saved in big endian format.
 	 */
+	uint32_t bsp_res;
+	/*
+	 * crypto_block_size_pointer and reserved 24 bits saved in big endian
+	 * format.
+	 */
 	uint32_t iv_offset:16;
 	/* Starting point for Initialisation Vector. */
 	struct mlx5_crypto_dek *dek; /* Pointer to dek struct. */
@@ -170,6 +175,26 @@ mlx5_crypto_sym_session_configure(struct rte_cryptodev *dev,
 			 MLX5_BSF_P_TYPE_CRYPTO << MLX5_BSF_P_TYPE_OFFSET |
 			 encryption_order << MLX5_ENCRYPTION_ORDER_OFFSET |
 			 MLX5_ENCRYPTION_STANDARD_AES_XTS);
+	switch (xform->cipher.dataunit_len) {
+	case RTE_CRYPTO_CIPHER_DATA_UNIT_LEN_512_BYTES:
+		sess_private_data->bsp_res = rte_cpu_to_be_32
+					     ((uint32_t)MLX5_BLOCK_SIZE_512B <<
+					     MLX5_BLOCK_SIZE_OFFSET);
+		break;
+	case RTE_CRYPTO_CIPHER_DATA_UNIT_LEN_4096_BYTES:
+		sess_private_data->bsp_res = rte_cpu_to_be_32
+					     ((uint32_t)MLX5_BLOCK_SIZE_4096B <<
+					     MLX5_BLOCK_SIZE_OFFSET);
+		break;
+	case RTE_CRYPTO_CIPHER_DATA_UNIT_LEN_1M_BYTES:
+		sess_private_data->bsp_res = rte_cpu_to_be_32
+					     ((uint32_t)MLX5_BLOCK_SIZE_1MB <<
+					     MLX5_BLOCK_SIZE_OFFSET);
+		break;
+	default:
+		DRV_LOG(ERR, "Cipher data unit length is not supported.");
+		return -ENOTSUP;
+	}
 	sess_private_data->iv_offset = cipher->iv.offset;
 	set_sym_session_private_data(session, dev->driver_id,
 				     sess_private_data);
-- 
2.21.0

Re: [dpdk-dev] [EXT] [PATCH 00/24] drivers: introduce mlx5 crypto PMD

[Akhil Goyal]

Hi Matan/Shiri
> Dekel Peled (13):
>   common/mlx5: remove redundant spaces in header file
>   common/mlx5: update GENEVE TLV OPT obj name
>   common/mlx5: optimize read of general obj type caps
>   common/mlx5: add HCA cap for AES-XTS crypto
>   common/mlx5: support general object DEK create op
>   common/mlx5: add crypto en field to MKEY context
>   common/mlx5: support umr en field in MKEY context
>   common/mlx5: support general obj IMPORT KEK create
>   common/mlx5: support general obj CRYPTO LOGIN create
>   common/mlx5: add crypto BSF struct and defines
>   common/mlx5: support general obj CREDENTIAL create
>   common/mlx5: add crypto register structs and defs
>   common/mlx5: support register write access
> 
> Shiri Kuzin (11):
>   common/mlx5: share hash list tool
>   common/mlx5: share get ib device match function
>   drivers: introduce mlx5 crypto PMD
>   crypto/mlx5: add DEK object management
>   crypto/mlx5: support session operations
>   crypto/mlx5: add basic operations
>   crypto/mlx5: support queue pairs operations
>   crypto/mlx5: add dev stop and start operations
>   crypto/mlx5: add memory region management
>   crypto/mlx5: create login object using DevX
>   crypto/mlx5: adjust to the multiple data unit API
> 
I would recommend this series be split into 2.
1. non-crypto code for common/mlx5 for next-net-mlx tree.
Eg: common/mlx5: remove redundant spaces in header file
       common/mlx5: update GENEVE TLV OPT obj name
       common/mlx5: optimize read of general obj type caps
       common/mlx5: support register write access
2. crypto PMD and crypto related changes in common/mlx5 for crypto tree.

Regards,
Akhil

[dpdk-dev] [PATCH v2 00/16] mlx5 common part for crypto driver

[Matan Azrad]

The crypto PMD will be supported on Nvidia ConnectX6
The crypto PMD will add the support of encryption and decryption using the AES-XTS symmetric algorithm.
The crypto PMD requires rdma-core and uses mlx5 DevX.
This is the mlx5 common part that added support for DevX commands needed for crypto driver.
Also includes PRM definitions.

v2:
Add data-path part.

Dekel Peled (12):
  common/mlx5: remove redundant spaces in header file
  common/mlx5: update GENEVE TLV OPT obj name
  common/mlx5: optimize read of general obj type caps
  common/mlx5: add HCA cap for AES-XTS crypto
  common/mlx5: support general object DEK create op
  common/mlx5: adjust DevX MKEY fields for crypto
  common/mlx5: support general obj IMPORT KEK create
  common/mlx5: support general obj CRYPTO LOGIN create
  common/mlx5: add crypto BSF struct and defines
  common/mlx5: support general obj CREDENTIAL create
  common/mlx5: add crypto register structs and defs
  common/mlx5: support register write access

Shiri Kuzin (2):
  common/mlx5: share hash list tool
  common/mlx5: share get ib device match function

Suanming Mou (2):
  common/mlx5: fix cypto bsf attr
  common/mlx5: add UMR and RDMA write WQE defines

 drivers/common/mlx5/linux/mlx5_common_os.c   |  30 +-
 drivers/common/mlx5/linux/mlx5_common_os.h   |   5 +
 drivers/common/mlx5/linux/mlx5_nl.c          |   2 +-
 drivers/common/mlx5/meson.build              |  15 +-
 drivers/common/mlx5/mlx5_common.c            |   2 +-
 drivers/common/mlx5/mlx5_common.h            |   2 +
 drivers/common/mlx5/mlx5_common_devx.c       |   2 +-
 drivers/common/mlx5/mlx5_common_log.h        |  21 +
 drivers/common/mlx5/mlx5_common_mp.c         |   2 +-
 drivers/common/mlx5/mlx5_common_mr.c         |   2 +-
 drivers/common/mlx5/mlx5_common_pci.c        |   4 +-
 drivers/common/mlx5/mlx5_common_utils.c      | 221 +++++++
 drivers/common/mlx5/mlx5_common_utils.h      | 202 +++++-
 drivers/common/mlx5/mlx5_devx_cmds.c         | 312 +++++++++-
 drivers/common/mlx5/mlx5_devx_cmds.h         |  66 ++
 drivers/common/mlx5/mlx5_malloc.c            |   2 +-
 drivers/common/mlx5/mlx5_prm.h               | 610 +++++++++++++------
 drivers/common/mlx5/version.map              |  24 +-
 drivers/common/mlx5/windows/mlx5_common_os.c |   2 +-
 drivers/common/mlx5/windows/mlx5_glue.c      |   2 +-
 drivers/compress/mlx5/mlx5_compress.c        |  30 +-
 drivers/net/mlx5/mlx5_utils.c                | 209 -------
 drivers/net/mlx5/mlx5_utils.h                | 194 +-----
 drivers/regex/mlx5/mlx5_regex.c              |  30 +-
 drivers/vdpa/mlx5/mlx5_vdpa.c                |  32 +-
 25 files changed, 1317 insertions(+), 706 deletions(-)
 create mode 100644 drivers/common/mlx5/mlx5_common_log.h
 create mode 100644 drivers/common/mlx5/mlx5_common_utils.c

-- 
2.25.1

[dpdk-dev] [PATCH v2 01/16] common/mlx5: remove redundant spaces in header file

[Matan Azrad]

From: Dekel Peled <dekelp@nvidia.com>

File drivers/common/mlx5/mlx5_prm.h includes structs representing
data items as defined in PRM document.
Some of these structs were copied as-is from kernel file mlx5_ifc.h.
As result the structs are not all aligned with the same spacing.

This patch removes redundant spaces and new lines from several structs,
to align all structs in mlx5_prm.h to the same format.

Signed-off-by: Dekel Peled <dekelp@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/mlx5_prm.h | 334 +++++++++++++++------------------
 1 file changed, 155 insertions(+), 179 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h
index efa5ae67bf..da1510ac1e 100644
--- a/drivers/common/mlx5/mlx5_prm.h
+++ b/drivers/common/mlx5/mlx5_prm.h
@@ -953,163 +953,139 @@ enum {
 
 /* Flow counters. */
 struct mlx5_ifc_alloc_flow_counter_out_bits {
-	u8         status[0x8];
-	u8         reserved_at_8[0x18];
-	u8         syndrome[0x20];
-	u8         flow_counter_id[0x20];
-	u8         reserved_at_60[0x20];
+	u8 status[0x8];
+	u8 reserved_at_8[0x18];
+	u8 syndrome[0x20];
+	u8 flow_counter_id[0x20];
+	u8 reserved_at_60[0x20];
 };
 
 struct mlx5_ifc_alloc_flow_counter_in_bits {
-	u8         opcode[0x10];
-	u8         reserved_at_10[0x10];
-	u8         reserved_at_20[0x10];
-	u8         op_mod[0x10];
-	u8         flow_counter_id[0x20];
-	u8         reserved_at_40[0x18];
-	u8         flow_counter_bulk[0x8];
+	u8 opcode[0x10];
+	u8 reserved_at_10[0x10];
+	u8 reserved_at_20[0x10];
+	u8 op_mod[0x10];
+	u8 flow_counter_id[0x20];
+	u8 reserved_at_40[0x18];
+	u8 flow_counter_bulk[0x8];
 };
 
 struct mlx5_ifc_dealloc_flow_counter_out_bits {
-	u8         status[0x8];
-	u8         reserved_at_8[0x18];
-	u8         syndrome[0x20];
-	u8         reserved_at_40[0x40];
+	u8 status[0x8];
+	u8 reserved_at_8[0x18];
+	u8 syndrome[0x20];
+	u8 reserved_at_40[0x40];
 };
 
 struct mlx5_ifc_dealloc_flow_counter_in_bits {
-	u8         opcode[0x10];
-	u8         reserved_at_10[0x10];
-	u8         reserved_at_20[0x10];
-	u8         op_mod[0x10];
-	u8         flow_counter_id[0x20];
-	u8         reserved_at_60[0x20];
+	u8 opcode[0x10];
+	u8 reserved_at_10[0x10];
+	u8 reserved_at_20[0x10];
+	u8 op_mod[0x10];
+	u8 flow_counter_id[0x20];
+	u8 reserved_at_60[0x20];
 };
 
 struct mlx5_ifc_traffic_counter_bits {
-	u8         packets[0x40];
-	u8         octets[0x40];
+	u8 packets[0x40];
+	u8 octets[0x40];
 };
 
 struct mlx5_ifc_query_flow_counter_out_bits {
-	u8         status[0x8];
-	u8         reserved_at_8[0x18];
-	u8         syndrome[0x20];
-	u8         reserved_at_40[0x40];
+	u8 status[0x8];
+	u8 reserved_at_8[0x18];
+	u8 syndrome[0x20];
+	u8 reserved_at_40[0x40];
 	struct mlx5_ifc_traffic_counter_bits flow_statistics[];
 };
 
 struct mlx5_ifc_query_flow_counter_in_bits {
-	u8         opcode[0x10];
-	u8         reserved_at_10[0x10];
-	u8         reserved_at_20[0x10];
-	u8         op_mod[0x10];
-	u8         reserved_at_40[0x20];
-	u8         mkey[0x20];
-	u8         address[0x40];
-	u8         clear[0x1];
-	u8         dump_to_memory[0x1];
-	u8         num_of_counters[0x1e];
-	u8         flow_counter_id[0x20];
+	u8 opcode[0x10];
+	u8 reserved_at_10[0x10];
+	u8 reserved_at_20[0x10];
+	u8 op_mod[0x10];
+	u8 reserved_at_40[0x20];
+	u8 mkey[0x20];
+	u8 address[0x40];
+	u8 clear[0x1];
+	u8 dump_to_memory[0x1];
+	u8 num_of_counters[0x1e];
+	u8 flow_counter_id[0x20];
 };
 
 #define MLX5_MAX_KLM_BYTE_COUNT 0x80000000u
 #define MLX5_MIN_KLM_FIXED_BUFFER_SIZE 0x1000u
 
-
 struct mlx5_ifc_klm_bits {
-	u8         byte_count[0x20];
-	u8         mkey[0x20];
-	u8         address[0x40];
+	u8 byte_count[0x20];
+	u8 mkey[0x20];
+	u8 address[0x40];
 };
 
 struct mlx5_ifc_mkc_bits {
-	u8         reserved_at_0[0x1];
-	u8         free[0x1];
-	u8         reserved_at_2[0x1];
-	u8         access_mode_4_2[0x3];
-	u8         reserved_at_6[0x7];
-	u8         relaxed_ordering_write[0x1];
-	u8         reserved_at_e[0x1];
-	u8         small_fence_on_rdma_read_response[0x1];
-	u8         umr_en[0x1];
-	u8         a[0x1];
-	u8         rw[0x1];
-	u8         rr[0x1];
-	u8         lw[0x1];
-	u8         lr[0x1];
-	u8         access_mode_1_0[0x2];
-	u8         reserved_at_18[0x8];
-
-	u8         qpn[0x18];
-	u8         mkey_7_0[0x8];
-
-	u8         reserved_at_40[0x20];
-
-	u8         length64[0x1];
-	u8         bsf_en[0x1];
-	u8         sync_umr[0x1];
-	u8         reserved_at_63[0x2];
-	u8         expected_sigerr_count[0x1];
-	u8         reserved_at_66[0x1];
-	u8         en_rinval[0x1];
-	u8         pd[0x18];
-
-	u8         start_addr[0x40];
-
-	u8         len[0x40];
-
-	u8         bsf_octword_size[0x20];
-
-	u8         reserved_at_120[0x80];
-
-	u8         translations_octword_size[0x20];
-
-	u8         reserved_at_1c0[0x19];
-	u8		   relaxed_ordering_read[0x1];
-	u8		   reserved_at_1da[0x1];
-	u8         log_page_size[0x5];
-
-	u8         reserved_at_1e0[0x20];
+	u8 reserved_at_0[0x1];
+	u8 free[0x1];
+	u8 reserved_at_2[0x1];
+	u8 access_mode_4_2[0x3];
+	u8 reserved_at_6[0x7];
+	u8 relaxed_ordering_write[0x1];
+	u8 reserved_at_e[0x1];
+	u8 small_fence_on_rdma_read_response[0x1];
+	u8 umr_en[0x1];
+	u8 a[0x1];
+	u8 rw[0x1];
+	u8 rr[0x1];
+	u8 lw[0x1];
+	u8 lr[0x1];
+	u8 access_mode_1_0[0x2];
+	u8 reserved_at_18[0x8];
+	u8 qpn[0x18];
+	u8 mkey_7_0[0x8];
+	u8 reserved_at_40[0x20];
+	u8 length64[0x1];
+	u8 bsf_en[0x1];
+	u8 sync_umr[0x1];
+	u8 reserved_at_63[0x2];
+	u8 expected_sigerr_count[0x1];
+	u8 reserved_at_66[0x1];
+	u8 en_rinval[0x1];
+	u8 pd[0x18];
+	u8 start_addr[0x40];
+	u8 len[0x40];
+	u8 bsf_octword_size[0x20];
+	u8 reserved_at_120[0x80];
+	u8 translations_octword_size[0x20];
+	u8 reserved_at_1c0[0x19];
+	u8 relaxed_ordering_read[0x1];
+	u8 reserved_at_1da[0x1];
+	u8 log_page_size[0x5];
+	u8 reserved_at_1e0[0x20];
 };
 
 struct mlx5_ifc_create_mkey_out_bits {
-	u8         status[0x8];
-	u8         reserved_at_8[0x18];
-
-	u8         syndrome[0x20];
-
-	u8         reserved_at_40[0x8];
-	u8         mkey_index[0x18];
-
-	u8         reserved_at_60[0x20];
+	u8 status[0x8];
+	u8 reserved_at_8[0x18];
+	u8 syndrome[0x20];
+	u8 reserved_at_40[0x8];
+	u8 mkey_index[0x18];
+	u8 reserved_at_60[0x20];
 };
 
 struct mlx5_ifc_create_mkey_in_bits {
-	u8         opcode[0x10];
-	u8         reserved_at_10[0x10];
-
-	u8         reserved_at_20[0x10];
-	u8         op_mod[0x10];
-
-	u8         reserved_at_40[0x20];
-
-	u8         pg_access[0x1];
-	u8         reserved_at_61[0x1f];
-
+	u8 opcode[0x10];
+	u8 reserved_at_10[0x10];
+	u8 reserved_at_20[0x10];
+	u8 op_mod[0x10];
+	u8 reserved_at_40[0x20];
+	u8 pg_access[0x1];
+	u8 reserved_at_61[0x1f];
 	struct mlx5_ifc_mkc_bits memory_key_mkey_entry;
-
-	u8         reserved_at_280[0x80];
-
-	u8         translations_octword_actual_size[0x20];
-
-	u8         mkey_umem_id[0x20];
-
-	u8         mkey_umem_offset[0x40];
-
-	u8         reserved_at_380[0x500];
-
-	u8         klm_pas_mtt[][0x20];
+	u8 reserved_at_280[0x80];
+	u8 translations_octword_actual_size[0x20];
+	u8 mkey_umem_id[0x20];
+	u8 mkey_umem_offset[0x40];
+	u8 reserved_at_380[0x500];
+	u8 klm_pas_mtt[][0x20];
 };
 
 enum {
@@ -2272,27 +2248,27 @@ enum {
 };
 
 struct mlx5_ifc_flow_meter_parameters_bits {
-	u8         valid[0x1];			// 00h
-	u8         bucket_overflow[0x1];
-	u8         start_color[0x2];
-	u8         both_buckets_on_green[0x1];
-	u8         meter_mode[0x2];
-	u8         reserved_at_1[0x19];
-	u8         reserved_at_2[0x20]; //04h
-	u8         reserved_at_3[0x3];
-	u8         cbs_exponent[0x5];		// 08h
-	u8         cbs_mantissa[0x8];
-	u8         reserved_at_4[0x3];
-	u8         cir_exponent[0x5];
-	u8         cir_mantissa[0x8];
-	u8         reserved_at_5[0x20];		// 0Ch
-	u8         reserved_at_6[0x3];
-	u8         ebs_exponent[0x5];		// 10h
-	u8         ebs_mantissa[0x8];
-	u8         reserved_at_7[0x3];
-	u8         eir_exponent[0x5];
-	u8         eir_mantissa[0x8];
-	u8         reserved_at_8[0x60];		// 14h-1Ch
+	u8 valid[0x1];
+	u8 bucket_overflow[0x1];
+	u8 start_color[0x2];
+	u8 both_buckets_on_green[0x1];
+	u8 meter_mode[0x2];
+	u8 reserved_at_1[0x19];
+	u8 reserved_at_2[0x20];
+	u8 reserved_at_3[0x3];
+	u8 cbs_exponent[0x5];
+	u8 cbs_mantissa[0x8];
+	u8 reserved_at_4[0x3];
+	u8 cir_exponent[0x5];
+	u8 cir_mantissa[0x8];
+	u8 reserved_at_5[0x20];
+	u8 reserved_at_6[0x3];
+	u8 ebs_exponent[0x5];
+	u8 ebs_mantissa[0x8];
+	u8 reserved_at_7[0x3];
+	u8 eir_exponent[0x5];
+	u8 eir_mantissa[0x8];
+	u8 reserved_at_8[0x60];
 };
 #define MLX5_IFC_FLOW_METER_PARAM_MASK UINT64_C(0x80FFFFFF)
 #define MLX5_IFC_FLOW_METER_DISABLE_CBS_CIR_VAL 0x14BF00C8
@@ -2359,46 +2335,46 @@ struct mlx5_ifc_cqc_bits {
 };
 
 struct mlx5_ifc_health_buffer_bits {
-	u8         reserved_0[0x100];
-	u8         assert_existptr[0x20];
-	u8         assert_callra[0x20];
-	u8         reserved_1[0x40];
-	u8         fw_version[0x20];
-	u8         hw_id[0x20];
-	u8         reserved_2[0x20];
-	u8         irisc_index[0x8];
-	u8         synd[0x8];
-	u8         ext_synd[0x10];
+	u8 reserved_0[0x100];
+	u8 assert_existptr[0x20];
+	u8 assert_callra[0x20];
+	u8 reserved_1[0x40];
+	u8 fw_version[0x20];
+	u8 hw_id[0x20];
+	u8 reserved_2[0x20];
+	u8 irisc_index[0x8];
+	u8 synd[0x8];
+	u8 ext_synd[0x10];
 };
 
 struct mlx5_ifc_initial_seg_bits {
-	u8         fw_rev_minor[0x10];
-	u8         fw_rev_major[0x10];
-	u8         cmd_interface_rev[0x10];
-	u8         fw_rev_subminor[0x10];
-	u8         reserved_0[0x40];
-	u8         cmdq_phy_addr_63_32[0x20];
-	u8         cmdq_phy_addr_31_12[0x14];
-	u8         reserved_1[0x2];
-	u8         nic_interface[0x2];
-	u8         log_cmdq_size[0x4];
-	u8         log_cmdq_stride[0x4];
-	u8         command_doorbell_vector[0x20];
-	u8         reserved_2[0xf00];
-	u8         initializing[0x1];
-	u8         nic_interface_supported[0x7];
-	u8         reserved_4[0x18];
+	u8 fw_rev_minor[0x10];
+	u8 fw_rev_major[0x10];
+	u8 cmd_interface_rev[0x10];
+	u8 fw_rev_subminor[0x10];
+	u8 reserved_0[0x40];
+	u8 cmdq_phy_addr_63_32[0x20];
+	u8 cmdq_phy_addr_31_12[0x14];
+	u8 reserved_1[0x2];
+	u8 nic_interface[0x2];
+	u8 log_cmdq_size[0x4];
+	u8 log_cmdq_stride[0x4];
+	u8 command_doorbell_vector[0x20];
+	u8 reserved_2[0xf00];
+	u8 initializing[0x1];
+	u8 nic_interface_supported[0x7];
+	u8 reserved_4[0x18];
 	struct mlx5_ifc_health_buffer_bits health_buffer;
-	u8         no_dram_nic_offset[0x20];
-	u8         reserved_5[0x6de0];
-	u8         internal_timer_h[0x20];
-	u8         internal_timer_l[0x20];
-	u8         reserved_6[0x20];
-	u8         reserved_7[0x1f];
-	u8         clear_int[0x1];
-	u8         health_syndrome[0x8];
-	u8         health_counter[0x18];
-	u8         reserved_8[0x17fc0];
+	u8 no_dram_nic_offset[0x20];
+	u8 reserved_5[0x6de0];
+	u8 internal_timer_h[0x20];
+	u8 internal_timer_l[0x20];
+	u8 reserved_6[0x20];
+	u8 reserved_7[0x1f];
+	u8 clear_int[0x1];
+	u8 health_syndrome[0x8];
+	u8 health_counter[0x18];
+	u8 reserved_8[0x17fc0];
 };
 
 struct mlx5_ifc_create_cq_out_bits {
-- 
2.25.1

[dpdk-dev] [PATCH v2 02/16] common/mlx5: update GENEVE TLV OPT obj name

[Matan Azrad]

From: Dekel Peled <dekelp@nvidia.com>

Rename MLX5_OBJ_TYPE_GENEVE_TLV_OPT as
MLX5_GENERAL_OBJ_TYPE_GENEVE_TLV_OPT, to align with other general
objects names.

Signed-off-by: Dekel Peled <dekelp@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/mlx5_devx_cmds.c | 2 +-
 drivers/common/mlx5/mlx5_prm.h       | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c
index 79fff6457c..831175efc5 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.c
+++ b/drivers/common/mlx5/mlx5_devx_cmds.c
@@ -2265,7 +2265,7 @@ mlx5_devx_cmd_create_geneve_tlv_option(void *ctx,
 	MLX5_SET(general_obj_in_cmd_hdr, hdr, opcode,
 			MLX5_CMD_OP_CREATE_GENERAL_OBJECT);
 	MLX5_SET(general_obj_in_cmd_hdr, hdr, obj_type,
-			MLX5_OBJ_TYPE_GENEVE_TLV_OPT);
+		 MLX5_GENERAL_OBJ_TYPE_GENEVE_TLV_OPT);
 	MLX5_SET(geneve_tlv_option, opt, option_class,
 			rte_be_to_cpu_16(class));
 	MLX5_SET(geneve_tlv_option, opt, option_type, type);
diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h
index da1510ac1e..2e5e42f6e9 100644
--- a/drivers/common/mlx5/mlx5_prm.h
+++ b/drivers/common/mlx5/mlx5_prm.h
@@ -1108,7 +1108,7 @@ enum {
 #define MLX5_GENERAL_OBJ_TYPES_CAP_FLOW_METER_ASO \
 			(1ULL << MLX5_GENERAL_OBJ_TYPE_FLOW_METER_ASO)
 #define MLX5_GENERAL_OBJ_TYPES_CAP_GENEVE_TLV_OPT \
-			(1ULL << MLX5_OBJ_TYPE_GENEVE_TLV_OPT)
+			(1ULL << MLX5_GENERAL_OBJ_TYPE_GENEVE_TLV_OPT)
 
 enum {
 	MLX5_HCA_CAP_OPMOD_GET_MAX   = 0,
@@ -2402,7 +2402,7 @@ struct mlx5_ifc_create_cq_in_bits {
 };
 
 enum {
-	MLX5_OBJ_TYPE_GENEVE_TLV_OPT = 0x000b,
+	MLX5_GENERAL_OBJ_TYPE_GENEVE_TLV_OPT = 0x000b,
 	MLX5_GENERAL_OBJ_TYPE_VIRTQ = 0x000d,
 	MLX5_GENERAL_OBJ_TYPE_VIRTIO_Q_COUNTERS = 0x001c,
 	MLX5_GENERAL_OBJ_TYPE_FLEX_PARSE_GRAPH = 0x0022,
-- 
2.25.1

[dpdk-dev] [PATCH v2 03/16] common/mlx5: optimize read of general obj type caps

[Matan Azrad]

From: Dekel Peled <dekelp@nvidia.com>

General object types support is indicated in bitmap general_obj_types,
which is part of HCA capabilities list.
Currently this bitmap is read multiple times, and each time a different
bit is extracted.

This patch optimizes the code, reading the bitmap once into a local
variable, and then extracting the required bits.

Signed-off-by: Dekel Peled <dekelp@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/mlx5_devx_cmds.c | 21 ++++++++++++++++-----
 1 file changed, 16 insertions(+), 5 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c
index 831175efc5..a0bf0d3009 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.c
+++ b/drivers/common/mlx5/mlx5_devx_cmds.c
@@ -647,6 +647,7 @@ mlx5_devx_cmd_query_hca_attr(void *ctx,
 	uint32_t out[MLX5_ST_SZ_DW(query_hca_cap_out)] = {0};
 	void *hcattr;
 	int status, syndrome, rc, i;
+	uint64_t general_obj_types_supported = 0;
 
 	MLX5_SET(query_hca_cap_in, in, opcode, MLX5_CMD_OP_QUERY_HCA_CAP);
 	MLX5_SET(query_hca_cap_in, in, op_mod,
@@ -725,12 +726,22 @@ mlx5_devx_cmd_query_hca_attr(void *ctx,
 	attr->regex = MLX5_GET(cmd_hca_cap, hcattr, regexp);
 	attr->regexp_num_of_engines = MLX5_GET(cmd_hca_cap, hcattr,
 					       regexp_num_of_engines);
-	attr->flow_hit_aso = !!(MLX5_GET64(cmd_hca_cap, hcattr,
-					   general_obj_types) &
+	/* Read the general_obj_types bitmap and extract the relevant bits. */
+	general_obj_types_supported = MLX5_GET64(cmd_hca_cap, hcattr,
+						 general_obj_types);
+	attr->vdpa.valid = !!(general_obj_types_supported &
+			      MLX5_GENERAL_OBJ_TYPES_CAP_VIRTQ_NET_Q);
+	attr->vdpa.queue_counters_valid =
+			!!(general_obj_types_supported &
+			   MLX5_GENERAL_OBJ_TYPES_CAP_VIRTIO_Q_COUNTERS);
+	attr->parse_graph_flex_node =
+			!!(general_obj_types_supported &
+			   MLX5_GENERAL_OBJ_TYPES_CAP_PARSE_GRAPH_FLEX_NODE);
+	attr->flow_hit_aso = !!(general_obj_types_supported &
 				MLX5_GENERAL_OBJ_TYPES_CAP_FLOW_HIT_ASO);
-	attr->geneve_tlv_opt = !!(MLX5_GET64(cmd_hca_cap, hcattr,
-					   general_obj_types) &
-				MLX5_GENERAL_OBJ_TYPES_CAP_GENEVE_TLV_OPT);
+	attr->geneve_tlv_opt = !!(general_obj_types_supported &
+				  MLX5_GENERAL_OBJ_TYPES_CAP_GENEVE_TLV_OPT);
+	/* Add reading of other GENERAL_OBJ_TYPES_CAP bits above this line. */
 	attr->log_max_cq = MLX5_GET(cmd_hca_cap, hcattr, log_max_cq);
 	attr->log_max_qp = MLX5_GET(cmd_hca_cap, hcattr, log_max_qp);
 	attr->log_max_cq_sz = MLX5_GET(cmd_hca_cap, hcattr, log_max_cq_sz);
-- 
2.25.1

[dpdk-dev] [PATCH v2 04/16] common/mlx5: add HCA cap for AES-XTS crypto

[Matan Azrad]

From: Dekel Peled <dekelp@nvidia.com>

Update the PRM structure and HCA capabilities reading, to include
relevant capabilities for AES-XTS crypto.

Signed-off-by: Dekel Peled <dekelp@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/mlx5_devx_cmds.c | 3 +++
 drivers/common/mlx5/mlx5_devx_cmds.h | 2 ++
 drivers/common/mlx5/mlx5_prm.h       | 5 ++++-
 3 files changed, 9 insertions(+), 1 deletion(-)

diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c
index a0bf0d3009..7ca767944e 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.c
+++ b/drivers/common/mlx5/mlx5_devx_cmds.c
@@ -771,6 +771,9 @@ mlx5_devx_cmd_query_hca_attr(void *ctx,
 		MLX5_GET(cmd_hca_cap, hcattr, umr_indirect_mkey_disabled);
 	attr->umr_modify_entity_size_disabled =
 		MLX5_GET(cmd_hca_cap, hcattr, umr_modify_entity_size_disabled);
+	attr->crypto = MLX5_GET(cmd_hca_cap, hcattr, crypto);
+	if (attr->crypto)
+		attr->aes_xts = MLX5_GET(cmd_hca_cap, hcattr, aes_xts);
 	if (attr->qos.sup) {
 		MLX5_SET(query_hca_cap_in, in, op_mod,
 			 MLX5_GET_HCA_CAP_OP_MOD_QOS_CAP |
diff --git a/drivers/common/mlx5/mlx5_devx_cmds.h b/drivers/common/mlx5/mlx5_devx_cmds.h
index 870bdb6b30..28ade5bbc4 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.h
+++ b/drivers/common/mlx5/mlx5_devx_cmds.h
@@ -137,6 +137,8 @@ struct mlx5_hca_attr {
 	uint32_t qp_ts_format:2;
 	uint32_t regex:1;
 	uint32_t reg_c_preserve:1;
+	uint32_t crypto:1; /* Crypto engine is supported. */
+	uint32_t aes_xts:1; /* AES-XTS crypto is supported. */
 	uint32_t regexp_num_of_engines;
 	uint32_t log_max_ft_sampler_num:8;
 	uint32_t geneve_tlv_opt;
diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h
index 2e5e42f6e9..a8fbfbb0f5 100644
--- a/drivers/common/mlx5/mlx5_prm.h
+++ b/drivers/common/mlx5/mlx5_prm.h
@@ -1427,7 +1427,10 @@ struct mlx5_ifc_cmd_hca_cap_bits {
 	u8 sq_ts_format[0x2];
 	u8 rq_ts_format[0x2];
 	u8 reserved_at_444[0x1C];
-	u8 reserved_at_460[0x10];
+	u8 reserved_at_460[0x8];
+	u8 aes_xts[0x1];
+	u8 crypto[0x1];
+	u8 reserved_at_46a[0x6];
 	u8 max_num_eqs[0x10];
 	u8 reserved_at_480[0x3];
 	u8 log_max_l2_table[0x5];
-- 
2.25.1

[dpdk-dev] [PATCH v2 05/16] common/mlx5: support general object DEK create op

[Matan Azrad]

From: Dekel Peled <dekelp@nvidia.com>

Data Encryption Keys (DEKs) are the keys used for data
encryption/decryption operations.

Add reading of DEK support capability.
Add function to create general object type DEK, using DevX API.

Signed-off-by: Dekel Peled <dekelp@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/mlx5_devx_cmds.c | 53 ++++++++++++++++++++++++++++
 drivers/common/mlx5/mlx5_devx_cmds.h | 17 +++++++++
 drivers/common/mlx5/mlx5_prm.h       | 39 ++++++++++++++++++++
 drivers/common/mlx5/version.map      |  1 +
 4 files changed, 110 insertions(+)

diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c
index 7ca767944e..742c82cca4 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.c
+++ b/drivers/common/mlx5/mlx5_devx_cmds.c
@@ -741,6 +741,8 @@ mlx5_devx_cmd_query_hca_attr(void *ctx,
 				MLX5_GENERAL_OBJ_TYPES_CAP_FLOW_HIT_ASO);
 	attr->geneve_tlv_opt = !!(general_obj_types_supported &
 				  MLX5_GENERAL_OBJ_TYPES_CAP_GENEVE_TLV_OPT);
+	attr->dek = !!(general_obj_types_supported &
+		       MLX5_GENERAL_OBJ_TYPES_CAP_DEK);
 	/* Add reading of other GENERAL_OBJ_TYPES_CAP bits above this line. */
 	attr->log_max_cq = MLX5_GET(cmd_hca_cap, hcattr, log_max_cq);
 	attr->log_max_qp = MLX5_GET(cmd_hca_cap, hcattr, log_max_qp);
@@ -2397,3 +2399,54 @@ mlx5_devx_cmd_queue_counter_query(struct mlx5_devx_obj *dcs, int clear,
 	*out_of_buffers = MLX5_GET(query_q_counter_out, out, out_of_buffer);
 	return 0;
 }
+
+/**
+ * Create general object of type DEK using DevX API.
+ *
+ * @param[in] ctx
+ *   Context returned from mlx5 open_device() glue function.
+ * @param [in] attr
+ *   Pointer to DEK attributes structure.
+ *
+ * @return
+ *   The DevX object created, NULL otherwise and rte_errno is set.
+ */
+struct mlx5_devx_obj *
+mlx5_devx_cmd_create_dek_obj(void *ctx, struct mlx5_devx_dek_attr *attr)
+{
+	uint32_t in[MLX5_ST_SZ_DW(create_dek_in)] = {0};
+	uint32_t out[MLX5_ST_SZ_DW(general_obj_out_cmd_hdr)] = {0};
+	struct mlx5_devx_obj *dek_obj = NULL;
+	void *ptr = NULL, *key_addr = NULL;
+
+	dek_obj = mlx5_malloc(MLX5_MEM_ZERO, sizeof(*dek_obj),
+			      0, SOCKET_ID_ANY);
+	if (dek_obj == NULL) {
+		DRV_LOG(ERR, "Failed to allocate DEK object data");
+		rte_errno = ENOMEM;
+		return NULL;
+	}
+	ptr = MLX5_ADDR_OF(create_dek_in, in, hdr);
+	MLX5_SET(general_obj_in_cmd_hdr, ptr, opcode,
+		 MLX5_CMD_OP_CREATE_GENERAL_OBJECT);
+	MLX5_SET(general_obj_in_cmd_hdr, ptr, obj_type,
+		 MLX5_GENERAL_OBJ_TYPE_DEK);
+	ptr = MLX5_ADDR_OF(create_dek_in, in, dek);
+	MLX5_SET(dek, ptr, key_size, attr->key_size);
+	MLX5_SET(dek, ptr, has_keytag, attr->has_keytag);
+	MLX5_SET(dek, ptr, key_purpose, attr->key_purpose);
+	MLX5_SET(dek, ptr, pd, attr->pd);
+	MLX5_SET64(dek, ptr, opaque, attr->opaque);
+	key_addr = MLX5_ADDR_OF(dek, ptr, key);
+	memcpy(key_addr, (void *)(attr->key), MLX5_CRYPTO_KEY_MAX_SIZE);
+	dek_obj->obj = mlx5_glue->devx_obj_create(ctx, in, sizeof(in),
+						  out, sizeof(out));
+	if (dek_obj->obj == NULL) {
+		rte_errno = errno;
+		DRV_LOG(ERR, "Failed to create DEK obj using DevX.");
+		mlx5_free(dek_obj);
+		return NULL;
+	}
+	dek_obj->id = MLX5_GET(general_obj_out_cmd_hdr, out, obj_id);
+	return dek_obj;
+}
diff --git a/drivers/common/mlx5/mlx5_devx_cmds.h b/drivers/common/mlx5/mlx5_devx_cmds.h
index 28ade5bbc4..b9ff7ab87d 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.h
+++ b/drivers/common/mlx5/mlx5_devx_cmds.h
@@ -139,6 +139,7 @@ struct mlx5_hca_attr {
 	uint32_t reg_c_preserve:1;
 	uint32_t crypto:1; /* Crypto engine is supported. */
 	uint32_t aes_xts:1; /* AES-XTS crypto is supported. */
+	uint32_t dek:1; /* General obj type DEK is supported. */
 	uint32_t regexp_num_of_engines;
 	uint32_t log_max_ft_sampler_num:8;
 	uint32_t geneve_tlv_opt;
@@ -435,6 +436,18 @@ struct mlx5_devx_graph_node_attr {
 	struct mlx5_devx_graph_arc_attr out[MLX5_GRAPH_NODE_ARC_NUM];
 };
 
+/* Encryption key size is up to 1024 bit, 128 bytes. */
+#define MLX5_CRYPTO_KEY_MAX_SIZE	128
+
+struct mlx5_devx_dek_attr {
+	uint32_t key_size:4;
+	uint32_t has_keytag:1;
+	uint32_t key_purpose:4;
+	uint32_t pd:24;
+	uint64_t opaque;
+	uint8_t key[MLX5_CRYPTO_KEY_MAX_SIZE];
+};
+
 /* mlx5_devx_cmds.c */
 
 __rte_internal
@@ -587,4 +600,8 @@ int mlx5_devx_cmd_queue_counter_query(struct mlx5_devx_obj *dcs, int clear,
 __rte_internal
 struct mlx5_devx_obj *mlx5_devx_cmd_create_flow_meter_aso_obj(void *ctx,
 					uint32_t pd, uint32_t log_obj_size);
+__rte_internal
+struct mlx5_devx_obj *
+mlx5_devx_cmd_create_dek_obj(void *ctx, struct mlx5_devx_dek_attr *attr);
+
 #endif /* RTE_PMD_MLX5_DEVX_CMDS_H_ */
diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h
index a8fbfbb0f5..bc9f58ad03 100644
--- a/drivers/common/mlx5/mlx5_prm.h
+++ b/drivers/common/mlx5/mlx5_prm.h
@@ -1109,6 +1109,8 @@ enum {
 			(1ULL << MLX5_GENERAL_OBJ_TYPE_FLOW_METER_ASO)
 #define MLX5_GENERAL_OBJ_TYPES_CAP_GENEVE_TLV_OPT \
 			(1ULL << MLX5_GENERAL_OBJ_TYPE_GENEVE_TLV_OPT)
+#define MLX5_GENERAL_OBJ_TYPES_CAP_DEK \
+			(1ULL << MLX5_GENERAL_OBJ_TYPE_DEK)
 
 enum {
 	MLX5_HCA_CAP_OPMOD_GET_MAX   = 0,
@@ -2406,6 +2408,7 @@ struct mlx5_ifc_create_cq_in_bits {
 
 enum {
 	MLX5_GENERAL_OBJ_TYPE_GENEVE_TLV_OPT = 0x000b,
+	MLX5_GENERAL_OBJ_TYPE_DEK = 0x000c,
 	MLX5_GENERAL_OBJ_TYPE_VIRTQ = 0x000d,
 	MLX5_GENERAL_OBJ_TYPE_VIRTIO_Q_COUNTERS = 0x001c,
 	MLX5_GENERAL_OBJ_TYPE_FLEX_PARSE_GRAPH = 0x0022,
@@ -2469,6 +2472,42 @@ struct mlx5_ifc_create_geneve_tlv_option_in_bits {
 	struct mlx5_ifc_geneve_tlv_option_bits geneve_tlv_opt;
 };
 
+enum {
+	MLX5_CRYPTO_KEY_SIZE_128b = 0x0,
+	MLX5_CRYPTO_KEY_SIZE_256b = 0x1,
+};
+
+enum {
+	MLX5_CRYPTO_KEY_PURPOSE_TLS	= 0x1,
+	MLX5_CRYPTO_KEY_PURPOSE_IPSEC	= 0x2,
+	MLX5_CRYPTO_KEY_PURPOSE_AES_XTS	= 0x3,
+	MLX5_CRYPTO_KEY_PURPOSE_MACSEC	= 0x4,
+	MLX5_CRYPTO_KEY_PURPOSE_GCM	= 0x5,
+	MLX5_CRYPTO_KEY_PURPOSE_PSP	= 0x6,
+};
+
+struct mlx5_ifc_dek_bits {
+	u8 modify_field_select[0x40];
+	u8 state[0x8];
+	u8 reserved_at_48[0xc];
+	u8 key_size[0x4];
+	u8 has_keytag[0x1];
+	u8 reserved_at_59[0x3];
+	u8 key_purpose[0x4];
+	u8 reserved_at_60[0x8];
+	u8 pd[0x18];
+	u8 reserved_at_80[0x100];
+	u8 opaque[0x40];
+	u8 reserved_at_1c0[0x40];
+	u8 key[0x400];
+	u8 reserved_at_600[0x200];
+};
+
+struct mlx5_ifc_create_dek_in_bits {
+	struct mlx5_ifc_general_obj_in_cmd_hdr_bits hdr;
+	struct mlx5_ifc_dek_bits dek;
+};
+
 enum {
 	MLX5_VIRTQ_STATE_INIT = 0,
 	MLX5_VIRTQ_STATE_RDY = 1,
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index 18dc96276d..2976edce0b 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -27,6 +27,7 @@ INTERNAL {
 	mlx5_devx_cmd_create_flow_hit_aso_obj;
 	mlx5_devx_cmd_create_flow_meter_aso_obj;
 	mlx5_devx_cmd_create_geneve_tlv_option;
+        mlx5_devx_cmd_create_dek_obj;
 	mlx5_devx_cmd_destroy;
 	mlx5_devx_cmd_flow_counter_alloc;
 	mlx5_devx_cmd_flow_counter_query;
-- 
2.25.1

[dpdk-dev] [PATCH v2 06/16] common/mlx5: adjust DevX MKEY fields for crypto

[Matan Azrad]

From: Dekel Peled <dekelp@nvidia.com>

MKEY that will be used for crypto purposes must be created with
crypto_en and remote access attributes.
This patch adds support for them in the DevX MKEY context.

Signed-off-by: Dekel Peled <dekelp@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/mlx5_devx_cmds.c |  5 +++++
 drivers/common/mlx5/mlx5_devx_cmds.h |  2 ++
 drivers/common/mlx5/mlx5_prm.h       | 10 +++++++++-
 3 files changed, 16 insertions(+), 1 deletion(-)

diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c
index 742c82cca4..68a10b149a 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.c
+++ b/drivers/common/mlx5/mlx5_devx_cmds.c
@@ -263,6 +263,10 @@ mlx5_devx_cmd_mkey_create(void *ctx,
 	MLX5_SET(create_mkey_in, in, pg_access, attr->pg_access);
 	MLX5_SET(mkc, mkc, lw, 0x1);
 	MLX5_SET(mkc, mkc, lr, 0x1);
+	if (attr->set_remote_rw) {
+		MLX5_SET(mkc, mkc, rw, 0x1);
+		MLX5_SET(mkc, mkc, rr, 0x1);
+	}
 	MLX5_SET(mkc, mkc, qpn, 0xffffff);
 	MLX5_SET(mkc, mkc, pd, attr->pd);
 	MLX5_SET(mkc, mkc, mkey_7_0, attr->umem_id & 0xFF);
@@ -273,6 +277,7 @@ mlx5_devx_cmd_mkey_create(void *ctx,
 	MLX5_SET(mkc, mkc, relaxed_ordering_read, attr->relaxed_ordering_read);
 	MLX5_SET64(mkc, mkc, start_addr, attr->addr);
 	MLX5_SET64(mkc, mkc, len, attr->size);
+	MLX5_SET(mkc, mkc, crypto_en, attr->crypto_en);
 	mkey->obj = mlx5_glue->devx_obj_create(ctx, in, in_size_dw * 4, out,
 					       sizeof(out));
 	if (!mkey->obj) {
diff --git a/drivers/common/mlx5/mlx5_devx_cmds.h b/drivers/common/mlx5/mlx5_devx_cmds.h
index b9ff7ab87d..600577f18a 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.h
+++ b/drivers/common/mlx5/mlx5_devx_cmds.h
@@ -32,6 +32,8 @@ struct mlx5_devx_mkey_attr {
 	uint32_t relaxed_ordering_write:1;
 	uint32_t relaxed_ordering_read:1;
 	uint32_t umr_en:1;
+	uint32_t crypto_en:2;
+	uint32_t set_remote_rw:1;
 	struct mlx5_klm *klm_array;
 	int klm_num;
 };
diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h
index bc9f58ad03..25f6f8ff00 100644
--- a/drivers/common/mlx5/mlx5_prm.h
+++ b/drivers/common/mlx5/mlx5_prm.h
@@ -1059,7 +1059,15 @@ struct mlx5_ifc_mkc_bits {
 	u8 relaxed_ordering_read[0x1];
 	u8 reserved_at_1da[0x1];
 	u8 log_page_size[0x5];
-	u8 reserved_at_1e0[0x20];
+	u8 reserved_at_1e0[0x3];
+	u8 crypto_en[0x2];
+	u8 reserved_at_1e5[0x1b];
+};
+
+/* Range of values for MKEY context crypto_en field. */
+enum {
+	MLX5_MKEY_CRYPTO_DISABLED = 0x0,
+	MLX5_MKEY_CRYPTO_ENABLED = 0x1,
 };
 
 struct mlx5_ifc_create_mkey_out_bits {
-- 
2.25.1

[dpdk-dev] [PATCH v2 07/16] common/mlx5: fix cypto bsf attr

[Matan Azrad]

From: Suanming Mou <suanmingm@nvidia.com>

This commit should be squash to crypto en commit.

Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
---
 drivers/common/mlx5/mlx5_devx_cmds.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c
index 68a10b149a..7e3c8b55fa 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.c
+++ b/drivers/common/mlx5/mlx5_devx_cmds.c
@@ -278,6 +278,10 @@ mlx5_devx_cmd_mkey_create(void *ctx,
 	MLX5_SET64(mkc, mkc, start_addr, attr->addr);
 	MLX5_SET64(mkc, mkc, len, attr->size);
 	MLX5_SET(mkc, mkc, crypto_en, attr->crypto_en);
+	if (attr->crypto_en) {
+		MLX5_SET(mkc, mkc, bsf_en, attr->crypto_en);
+		MLX5_SET(mkc, mkc, bsf_octword_size, 4);
+	}
 	mkey->obj = mlx5_glue->devx_obj_create(ctx, in, in_size_dw * 4, out,
 					       sizeof(out));
 	if (!mkey->obj) {
-- 
2.25.1

[dpdk-dev] [PATCH v2 08/16] common/mlx5: support general obj IMPORT KEK create

[Matan Azrad]

From: Dekel Peled <dekelp@nvidia.com>

IMPORT_KEK object is used to wrap (encrypt) critical security
parameters, such as other keys and credentials, when those need
to be passed between the device and the software.

This patch add support of IMPORT_KEK object create operation.
Add reading of IMPORT_KEK support capability.
Add function to create general object type IMPORT_KEK, using DevX API.

Signed-off-by: Dekel Peled <dekelp@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/mlx5_devx_cmds.c | 50 ++++++++++++++++++++++++++++
 drivers/common/mlx5/mlx5_devx_cmds.h | 13 ++++++++
 drivers/common/mlx5/mlx5_prm.h       | 18 ++++++++++
 drivers/common/mlx5/version.map      |  1 +
 4 files changed, 82 insertions(+)

diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c
index 7e3c8b55fa..afef7a5f63 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.c
+++ b/drivers/common/mlx5/mlx5_devx_cmds.c
@@ -752,6 +752,8 @@ mlx5_devx_cmd_query_hca_attr(void *ctx,
 				  MLX5_GENERAL_OBJ_TYPES_CAP_GENEVE_TLV_OPT);
 	attr->dek = !!(general_obj_types_supported &
 		       MLX5_GENERAL_OBJ_TYPES_CAP_DEK);
+	attr->import_kek = !!(general_obj_types_supported &
+			      MLX5_GENERAL_OBJ_TYPES_CAP_IMPORT_KEK);
 	/* Add reading of other GENERAL_OBJ_TYPES_CAP bits above this line. */
 	attr->log_max_cq = MLX5_GET(cmd_hca_cap, hcattr, log_max_cq);
 	attr->log_max_qp = MLX5_GET(cmd_hca_cap, hcattr, log_max_qp);
@@ -2459,3 +2461,51 @@ mlx5_devx_cmd_create_dek_obj(void *ctx, struct mlx5_devx_dek_attr *attr)
 	dek_obj->id = MLX5_GET(general_obj_out_cmd_hdr, out, obj_id);
 	return dek_obj;
 }
+
+/**
+ * Create general object of type IMPORT_KEK using DevX API.
+ *
+ * @param[in] ctx
+ *   Context returned from mlx5 open_device() glue function.
+ * @param [in] attr
+ *   Pointer to IMPORT_KEK attributes structure.
+ *
+ * @return
+ *   The DevX object created, NULL otherwise and rte_errno is set.
+ */
+struct mlx5_devx_obj *
+mlx5_devx_cmd_create_import_kek_obj(void *ctx,
+				    struct mlx5_devx_import_kek_attr *attr)
+{
+	uint32_t in[MLX5_ST_SZ_DW(create_import_kek_in)] = {0};
+	uint32_t out[MLX5_ST_SZ_DW(general_obj_out_cmd_hdr)] = {0};
+	struct mlx5_devx_obj *import_kek_obj = NULL;
+	void *ptr = NULL, *key_addr = NULL;
+
+	import_kek_obj = mlx5_malloc(MLX5_MEM_ZERO, sizeof(*import_kek_obj),
+				     0, SOCKET_ID_ANY);
+	if (import_kek_obj == NULL) {
+		DRV_LOG(ERR, "Failed to allocate IMPORT_KEK object data");
+		rte_errno = ENOMEM;
+		return NULL;
+	}
+	ptr = MLX5_ADDR_OF(create_import_kek_in, in, hdr);
+	MLX5_SET(general_obj_in_cmd_hdr, ptr, opcode,
+		 MLX5_CMD_OP_CREATE_GENERAL_OBJECT);
+	MLX5_SET(general_obj_in_cmd_hdr, ptr, obj_type,
+		 MLX5_GENERAL_OBJ_TYPE_IMPORT_KEK);
+	ptr = MLX5_ADDR_OF(create_import_kek_in, in, import_kek);
+	MLX5_SET(import_kek, ptr, key_size, attr->key_size);
+	key_addr = MLX5_ADDR_OF(import_kek, ptr, key);
+	memcpy(key_addr, (void *)(attr->key), MLX5_CRYPTO_KEY_MAX_SIZE);
+	import_kek_obj->obj = mlx5_glue->devx_obj_create(ctx, in, sizeof(in),
+							 out, sizeof(out));
+	if (import_kek_obj->obj == NULL) {
+		rte_errno = errno;
+		DRV_LOG(ERR, "Failed to create IMPORT_KEK object using DevX.");
+		mlx5_free(import_kek_obj);
+		return NULL;
+	}
+	import_kek_obj->id = MLX5_GET(general_obj_out_cmd_hdr, out, obj_id);
+	return import_kek_obj;
+}
diff --git a/drivers/common/mlx5/mlx5_devx_cmds.h b/drivers/common/mlx5/mlx5_devx_cmds.h
index 600577f18a..6423610dae 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.h
+++ b/drivers/common/mlx5/mlx5_devx_cmds.h
@@ -142,6 +142,7 @@ struct mlx5_hca_attr {
 	uint32_t crypto:1; /* Crypto engine is supported. */
 	uint32_t aes_xts:1; /* AES-XTS crypto is supported. */
 	uint32_t dek:1; /* General obj type DEK is supported. */
+	uint32_t import_kek:1; /* General obj type IMPORT_KEK supported. */
 	uint32_t regexp_num_of_engines;
 	uint32_t log_max_ft_sampler_num:8;
 	uint32_t geneve_tlv_opt;
@@ -450,6 +451,13 @@ struct mlx5_devx_dek_attr {
 	uint8_t key[MLX5_CRYPTO_KEY_MAX_SIZE];
 };
 
+struct mlx5_devx_import_kek_attr {
+	uint64_t modify_field_select;
+	uint32_t state:8;
+	uint32_t key_size:4;
+	uint8_t key[MLX5_CRYPTO_KEY_MAX_SIZE];
+};
+
 /* mlx5_devx_cmds.c */
 
 __rte_internal
@@ -606,4 +614,9 @@ __rte_internal
 struct mlx5_devx_obj *
 mlx5_devx_cmd_create_dek_obj(void *ctx, struct mlx5_devx_dek_attr *attr);
 
+__rte_internal
+struct mlx5_devx_obj *
+mlx5_devx_cmd_create_import_kek_obj(void *ctx,
+				    struct mlx5_devx_import_kek_attr *attr);
+
 #endif /* RTE_PMD_MLX5_DEVX_CMDS_H_ */
diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h
index 25f6f8ff00..bc339566a6 100644
--- a/drivers/common/mlx5/mlx5_prm.h
+++ b/drivers/common/mlx5/mlx5_prm.h
@@ -1119,6 +1119,8 @@ enum {
 			(1ULL << MLX5_GENERAL_OBJ_TYPE_GENEVE_TLV_OPT)
 #define MLX5_GENERAL_OBJ_TYPES_CAP_DEK \
 			(1ULL << MLX5_GENERAL_OBJ_TYPE_DEK)
+#define MLX5_GENERAL_OBJ_TYPES_CAP_IMPORT_KEK \
+			(1ULL << MLX5_GENERAL_OBJ_TYPE_IMPORT_KEK)
 
 enum {
 	MLX5_HCA_CAP_OPMOD_GET_MAX   = 0,
@@ -2419,6 +2421,7 @@ enum {
 	MLX5_GENERAL_OBJ_TYPE_DEK = 0x000c,
 	MLX5_GENERAL_OBJ_TYPE_VIRTQ = 0x000d,
 	MLX5_GENERAL_OBJ_TYPE_VIRTIO_Q_COUNTERS = 0x001c,
+	MLX5_GENERAL_OBJ_TYPE_IMPORT_KEK = 0x001d,
 	MLX5_GENERAL_OBJ_TYPE_FLEX_PARSE_GRAPH = 0x0022,
 	MLX5_GENERAL_OBJ_TYPE_FLOW_METER_ASO = 0x0024,
 	MLX5_GENERAL_OBJ_TYPE_FLOW_HIT_ASO = 0x0025,
@@ -2516,6 +2519,21 @@ struct mlx5_ifc_create_dek_in_bits {
 	struct mlx5_ifc_dek_bits dek;
 };
 
+struct mlx5_ifc_import_kek_bits {
+	u8 modify_field_select[0x40];
+	u8 state[0x8];
+	u8 reserved_at_48[0xc];
+	u8 key_size[0x4];
+	u8 reserved_at_58[0x1a8];
+	u8 key[0x400];
+	u8 reserved_at_600[0x200];
+};
+
+struct mlx5_ifc_create_import_kek_in_bits {
+	struct mlx5_ifc_general_obj_in_cmd_hdr_bits hdr;
+	struct mlx5_ifc_import_kek_bits import_kek;
+};
+
 enum {
 	MLX5_VIRTQ_STATE_INIT = 0,
 	MLX5_VIRTQ_STATE_RDY = 1,
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index 2976edce0b..56a1b77b0a 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -28,6 +28,7 @@ INTERNAL {
 	mlx5_devx_cmd_create_flow_meter_aso_obj;
 	mlx5_devx_cmd_create_geneve_tlv_option;
         mlx5_devx_cmd_create_dek_obj;
+        mlx5_devx_cmd_create_import_kek_obj;
 	mlx5_devx_cmd_destroy;
 	mlx5_devx_cmd_flow_counter_alloc;
 	mlx5_devx_cmd_flow_counter_query;
-- 
2.25.1

[dpdk-dev] [PATCH v2 09/16] common/mlx5: support general obj CRYPTO LOGIN create

[Matan Azrad]

From: Dekel Peled <dekelp@nvidia.com>

CRYPTO_LOGIN Object is used to login to the device as crypto user
or crypto officer.
Required in order to perform any crypto related control operations.

This patch adds support of CRYPTO_LOGIN object create operation.
Add reading of CRYPTO_LOGIN support capability.
Add function to create general object type CRYPTO_LOGIN, using DevX API.

Signed-off-by: Dekel Peled <dekelp@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/mlx5_devx_cmds.c | 54 ++++++++++++++++++++++++++++
 drivers/common/mlx5/mlx5_devx_cmds.h | 15 ++++++++
 drivers/common/mlx5/mlx5_prm.h       | 19 ++++++++++
 drivers/common/mlx5/version.map      |  1 +
 4 files changed, 89 insertions(+)

diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c
index afef7a5f63..5e082ebb78 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.c
+++ b/drivers/common/mlx5/mlx5_devx_cmds.c
@@ -754,6 +754,8 @@ mlx5_devx_cmd_query_hca_attr(void *ctx,
 		       MLX5_GENERAL_OBJ_TYPES_CAP_DEK);
 	attr->import_kek = !!(general_obj_types_supported &
 			      MLX5_GENERAL_OBJ_TYPES_CAP_IMPORT_KEK);
+	attr->crypto_login = !!(general_obj_types_supported &
+				MLX5_GENERAL_OBJ_TYPES_CAP_CRYPTO_LOGIN);
 	/* Add reading of other GENERAL_OBJ_TYPES_CAP bits above this line. */
 	attr->log_max_cq = MLX5_GET(cmd_hca_cap, hcattr, log_max_cq);
 	attr->log_max_qp = MLX5_GET(cmd_hca_cap, hcattr, log_max_qp);
@@ -2509,3 +2511,55 @@ mlx5_devx_cmd_create_import_kek_obj(void *ctx,
 	import_kek_obj->id = MLX5_GET(general_obj_out_cmd_hdr, out, obj_id);
 	return import_kek_obj;
 }
+
+/**
+ * Create general object of type CRYPTO_LOGIN using DevX API.
+ *
+ * @param[in] ctx
+ *   Context returned from mlx5 open_device() glue function.
+ * @param [in] attr
+ *   Pointer to CRYPTO_LOGIN attributes structure.
+ *
+ * @return
+ *   The DevX object created, NULL otherwise and rte_errno is set.
+ */
+struct mlx5_devx_obj *
+mlx5_devx_cmd_create_crypto_login_obj(void *ctx,
+				      struct mlx5_devx_crypto_login_attr *attr)
+{
+	uint32_t in[MLX5_ST_SZ_DW(create_crypto_login_in)] = {0};
+	uint32_t out[MLX5_ST_SZ_DW(general_obj_out_cmd_hdr)] = {0};
+	struct mlx5_devx_obj *crypto_login_obj = NULL;
+	void *ptr = NULL, *credential_addr = NULL;
+
+	crypto_login_obj = mlx5_malloc(MLX5_MEM_ZERO, sizeof(*crypto_login_obj),
+				       0, SOCKET_ID_ANY);
+	if (crypto_login_obj == NULL) {
+		DRV_LOG(ERR, "Failed to allocate CRYPTO_LOGIN object data");
+		rte_errno = ENOMEM;
+		return NULL;
+	}
+	ptr = MLX5_ADDR_OF(create_crypto_login_in, in, hdr);
+	MLX5_SET(general_obj_in_cmd_hdr, ptr, opcode,
+		 MLX5_CMD_OP_CREATE_GENERAL_OBJECT);
+	MLX5_SET(general_obj_in_cmd_hdr, ptr, obj_type,
+		 MLX5_GENERAL_OBJ_TYPE_CRYPTO_LOGIN);
+	ptr = MLX5_ADDR_OF(create_crypto_login_in, in, crypto_login);
+	MLX5_SET(crypto_login, ptr, credential_pointer,
+		 attr->credential_pointer);
+	MLX5_SET(crypto_login, ptr, session_import_kek_ptr,
+		 attr->session_import_kek_ptr);
+	credential_addr = MLX5_ADDR_OF(crypto_login, ptr, credential);
+	memcpy(credential_addr, (void *)(attr->credential),
+	       MLX5_CRYPTO_LOGIN_CREDENTIAL_SIZE);
+	crypto_login_obj->obj = mlx5_glue->devx_obj_create(ctx, in, sizeof(in),
+							   out, sizeof(out));
+	if (crypto_login_obj->obj == NULL) {
+		rte_errno = errno;
+		DRV_LOG(ERR, "Failed to create CRYPTO_LOGIN obj using DevX.");
+		mlx5_free(crypto_login_obj);
+		return NULL;
+	}
+	crypto_login_obj->id = MLX5_GET(general_obj_out_cmd_hdr, out, obj_id);
+	return crypto_login_obj;
+}
diff --git a/drivers/common/mlx5/mlx5_devx_cmds.h b/drivers/common/mlx5/mlx5_devx_cmds.h
index 6423610dae..709e28bfba 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.h
+++ b/drivers/common/mlx5/mlx5_devx_cmds.h
@@ -143,6 +143,7 @@ struct mlx5_hca_attr {
 	uint32_t aes_xts:1; /* AES-XTS crypto is supported. */
 	uint32_t dek:1; /* General obj type DEK is supported. */
 	uint32_t import_kek:1; /* General obj type IMPORT_KEK supported. */
+	uint32_t crypto_login:1; /* General obj type CRYPTO_LOGIN supported. */
 	uint32_t regexp_num_of_engines;
 	uint32_t log_max_ft_sampler_num:8;
 	uint32_t geneve_tlv_opt;
@@ -458,6 +459,15 @@ struct mlx5_devx_import_kek_attr {
 	uint8_t key[MLX5_CRYPTO_KEY_MAX_SIZE];
 };
 
+#define MLX5_CRYPTO_LOGIN_CREDENTIAL_SIZE	48
+
+struct mlx5_devx_crypto_login_attr {
+	uint64_t modify_field_select;
+	uint32_t credential_pointer:24;
+	uint32_t session_import_kek_ptr:24;
+	uint8_t credential[MLX5_CRYPTO_LOGIN_CREDENTIAL_SIZE];
+};
+
 /* mlx5_devx_cmds.c */
 
 __rte_internal
@@ -619,4 +629,9 @@ struct mlx5_devx_obj *
 mlx5_devx_cmd_create_import_kek_obj(void *ctx,
 				    struct mlx5_devx_import_kek_attr *attr);
 
+__rte_internal
+struct mlx5_devx_obj *
+mlx5_devx_cmd_create_crypto_login_obj(void *ctx,
+				      struct mlx5_devx_crypto_login_attr *attr);
+
 #endif /* RTE_PMD_MLX5_DEVX_CMDS_H_ */
diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h
index bc339566a6..a2437faec0 100644
--- a/drivers/common/mlx5/mlx5_prm.h
+++ b/drivers/common/mlx5/mlx5_prm.h
@@ -1121,6 +1121,8 @@ enum {
 			(1ULL << MLX5_GENERAL_OBJ_TYPE_DEK)
 #define MLX5_GENERAL_OBJ_TYPES_CAP_IMPORT_KEK \
 			(1ULL << MLX5_GENERAL_OBJ_TYPE_IMPORT_KEK)
+#define MLX5_GENERAL_OBJ_TYPES_CAP_CRYPTO_LOGIN \
+			(1ULL << MLX5_GENERAL_OBJ_TYPE_CRYPTO_LOGIN)
 
 enum {
 	MLX5_HCA_CAP_OPMOD_GET_MAX   = 0,
@@ -2422,6 +2424,7 @@ enum {
 	MLX5_GENERAL_OBJ_TYPE_VIRTQ = 0x000d,
 	MLX5_GENERAL_OBJ_TYPE_VIRTIO_Q_COUNTERS = 0x001c,
 	MLX5_GENERAL_OBJ_TYPE_IMPORT_KEK = 0x001d,
+	MLX5_GENERAL_OBJ_TYPE_CRYPTO_LOGIN = 0x001f,
 	MLX5_GENERAL_OBJ_TYPE_FLEX_PARSE_GRAPH = 0x0022,
 	MLX5_GENERAL_OBJ_TYPE_FLOW_METER_ASO = 0x0024,
 	MLX5_GENERAL_OBJ_TYPE_FLOW_HIT_ASO = 0x0025,
@@ -2534,6 +2537,22 @@ struct mlx5_ifc_create_import_kek_in_bits {
 	struct mlx5_ifc_import_kek_bits import_kek;
 };
 
+struct mlx5_ifc_crypto_login_bits {
+	u8 modify_field_select[0x40];
+	u8 reserved_at_40[0x48];
+	u8 credential_pointer[0x18];
+	u8 reserved_at_a0[0x8];
+	u8 session_import_kek_ptr[0x18];
+	u8 reserved_at_c0[0x140];
+	u8 credential[0x180];
+	u8 reserved_at_380[0x480];
+};
+
+struct mlx5_ifc_create_crypto_login_in_bits {
+	struct mlx5_ifc_general_obj_in_cmd_hdr_bits hdr;
+	struct mlx5_ifc_crypto_login_bits crypto_login;
+};
+
 enum {
 	MLX5_VIRTQ_STATE_INIT = 0,
 	MLX5_VIRTQ_STATE_RDY = 1,
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index 56a1b77b0a..bbef436fde 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -29,6 +29,7 @@ INTERNAL {
 	mlx5_devx_cmd_create_geneve_tlv_option;
         mlx5_devx_cmd_create_dek_obj;
         mlx5_devx_cmd_create_import_kek_obj;
+        mlx5_devx_cmd_create_crypto_login_obj;
 	mlx5_devx_cmd_destroy;
 	mlx5_devx_cmd_flow_counter_alloc;
 	mlx5_devx_cmd_flow_counter_query;
-- 
2.25.1

[dpdk-dev] [PATCH v2 10/16] common/mlx5: add crypto BSF struct and defines

[Matan Azrad]

From: Dekel Peled <dekelp@nvidia.com>

This patch adds the struct defining crypto BSF segment of UMR WQE,
and the related value definitions and offsets.

Signed-off-by: Dekel Peled <dekelp@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/mlx5_prm.h | 66 ++++++++++++++++++++++++++++++++++
 1 file changed, 66 insertions(+)

diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h
index a2437faec0..a9dcbfa63c 100644
--- a/drivers/common/mlx5/mlx5_prm.h
+++ b/drivers/common/mlx5/mlx5_prm.h
@@ -1096,6 +1096,72 @@ struct mlx5_ifc_create_mkey_in_bits {
 	u8 klm_pas_mtt[][0x20];
 };
 
+enum {
+	MLX5_BSF_SIZE_16B = 0x0,
+	MLX5_BSF_SIZE_32B = 0x1,
+	MLX5_BSF_SIZE_64B = 0x2,
+	MLX5_BSF_SIZE_128B = 0x3,
+};
+
+enum {
+	MLX5_BSF_P_TYPE_SIGNATURE = 0x0,
+	MLX5_BSF_P_TYPE_CRYPTO = 0x1,
+};
+
+enum {
+	MLX5_ENCRYPTION_ORDER_ENCRYPTED_WIRE_SIGNATURE = 0x0,
+	MLX5_ENCRYPTION_ORDER_ENCRYPTED_MEMORY_SIGNATURE = 0x1,
+	MLX5_ENCRYPTION_ORDER_ENCRYPTED_RAW_WIRE = 0x2,
+	MLX5_ENCRYPTION_ORDER_ENCRYPTED_RAW_MEMORY = 0x3,
+};
+
+enum {
+	MLX5_ENCRYPTION_STANDARD_AES_XTS = 0x0,
+};
+
+enum {
+	MLX5_BLOCK_SIZE_512B	= 0x1,
+	MLX5_BLOCK_SIZE_520B	= 0x2,
+	MLX5_BLOCK_SIZE_4096B	= 0x3,
+	MLX5_BLOCK_SIZE_4160B	= 0x4,
+	MLX5_BLOCK_SIZE_1MB	= 0x5,
+	MLX5_BLOCK_SIZE_4048B	= 0x6,
+};
+
+#define MLX5_BSF_SIZE_OFFSET		30
+#define MLX5_BSF_P_TYPE_OFFSET		24
+#define MLX5_ENCRYPTION_ORDER_OFFSET	16
+#define MLX5_BLOCK_SIZE_OFFSET		24
+
+struct mlx5_wqe_umr_bsf_seg {
+	/*
+	 * bs_bpt_eo_es contains:
+	 * bs	bsf_size		2 bits at MLX5_BSF_SIZE_OFFSET
+	 * bpt	bsf_p_type		2 bits at MLX5_BSF_P_TYPE_OFFSET
+	 * eo	encryption_order	4 bits at MLX5_ENCRYPTION_ORDER_OFFSET
+	 * es	encryption_standard	4 bits at offset 0
+	 */
+	uint32_t bs_bpt_eo_es;
+	uint32_t raw_data_size;
+	/*
+	 * bsp_res contains:
+	 * bsp	crypto_block_size_pointer	8 bits at MLX5_BLOCK_SIZE_OFFSET
+	 * res	reserved 24 bits
+	 */
+	uint32_t bsp_res;
+	uint32_t reserved0;
+	uint8_t xts_initial_tweak[16];
+	/*
+	 * res_dp contains:
+	 * res	reserved 8 bits
+	 * dp	dek_pointer		24 bits at offset 0
+	 */
+	uint32_t res_dp;
+	uint32_t reserved1;
+	uint64_t keytag;
+	uint32_t reserved2[4];
+} __rte_packed;
+
 enum {
 	MLX5_GET_HCA_CAP_OP_MOD_GENERAL_DEVICE = 0x0 << 1,
 	MLX5_GET_HCA_CAP_OP_MOD_ETHERNET_OFFLOAD_CAPS = 0x1 << 1,
-- 
2.25.1

[dpdk-dev] [PATCH v2 11/16] common/mlx5: share hash list tool

[Matan Azrad]

From: Shiri Kuzin <shirik@nvidia.com>

In order to use the hash list defined in net in other drivers, the
hash list is moved to common utilities.

In addition, the log definition was moved from the common utilities to
a dedicated new log file in common in order to prevent a conflict.

Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/linux/mlx5_common_os.c   |   2 +-
 drivers/common/mlx5/linux/mlx5_nl.c          |   2 +-
 drivers/common/mlx5/meson.build              |  15 +-
 drivers/common/mlx5/mlx5_common.c            |   2 +-
 drivers/common/mlx5/mlx5_common_devx.c       |   2 +-
 drivers/common/mlx5/mlx5_common_log.h        |  21 ++
 drivers/common/mlx5/mlx5_common_mp.c         |   2 +-
 drivers/common/mlx5/mlx5_common_mr.c         |   2 +-
 drivers/common/mlx5/mlx5_common_pci.c        |   4 +-
 drivers/common/mlx5/mlx5_common_utils.c      | 221 +++++++++++++++++++
 drivers/common/mlx5/mlx5_common_utils.h      | 202 ++++++++++++++++-
 drivers/common/mlx5/mlx5_devx_cmds.c         |   2 +-
 drivers/common/mlx5/mlx5_malloc.c            |   2 +-
 drivers/common/mlx5/version.map              |   6 +
 drivers/common/mlx5/windows/mlx5_common_os.c |   2 +-
 drivers/common/mlx5/windows/mlx5_glue.c      |   2 +-
 drivers/net/mlx5/mlx5_utils.c                | 209 ------------------
 drivers/net/mlx5/mlx5_utils.h                | 194 +---------------
 18 files changed, 465 insertions(+), 427 deletions(-)
 create mode 100644 drivers/common/mlx5/mlx5_common_log.h
 create mode 100644 drivers/common/mlx5/mlx5_common_utils.c

diff --git a/drivers/common/mlx5/linux/mlx5_common_os.c b/drivers/common/mlx5/linux/mlx5_common_os.c
index 5cf9576921..fba8245b8b 100644
--- a/drivers/common/mlx5/linux/mlx5_common_os.c
+++ b/drivers/common/mlx5/linux/mlx5_common_os.c
@@ -15,7 +15,7 @@
 #include <rte_string_fns.h>
 
 #include "mlx5_common.h"
-#include "mlx5_common_utils.h"
+#include "mlx5_common_log.h"
 #include "mlx5_glue.h"
 
 #ifdef MLX5_GLUE
diff --git a/drivers/common/mlx5/linux/mlx5_nl.c b/drivers/common/mlx5/linux/mlx5_nl.c
index 752c57b33d..f0d04f9473 100644
--- a/drivers/common/mlx5/linux/mlx5_nl.c
+++ b/drivers/common/mlx5/linux/mlx5_nl.c
@@ -20,7 +20,7 @@
 #include <rte_errno.h>
 
 #include "mlx5_nl.h"
-#include "mlx5_common_utils.h"
+#include "../mlx5_common_log.h"
 #include "mlx5_malloc.h"
 #ifdef HAVE_DEVLINK
 #include <linux/devlink.h>
diff --git a/drivers/common/mlx5/meson.build b/drivers/common/mlx5/meson.build
index b5585d2cc3..cd371fe6e1 100644
--- a/drivers/common/mlx5/meson.build
+++ b/drivers/common/mlx5/meson.build
@@ -9,13 +9,14 @@ endif
 
 deps += ['hash', 'pci', 'bus_pci', 'net', 'eal', 'kvargs']
 sources += files(
-        'mlx5_devx_cmds.c',
-        'mlx5_common.c',
-        'mlx5_common_mp.c',
-        'mlx5_common_mr.c',
-        'mlx5_malloc.c',
-        'mlx5_common_pci.c',
-        'mlx5_common_devx.c',
+	'mlx5_devx_cmds.c',
+	'mlx5_common.c',
+	'mlx5_common_mp.c',
+	'mlx5_common_mr.c',
+	'mlx5_malloc.c',
+	'mlx5_common_pci.c',
+	'mlx5_common_devx.c',
+	'mlx5_common_utils.c',
 )
 
 cflags_options = [
diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index f92f05bda5..d397459a3d 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -11,7 +11,7 @@
 
 #include "mlx5_common.h"
 #include "mlx5_common_os.h"
-#include "mlx5_common_utils.h"
+#include "mlx5_common_log.h"
 #include "mlx5_common_pci.h"
 
 uint8_t haswell_broadwell_cpu;
diff --git a/drivers/common/mlx5/mlx5_common_devx.c b/drivers/common/mlx5/mlx5_common_devx.c
index d19be122bd..22c8d356c4 100644
--- a/drivers/common/mlx5/mlx5_common_devx.c
+++ b/drivers/common/mlx5/mlx5_common_devx.c
@@ -12,7 +12,7 @@
 
 #include "mlx5_prm.h"
 #include "mlx5_devx_cmds.h"
-#include "mlx5_common_utils.h"
+#include "mlx5_common_log.h"
 #include "mlx5_malloc.h"
 #include "mlx5_common.h"
 #include "mlx5_common_devx.h"
diff --git a/drivers/common/mlx5/mlx5_common_log.h b/drivers/common/mlx5/mlx5_common_log.h
new file mode 100644
index 0000000000..26b13fedaf
--- /dev/null
+++ b/drivers/common/mlx5/mlx5_common_log.h
@@ -0,0 +1,21 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2019 Mellanox Technologies, Ltd
+ */
+
+#ifndef RTE_PMD_MLX5_COMMON_LOG_H_
+#define RTE_PMD_MLX5_COMMON_LOG_H_
+
+#include "mlx5_common.h"
+
+
+extern int mlx5_common_logtype;
+
+#define MLX5_COMMON_LOG_PREFIX "mlx5_common"
+/* Generic printf()-like logging macro with automatic line feed. */
+#define DRV_LOG(level, ...) \
+	PMD_DRV_LOG_(level, mlx5_common_logtype, MLX5_COMMON_LOG_PREFIX, \
+		__VA_ARGS__ PMD_DRV_LOG_STRIP PMD_DRV_LOG_OPAREN, \
+		PMD_DRV_LOG_CPAREN)
+
+#endif /* RTE_PMD_MLX5_COMMON_LOG_H_ */
+
diff --git a/drivers/common/mlx5/mlx5_common_mp.c b/drivers/common/mlx5/mlx5_common_mp.c
index 40e3956e45..673a7c31de 100644
--- a/drivers/common/mlx5/mlx5_common_mp.c
+++ b/drivers/common/mlx5/mlx5_common_mp.c
@@ -10,7 +10,7 @@
 #include <rte_errno.h>
 
 #include "mlx5_common_mp.h"
-#include "mlx5_common_utils.h"
+#include "mlx5_common_log.h"
 #include "mlx5_malloc.h"
 
 /**
diff --git a/drivers/common/mlx5/mlx5_common_mr.c b/drivers/common/mlx5/mlx5_common_mr.c
index e1ed0caf3a..afb5b3d0a7 100644
--- a/drivers/common/mlx5/mlx5_common_mr.c
+++ b/drivers/common/mlx5/mlx5_common_mr.c
@@ -11,7 +11,7 @@
 #include "mlx5_glue.h"
 #include "mlx5_common_mp.h"
 #include "mlx5_common_mr.h"
-#include "mlx5_common_utils.h"
+#include "mlx5_common_log.h"
 #include "mlx5_malloc.h"
 
 struct mr_find_contig_memsegs_data {
diff --git a/drivers/common/mlx5/mlx5_common_pci.c b/drivers/common/mlx5/mlx5_common_pci.c
index 5a73ffa60a..3f16cd21cf 100644
--- a/drivers/common/mlx5/mlx5_common_pci.c
+++ b/drivers/common/mlx5/mlx5_common_pci.c
@@ -4,7 +4,9 @@
 
 #include <stdlib.h>
 #include <rte_malloc.h>
-#include "mlx5_common_utils.h"
+#include <rte_class.h>
+
+#include "mlx5_common_log.h"
 #include "mlx5_common_pci.h"
 
 struct mlx5_pci_device {
diff --git a/drivers/common/mlx5/mlx5_common_utils.c b/drivers/common/mlx5/mlx5_common_utils.c
new file mode 100644
index 0000000000..ad2011e858
--- /dev/null
+++ b/drivers/common/mlx5/mlx5_common_utils.c
@@ -0,0 +1,221 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2019 Mellanox Technologies, Ltd
+ */
+
+#include <rte_malloc.h>
+#include <rte_hash_crc.h>
+#include <rte_errno.h>
+
+#include <mlx5_malloc.h>
+
+#include "mlx5_common_utils.h"
+#include "mlx5_common_log.h"
+
+/********************* Hash List **********************/
+
+static struct mlx5_hlist_entry *
+mlx5_hlist_default_create_cb(struct mlx5_hlist *h, uint64_t key __rte_unused,
+			     void *ctx __rte_unused)
+{
+	return mlx5_malloc(MLX5_MEM_ZERO, h->entry_sz, 0, SOCKET_ID_ANY);
+}
+
+static void
+mlx5_hlist_default_remove_cb(struct mlx5_hlist *h __rte_unused,
+			     struct mlx5_hlist_entry *entry)
+{
+	mlx5_free(entry);
+}
+
+struct mlx5_hlist *
+mlx5_hlist_create(const char *name, uint32_t size, uint32_t entry_size,
+		  uint32_t flags, mlx5_hlist_create_cb cb_create,
+		  mlx5_hlist_match_cb cb_match, mlx5_hlist_remove_cb cb_remove)
+{
+	struct mlx5_hlist *h;
+	uint32_t act_size;
+	uint32_t alloc_size;
+	uint32_t i;
+
+	if (!size || !cb_match || (!cb_create ^ !cb_remove))
+		return NULL;
+	/* Align to the next power of 2, 32bits integer is enough now. */
+	if (!rte_is_power_of_2(size)) {
+		act_size = rte_align32pow2(size);
+		DRV_LOG(DEBUG, "Size 0x%" PRIX32 " is not power of 2, "
+			"will be aligned to 0x%" PRIX32 ".", size, act_size);
+	} else {
+		act_size = size;
+	}
+	alloc_size = sizeof(struct mlx5_hlist) +
+		     sizeof(struct mlx5_hlist_bucket) * act_size;
+	/* Using zmalloc, then no need to initialize the heads. */
+	h = mlx5_malloc(MLX5_MEM_ZERO, alloc_size, RTE_CACHE_LINE_SIZE,
+			SOCKET_ID_ANY);
+	if (!h) {
+		DRV_LOG(ERR, "No memory for hash list %s creation",
+			name ? name : "None");
+		return NULL;
+	}
+	if (name)
+		snprintf(h->name, MLX5_HLIST_NAMESIZE, "%s", name);
+	h->table_sz = act_size;
+	h->mask = act_size - 1;
+	h->entry_sz = entry_size;
+	h->direct_key = !!(flags & MLX5_HLIST_DIRECT_KEY);
+	h->write_most = !!(flags & MLX5_HLIST_WRITE_MOST);
+	h->cb_create = cb_create ? cb_create : mlx5_hlist_default_create_cb;
+	h->cb_match = cb_match;
+	h->cb_remove = cb_remove ? cb_remove : mlx5_hlist_default_remove_cb;
+	for (i = 0; i < act_size; i++)
+		rte_rwlock_init(&h->buckets[i].lock);
+	DRV_LOG(DEBUG, "Hash list with %s size 0x%" PRIX32 " is created.",
+		h->name, act_size);
+	return h;
+}
+
+static struct mlx5_hlist_entry *
+__hlist_lookup(struct mlx5_hlist *h, uint64_t key, uint32_t idx,
+	       void *ctx, bool reuse)
+{
+	struct mlx5_hlist_head *first;
+	struct mlx5_hlist_entry *node;
+
+	MLX5_ASSERT(h);
+	first = &h->buckets[idx].head;
+	LIST_FOREACH(node, first, next) {
+		if (!h->cb_match(h, node, key, ctx)) {
+			if (reuse) {
+				__atomic_add_fetch(&node->ref_cnt, 1,
+						   __ATOMIC_RELAXED);
+				DRV_LOG(DEBUG, "Hash list %s entry %p "
+					"reuse: %u.",
+					h->name, (void *)node, node->ref_cnt);
+			}
+			break;
+		}
+	}
+	return node;
+}
+
+static struct mlx5_hlist_entry *
+hlist_lookup(struct mlx5_hlist *h, uint64_t key, uint32_t idx,
+	     void *ctx, bool reuse)
+{
+	struct mlx5_hlist_entry *node;
+
+	MLX5_ASSERT(h);
+	rte_rwlock_read_lock(&h->buckets[idx].lock);
+	node = __hlist_lookup(h, key, idx, ctx, reuse);
+	rte_rwlock_read_unlock(&h->buckets[idx].lock);
+	return node;
+}
+
+struct mlx5_hlist_entry *
+mlx5_hlist_lookup(struct mlx5_hlist *h, uint64_t key, void *ctx)
+{
+	uint32_t idx;
+
+	if (h->direct_key)
+		idx = (uint32_t)(key & h->mask);
+	else
+		idx = rte_hash_crc_8byte(key, 0) & h->mask;
+	return hlist_lookup(h, key, idx, ctx, false);
+}
+
+struct mlx5_hlist_entry*
+mlx5_hlist_register(struct mlx5_hlist *h, uint64_t key, void *ctx)
+{
+	uint32_t idx;
+	struct mlx5_hlist_head *first;
+	struct mlx5_hlist_bucket *b;
+	struct mlx5_hlist_entry *entry;
+	uint32_t prev_gen_cnt = 0;
+
+	if (h->direct_key)
+		idx = (uint32_t)(key & h->mask);
+	else
+		idx = rte_hash_crc_8byte(key, 0) & h->mask;
+	MLX5_ASSERT(h);
+	b = &h->buckets[idx];
+	/* Use write lock directly for write-most list. */
+	if (!h->write_most) {
+		prev_gen_cnt = __atomic_load_n(&b->gen_cnt, __ATOMIC_ACQUIRE);
+		entry = hlist_lookup(h, key, idx, ctx, true);
+		if (entry)
+			return entry;
+	}
+	rte_rwlock_write_lock(&b->lock);
+	/* Check if the list changed by other threads. */
+	if (h->write_most ||
+	    prev_gen_cnt != __atomic_load_n(&b->gen_cnt, __ATOMIC_ACQUIRE)) {
+		entry = __hlist_lookup(h, key, idx, ctx, true);
+		if (entry)
+			goto done;
+	}
+	first = &b->head;
+	entry = h->cb_create(h, key, ctx);
+	if (!entry) {
+		rte_errno = ENOMEM;
+		DRV_LOG(DEBUG, "Can't allocate hash list %s entry.", h->name);
+		goto done;
+	}
+	entry->idx = idx;
+	entry->ref_cnt = 1;
+	LIST_INSERT_HEAD(first, entry, next);
+	__atomic_add_fetch(&b->gen_cnt, 1, __ATOMIC_ACQ_REL);
+	DRV_LOG(DEBUG, "Hash list %s entry %p new: %u.",
+		h->name, (void *)entry, entry->ref_cnt);
+done:
+	rte_rwlock_write_unlock(&b->lock);
+	return entry;
+}
+
+int
+mlx5_hlist_unregister(struct mlx5_hlist *h, struct mlx5_hlist_entry *entry)
+{
+	uint32_t idx = entry->idx;
+
+	rte_rwlock_write_lock(&h->buckets[idx].lock);
+	MLX5_ASSERT(entry && entry->ref_cnt && entry->next.le_prev);
+	DRV_LOG(DEBUG, "Hash list %s entry %p deref: %u.",
+		h->name, (void *)entry, entry->ref_cnt);
+	if (--entry->ref_cnt) {
+		rte_rwlock_write_unlock(&h->buckets[idx].lock);
+		return 1;
+	}
+	LIST_REMOVE(entry, next);
+	/* Set to NULL to get rid of removing action for more than once. */
+	entry->next.le_prev = NULL;
+	h->cb_remove(h, entry);
+	rte_rwlock_write_unlock(&h->buckets[idx].lock);
+	DRV_LOG(DEBUG, "Hash list %s entry %p removed.",
+		h->name, (void *)entry);
+	return 0;
+}
+
+void
+mlx5_hlist_destroy(struct mlx5_hlist *h)
+{
+	uint32_t idx;
+	struct mlx5_hlist_entry *entry;
+
+	MLX5_ASSERT(h);
+	for (idx = 0; idx < h->table_sz; ++idx) {
+		/* No LIST_FOREACH_SAFE, using while instead. */
+		while (!LIST_EMPTY(&h->buckets[idx].head)) {
+			entry = LIST_FIRST(&h->buckets[idx].head);
+			LIST_REMOVE(entry, next);
+			/*
+			 * The owner of whole element which contains data entry
+			 * is the user, so it's the user's duty to do the clean
+			 * up and the free work because someone may not put the
+			 * hlist entry at the beginning(suggested to locate at
+			 * the beginning). Or else the default free function
+			 * will be used.
+			 */
+			h->cb_remove(h, entry);
+		}
+	}
+	mlx5_free(h);
+}
diff --git a/drivers/common/mlx5/mlx5_common_utils.h b/drivers/common/mlx5/mlx5_common_utils.h
index 6cba39c8cc..ed378ce9bd 100644
--- a/drivers/common/mlx5/mlx5_common_utils.h
+++ b/drivers/common/mlx5/mlx5_common_utils.h
@@ -7,14 +7,202 @@
 
 #include "mlx5_common.h"
 
+#define MLX5_HLIST_DIRECT_KEY 0x0001 /* Use the key directly as hash index. */
+#define MLX5_HLIST_WRITE_MOST 0x0002 /* List mostly used for append new. */
 
-extern int mlx5_common_logtype;
+/** Maximum size of string for naming the hlist table. */
+#define MLX5_HLIST_NAMESIZE			32
 
-#define MLX5_COMMON_LOG_PREFIX "mlx5_common"
-/* Generic printf()-like logging macro with automatic line feed. */
-#define DRV_LOG(level, ...) \
-	PMD_DRV_LOG_(level, mlx5_common_logtype, MLX5_COMMON_LOG_PREFIX, \
-		__VA_ARGS__ PMD_DRV_LOG_STRIP PMD_DRV_LOG_OPAREN, \
-		PMD_DRV_LOG_CPAREN)
+struct mlx5_hlist;
+
+/**
+ * Structure of the entry in the hash list, user should define its own struct
+ * that contains this in order to store the data. The 'key' is 64-bits right
+ * now and its user's responsibility to guarantee there is no collision.
+ */
+struct mlx5_hlist_entry {
+	LIST_ENTRY(mlx5_hlist_entry) next; /* entry pointers in the list. */
+	uint32_t idx; /* Bucket index the entry belongs to. */
+	uint32_t ref_cnt; /* Reference count. */
+};
+
+/** Structure for hash head. */
+LIST_HEAD(mlx5_hlist_head, mlx5_hlist_entry);
+
+/**
+ * Type of callback function for entry removal.
+ *
+ * @param list
+ *   The hash list.
+ * @param entry
+ *   The entry in the list.
+ */
+typedef void (*mlx5_hlist_remove_cb)(struct mlx5_hlist *list,
+				     struct mlx5_hlist_entry *entry);
+
+/**
+ * Type of function for user defined matching.
+ *
+ * @param list
+ *   The hash list.
+ * @param entry
+ *   The entry in the list.
+ * @param key
+ *   The new entry key.
+ * @param ctx
+ *   The pointer to new entry context.
+ *
+ * @return
+ *   0 if matching, non-zero number otherwise.
+ */
+typedef int (*mlx5_hlist_match_cb)(struct mlx5_hlist *list,
+				   struct mlx5_hlist_entry *entry,
+				   uint64_t key, void *ctx);
+
+/**
+ * Type of function for user defined hash list entry creation.
+ *
+ * @param list
+ *   The hash list.
+ * @param key
+ *   The key of the new entry.
+ * @param ctx
+ *   The pointer to new entry context.
+ *
+ * @return
+ *   Pointer to allocated entry on success, NULL otherwise.
+ */
+typedef struct mlx5_hlist_entry *(*mlx5_hlist_create_cb)
+				  (struct mlx5_hlist *list,
+				   uint64_t key, void *ctx);
+
+/* Hash list bucket head. */
+struct mlx5_hlist_bucket {
+	struct mlx5_hlist_head head; /* List head. */
+	rte_rwlock_t lock; /* Bucket lock. */
+	uint32_t gen_cnt; /* List modification will update generation count. */
+} __rte_cache_aligned;
+
+/**
+ * Hash list table structure
+ *
+ * Entry in hash list could be reused if entry already exists, reference
+ * count will increase and the existing entry returns.
+ *
+ * When destroy an entry from list, decrease reference count and only
+ * destroy when no further reference.
+ */
+struct mlx5_hlist {
+	char name[MLX5_HLIST_NAMESIZE]; /**< Name of the hash list. */
+	/**< number of heads, need to be power of 2. */
+	uint32_t table_sz;
+	uint32_t entry_sz; /**< Size of entry, used to allocate entry. */
+	/**< mask to get the index of the list heads. */
+	uint32_t mask;
+	bool direct_key; /* Use the new entry key directly as hash index. */
+	bool write_most; /* List mostly used for append new or destroy. */
+	void *ctx;
+	mlx5_hlist_create_cb cb_create; /**< entry create callback. */
+	mlx5_hlist_match_cb cb_match; /**< entry match callback. */
+	mlx5_hlist_remove_cb cb_remove; /**< entry remove callback. */
+	struct mlx5_hlist_bucket buckets[] __rte_cache_aligned;
+	/**< list bucket arrays. */
+};
+
+/**
+ * Create a hash list table, the user can specify the list heads array size
+ * of the table, now the size should be a power of 2 in order to get better
+ * distribution for the entries. Each entry is a part of the whole data element
+ * and the caller should be responsible for the data element's allocation and
+ * cleanup / free. Key of each entry will be calculated with CRC in order to
+ * generate a little fairer distribution.
+ *
+ * @param name
+ *   Name of the hash list(optional).
+ * @param size
+ *   Heads array size of the hash list.
+ * @param entry_size
+ *   Entry size to allocate if cb_create not specified.
+ * @param flags
+ *   The hash list attribute flags.
+ * @param cb_create
+ *   Callback function for entry create.
+ * @param cb_match
+ *   Callback function for entry match.
+ * @param cb_destroy
+ *   Callback function for entry destroy.
+ * @return
+ *   Pointer of the hash list table created, NULL on failure.
+ */
+__rte_internal
+struct mlx5_hlist *mlx5_hlist_create(const char *name, uint32_t size,
+				     uint32_t entry_size, uint32_t flags,
+				     mlx5_hlist_create_cb cb_create,
+				     mlx5_hlist_match_cb cb_match,
+				     mlx5_hlist_remove_cb cb_destroy);
+
+/**
+ * Search an entry matching the key.
+ *
+ * Result returned might be destroyed by other thread, must use
+ * this function only in main thread.
+ *
+ * @param h
+ *   Pointer to the hast list table.
+ * @param key
+ *   Key for the searching entry.
+ * @param ctx
+ *   Common context parameter used by entry callback function.
+ *
+ * @return
+ *   Pointer of the hlist entry if found, NULL otherwise.
+ */
+__rte_internal
+struct mlx5_hlist_entry *mlx5_hlist_lookup(struct mlx5_hlist *h, uint64_t key,
+					   void *ctx);
+
+/**
+ * Insert an entry to the hash list table, the entry is only part of whole data
+ * element and a 64B key is used for matching. User should construct the key or
+ * give a calculated hash signature and guarantee there is no collision.
+ *
+ * @param h
+ *   Pointer to the hast list table.
+ * @param entry
+ *   Entry to be inserted into the hash list table.
+ * @param ctx
+ *   Common context parameter used by callback function.
+ *
+ * @return
+ *   registered entry on success, NULL otherwise
+ */
+__rte_internal
+struct mlx5_hlist_entry *mlx5_hlist_register(struct mlx5_hlist *h, uint64_t key,
+					     void *ctx);
+
+/**
+ * Remove an entry from the hash list table. User should guarantee the validity
+ * of the entry.
+ *
+ * @param h
+ *   Pointer to the hast list table. (not used)
+ * @param entry
+ *   Entry to be removed from the hash list table.
+ * @return
+ *   0 on entry removed, 1 on entry still referenced.
+ */
+__rte_internal
+int mlx5_hlist_unregister(struct mlx5_hlist *h, struct mlx5_hlist_entry *entry);
+
+/**
+ * Destroy the hash list table, all the entries already inserted into the lists
+ * will be handled by the callback function provided by the user (including
+ * free if needed) before the table is freed.
+ *
+ * @param h
+ *   Pointer to the hast list table.
+ */
+__rte_internal
+void mlx5_hlist_destroy(struct mlx5_hlist *h);
 
 #endif /* RTE_PMD_MLX5_COMMON_UTILS_H_ */
diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c
index 5e082ebb78..c0061741e8 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.c
+++ b/drivers/common/mlx5/mlx5_devx_cmds.c
@@ -9,7 +9,7 @@
 
 #include "mlx5_prm.h"
 #include "mlx5_devx_cmds.h"
-#include "mlx5_common_utils.h"
+#include "mlx5_common_log.h"
 #include "mlx5_malloc.h"
 
 
diff --git a/drivers/common/mlx5/mlx5_malloc.c b/drivers/common/mlx5/mlx5_malloc.c
index 9d30cedbaa..b19501e1bc 100644
--- a/drivers/common/mlx5/mlx5_malloc.c
+++ b/drivers/common/mlx5/mlx5_malloc.c
@@ -8,7 +8,7 @@
 #include <stdbool.h>
 #include <string.h>
 
-#include "mlx5_common_utils.h"
+#include "mlx5_common_log.h"
 #include "mlx5_common_os.h"
 #include "mlx5_malloc.h"
 
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index bbef436fde..d16e484ffa 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -121,4 +121,10 @@ INTERNAL {
 	mlx5_free;
 
 	mlx5_pci_driver_register;
+
+	mlx5_hlist_create;
+	mlx5_hlist_lookup;
+	mlx5_hlist_register;
+	mlx5_hlist_unregister;
+	mlx5_hlist_destroy;
 };
diff --git a/drivers/common/mlx5/windows/mlx5_common_os.c b/drivers/common/mlx5/windows/mlx5_common_os.c
index f2d781a965..2e6e172a96 100644
--- a/drivers/common/mlx5/windows/mlx5_common_os.c
+++ b/drivers/common/mlx5/windows/mlx5_common_os.c
@@ -11,7 +11,7 @@
 #include <rte_errno.h>
 
 #include "mlx5_devx_cmds.h"
-#include "mlx5_common_utils.h"
+#include "../mlx5_common_log.h"
 #include "mlx5_common.h"
 #include "mlx5_common_os.h"
 #include "mlx5_malloc.h"
diff --git a/drivers/common/mlx5/windows/mlx5_glue.c b/drivers/common/mlx5/windows/mlx5_glue.c
index aef6d3b5f4..535487a8d4 100644
--- a/drivers/common/mlx5/windows/mlx5_glue.c
+++ b/drivers/common/mlx5/windows/mlx5_glue.c
@@ -12,7 +12,7 @@
 #include <rte_malloc.h>
 
 #include "mlx5_glue.h"
-#include "mlx5_common_utils.h"
+#include "../mlx5_common_log.h"
 #include "mlx5_win_ext.h"
 
 /*
diff --git a/drivers/net/mlx5/mlx5_utils.c b/drivers/net/mlx5/mlx5_utils.c
index a39b5edddc..18fe23e4fb 100644
--- a/drivers/net/mlx5/mlx5_utils.c
+++ b/drivers/net/mlx5/mlx5_utils.c
@@ -3,220 +3,11 @@
  */
 
 #include <rte_malloc.h>
-#include <rte_hash_crc.h>
 
 #include <mlx5_malloc.h>
 
 #include "mlx5_utils.h"
 
-/********************* Hash List **********************/
-
-static struct mlx5_hlist_entry *
-mlx5_hlist_default_create_cb(struct mlx5_hlist *h, uint64_t key __rte_unused,
-			     void *ctx __rte_unused)
-{
-	return mlx5_malloc(MLX5_MEM_ZERO, h->entry_sz, 0, SOCKET_ID_ANY);
-}
-
-static void
-mlx5_hlist_default_remove_cb(struct mlx5_hlist *h __rte_unused,
-			     struct mlx5_hlist_entry *entry)
-{
-	mlx5_free(entry);
-}
-
-struct mlx5_hlist *
-mlx5_hlist_create(const char *name, uint32_t size, uint32_t entry_size,
-		  uint32_t flags, mlx5_hlist_create_cb cb_create,
-		  mlx5_hlist_match_cb cb_match, mlx5_hlist_remove_cb cb_remove)
-{
-	struct mlx5_hlist *h;
-	uint32_t act_size;
-	uint32_t alloc_size;
-	uint32_t i;
-
-	if (!size || !cb_match || (!cb_create ^ !cb_remove))
-		return NULL;
-	/* Align to the next power of 2, 32bits integer is enough now. */
-	if (!rte_is_power_of_2(size)) {
-		act_size = rte_align32pow2(size);
-		DRV_LOG(DEBUG, "Size 0x%" PRIX32 " is not power of 2, "
-			"will be aligned to 0x%" PRIX32 ".", size, act_size);
-	} else {
-		act_size = size;
-	}
-	alloc_size = sizeof(struct mlx5_hlist) +
-		     sizeof(struct mlx5_hlist_bucket) * act_size;
-	/* Using zmalloc, then no need to initialize the heads. */
-	h = mlx5_malloc(MLX5_MEM_ZERO, alloc_size, RTE_CACHE_LINE_SIZE,
-			SOCKET_ID_ANY);
-	if (!h) {
-		DRV_LOG(ERR, "No memory for hash list %s creation",
-			name ? name : "None");
-		return NULL;
-	}
-	if (name)
-		snprintf(h->name, MLX5_HLIST_NAMESIZE, "%s", name);
-	h->table_sz = act_size;
-	h->mask = act_size - 1;
-	h->entry_sz = entry_size;
-	h->direct_key = !!(flags & MLX5_HLIST_DIRECT_KEY);
-	h->write_most = !!(flags & MLX5_HLIST_WRITE_MOST);
-	h->cb_create = cb_create ? cb_create : mlx5_hlist_default_create_cb;
-	h->cb_match = cb_match;
-	h->cb_remove = cb_remove ? cb_remove : mlx5_hlist_default_remove_cb;
-	for (i = 0; i < act_size; i++)
-		rte_rwlock_init(&h->buckets[i].lock);
-	DRV_LOG(DEBUG, "Hash list with %s size 0x%" PRIX32 " is created.",
-		h->name, act_size);
-	return h;
-}
-
-static struct mlx5_hlist_entry *
-__hlist_lookup(struct mlx5_hlist *h, uint64_t key, uint32_t idx,
-	       void *ctx, bool reuse)
-{
-	struct mlx5_hlist_head *first;
-	struct mlx5_hlist_entry *node;
-
-	MLX5_ASSERT(h);
-	first = &h->buckets[idx].head;
-	LIST_FOREACH(node, first, next) {
-		if (!h->cb_match(h, node, key, ctx)) {
-			if (reuse) {
-				__atomic_add_fetch(&node->ref_cnt, 1,
-						   __ATOMIC_RELAXED);
-				DRV_LOG(DEBUG, "Hash list %s entry %p "
-					"reuse: %u.",
-					h->name, (void *)node, node->ref_cnt);
-			}
-			break;
-		}
-	}
-	return node;
-}
-
-static struct mlx5_hlist_entry *
-hlist_lookup(struct mlx5_hlist *h, uint64_t key, uint32_t idx,
-	     void *ctx, bool reuse)
-{
-	struct mlx5_hlist_entry *node;
-
-	MLX5_ASSERT(h);
-	rte_rwlock_read_lock(&h->buckets[idx].lock);
-	node = __hlist_lookup(h, key, idx, ctx, reuse);
-	rte_rwlock_read_unlock(&h->buckets[idx].lock);
-	return node;
-}
-
-struct mlx5_hlist_entry *
-mlx5_hlist_lookup(struct mlx5_hlist *h, uint64_t key, void *ctx)
-{
-	uint32_t idx;
-
-	if (h->direct_key)
-		idx = (uint32_t)(key & h->mask);
-	else
-		idx = rte_hash_crc_8byte(key, 0) & h->mask;
-	return hlist_lookup(h, key, idx, ctx, false);
-}
-
-struct mlx5_hlist_entry*
-mlx5_hlist_register(struct mlx5_hlist *h, uint64_t key, void *ctx)
-{
-	uint32_t idx;
-	struct mlx5_hlist_head *first;
-	struct mlx5_hlist_bucket *b;
-	struct mlx5_hlist_entry *entry;
-	uint32_t prev_gen_cnt = 0;
-
-	if (h->direct_key)
-		idx = (uint32_t)(key & h->mask);
-	else
-		idx = rte_hash_crc_8byte(key, 0) & h->mask;
-	MLX5_ASSERT(h);
-	b = &h->buckets[idx];
-	/* Use write lock directly for write-most list. */
-	if (!h->write_most) {
-		prev_gen_cnt = __atomic_load_n(&b->gen_cnt, __ATOMIC_ACQUIRE);
-		entry = hlist_lookup(h, key, idx, ctx, true);
-		if (entry)
-			return entry;
-	}
-	rte_rwlock_write_lock(&b->lock);
-	/* Check if the list changed by other threads. */
-	if (h->write_most ||
-	    prev_gen_cnt != __atomic_load_n(&b->gen_cnt, __ATOMIC_ACQUIRE)) {
-		entry = __hlist_lookup(h, key, idx, ctx, true);
-		if (entry)
-			goto done;
-	}
-	first = &b->head;
-	entry = h->cb_create(h, key, ctx);
-	if (!entry) {
-		rte_errno = ENOMEM;
-		DRV_LOG(DEBUG, "Can't allocate hash list %s entry.", h->name);
-		goto done;
-	}
-	entry->idx = idx;
-	entry->ref_cnt = 1;
-	LIST_INSERT_HEAD(first, entry, next);
-	__atomic_add_fetch(&b->gen_cnt, 1, __ATOMIC_ACQ_REL);
-	DRV_LOG(DEBUG, "Hash list %s entry %p new: %u.",
-		h->name, (void *)entry, entry->ref_cnt);
-done:
-	rte_rwlock_write_unlock(&b->lock);
-	return entry;
-}
-
-int
-mlx5_hlist_unregister(struct mlx5_hlist *h, struct mlx5_hlist_entry *entry)
-{
-	uint32_t idx = entry->idx;
-
-	rte_rwlock_write_lock(&h->buckets[idx].lock);
-	MLX5_ASSERT(entry && entry->ref_cnt && entry->next.le_prev);
-	DRV_LOG(DEBUG, "Hash list %s entry %p deref: %u.",
-		h->name, (void *)entry, entry->ref_cnt);
-	if (--entry->ref_cnt) {
-		rte_rwlock_write_unlock(&h->buckets[idx].lock);
-		return 1;
-	}
-	LIST_REMOVE(entry, next);
-	/* Set to NULL to get rid of removing action for more than once. */
-	entry->next.le_prev = NULL;
-	h->cb_remove(h, entry);
-	rte_rwlock_write_unlock(&h->buckets[idx].lock);
-	DRV_LOG(DEBUG, "Hash list %s entry %p removed.",
-		h->name, (void *)entry);
-	return 0;
-}
-
-void
-mlx5_hlist_destroy(struct mlx5_hlist *h)
-{
-	uint32_t idx;
-	struct mlx5_hlist_entry *entry;
-
-	MLX5_ASSERT(h);
-	for (idx = 0; idx < h->table_sz; ++idx) {
-		/* No LIST_FOREACH_SAFE, using while instead. */
-		while (!LIST_EMPTY(&h->buckets[idx].head)) {
-			entry = LIST_FIRST(&h->buckets[idx].head);
-			LIST_REMOVE(entry, next);
-			/*
-			 * The owner of whole element which contains data entry
-			 * is the user, so it's the user's duty to do the clean
-			 * up and the free work because someone may not put the
-			 * hlist entry at the beginning(suggested to locate at
-			 * the beginning). Or else the default free function
-			 * will be used.
-			 */
-			h->cb_remove(h, entry);
-		}
-	}
-	mlx5_free(h);
-}
 
 /********************* Cache list ************************/
 
diff --git a/drivers/net/mlx5/mlx5_utils.h b/drivers/net/mlx5/mlx5_utils.h
index 289941cebc..b54517c6df 100644
--- a/drivers/net/mlx5/mlx5_utils.h
+++ b/drivers/net/mlx5/mlx5_utils.h
@@ -18,6 +18,7 @@
 #include <rte_bitmap.h>
 
 #include <mlx5_common.h>
+#include <mlx5_common_utils.h>
 
 #include "mlx5_defs.h"
 
@@ -261,199 +262,6 @@ log2above(unsigned int v)
 	return l + r;
 }
 
-#define MLX5_HLIST_DIRECT_KEY 0x0001 /* Use the key directly as hash index. */
-#define MLX5_HLIST_WRITE_MOST 0x0002 /* List mostly used for append new. */
-
-/** Maximum size of string for naming the hlist table. */
-#define MLX5_HLIST_NAMESIZE			32
-
-struct mlx5_hlist;
-
-/**
- * Structure of the entry in the hash list, user should define its own struct
- * that contains this in order to store the data. The 'key' is 64-bits right
- * now and its user's responsibility to guarantee there is no collision.
- */
-struct mlx5_hlist_entry {
-	LIST_ENTRY(mlx5_hlist_entry) next; /* entry pointers in the list. */
-	uint32_t idx; /* Bucket index the entry belongs to. */
-	uint32_t ref_cnt; /* Reference count. */
-};
-
-/** Structure for hash head. */
-LIST_HEAD(mlx5_hlist_head, mlx5_hlist_entry);
-
-/**
- * Type of callback function for entry removal.
- *
- * @param list
- *   The hash list.
- * @param entry
- *   The entry in the list.
- */
-typedef void (*mlx5_hlist_remove_cb)(struct mlx5_hlist *list,
-				     struct mlx5_hlist_entry *entry);
-
-/**
- * Type of function for user defined matching.
- *
- * @param list
- *   The hash list.
- * @param entry
- *   The entry in the list.
- * @param key
- *   The new entry key.
- * @param ctx
- *   The pointer to new entry context.
- *
- * @return
- *   0 if matching, non-zero number otherwise.
- */
-typedef int (*mlx5_hlist_match_cb)(struct mlx5_hlist *list,
-				   struct mlx5_hlist_entry *entry,
-				   uint64_t key, void *ctx);
-
-/**
- * Type of function for user defined hash list entry creation.
- *
- * @param list
- *   The hash list.
- * @param key
- *   The key of the new entry.
- * @param ctx
- *   The pointer to new entry context.
- *
- * @return
- *   Pointer to allocated entry on success, NULL otherwise.
- */
-typedef struct mlx5_hlist_entry *(*mlx5_hlist_create_cb)
-				  (struct mlx5_hlist *list,
-				   uint64_t key, void *ctx);
-
-/* Hash list bucket head. */
-struct mlx5_hlist_bucket {
-	struct mlx5_hlist_head head; /* List head. */
-	rte_rwlock_t lock; /* Bucket lock. */
-	uint32_t gen_cnt; /* List modification will update generation count. */
-} __rte_cache_aligned;
-
-/**
- * Hash list table structure
- *
- * Entry in hash list could be reused if entry already exists, reference
- * count will increase and the existing entry returns.
- *
- * When destroy an entry from list, decrease reference count and only
- * destroy when no further reference.
- */
-struct mlx5_hlist {
-	char name[MLX5_HLIST_NAMESIZE]; /**< Name of the hash list. */
-	/**< number of heads, need to be power of 2. */
-	uint32_t table_sz;
-	uint32_t entry_sz; /**< Size of entry, used to allocate entry. */
-	/**< mask to get the index of the list heads. */
-	uint32_t mask;
-	bool direct_key; /* Use the new entry key directly as hash index. */
-	bool write_most; /* List mostly used for append new or destroy. */
-	void *ctx;
-	mlx5_hlist_create_cb cb_create; /**< entry create callback. */
-	mlx5_hlist_match_cb cb_match; /**< entry match callback. */
-	mlx5_hlist_remove_cb cb_remove; /**< entry remove callback. */
-	struct mlx5_hlist_bucket buckets[] __rte_cache_aligned;
-	/**< list bucket arrays. */
-};
-
-/**
- * Create a hash list table, the user can specify the list heads array size
- * of the table, now the size should be a power of 2 in order to get better
- * distribution for the entries. Each entry is a part of the whole data element
- * and the caller should be responsible for the data element's allocation and
- * cleanup / free. Key of each entry will be calculated with CRC in order to
- * generate a little fairer distribution.
- *
- * @param name
- *   Name of the hash list(optional).
- * @param size
- *   Heads array size of the hash list.
- * @param entry_size
- *   Entry size to allocate if cb_create not specified.
- * @param flags
- *   The hash list attribute flags.
- * @param cb_create
- *   Callback function for entry create.
- * @param cb_match
- *   Callback function for entry match.
- * @param cb_destroy
- *   Callback function for entry destroy.
- * @return
- *   Pointer of the hash list table created, NULL on failure.
- */
-struct mlx5_hlist *mlx5_hlist_create(const char *name, uint32_t size,
-				     uint32_t entry_size, uint32_t flags,
-				     mlx5_hlist_create_cb cb_create,
-				     mlx5_hlist_match_cb cb_match,
-				     mlx5_hlist_remove_cb cb_destroy);
-
-/**
- * Search an entry matching the key.
- *
- * Result returned might be destroyed by other thread, must use
- * this function only in main thread.
- *
- * @param h
- *   Pointer to the hast list table.
- * @param key
- *   Key for the searching entry.
- * @param ctx
- *   Common context parameter used by entry callback function.
- *
- * @return
- *   Pointer of the hlist entry if found, NULL otherwise.
- */
-struct mlx5_hlist_entry *mlx5_hlist_lookup(struct mlx5_hlist *h, uint64_t key,
-					   void *ctx);
-
-/**
- * Insert an entry to the hash list table, the entry is only part of whole data
- * element and a 64B key is used for matching. User should construct the key or
- * give a calculated hash signature and guarantee there is no collision.
- *
- * @param h
- *   Pointer to the hast list table.
- * @param entry
- *   Entry to be inserted into the hash list table.
- * @param ctx
- *   Common context parameter used by callback function.
- *
- * @return
- *   registered entry on success, NULL otherwise
- */
-struct mlx5_hlist_entry *mlx5_hlist_register(struct mlx5_hlist *h, uint64_t key,
-					     void *ctx);
-
-/**
- * Remove an entry from the hash list table. User should guarantee the validity
- * of the entry.
- *
- * @param h
- *   Pointer to the hast list table. (not used)
- * @param entry
- *   Entry to be removed from the hash list table.
- * @return
- *   0 on entry removed, 1 on entry still referenced.
- */
-int mlx5_hlist_unregister(struct mlx5_hlist *h, struct mlx5_hlist_entry *entry);
-
-/**
- * Destroy the hash list table, all the entries already inserted into the lists
- * will be handled by the callback function provided by the user (including
- * free if needed) before the table is freed.
- *
- * @param h
- *   Pointer to the hast list table.
- */
-void mlx5_hlist_destroy(struct mlx5_hlist *h);
-
 /************************ cache list *****************************/
 
 /** Maximum size of string for naming. */
-- 
2.25.1

[dpdk-dev] [PATCH v2 12/16] common/mlx5: share get ib device match function

[Matan Azrad]

From: Shiri Kuzin <shirik@nvidia.com>

The get_ib_device_match function iterates over the list of ib devices
returned by the get_device_list glue function and returns the ib device
matching the provided address.

Since this function is in use by several drivers, in this patch we
share the function in common part.

Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/linux/mlx5_common_os.c | 28 +++++++++++++++++++
 drivers/common/mlx5/linux/mlx5_common_os.h |  5 ++++
 drivers/common/mlx5/mlx5_common.h          |  2 ++
 drivers/common/mlx5/version.map            |  1 +
 drivers/compress/mlx5/mlx5_compress.c      | 30 +-------------------
 drivers/regex/mlx5/mlx5_regex.c            | 30 ++------------------
 drivers/vdpa/mlx5/mlx5_vdpa.c              | 32 ++--------------------
 7 files changed, 41 insertions(+), 87 deletions(-)

diff --git a/drivers/common/mlx5/linux/mlx5_common_os.c b/drivers/common/mlx5/linux/mlx5_common_os.c
index fba8245b8b..037147fe31 100644
--- a/drivers/common/mlx5/linux/mlx5_common_os.c
+++ b/drivers/common/mlx5/linux/mlx5_common_os.c
@@ -16,6 +16,7 @@
 
 #include "mlx5_common.h"
 #include "mlx5_common_log.h"
+#include "mlx5_common_os.h"
 #include "mlx5_glue.h"
 
 #ifdef MLX5_GLUE
@@ -423,3 +424,30 @@ mlx5_glue_constructor(void)
 	mlx5_glue = NULL;
 }
 
+struct ibv_device *
+mlx5_os_get_ib_device_match(struct rte_pci_addr *addr)
+{
+	int n;
+	struct ibv_device **ibv_list = mlx5_glue->get_device_list(&n);
+	struct ibv_device *ibv_match = NULL;
+
+	if (ibv_list == NULL) {
+		rte_errno = ENOSYS;
+		return NULL;
+	}
+	while (n-- > 0) {
+		struct rte_pci_addr paddr;
+
+		DRV_LOG(DEBUG, "Checking device \"%s\"..", ibv_list[n]->name);
+		if (mlx5_dev_to_pci_addr(ibv_list[n]->ibdev_path, &paddr) != 0)
+			continue;
+		if (rte_pci_addr_cmp(addr, &paddr) != 0)
+			continue;
+		ibv_match = ibv_list[n];
+		break;
+	}
+	if (ibv_match == NULL)
+		rte_errno = ENOENT;
+	mlx5_glue->free_device_list(ibv_list);
+	return ibv_match;
+}
diff --git a/drivers/common/mlx5/linux/mlx5_common_os.h b/drivers/common/mlx5/linux/mlx5_common_os.h
index d1c7e3dce0..9fff9cdb83 100644
--- a/drivers/common/mlx5/linux/mlx5_common_os.h
+++ b/drivers/common/mlx5/linux/mlx5_common_os.h
@@ -288,4 +288,9 @@ mlx5_os_free(void *addr)
 {
 	free(addr);
 }
+
+__rte_internal
+struct ibv_device *
+mlx5_os_get_ib_device_match(struct rte_pci_addr *addr);
+
 #endif /* RTE_PMD_MLX5_COMMON_OS_H_ */
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index f3c6beb23b..89aca32305 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -18,6 +18,7 @@
 
 #include "mlx5_prm.h"
 #include "mlx5_devx_cmds.h"
+#include "mlx5_common_os.h"
 
 /* Reported driver name. */
 #define MLX5_PCI_DRIVER_NAME "mlx5_pci"
@@ -215,6 +216,7 @@ enum mlx5_class {
 	MLX5_CLASS_VDPA = RTE_BIT64(1),
 	MLX5_CLASS_REGEX = RTE_BIT64(2),
 	MLX5_CLASS_COMPRESS = RTE_BIT64(3),
+	MLX5_CLASS_CRYPTO = RTE_BIT64(4),
 };
 
 #define MLX5_DBR_SIZE RTE_CACHE_LINE_SIZE
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index d16e484ffa..00df37e81a 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -111,6 +111,7 @@ INTERNAL {
 	mlx5_os_reg_mr;
 	mlx5_os_umem_reg;
 	mlx5_os_umem_dereg;
+	mlx5_os_get_ib_device_match; # WINDOWS_NO_EXPORT
 
 	mlx5_translate_port_name; # WINDOWS_NO_EXPORT
 
diff --git a/drivers/compress/mlx5/mlx5_compress.c b/drivers/compress/mlx5/mlx5_compress.c
index ec3c237512..ff988d26ef 100644
--- a/drivers/compress/mlx5/mlx5_compress.c
+++ b/drivers/compress/mlx5/mlx5_compress.c
@@ -647,34 +647,6 @@ mlx5_compress_dequeue_burst(void *queue_pair, struct rte_comp_op **ops,
 	return i;
 }
 
-static struct ibv_device *
-mlx5_compress_get_ib_device_match(struct rte_pci_addr *addr)
-{
-	int n;
-	struct ibv_device **ibv_list = mlx5_glue->get_device_list(&n);
-	struct ibv_device *ibv_match = NULL;
-
-	if (ibv_list == NULL) {
-		rte_errno = ENOSYS;
-		return NULL;
-	}
-	while (n-- > 0) {
-		struct rte_pci_addr paddr;
-
-		DRV_LOG(DEBUG, "Checking device \"%s\"..", ibv_list[n]->name);
-		if (mlx5_dev_to_pci_addr(ibv_list[n]->ibdev_path, &paddr) != 0)
-			continue;
-		if (rte_pci_addr_cmp(addr, &paddr) != 0)
-			continue;
-		ibv_match = ibv_list[n];
-		break;
-	}
-	if (ibv_match == NULL)
-		rte_errno = ENOENT;
-	mlx5_glue->free_device_list(ibv_list);
-	return ibv_match;
-}
-
 static void
 mlx5_compress_hw_global_release(struct mlx5_compress_priv *priv)
 {
@@ -774,7 +746,7 @@ mlx5_compress_pci_probe(struct rte_pci_driver *pci_drv,
 		rte_errno = ENOTSUP;
 		return -rte_errno;
 	}
-	ibv = mlx5_compress_get_ib_device_match(&pci_dev->addr);
+	ibv = mlx5_os_get_ib_device_match(&pci_dev->addr);
 	if (ibv == NULL) {
 		DRV_LOG(ERR, "No matching IB device for PCI slot "
 			PCI_PRI_FMT ".", pci_dev->addr.domain,
diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
index 82c485e50c..5b81666d21 100644
--- a/drivers/regex/mlx5/mlx5_regex.c
+++ b/drivers/regex/mlx5/mlx5_regex.c
@@ -11,6 +11,7 @@
 #include <rte_regexdev_driver.h>
 
 #include <mlx5_common_pci.h>
+#include <mlx5_common.h>
 #include <mlx5_glue.h>
 #include <mlx5_devx_cmds.h>
 #include <mlx5_prm.h>
@@ -52,33 +53,6 @@ mlx5_regex_close(struct rte_regexdev *dev __rte_unused)
 	return 0;
 }
 
-static struct ibv_device *
-mlx5_regex_get_ib_device_match(struct rte_pci_addr *addr)
-{
-	int n;
-	struct ibv_device **ibv_list = mlx5_glue->get_device_list(&n);
-	struct ibv_device *ibv_match = NULL;
-
-	if (!ibv_list) {
-		rte_errno = ENOSYS;
-		return NULL;
-	}
-	while (n-- > 0) {
-		struct rte_pci_addr pci_addr;
-
-		DRV_LOG(DEBUG, "Checking device \"%s\"..", ibv_list[n]->name);
-		if (mlx5_dev_to_pci_addr(ibv_list[n]->ibdev_path, &pci_addr))
-			continue;
-		if (rte_pci_addr_cmp(addr, &pci_addr))
-			continue;
-		ibv_match = ibv_list[n];
-		break;
-	}
-	if (!ibv_match)
-		rte_errno = ENOENT;
-	mlx5_glue->free_device_list(ibv_list);
-	return ibv_match;
-}
 static int
 mlx5_regex_engines_status(struct ibv_context *ctx, int num_engines)
 {
@@ -121,7 +95,7 @@ mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	int ret;
 	uint32_t val;
 
-	ibv = mlx5_regex_get_ib_device_match(&pci_dev->addr);
+	ibv = mlx5_os_get_ib_device_match(&pci_dev->addr);
 	if (!ibv) {
 		DRV_LOG(ERR, "No matching IB device for PCI slot "
 			PCI_PRI_FMT ".", pci_dev->addr.domain,
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 898e50f807..9a946ac8a7 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -472,34 +472,6 @@ static struct rte_vdpa_dev_ops mlx5_vdpa_ops = {
 	.reset_stats = mlx5_vdpa_reset_stats,
 };
 
-static struct ibv_device *
-mlx5_vdpa_get_ib_device_match(struct rte_pci_addr *addr)
-{
-	int n;
-	struct ibv_device **ibv_list = mlx5_glue->get_device_list(&n);
-	struct ibv_device *ibv_match = NULL;
-
-	if (!ibv_list) {
-		rte_errno = ENOSYS;
-		return NULL;
-	}
-	while (n-- > 0) {
-		struct rte_pci_addr pci_addr;
-
-		DRV_LOG(DEBUG, "Checking device \"%s\"..", ibv_list[n]->name);
-		if (mlx5_dev_to_pci_addr(ibv_list[n]->ibdev_path, &pci_addr))
-			continue;
-		if (rte_pci_addr_cmp(addr, &pci_addr))
-			continue;
-		ibv_match = ibv_list[n];
-		break;
-	}
-	if (!ibv_match)
-		rte_errno = ENOENT;
-	mlx5_glue->free_device_list(ibv_list);
-	return ibv_match;
-}
-
 /* Try to disable ROCE by Netlink\Devlink. */
 static int
 mlx5_vdpa_nl_roce_disable(const char *addr)
@@ -595,7 +567,7 @@ mlx5_vdpa_roce_disable(struct rte_pci_addr *addr, struct ibv_device **ibv)
 		struct ibv_device *ibv_new;
 
 		for (r = MLX5_VDPA_MAX_RETRIES; r; r--) {
-			ibv_new = mlx5_vdpa_get_ib_device_match(addr);
+			ibv_new = mlx5_os_get_ib_device_match(addr);
 			if (ibv_new) {
 				*ibv = ibv_new;
 				return 0;
@@ -698,7 +670,7 @@ mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	struct mlx5_hca_attr attr;
 	int ret;
 
-	ibv = mlx5_vdpa_get_ib_device_match(&pci_dev->addr);
+	ibv = mlx5_os_get_ib_device_match(&pci_dev->addr);
 	if (!ibv) {
 		DRV_LOG(ERR, "No matching IB device for PCI slot "
 			PCI_PRI_FMT ".", pci_dev->addr.domain,
-- 
2.25.1

[dpdk-dev] [PATCH v2 13/16] common/mlx5: support general obj CREDENTIAL create

[Matan Azrad]

From: Dekel Peled <dekelp@nvidia.com>

CREDENTIAL object is used for any crypto operation in wrapped mode.

This patch add support of CREDENTIAL object create operation.
Add reading of CREDENTIAL support capability.
Add function to create general object type CREDENTIAL, using DevX API.

Signed-off-by: Dekel Peled <dekelp@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/mlx5_devx_cmds.c | 53 +++++++++++++++++++++++++++-
 drivers/common/mlx5/mlx5_devx_cmds.h | 17 +++++++--
 drivers/common/mlx5/mlx5_prm.h       | 23 ++++++++++++
 drivers/common/mlx5/version.map      |  1 +
 4 files changed, 91 insertions(+), 3 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c
index c0061741e8..c0a0853c3a 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.c
+++ b/drivers/common/mlx5/mlx5_devx_cmds.c
@@ -754,6 +754,8 @@ mlx5_devx_cmd_query_hca_attr(void *ctx,
 		       MLX5_GENERAL_OBJ_TYPES_CAP_DEK);
 	attr->import_kek = !!(general_obj_types_supported &
 			      MLX5_GENERAL_OBJ_TYPES_CAP_IMPORT_KEK);
+	attr->credential = !!(general_obj_types_supported &
+			      MLX5_GENERAL_OBJ_TYPES_CAP_CREDENTIAL);
 	attr->crypto_login = !!(general_obj_types_supported &
 				MLX5_GENERAL_OBJ_TYPES_CAP_CRYPTO_LOGIN);
 	/* Add reading of other GENERAL_OBJ_TYPES_CAP bits above this line. */
@@ -2512,6 +2514,55 @@ mlx5_devx_cmd_create_import_kek_obj(void *ctx,
 	return import_kek_obj;
 }
 
+/**
+ * Create general object of type CREDENTIAL using DevX API.
+ *
+ * @param[in] ctx
+ *   Context returned from mlx5 open_device() glue function.
+ * @param [in] attr
+ *   Pointer to CREDENTIAL attributes structure.
+ *
+ * @return
+ *   The DevX object created, NULL otherwise and rte_errno is set.
+ */
+struct mlx5_devx_obj *
+mlx5_devx_cmd_create_credential_obj(void *ctx,
+				    struct mlx5_devx_credential_attr *attr)
+{
+	uint32_t in[MLX5_ST_SZ_DW(create_credential_in)] = {0};
+	uint32_t out[MLX5_ST_SZ_DW(general_obj_out_cmd_hdr)] = {0};
+	struct mlx5_devx_obj *credential_obj = NULL;
+	void *ptr = NULL, *credential_addr = NULL;
+
+	credential_obj = mlx5_malloc(MLX5_MEM_ZERO, sizeof(*credential_obj),
+				     0, SOCKET_ID_ANY);
+	if (credential_obj == NULL) {
+		DRV_LOG(ERR, "Failed to allocate CREDENTIAL object data");
+		rte_errno = ENOMEM;
+		return NULL;
+	}
+	ptr = MLX5_ADDR_OF(create_credential_in, in, hdr);
+	MLX5_SET(general_obj_in_cmd_hdr, ptr, opcode,
+		 MLX5_CMD_OP_CREATE_GENERAL_OBJECT);
+	MLX5_SET(general_obj_in_cmd_hdr, ptr, obj_type,
+		 MLX5_GENERAL_OBJ_TYPE_CREDENTIAL);
+	ptr = MLX5_ADDR_OF(create_credential_in, in, credential);
+	MLX5_SET(credential, ptr, credential_role, attr->credential_role);
+	credential_addr = MLX5_ADDR_OF(credential, ptr, credential);
+	memcpy(credential_addr, (void *)(attr->credential),
+	       MLX5_CRYPTO_CREDENTIAL_SIZE);
+	credential_obj->obj = mlx5_glue->devx_obj_create(ctx, in, sizeof(in),
+							 out, sizeof(out));
+	if (credential_obj->obj == NULL) {
+		rte_errno = errno;
+		DRV_LOG(ERR, "Failed to create CREDENTIAL object using DevX.");
+		mlx5_free(credential_obj);
+		return NULL;
+	}
+	credential_obj->id = MLX5_GET(general_obj_out_cmd_hdr, out, obj_id);
+	return credential_obj;
+}
+
 /**
  * Create general object of type CRYPTO_LOGIN using DevX API.
  *
@@ -2551,7 +2602,7 @@ mlx5_devx_cmd_create_crypto_login_obj(void *ctx,
 		 attr->session_import_kek_ptr);
 	credential_addr = MLX5_ADDR_OF(crypto_login, ptr, credential);
 	memcpy(credential_addr, (void *)(attr->credential),
-	       MLX5_CRYPTO_LOGIN_CREDENTIAL_SIZE);
+	       MLX5_CRYPTO_CREDENTIAL_SIZE);
 	crypto_login_obj->obj = mlx5_glue->devx_obj_create(ctx, in, sizeof(in),
 							   out, sizeof(out));
 	if (crypto_login_obj->obj == NULL) {
diff --git a/drivers/common/mlx5/mlx5_devx_cmds.h b/drivers/common/mlx5/mlx5_devx_cmds.h
index 709e28bfba..811e7a1462 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.h
+++ b/drivers/common/mlx5/mlx5_devx_cmds.h
@@ -143,6 +143,7 @@ struct mlx5_hca_attr {
 	uint32_t aes_xts:1; /* AES-XTS crypto is supported. */
 	uint32_t dek:1; /* General obj type DEK is supported. */
 	uint32_t import_kek:1; /* General obj type IMPORT_KEK supported. */
+	uint32_t credential:1; /* General obj type CREDENTIAL supported. */
 	uint32_t crypto_login:1; /* General obj type CRYPTO_LOGIN supported. */
 	uint32_t regexp_num_of_engines;
 	uint32_t log_max_ft_sampler_num:8;
@@ -459,13 +460,20 @@ struct mlx5_devx_import_kek_attr {
 	uint8_t key[MLX5_CRYPTO_KEY_MAX_SIZE];
 };
 
-#define MLX5_CRYPTO_LOGIN_CREDENTIAL_SIZE	48
+#define MLX5_CRYPTO_CREDENTIAL_SIZE	48
+
+struct mlx5_devx_credential_attr {
+	uint64_t modify_field_select;
+	uint32_t state:8;
+	uint32_t credential_role:8;
+	uint8_t credential[MLX5_CRYPTO_CREDENTIAL_SIZE];
+};
 
 struct mlx5_devx_crypto_login_attr {
 	uint64_t modify_field_select;
 	uint32_t credential_pointer:24;
 	uint32_t session_import_kek_ptr:24;
-	uint8_t credential[MLX5_CRYPTO_LOGIN_CREDENTIAL_SIZE];
+	uint8_t credential[MLX5_CRYPTO_CREDENTIAL_SIZE];
 };
 
 /* mlx5_devx_cmds.c */
@@ -629,6 +637,11 @@ struct mlx5_devx_obj *
 mlx5_devx_cmd_create_import_kek_obj(void *ctx,
 				    struct mlx5_devx_import_kek_attr *attr);
 
+__rte_internal
+struct mlx5_devx_obj *
+mlx5_devx_cmd_create_credential_obj(void *ctx,
+				    struct mlx5_devx_credential_attr *attr);
+
 __rte_internal
 struct mlx5_devx_obj *
 mlx5_devx_cmd_create_crypto_login_obj(void *ctx,
diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h
index a9dcbfa63c..432c8fdb63 100644
--- a/drivers/common/mlx5/mlx5_prm.h
+++ b/drivers/common/mlx5/mlx5_prm.h
@@ -1187,6 +1187,8 @@ enum {
 			(1ULL << MLX5_GENERAL_OBJ_TYPE_DEK)
 #define MLX5_GENERAL_OBJ_TYPES_CAP_IMPORT_KEK \
 			(1ULL << MLX5_GENERAL_OBJ_TYPE_IMPORT_KEK)
+#define MLX5_GENERAL_OBJ_TYPES_CAP_CREDENTIAL \
+			(1ULL << MLX5_GENERAL_OBJ_TYPE_CREDENTIAL)
 #define MLX5_GENERAL_OBJ_TYPES_CAP_CRYPTO_LOGIN \
 			(1ULL << MLX5_GENERAL_OBJ_TYPE_CRYPTO_LOGIN)
 
@@ -2490,6 +2492,7 @@ enum {
 	MLX5_GENERAL_OBJ_TYPE_VIRTQ = 0x000d,
 	MLX5_GENERAL_OBJ_TYPE_VIRTIO_Q_COUNTERS = 0x001c,
 	MLX5_GENERAL_OBJ_TYPE_IMPORT_KEK = 0x001d,
+	MLX5_GENERAL_OBJ_TYPE_CREDENTIAL = 0x001e,
 	MLX5_GENERAL_OBJ_TYPE_CRYPTO_LOGIN = 0x001f,
 	MLX5_GENERAL_OBJ_TYPE_FLEX_PARSE_GRAPH = 0x0022,
 	MLX5_GENERAL_OBJ_TYPE_FLOW_METER_ASO = 0x0024,
@@ -2603,6 +2606,26 @@ struct mlx5_ifc_create_import_kek_in_bits {
 	struct mlx5_ifc_import_kek_bits import_kek;
 };
 
+enum {
+	MLX5_CREDENTIAL_ROLE_OFFICER = 0x0,
+	MLX5_CREDENTIAL_ROLE_USER = 0x1,
+};
+
+struct mlx5_ifc_credential_bits {
+	u8 modify_field_select[0x40];
+	u8 state[0x8];
+	u8 reserved_at_48[0x10];
+	u8 credential_role[0x8];
+	u8 reserved_at_60[0x1a0];
+	u8 credential[0x180];
+	u8 reserved_at_380[0x480];
+};
+
+struct mlx5_ifc_create_credential_in_bits {
+	struct mlx5_ifc_general_obj_in_cmd_hdr_bits hdr;
+	struct mlx5_ifc_credential_bits credential;
+};
+
 struct mlx5_ifc_crypto_login_bits {
 	u8 modify_field_select[0x40];
 	u8 reserved_at_40[0x48];
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index 00df37e81a..1885cb8f6a 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -29,6 +29,7 @@ INTERNAL {
 	mlx5_devx_cmd_create_geneve_tlv_option;
         mlx5_devx_cmd_create_dek_obj;
         mlx5_devx_cmd_create_import_kek_obj;
+        mlx5_devx_cmd_create_credential_obj;
         mlx5_devx_cmd_create_crypto_login_obj;
 	mlx5_devx_cmd_destroy;
 	mlx5_devx_cmd_flow_counter_alloc;
-- 
2.25.1

[dpdk-dev] [PATCH v2 14/16] common/mlx5: add crypto register structs and defs

[Matan Azrad]

From: Dekel Peled <dekelp@nvidia.com>

Encryption key management requires use of several related registers.
This patch adds the relevant structs and values, according to PRM
definitions.

Signed-off-by: Dekel Peled <dekelp@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/mlx5_prm.h | 41 ++++++++++++++++++++++++++++++++++
 1 file changed, 41 insertions(+)

diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h
index 432c8fdb63..c2cd2d9f70 100644
--- a/drivers/common/mlx5/mlx5_prm.h
+++ b/drivers/common/mlx5/mlx5_prm.h
@@ -3307,6 +3307,10 @@ enum {
 
 enum {
 	MLX5_REGISTER_ID_MTUTC  = 0x9055,
+	MLX5_CRYPTO_OPERATIONAL_REGISTER_ID = 0xC002,
+	MLX5_CRYPTO_COMMISSIONING_REGISTER_ID = 0xC003,
+	MLX5_IMPORT_KEK_HANDLE_REGISTER_ID = 0xC004,
+	MLX5_CREDENTIAL_HANDLE_REGISTER_ID = 0xC005,
 };
 
 struct mlx5_ifc_register_mtutc_bits {
@@ -3324,6 +3328,43 @@ struct mlx5_ifc_register_mtutc_bits {
 #define MLX5_MTUTC_TIMESTAMP_MODE_INTERNAL_TIMER 0
 #define MLX5_MTUTC_TIMESTAMP_MODE_REAL_TIME 1
 
+struct mlx5_ifc_crypto_operational_register_bits {
+	u8 wrapped_crypto_operational[0x1];
+	u8 reserved_at_1[0x1b];
+	u8 kek_size[0x4];
+	u8 reserved_at_20[0x20];
+	u8 credential[0x140];
+	u8 kek[0x100];
+	u8 reserved_at_280[0x180];
+};
+
+struct mlx5_ifc_crypto_commissioning_register_bits {
+	u8 token[0x1]; /* TODO: add size after PRM update */
+};
+
+struct mlx5_ifc_import_kek_handle_register_bits {
+	struct mlx5_ifc_crypto_login_bits crypto_login_object;
+	struct mlx5_ifc_import_kek_bits import_kek_object;
+	u8 reserved_at_200[0x4];
+	u8 write_operation[0x4];
+	u8 import_kek_id[0x18];
+	u8 reserved_at_220[0xe0];
+};
+
+struct mlx5_ifc_credential_handle_register_bits {
+	struct mlx5_ifc_crypto_login_bits crypto_login_object;
+	struct mlx5_ifc_credential_bits credential_object;
+	u8 reserved_at_200[0x4];
+	u8 write_operation[0x4];
+	u8 credential_id[0x18];
+	u8 reserved_at_220[0xe0];
+};
+
+enum {
+	MLX5_REGISTER_ADD_OPERATION = 0x1,
+	MLX5_REGISTER_DELETE_OPERATION = 0x2,
+};
+
 struct mlx5_ifc_parse_graph_arc_bits {
 	u8 start_inner_tunnel[0x1];
 	u8 reserved_at_1[0x7];
-- 
2.25.1

[dpdk-dev] [PATCH v2 15/16] common/mlx5: support register write access

[Matan Azrad]

From: Dekel Peled <dekelp@nvidia.com>

This patch adds support of write operation to NIC registers.

Signed-off-by: Dekel Peled <dekelp@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/mlx5_devx_cmds.c | 67 +++++++++++++++++++++++++++-
 drivers/common/mlx5/mlx5_devx_cmds.h |  4 ++
 drivers/common/mlx5/version.map      | 21 ++++++---
 3 files changed, 83 insertions(+), 9 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c
index c0a0853c3a..0b421933ce 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.c
+++ b/drivers/common/mlx5/mlx5_devx_cmds.c
@@ -12,7 +12,6 @@
 #include "mlx5_common_log.h"
 #include "mlx5_malloc.h"
 
-
 /**
  * Perform read access to the registers. Reads data from register
  * and writes ones to the specified buffer.
@@ -61,7 +60,7 @@ mlx5_devx_cmd_register_read(void *ctx, uint16_t reg_id, uint32_t arg,
 	if (status) {
 		int syndrome = MLX5_GET(access_register_out, out, syndrome);
 
-		DRV_LOG(DEBUG, "Failed to access NIC register 0x%X, "
+		DRV_LOG(DEBUG, "Failed to read access NIC register 0x%X, "
 			       "status %x, syndrome = %x",
 			       reg_id, status, syndrome);
 		return -1;
@@ -74,6 +73,70 @@ mlx5_devx_cmd_register_read(void *ctx, uint16_t reg_id, uint32_t arg,
 	return rc;
 }
 
+/**
+ * Perform write access to the registers.
+ *
+ * @param[in] ctx
+ *   Context returned from mlx5 open_device() glue function.
+ * @param[in] reg_id
+ *   Register identifier according to the PRM.
+ * @param[in] arg
+ *   Register access auxiliary parameter according to the PRM.
+ * @param[out] data
+ *   Pointer to the buffer containing data to write.
+ * @param[in] dw_cnt
+ *   Buffer size in double words (32bit units).
+ *
+ * @return
+ *   0 on success, a negative value otherwise.
+ */
+int
+mlx5_devx_cmd_register_write(void *ctx, uint16_t reg_id, uint32_t arg,
+			     uint32_t *data, uint32_t dw_cnt)
+{
+	uint32_t in[MLX5_ST_SZ_DW(access_register_in) +
+		    MLX5_ACCESS_REGISTER_DATA_DWORD_MAX] = {0};
+	uint32_t out[MLX5_ST_SZ_DW(access_register_out)] = {0};
+	int status, rc;
+	void *ptr;
+
+	MLX5_ASSERT(data && dw_cnt);
+	MLX5_ASSERT(dw_cnt <= MLX5_ACCESS_REGISTER_DATA_DWORD_MAX);
+	if (dw_cnt > MLX5_ACCESS_REGISTER_DATA_DWORD_MAX) {
+		DRV_LOG(ERR, "Data to write exceeds max size");
+		return -1;
+	}
+	MLX5_SET(access_register_in, in, opcode,
+		 MLX5_CMD_OP_ACCESS_REGISTER_USER);
+	MLX5_SET(access_register_in, in, op_mod,
+		 MLX5_ACCESS_REGISTER_IN_OP_MOD_WRITE);
+	MLX5_SET(access_register_in, in, register_id, reg_id);
+	MLX5_SET(access_register_in, in, argument, arg);
+	ptr = MLX5_ADDR_OF(access_register_in, in, register_data);
+	memcpy(ptr, data, dw_cnt * sizeof(uint32_t));
+	rc = mlx5_glue->devx_general_cmd(ctx, in, sizeof(in), out, sizeof(out));
+
+	rc = mlx5_glue->devx_general_cmd(ctx, in,
+					 MLX5_ST_SZ_BYTES(access_register_in) +
+					 dw_cnt * sizeof(uint32_t),
+					 out, sizeof(out));
+	if (rc)
+		goto error;
+	status = MLX5_GET(access_register_out, out, status);
+	if (status) {
+		int syndrome = MLX5_GET(access_register_out, out, syndrome);
+
+		DRV_LOG(DEBUG, "Failed to write access NIC register 0x%X, "
+			       "status %x, syndrome = %x",
+			       reg_id, status, syndrome);
+		return -1;
+	}
+	return 0;
+error:
+	rc = (rc > 0) ? -rc : rc;
+	return rc;
+}
+
 /**
  * Allocate flow counters via devx interface.
  *
diff --git a/drivers/common/mlx5/mlx5_devx_cmds.h b/drivers/common/mlx5/mlx5_devx_cmds.h
index 811e7a1462..ce570ad28a 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.h
+++ b/drivers/common/mlx5/mlx5_devx_cmds.h
@@ -566,6 +566,10 @@ __rte_internal
 int mlx5_devx_cmd_register_read(void *ctx, uint16_t reg_id,
 				uint32_t arg, uint32_t *data, uint32_t dw_cnt);
 
+__rte_internal
+int mlx5_devx_cmd_register_write(void *ctx, uint16_t reg_id,
+				 uint32_t arg, uint32_t *data, uint32_t dw_cnt);
+
 __rte_internal
 struct mlx5_devx_obj *
 mlx5_devx_cmd_create_geneve_tlv_option(void *ctx,
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index 1885cb8f6a..1dc2d063ff 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -13,8 +13,17 @@ INTERNAL {
 	mlx5_dev_to_pci_addr; # WINDOWS_NO_EXPORT
 
 	mlx5_devx_cmd_alloc_pd;
+	mlx5_devx_alloc_uar;
+
 	mlx5_devx_cmd_create_cq;
+	mlx5_devx_cmd_create_credential_obj;
+	mlx5_devx_cmd_create_crypto_login_obj;
+	mlx5_devx_cmd_create_dek_obj;
 	mlx5_devx_cmd_create_flex_parser;
+	mlx5_devx_cmd_create_flow_hit_aso_obj;
+	mlx5_devx_cmd_create_flow_meter_aso_obj;
+	mlx5_devx_cmd_create_geneve_tlv_option;
+	mlx5_devx_cmd_create_import_kek_obj;
 	mlx5_devx_cmd_create_qp;
 	mlx5_devx_cmd_create_rq;
 	mlx5_devx_cmd_create_rqt;
@@ -24,13 +33,6 @@ INTERNAL {
 	mlx5_devx_cmd_create_tis;
 	mlx5_devx_cmd_create_virtio_q_counters; # WINDOWS_NO_EXPORT
 	mlx5_devx_cmd_create_virtq;
-	mlx5_devx_cmd_create_flow_hit_aso_obj;
-	mlx5_devx_cmd_create_flow_meter_aso_obj;
-	mlx5_devx_cmd_create_geneve_tlv_option;
-        mlx5_devx_cmd_create_dek_obj;
-        mlx5_devx_cmd_create_import_kek_obj;
-        mlx5_devx_cmd_create_credential_obj;
-        mlx5_devx_cmd_create_crypto_login_obj;
 	mlx5_devx_cmd_destroy;
 	mlx5_devx_cmd_flow_counter_alloc;
 	mlx5_devx_cmd_flow_counter_query;
@@ -51,12 +53,17 @@ INTERNAL {
 	mlx5_devx_cmd_queue_counter_alloc; # WINDOWS_NO_EXPORT
 	mlx5_devx_cmd_queue_counter_query; # WINDOWS_NO_EXPORT
 	mlx5_devx_cmd_register_read;
+	mlx5_devx_cmd_register_write;
 	mlx5_devx_cmd_wq_query; # WINDOWS_NO_EXPORT
 	mlx5_devx_get_out_command_status;
 	mlx5_devx_alloc_uar; # WINDOWS_NO_EXPORT
 
+
 	mlx5_devx_cq_create;
 	mlx5_devx_cq_destroy;
+
+	mlx5_devx_get_out_command_status;
+
 	mlx5_devx_rq_create;
 	mlx5_devx_rq_destroy;
 	mlx5_devx_sq_create;
-- 
2.25.1

[dpdk-dev] [PATCH v2 16/16] common/mlx5: add UMR and RDMA write WQE defines

[Matan Azrad]

From: Suanming Mou <suanmingm@nvidia.com>

This patch adds the struct defining UMR and RDMA write WQEs.

Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Signed-off-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/mlx5_prm.h | 187 +++++++++++++++++++++------------
 1 file changed, 121 insertions(+), 66 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h
index c2cd2d9f70..1ffee5fd56 100644
--- a/drivers/common/mlx5/mlx5_prm.h
+++ b/drivers/common/mlx5/mlx5_prm.h
@@ -412,6 +412,127 @@ struct mlx5_cqe_ts {
 	uint8_t op_own;
 };
 
+struct mlx5_wqe_rseg {
+	uint64_t raddr;
+	uint32_t rkey;
+	uint32_t reserved;
+} __rte_packed;
+
+#define MLX5_UMRC_IF_OFFSET 31u
+#define MLX5_UMRC_KO_OFFSET 16u
+#define MLX5_UMRC_TO_BS_OFFSET 0u
+
+struct mlx5_wqe_umr_cseg {
+	uint32_t if_cf_toe_cq_res;
+	uint32_t ko_to_bs;
+	uint64_t mkey_mask;
+	uint32_t rsvd1[8];
+} __rte_packed;
+
+struct mlx5_wqe_mkey_cseg {
+	uint32_t fr_res_af_sf;
+	uint32_t qpn_mkey;
+	uint32_t reserved2;
+	uint32_t flags_pd;
+	uint64_t start_addr;
+	uint64_t len;
+	uint32_t bsf_octword_size;
+	uint32_t reserved3[4];
+	uint32_t translations_octword_size;
+	uint32_t res4_lps;
+	uint32_t reserved;
+} __rte_packed;
+
+enum {
+	MLX5_BSF_SIZE_16B = 0x0,
+	MLX5_BSF_SIZE_32B = 0x1,
+	MLX5_BSF_SIZE_64B = 0x2,
+	MLX5_BSF_SIZE_128B = 0x3,
+};
+
+enum {
+	MLX5_BSF_P_TYPE_SIGNATURE = 0x0,
+	MLX5_BSF_P_TYPE_CRYPTO = 0x1,
+};
+
+enum {
+	MLX5_ENCRYPTION_ORDER_ENCRYPTED_WIRE_SIGNATURE = 0x0,
+	MLX5_ENCRYPTION_ORDER_ENCRYPTED_MEMORY_SIGNATURE = 0x1,
+	MLX5_ENCRYPTION_ORDER_ENCRYPTED_RAW_WIRE = 0x2,
+	MLX5_ENCRYPTION_ORDER_ENCRYPTED_RAW_MEMORY = 0x3,
+};
+
+enum {
+	MLX5_ENCRYPTION_STANDARD_AES_XTS = 0x0,
+};
+
+enum {
+	MLX5_BLOCK_SIZE_512B	= 0x1,
+	MLX5_BLOCK_SIZE_520B	= 0x2,
+	MLX5_BLOCK_SIZE_4096B	= 0x3,
+	MLX5_BLOCK_SIZE_4160B	= 0x4,
+	MLX5_BLOCK_SIZE_1MB	= 0x5,
+	MLX5_BLOCK_SIZE_4048B	= 0x6,
+};
+
+#define MLX5_BSF_SIZE_OFFSET		30
+#define MLX5_BSF_P_TYPE_OFFSET		24
+#define MLX5_ENCRYPTION_ORDER_OFFSET	16
+#define MLX5_BLOCK_SIZE_OFFSET		24
+
+struct mlx5_wqe_umr_bsf_seg {
+	/*
+	 * bs_bpt_eo_es contains:
+	 * bs	bsf_size		2 bits at MLX5_BSF_SIZE_OFFSET
+	 * bpt	bsf_p_type		2 bits at MLX5_BSF_P_TYPE_OFFSET
+	 * eo	encryption_order	4 bits at MLX5_ENCRYPTION_ORDER_OFFSET
+	 * es	encryption_standard	4 bits at offset 0
+	 */
+	uint32_t bs_bpt_eo_es;
+	uint32_t raw_data_size;
+	/*
+	 * bsp_res contains:
+	 * bsp	crypto_block_size_pointer	8 bits at MLX5_BLOCK_SIZE_OFFSET
+	 * res	reserved 24 bits
+	 */
+	uint32_t bsp_res;
+	uint32_t reserved0;
+	uint8_t xts_initial_tweak[16];
+	/*
+	 * res_dp contains:
+	 * res	reserved 8 bits
+	 * dp	dek_pointer		24 bits at offset 0
+	 */
+	uint32_t res_dp;
+	uint32_t reserved1;
+	uint64_t keytag;
+	uint32_t reserved2[4];
+} __rte_packed;
+
+#ifdef PEDANTIC
+#pragma GCC diagnostic ignored "-Wpedantic"
+#endif
+
+struct mlx5_umr_wqe {
+	struct mlx5_wqe_cseg ctr;
+	struct mlx5_wqe_umr_cseg ucseg;
+	struct mlx5_wqe_mkey_cseg mkc;
+	union {
+		struct mlx5_wqe_dseg kseg[0];
+		struct mlx5_wqe_umr_bsf_seg bsf[0];
+	};
+} __rte_packed;
+
+struct mlx5_rdma_write_wqe {
+	struct mlx5_wqe_cseg ctr;
+	struct mlx5_wqe_rseg rseg;
+	struct mlx5_wqe_dseg dseg[0];
+} __rte_packed;
+
+#ifdef PEDANTIC
+#pragma GCC diagnostic error "-Wpedantic"
+#endif
+
 /* GGA */
 /* MMO metadata segment */
 
@@ -1096,72 +1217,6 @@ struct mlx5_ifc_create_mkey_in_bits {
 	u8 klm_pas_mtt[][0x20];
 };
 
-enum {
-	MLX5_BSF_SIZE_16B = 0x0,
-	MLX5_BSF_SIZE_32B = 0x1,
-	MLX5_BSF_SIZE_64B = 0x2,
-	MLX5_BSF_SIZE_128B = 0x3,
-};
-
-enum {
-	MLX5_BSF_P_TYPE_SIGNATURE = 0x0,
-	MLX5_BSF_P_TYPE_CRYPTO = 0x1,
-};
-
-enum {
-	MLX5_ENCRYPTION_ORDER_ENCRYPTED_WIRE_SIGNATURE = 0x0,
-	MLX5_ENCRYPTION_ORDER_ENCRYPTED_MEMORY_SIGNATURE = 0x1,
-	MLX5_ENCRYPTION_ORDER_ENCRYPTED_RAW_WIRE = 0x2,
-	MLX5_ENCRYPTION_ORDER_ENCRYPTED_RAW_MEMORY = 0x3,
-};
-
-enum {
-	MLX5_ENCRYPTION_STANDARD_AES_XTS = 0x0,
-};
-
-enum {
-	MLX5_BLOCK_SIZE_512B	= 0x1,
-	MLX5_BLOCK_SIZE_520B	= 0x2,
-	MLX5_BLOCK_SIZE_4096B	= 0x3,
-	MLX5_BLOCK_SIZE_4160B	= 0x4,
-	MLX5_BLOCK_SIZE_1MB	= 0x5,
-	MLX5_BLOCK_SIZE_4048B	= 0x6,
-};
-
-#define MLX5_BSF_SIZE_OFFSET		30
-#define MLX5_BSF_P_TYPE_OFFSET		24
-#define MLX5_ENCRYPTION_ORDER_OFFSET	16
-#define MLX5_BLOCK_SIZE_OFFSET		24
-
-struct mlx5_wqe_umr_bsf_seg {
-	/*
-	 * bs_bpt_eo_es contains:
-	 * bs	bsf_size		2 bits at MLX5_BSF_SIZE_OFFSET
-	 * bpt	bsf_p_type		2 bits at MLX5_BSF_P_TYPE_OFFSET
-	 * eo	encryption_order	4 bits at MLX5_ENCRYPTION_ORDER_OFFSET
-	 * es	encryption_standard	4 bits at offset 0
-	 */
-	uint32_t bs_bpt_eo_es;
-	uint32_t raw_data_size;
-	/*
-	 * bsp_res contains:
-	 * bsp	crypto_block_size_pointer	8 bits at MLX5_BLOCK_SIZE_OFFSET
-	 * res	reserved 24 bits
-	 */
-	uint32_t bsp_res;
-	uint32_t reserved0;
-	uint8_t xts_initial_tweak[16];
-	/*
-	 * res_dp contains:
-	 * res	reserved 8 bits
-	 * dp	dek_pointer		24 bits at offset 0
-	 */
-	uint32_t res_dp;
-	uint32_t reserved1;
-	uint64_t keytag;
-	uint32_t reserved2[4];
-} __rte_packed;
-
 enum {
 	MLX5_GET_HCA_CAP_OP_MOD_GENERAL_DEVICE = 0x0 << 1,
 	MLX5_GET_HCA_CAP_OP_MOD_ETHERNET_OFFLOAD_CAPS = 0x1 << 1,
-- 
2.25.1

[dpdk-dev] [PATCH v2 00/15] drivers: introduce mlx5 crypto PMD

[Matan Azrad]

Add a new PMD for Nvidia devices- crypto PMD.
The crypto PMD will be supported on Nvidia ConnectX6
The crypto PMD will add the support of encryption and decryption using the AES-XTS symmetric algorithm.
The crypto PMD requires rdma-core and uses mlx5 DevX.

v2:
Add data-path part.

Shiri Kuzin (10):
  drivers: introduce mlx5 crypto PMD
  crypto/mlx5: add DEK object management
  crypto/mlx5: support session operations
  crypto/mlx5: add basic operations
  crypto/mlx5: support queue pairs operations
  crypto/mlx5: add dev stop and start operations
  crypto/mlx5: add memory region management
  crypto/mlx5: create login object using DevX
  crypto/mlx5: adjust to the multiple data unit API
  crypto/mlx5: set feature flags and capabilities

Suanming Mou (5):
  crypto/mlx5: add keytag device argument
  crypto/mlx5: add maximum segments device argument
  crypto/mlx5: add WQE set initialization
  crypto/mlx5: add enqueue and dequeue operations
  crypto/mlx5: add statistic get and reset operations

 MAINTAINERS                             |    4 +
 doc/guides/cryptodevs/features/mlx5.ini |   37 +
 doc/guides/cryptodevs/index.rst         |    1 +
 doc/guides/cryptodevs/mlx5.rst          |  152 ++++
 doc/guides/rel_notes/release_21_05.rst  |    5 +
 drivers/common/mlx5/mlx5_common_pci.c   |   14 +
 drivers/common/mlx5/mlx5_common_pci.h   |   21 +-
 drivers/crypto/meson.build              |    1 +
 drivers/crypto/mlx5/meson.build         |   27 +
 drivers/crypto/mlx5/mlx5_crypto.c       | 1092 +++++++++++++++++++++++
 drivers/crypto/mlx5/mlx5_crypto.h       |   92 ++
 drivers/crypto/mlx5/mlx5_crypto_dek.c   |  136 +++
 drivers/crypto/mlx5/mlx5_crypto_utils.h |   19 +
 drivers/crypto/mlx5/version.map         |    3 +
 14 files changed, 1594 insertions(+), 10 deletions(-)
 create mode 100644 doc/guides/cryptodevs/features/mlx5.ini
 create mode 100644 doc/guides/cryptodevs/mlx5.rst
 create mode 100644 drivers/crypto/mlx5/meson.build
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto.c
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto.h
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto_dek.c
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto_utils.h
 create mode 100644 drivers/crypto/mlx5/version.map

-- 
2.25.1

[dpdk-dev] [PATCH v2 01/15] drivers: introduce mlx5 crypto PMD

[Matan Azrad]

From: Shiri Kuzin <shirik@nvidia.com>

Add a new PMD for Nvidia devices- crypto PMD.

The crypto PMD will be supported starting Nvidia ConnectX6 and
BlueField2.

The crypto PMD will add the support of encryption and decryption using
the AES-XTS symmetric algorithm.

The crypto PMD requires rdma-core and uses mlx5 DevX.

This patch adds the PCI probing, basic functions, build files and
log utility.

Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 MAINTAINERS                             |   4 +
 drivers/common/mlx5/mlx5_common_pci.c   |  14 ++
 drivers/common/mlx5/mlx5_common_pci.h   |  21 +-
 drivers/crypto/meson.build              |   1 +
 drivers/crypto/mlx5/meson.build         |  26 +++
 drivers/crypto/mlx5/mlx5_crypto.c       | 272 ++++++++++++++++++++++++
 drivers/crypto/mlx5/mlx5_crypto_utils.h |  19 ++
 drivers/crypto/mlx5/version.map         |   3 +
 8 files changed, 350 insertions(+), 10 deletions(-)
 create mode 100644 drivers/crypto/mlx5/meson.build
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto.c
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto_utils.h
 create mode 100644 drivers/crypto/mlx5/version.map

diff --git a/MAINTAINERS b/MAINTAINERS
index 44f3d322ed..707e5a43b4 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1081,6 +1081,10 @@ F: drivers/crypto/octeontx2/
 F: doc/guides/cryptodevs/octeontx2.rst
 F: doc/guides/cryptodevs/features/octeontx2.ini
 
+Mellanox mlx5
+M: Matan Azrad <matan@nvidia.com>
+F: drivers/crypto/mlx5/
+
 Null Crypto
 M: Declan Doherty <declan.doherty@intel.com>
 F: drivers/crypto/null/
diff --git a/drivers/common/mlx5/mlx5_common_pci.c b/drivers/common/mlx5/mlx5_common_pci.c
index 3f16cd21cf..8a47afee20 100644
--- a/drivers/common/mlx5/mlx5_common_pci.c
+++ b/drivers/common/mlx5/mlx5_common_pci.c
@@ -31,6 +31,7 @@ static const struct {
 	{ .name = "net", .driver_class = MLX5_CLASS_NET },
 	{ .name = "regex", .driver_class = MLX5_CLASS_REGEX },
 	{ .name = "compress", .driver_class = MLX5_CLASS_COMPRESS },
+	{ .name = "crypto", .driver_class = MLX5_CLASS_CRYPTO },
 };
 
 static const unsigned int mlx5_class_combinations[] = {
@@ -38,13 +39,26 @@ static const unsigned int mlx5_class_combinations[] = {
 	MLX5_CLASS_VDPA,
 	MLX5_CLASS_REGEX,
 	MLX5_CLASS_COMPRESS,
+	MLX5_CLASS_CRYPTO,
 	MLX5_CLASS_NET | MLX5_CLASS_REGEX,
 	MLX5_CLASS_VDPA | MLX5_CLASS_REGEX,
 	MLX5_CLASS_NET | MLX5_CLASS_COMPRESS,
 	MLX5_CLASS_VDPA | MLX5_CLASS_COMPRESS,
 	MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS,
+	MLX5_CLASS_NET | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_VDPA | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_REGEX | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_COMPRESS | MLX5_CLASS_CRYPTO,
 	MLX5_CLASS_NET | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS,
 	MLX5_CLASS_VDPA | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS,
+	MLX5_CLASS_NET | MLX5_CLASS_REGEX | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_VDPA | MLX5_CLASS_REGEX | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_NET | MLX5_CLASS_COMPRESS | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_VDPA | MLX5_CLASS_COMPRESS | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_NET | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS |
+	MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_VDPA | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS |
+	MLX5_CLASS_CRYPTO,
 	/* New class combination should be added here. */
 };
 
diff --git a/drivers/common/mlx5/mlx5_common_pci.h b/drivers/common/mlx5/mlx5_common_pci.h
index de89bb98bc..cb8d2f5f87 100644
--- a/drivers/common/mlx5/mlx5_common_pci.h
+++ b/drivers/common/mlx5/mlx5_common_pci.h
@@ -9,17 +9,18 @@
  * @file
  *
  * RTE Mellanox PCI Driver Interface
- * Mellanox ConnectX PCI device supports multiple class: net,vdpa,regex and
- * compress devices. This layer enables creating such multiple class of devices
- * on a single PCI device by allowing to bind multiple class specific device
- * driver to attach to mlx5_pci driver.
+ * Mellanox ConnectX PCI device supports multiple class: net,vdpa,regex,compress
+ * and crypto devices. This layer enables creating such multiple class of
+ * devices on a single PCI device by allowing to bind multiple class specific
+ * device driver to attach to mlx5_pci driver.
  *
- * -----------    ------------    -------------    ----------------
- * |   mlx5  |    |   mlx5   |    |   mlx5    |    |     mlx5     |
- * | net pmd |    | vdpa pmd |    | regex pmd |    | compress pmd |
- * -----------    ------------    -------------    ----------------
- *      \              \                    /              /
- *       \              \                  /              /
+ * --------    --------    ---------    ------------    ----------
+ * | mlx5 |    | mlx5 |    | mlx5  |    |   mlx5   |    |  mlx5  |
+ * | net  |    | vdpa |    | regex |    | compress |    | crypto |
+ * | pmd  |    | pmd  |    |  pmd  |    |   pmd    |    |  pmd   |
+ * --------    --------    ---------    ------------    ----------
+ *      \              \         |          /              /
+ *       \              \        |         /              /
  *        \              \_--------------_/              /
  *         \_______________|   mlx5     |_______________/
  *                         | pci common |
diff --git a/drivers/crypto/meson.build b/drivers/crypto/meson.build
index b9fdf9392f..7d9bd64cd1 100644
--- a/drivers/crypto/meson.build
+++ b/drivers/crypto/meson.build
@@ -16,6 +16,7 @@ drivers = [
         'dpaa2_sec',
         'kasumi',
         'mvsam',
+        'mlx5',
         'nitrox',
         'null',
         'octeontx',
diff --git a/drivers/crypto/mlx5/meson.build b/drivers/crypto/mlx5/meson.build
new file mode 100644
index 0000000000..5bf0912766
--- /dev/null
+++ b/drivers/crypto/mlx5/meson.build
@@ -0,0 +1,26 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright 2021 Mellanox Technologies, Ltd
+
+if not is_linux
+	build = false
+	reason = 'only supported on Linux'
+	subdir_done()
+endif
+
+fmt_name = 'mlx5_crypto'
+deps += ['common_mlx5', 'eal', 'cryptodev']
+sources = files(
+	'mlx5_crypto.c',
+)
+cflags_options = [
+	'-std=c11',
+	'-Wno-strict-prototypes',
+	'-D_BSD_SOURCE',
+	'-D_DEFAULT_SOURCE',
+	'-D_XOPEN_SOURCE=600'
+]
+foreach option:cflags_options
+	if cc.has_argument(option)
+		cflags += option
+	endif
+endforeach
\ No newline at end of file
diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
new file mode 100644
index 0000000000..8a3a307f7b
--- /dev/null
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -0,0 +1,272 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2021 Mellanox Technologies, Ltd
+ */
+
+#include <rte_malloc.h>
+#include <rte_log.h>
+#include <rte_errno.h>
+#include <rte_pci.h>
+#include <rte_crypto.h>
+#include <rte_cryptodev.h>
+#include <rte_cryptodev_pmd.h>
+
+#include <mlx5_glue.h>
+#include <mlx5_common.h>
+#include <mlx5_common_pci.h>
+#include <mlx5_devx_cmds.h>
+#include <mlx5_common_os.h>
+
+#include "mlx5_crypto_utils.h"
+
+#define MLX5_CRYPTO_DRIVER_NAME mlx5_crypto
+#define MLX5_CRYPTO_LOG_NAME    pmd.crypto.mlx5
+
+struct mlx5_crypto_priv {
+	TAILQ_ENTRY(mlx5_crypto_priv) next;
+	struct ibv_context *ctx; /* Device context. */
+	struct rte_pci_device *pci_dev;
+	struct rte_cryptodev *crypto_dev;
+	void *uar; /* User Access Region. */
+	uint32_t pdn; /* Protection Domain number. */
+	struct ibv_pd *pd;
+};
+
+TAILQ_HEAD(mlx5_crypto_privs, mlx5_crypto_priv) mlx5_crypto_priv_list =
+				TAILQ_HEAD_INITIALIZER(mlx5_crypto_priv_list);
+static pthread_mutex_t priv_list_lock = PTHREAD_MUTEX_INITIALIZER;
+
+int mlx5_crypto_logtype;
+
+uint8_t mlx5_crypto_driver_id;
+
+static const char mlx5_crypto_drv_name[] = RTE_STR(MLX5_CRYPTO_DRIVER_NAME);
+
+static const struct rte_driver mlx5_drv = {
+	.name = mlx5_crypto_drv_name,
+	.alias = mlx5_crypto_drv_name
+};
+
+static struct cryptodev_driver mlx5_cryptodev_driver;
+
+static struct rte_cryptodev_ops mlx5_crypto_ops = {
+	.dev_configure			= NULL,
+	.dev_start			= NULL,
+	.dev_stop			= NULL,
+	.dev_close			= NULL,
+	.dev_infos_get			= NULL,
+	.stats_get			= NULL,
+	.stats_reset			= NULL,
+	.queue_pair_setup		= NULL,
+	.queue_pair_release		= NULL,
+	.sym_session_get_size		= NULL,
+	.sym_session_configure		= NULL,
+	.sym_session_clear		= NULL,
+	.sym_get_raw_dp_ctx_size	= NULL,
+	.sym_configure_raw_dp_ctx	= NULL,
+};
+
+static void
+mlx5_crypto_hw_global_release(struct mlx5_crypto_priv *priv)
+{
+	if (priv->pd != NULL) {
+		claim_zero(mlx5_glue->dealloc_pd(priv->pd));
+		priv->pd = NULL;
+	}
+	if (priv->uar != NULL) {
+		mlx5_glue->devx_free_uar(priv->uar);
+		priv->uar = NULL;
+	}
+}
+
+static int
+mlx5_crypto_pd_create(struct mlx5_crypto_priv *priv)
+{
+#ifdef HAVE_IBV_FLOW_DV_SUPPORT
+	struct mlx5dv_obj obj;
+	struct mlx5dv_pd pd_info;
+	int ret;
+
+	priv->pd = mlx5_glue->alloc_pd(priv->ctx);
+	if (priv->pd == NULL) {
+		DRV_LOG(ERR, "Failed to allocate PD.");
+		return errno ? -errno : -ENOMEM;
+	}
+	obj.pd.in = priv->pd;
+	obj.pd.out = &pd_info;
+	ret = mlx5_glue->dv_init_obj(&obj, MLX5DV_OBJ_PD);
+	if (ret != 0) {
+		DRV_LOG(ERR, "Fail to get PD object info.");
+		mlx5_glue->dealloc_pd(priv->pd);
+		priv->pd = NULL;
+		return -errno;
+	}
+	priv->pdn = pd_info.pdn;
+	return 0;
+#else
+	(void)priv;
+	DRV_LOG(ERR, "Cannot get pdn - no DV support.");
+	return -ENOTSUP;
+#endif /* HAVE_IBV_FLOW_DV_SUPPORT */
+}
+
+static int
+mlx5_crypto_hw_global_prepare(struct mlx5_crypto_priv *priv)
+{
+	if (mlx5_crypto_pd_create(priv) != 0)
+		return -1;
+	priv->uar = mlx5_devx_alloc_uar(priv->ctx, -1);
+	if (priv->uar == NULL || mlx5_os_get_devx_uar_reg_addr(priv->uar) ==
+	    NULL) {
+		rte_errno = errno;
+		claim_zero(mlx5_glue->dealloc_pd(priv->pd));
+		DRV_LOG(ERR, "Failed to allocate UAR.");
+		return -1;
+	}
+	return 0;
+}
+
+/**
+ * DPDK callback to register a PCI device.
+ *
+ * This function spawns crypto device out of a given PCI device.
+ *
+ * @param[in] pci_drv
+ *   PCI driver structure (mlx5_crypto_driver).
+ * @param[in] pci_dev
+ *   PCI device information.
+ *
+ * @return
+ *   0 on success, 1 to skip this driver, a negative errno value otherwise
+ *   and rte_errno is set.
+ */
+static int
+mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
+			struct rte_pci_device *pci_dev)
+{
+	struct ibv_device *ibv;
+	struct rte_cryptodev *crypto_dev;
+	struct ibv_context *ctx;
+	struct mlx5_crypto_priv *priv;
+	struct mlx5_hca_attr attr = { 0 };
+	struct rte_cryptodev_pmd_init_params init_params = {
+		.name = "",
+		.private_data_size = sizeof(struct mlx5_crypto_priv),
+		.socket_id = pci_dev->device.numa_node,
+		.max_nb_queue_pairs =
+				RTE_CRYPTODEV_PMD_DEFAULT_MAX_NB_QUEUE_PAIRS,
+	};
+	RTE_SET_USED(pci_drv);
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		DRV_LOG(ERR, "Non-primary process type is not supported.");
+		rte_errno = ENOTSUP;
+		return -rte_errno;
+	}
+	ibv = mlx5_os_get_ib_device_match(&pci_dev->addr);
+	if (ibv == NULL) {
+		DRV_LOG(ERR, "No matching IB device for PCI slot "
+			PCI_PRI_FMT ".", pci_dev->addr.domain,
+			pci_dev->addr.bus, pci_dev->addr.devid,
+			pci_dev->addr.function);
+		return -rte_errno;
+	}
+	DRV_LOG(INFO, "PCI information matches for device \"%s\".", ibv->name);
+	ctx = mlx5_glue->dv_open_device(ibv);
+	if (ctx == NULL) {
+		DRV_LOG(ERR, "Failed to open IB device \"%s\".", ibv->name);
+		rte_errno = ENODEV;
+		return -rte_errno;
+	}
+	if (mlx5_devx_cmd_query_hca_attr(ctx, &attr) != 0 ||
+	    attr.crypto == 0 || attr.aes_xts == 0) {
+		DRV_LOG(ERR, "Not enough capabilities to support crypto "
+			"operations, maybe old FW/OFED version?");
+		claim_zero(mlx5_glue->close_device(ctx));
+		rte_errno = ENOTSUP;
+		return -ENOTSUP;
+	}
+	crypto_dev = rte_cryptodev_pmd_create(ibv->name, &pci_dev->device,
+					&init_params);
+	if (crypto_dev == NULL) {
+		DRV_LOG(ERR, "Failed to create device \"%s\".", ibv->name);
+		claim_zero(mlx5_glue->close_device(ctx));
+		return -ENODEV;
+	}
+	DRV_LOG(INFO,
+		"Crypto device %s was created successfully.", ibv->name);
+	crypto_dev->dev_ops = &mlx5_crypto_ops;
+	crypto_dev->dequeue_burst = NULL;
+	crypto_dev->enqueue_burst = NULL;
+	crypto_dev->feature_flags = RTE_CRYPTODEV_FF_HW_ACCELERATED;
+	crypto_dev->driver_id = mlx5_crypto_driver_id;
+	priv = crypto_dev->data->dev_private;
+	priv->ctx = ctx;
+	priv->pci_dev = pci_dev;
+	priv->crypto_dev = crypto_dev;
+	if (mlx5_crypto_hw_global_prepare(priv) != 0) {
+		rte_cryptodev_pmd_destroy(priv->crypto_dev);
+		claim_zero(mlx5_glue->close_device(priv->ctx));
+		return -1;
+	}
+	pthread_mutex_lock(&priv_list_lock);
+	TAILQ_INSERT_TAIL(&mlx5_crypto_priv_list, priv, next);
+	pthread_mutex_unlock(&priv_list_lock);
+	return 0;
+}
+
+static int
+mlx5_crypto_pci_remove(struct rte_pci_device *pdev)
+{
+	struct mlx5_crypto_priv *priv = NULL;
+
+	pthread_mutex_lock(&priv_list_lock);
+	TAILQ_FOREACH(priv, &mlx5_crypto_priv_list, next)
+		if (rte_pci_addr_cmp(&priv->pci_dev->addr, &pdev->addr) != 0)
+			break;
+	if (priv)
+		TAILQ_REMOVE(&mlx5_crypto_priv_list, priv, next);
+	pthread_mutex_unlock(&priv_list_lock);
+	if (priv) {
+		mlx5_crypto_hw_global_release(priv);
+		rte_cryptodev_pmd_destroy(priv->crypto_dev);
+		claim_zero(mlx5_glue->close_device(priv->ctx));
+	}
+	return 0;
+}
+
+static const struct rte_pci_id mlx5_crypto_pci_id_map[] = {
+		{
+			RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
+					PCI_DEVICE_ID_MELLANOX_CONNECTX6)
+		},
+		{
+			.vendor_id = 0
+		}
+	};
+
+static struct mlx5_pci_driver mlx5_crypto_driver = {
+	.driver_class = MLX5_CLASS_CRYPTO,
+	.pci_driver = {
+		.driver = {
+			.name = RTE_STR(MLX5_CRYPTO_DRIVER_NAME),
+		},
+		.id_table = mlx5_crypto_pci_id_map,
+		.probe = mlx5_crypto_pci_probe,
+		.remove = mlx5_crypto_pci_remove,
+		.drv_flags = 0,
+	},
+};
+
+RTE_INIT(rte_mlx5_crypto_init)
+{
+	mlx5_common_init();
+	if (mlx5_glue != NULL)
+		mlx5_pci_driver_register(&mlx5_crypto_driver);
+}
+
+RTE_PMD_REGISTER_CRYPTO_DRIVER(mlx5_cryptodev_driver, mlx5_drv,
+			       mlx5_crypto_driver_id);
+
+RTE_LOG_REGISTER(mlx5_crypto_logtype, MLX5_CRYPTO_LOG_NAME, NOTICE)
+RTE_PMD_EXPORT_NAME(MLX5_CRYPTO_DRIVER_NAME, __COUNTER__);
+RTE_PMD_REGISTER_PCI_TABLE(MLX5_CRYPTO_DRIVER_NAME, mlx5_crypto_pci_id_map);
+RTE_PMD_REGISTER_KMOD_DEP(MLX5_CRYPTO_DRIVER_NAME, "* ib_uverbs & mlx5_core & mlx5_ib");
diff --git a/drivers/crypto/mlx5/mlx5_crypto_utils.h b/drivers/crypto/mlx5/mlx5_crypto_utils.h
new file mode 100644
index 0000000000..cef4b07a36
--- /dev/null
+++ b/drivers/crypto/mlx5/mlx5_crypto_utils.h
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2021 Mellanox Technologies, Ltd
+ */
+
+#ifndef RTE_PMD_MLX5_CRYPTO_UTILS_H_
+#define RTE_PMD_MLX5_CRYPTO_UTILS_H_
+
+#include <mlx5_common.h>
+
+extern int mlx5_crypto_logtype;
+
+#define MLX5_CRYPTO_LOG_PREFIX "mlx5_crypto"
+/* Generic printf()-like logging macro with automatic line feed. */
+#define DRV_LOG(level, ...) \
+	PMD_DRV_LOG_(level, mlx5_crypto_logtype, MLX5_CRYPTO_LOG_PREFIX, \
+		__VA_ARGS__ PMD_DRV_LOG_STRIP PMD_DRV_LOG_OPAREN, \
+		PMD_DRV_LOG_CPAREN)
+
+#endif /* RTE_PMD_MLX5_CRYPTO_UTILS_H_ */
diff --git a/drivers/crypto/mlx5/version.map b/drivers/crypto/mlx5/version.map
new file mode 100644
index 0000000000..4a76d1d52d
--- /dev/null
+++ b/drivers/crypto/mlx5/version.map
@@ -0,0 +1,3 @@
+DPDK_21 {
+	local: *;
+};
-- 
2.25.1

[dpdk-dev] [PATCH v2 02/15] crypto/mlx5: add DEK object management

[Matan Azrad]

From: Shiri Kuzin <shirik@nvidia.com>

A DEK(Data encryption Key) is an mlx5 HW object which represents the
cipher algorithm key.
The DEKs are used during data encryption/decryption operations.

In symmetric algorithms like AES-STS, we use the same DEK for both
encryption and decryption.

Use the mlx5 hash-list tool to manage the DEK objects in the PMD.

Provide the compare, create and destroy functions to manage DEKs in
hash-list and introduce an internal API to setup and unset the DEK
management and to prepare and destroy specific DEK object.

The DEK hash-list will be created in dev_configure routine and
destroyed in dev_close routine.

Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/crypto/mlx5/meson.build       |   1 +
 drivers/crypto/mlx5/mlx5_crypto.c     |  44 +++++----
 drivers/crypto/mlx5/mlx5_crypto.h     |  51 ++++++++++
 drivers/crypto/mlx5/mlx5_crypto_dek.c | 136 ++++++++++++++++++++++++++
 4 files changed, 215 insertions(+), 17 deletions(-)
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto.h
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto_dek.c

diff --git a/drivers/crypto/mlx5/meson.build b/drivers/crypto/mlx5/meson.build
index 5bf0912766..0666c35094 100644
--- a/drivers/crypto/mlx5/meson.build
+++ b/drivers/crypto/mlx5/meson.build
@@ -11,6 +11,7 @@ fmt_name = 'mlx5_crypto'
 deps += ['common_mlx5', 'eal', 'cryptodev']
 sources = files(
 	'mlx5_crypto.c',
+	'mlx5_crypto_dek.c',
 )
 cflags_options = [
 	'-std=c11',
diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index 8a3a307f7b..e7c70c521f 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -3,12 +3,9 @@
  */
 
 #include <rte_malloc.h>
-#include <rte_log.h>
 #include <rte_errno.h>
+#include <rte_log.h>
 #include <rte_pci.h>
-#include <rte_crypto.h>
-#include <rte_cryptodev.h>
-#include <rte_cryptodev_pmd.h>
 
 #include <mlx5_glue.h>
 #include <mlx5_common.h>
@@ -17,19 +14,10 @@
 #include <mlx5_common_os.h>
 
 #include "mlx5_crypto_utils.h"
+#include "mlx5_crypto.h"
 
 #define MLX5_CRYPTO_DRIVER_NAME mlx5_crypto
-#define MLX5_CRYPTO_LOG_NAME    pmd.crypto.mlx5
-
-struct mlx5_crypto_priv {
-	TAILQ_ENTRY(mlx5_crypto_priv) next;
-	struct ibv_context *ctx; /* Device context. */
-	struct rte_pci_device *pci_dev;
-	struct rte_cryptodev *crypto_dev;
-	void *uar; /* User Access Region. */
-	uint32_t pdn; /* Protection Domain number. */
-	struct ibv_pd *pd;
-};
+#define MLX5_CRYPTO_LOG_NAME pmd.crypto.mlx5
 
 TAILQ_HEAD(mlx5_crypto_privs, mlx5_crypto_priv) mlx5_crypto_priv_list =
 				TAILQ_HEAD_INITIALIZER(mlx5_crypto_priv_list);
@@ -48,11 +36,33 @@ static const struct rte_driver mlx5_drv = {
 
 static struct cryptodev_driver mlx5_cryptodev_driver;
 
+static int
+mlx5_crypto_dev_configure(struct rte_cryptodev *dev,
+		struct rte_cryptodev_config *config __rte_unused)
+{
+	struct mlx5_crypto_priv *priv = dev->data->dev_private;
+
+	if (mlx5_crypto_dek_setup(priv) != 0) {
+		DRV_LOG(ERR, "Dek hash list creation has failed.");
+		return -ENOMEM;
+	}
+	return 0;
+}
+
+static int
+mlx5_crypto_dev_close(struct rte_cryptodev *dev)
+{
+	struct mlx5_crypto_priv *priv = dev->data->dev_private;
+
+	mlx5_crypto_dek_unset(priv);
+	return 0;
+}
+
 static struct rte_cryptodev_ops mlx5_crypto_ops = {
-	.dev_configure			= NULL,
+	.dev_configure			= mlx5_crypto_dev_configure,
 	.dev_start			= NULL,
 	.dev_stop			= NULL,
-	.dev_close			= NULL,
+	.dev_close			= mlx5_crypto_dev_close,
 	.dev_infos_get			= NULL,
 	.stats_get			= NULL,
 	.stats_reset			= NULL,
diff --git a/drivers/crypto/mlx5/mlx5_crypto.h b/drivers/crypto/mlx5/mlx5_crypto.h
new file mode 100644
index 0000000000..4ec67a7e0f
--- /dev/null
+++ b/drivers/crypto/mlx5/mlx5_crypto.h
@@ -0,0 +1,51 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2021 Mellanox Technologies, Ltd
+ */
+
+#ifndef MLX5_CRYPTO_H_
+#define MLX5_CRYPTO_H_
+
+#include <stdbool.h>
+
+#include <rte_cryptodev.h>
+#include <rte_cryptodev_pmd.h>
+
+#include <mlx5_common_utils.h>
+
+#define MLX5_CRYPTO_DEK_HTABLE_SZ (1 << 11)
+#define MLX5_CRYPTO_KEY_LENGTH 80
+
+struct mlx5_crypto_priv {
+	TAILQ_ENTRY(mlx5_crypto_priv) next;
+	struct ibv_context *ctx; /* Device context. */
+	struct rte_pci_device *pci_dev;
+	struct rte_cryptodev *crypto_dev;
+	void *uar; /* User Access Region. */
+	uint32_t pdn; /* Protection Domain number. */
+	struct ibv_pd *pd;
+	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
+};
+
+struct mlx5_crypto_dek {
+	struct mlx5_hlist_entry entry; /* Pointer to DEK hash list entry. */
+	struct mlx5_devx_obj *obj; /* Pointer to DEK DevX object. */
+	uint8_t data[MLX5_CRYPTO_KEY_LENGTH]; /* DEK key data. */
+	bool size_is_48; /* Whether the key\data size is 48 bytes or not. */
+};
+
+int
+mlx5_crypto_dek_destroy(struct mlx5_crypto_priv *priv,
+			struct mlx5_crypto_dek *dek);
+
+struct mlx5_crypto_dek *
+mlx5_crypto_dek_prepare(struct mlx5_crypto_priv *priv,
+			struct rte_crypto_cipher_xform *cipher);
+
+int
+mlx5_crypto_dek_setup(struct mlx5_crypto_priv *priv);
+
+void
+mlx5_crypto_dek_unset(struct mlx5_crypto_priv *priv);
+
+#endif /* MLX5_CRYPTO_H_ */
+
diff --git a/drivers/crypto/mlx5/mlx5_crypto_dek.c b/drivers/crypto/mlx5/mlx5_crypto_dek.c
new file mode 100644
index 0000000000..c76e208845
--- /dev/null
+++ b/drivers/crypto/mlx5/mlx5_crypto_dek.c
@@ -0,0 +1,136 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2018 Mellanox Technologies, Ltd
+ */
+
+#include <rte_ip.h>
+#include <rte_common.h>
+#include <rte_errno.h>
+#include <rte_log.h>
+
+#include <mlx5_prm.h>
+#include <mlx5_devx_cmds.h>
+
+#include "mlx5_crypto_utils.h"
+#include "mlx5_crypto.h"
+
+struct mlx5_crypto_dek_ctx {
+	struct rte_crypto_cipher_xform *cipher;
+	struct mlx5_crypto_priv *priv;
+};
+
+int
+mlx5_crypto_dek_destroy(struct mlx5_crypto_priv *priv,
+			struct mlx5_crypto_dek *dek)
+{
+	return mlx5_hlist_unregister(priv->dek_hlist, &dek->entry);
+}
+
+struct mlx5_crypto_dek *
+mlx5_crypto_dek_prepare(struct mlx5_crypto_priv *priv,
+			struct rte_crypto_cipher_xform *cipher)
+{
+	struct mlx5_hlist *dek_hlist = priv->dek_hlist;
+	struct mlx5_crypto_dek_ctx dek_ctx = {
+		.cipher = cipher,
+		.priv = priv,
+	};
+	struct rte_crypto_cipher_xform *cipher_ctx = cipher;
+	uint64_t key64 = __rte_raw_cksum(cipher_ctx->key.data,
+					 cipher_ctx->key.length, 0);
+	struct mlx5_hlist_entry *entry = mlx5_hlist_register(dek_hlist,
+							     key64, &dek_ctx);
+
+	return entry == NULL ? NULL :
+			     container_of(entry, struct mlx5_crypto_dek, entry);
+}
+
+static int
+mlx5_crypto_dek_match_cb(struct mlx5_hlist *list __rte_unused,
+			 struct mlx5_hlist_entry *entry,
+			 uint64_t key __rte_unused, void *cb_ctx)
+{
+	struct mlx5_crypto_dek_ctx *ctx = cb_ctx;
+	struct rte_crypto_cipher_xform *cipher_ctx = ctx->cipher;
+	struct mlx5_crypto_dek *dek =
+			container_of(entry, typeof(*dek), entry);
+	uint32_t key_len = dek->size_is_48 ? 48 : 80;
+
+	if (key_len != cipher_ctx->key.length)
+		return -1;
+	return memcmp(cipher_ctx->key.data, dek->data, key_len);
+}
+
+static struct mlx5_hlist_entry *
+mlx5_crypto_dek_create_cb(struct mlx5_hlist *list __rte_unused,
+			  uint64_t key __rte_unused, void *cb_ctx)
+{
+	struct mlx5_crypto_dek_ctx *ctx = cb_ctx;
+	struct rte_crypto_cipher_xform *cipher_ctx = ctx->cipher;
+	struct mlx5_crypto_dek *dek = rte_zmalloc(__func__, sizeof(*dek),
+						  RTE_CACHE_LINE_SIZE);
+	struct mlx5_devx_dek_attr dek_attr = {
+		.pd = ctx->priv->pdn,
+		.key_purpose = MLX5_CRYPTO_KEY_PURPOSE_AES_XTS,
+		.has_keytag = 1,
+	};
+
+	if (dek == NULL) {
+		DRV_LOG(ERR, "Failed to allocate dek memory.");
+		return NULL;
+	}
+	switch (cipher_ctx->key.length) {
+	case 48:
+		dek->size_is_48 = true;
+		dek_attr.key_size = MLX5_CRYPTO_KEY_SIZE_128b;
+		break;
+	case 80:
+		dek->size_is_48 = false;
+		dek_attr.key_size = MLX5_CRYPTO_KEY_SIZE_256b;
+		break;
+	default:
+		DRV_LOG(ERR, "Key size not supported.");
+		return NULL;
+	}
+	rte_memcpy(&dek_attr.key, cipher_ctx->key.data, cipher_ctx->key.length);
+	dek->obj = mlx5_devx_cmd_create_dek_obj(ctx->priv->ctx, &dek_attr);
+	if (dek->obj == NULL) {
+		rte_free(dek);
+		return NULL;
+	}
+	rte_memcpy(&dek->data, cipher_ctx->key.data, cipher_ctx->key.length);
+	return &dek->entry;
+}
+
+static void
+mlx5_crypto_dek_remove_cb(struct mlx5_hlist *list __rte_unused,
+			  struct mlx5_hlist_entry *entry)
+{
+	struct mlx5_crypto_dek *dek =
+		container_of(entry, typeof(*dek), entry);
+
+	claim_zero(mlx5_devx_cmd_destroy(dek->obj));
+	rte_free(dek);
+}
+
+
+int
+mlx5_crypto_dek_setup(struct mlx5_crypto_priv *priv)
+{
+	priv->dek_hlist = mlx5_hlist_create("dek_hlist",
+				 MLX5_CRYPTO_DEK_HTABLE_SZ,
+				 0, MLX5_HLIST_WRITE_MOST |
+				 MLX5_HLIST_DIRECT_KEY,
+				 mlx5_crypto_dek_create_cb,
+				 mlx5_crypto_dek_match_cb,
+				 mlx5_crypto_dek_remove_cb);
+	if (priv->dek_hlist == NULL)
+		return -1;
+	return 0;
+}
+
+void
+mlx5_crypto_dek_unset(struct mlx5_crypto_priv *priv)
+{
+	mlx5_hlist_destroy(priv->dek_hlist);
+	priv->dek_hlist = NULL;
+}
-- 
2.25.1

[dpdk-dev] [PATCH v2 03/15] crypto/mlx5: support session operations

[Matan Azrad]

From: Shiri Kuzin <shirik@nvidia.com>

Sessions are used in symmetric transformations in order to prepare
objects and data for packet processing stage.

A mlx5 session includes iv_offset, pointer to mlx5_crypto_dek struct,
bsf_size, bsf_p_type, encryption_order and encryption standard.

Implement the next session operations:
        mlx5_crypto_sym_session_get_size- returns the size of the mlx5
	session struct.
	mlx5_crypto_sym_session_configure- prepares the DEK hash-list
	and saves all the session data.
	mlx5_crypto_sym_session_clear - destroys the DEK hash-list.

Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/crypto/mlx5/mlx5_crypto.c | 96 ++++++++++++++++++++++++++++++-
 1 file changed, 93 insertions(+), 3 deletions(-)

diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index e7c70c521f..c494ed00d1 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -3,6 +3,7 @@
  */
 
 #include <rte_malloc.h>
+#include <rte_mempool.h>
 #include <rte_errno.h>
 #include <rte_log.h>
 #include <rte_pci.h>
@@ -36,6 +37,24 @@ static const struct rte_driver mlx5_drv = {
 
 static struct cryptodev_driver mlx5_cryptodev_driver;
 
+struct mlx5_crypto_session {
+	uint32_t bs_bpt_eo_es;
+	/*
+	 * bsf_size, bsf_p_type, encryption_order and encryption standard,
+	 * saved in big endian format.
+	 */
+	uint32_t iv_offset:16;
+	/* Starting point for Initialisation Vector. */
+	struct mlx5_crypto_dek *dek; /* Pointer to dek struct. */
+	uint32_t dek_id; /* DEK ID */
+} __rte_packed;
+
+static unsigned int
+mlx5_crypto_sym_session_get_size(struct rte_cryptodev *dev __rte_unused)
+{
+	return sizeof(struct mlx5_crypto_session);
+}
+
 static int
 mlx5_crypto_dev_configure(struct rte_cryptodev *dev,
 		struct rte_cryptodev_config *config __rte_unused)
@@ -58,6 +77,77 @@ mlx5_crypto_dev_close(struct rte_cryptodev *dev)
 	return 0;
 }
 
+static int
+mlx5_crypto_sym_session_configure(struct rte_cryptodev *dev,
+				  struct rte_crypto_sym_xform *xform,
+				  struct rte_cryptodev_sym_session *session,
+				  struct rte_mempool *mp)
+{
+	struct mlx5_crypto_priv *priv = dev->data->dev_private;
+	struct mlx5_crypto_session *sess_private_data;
+	struct rte_crypto_cipher_xform *cipher;
+	uint8_t encryption_order;
+	int ret;
+
+	if (unlikely(xform->next != NULL)) {
+		DRV_LOG(ERR, "Xform next is not supported.");
+		return -ENOTSUP;
+	}
+	if (unlikely((xform->type != RTE_CRYPTO_SYM_XFORM_CIPHER) ||
+		     (xform->cipher.algo != RTE_CRYPTO_CIPHER_AES_XTS))) {
+		DRV_LOG(ERR, "Only AES-XTS algorithm is supported.");
+		return -ENOTSUP;
+	}
+	ret = rte_mempool_get(mp, (void *)&sess_private_data);
+	if (ret != 0) {
+		DRV_LOG(ERR,
+			"Failed to get session %p private data from mempool.",
+			sess_private_data);
+		return -ENOMEM;
+	}
+	cipher = &xform->cipher;
+	sess_private_data->dek = mlx5_crypto_dek_prepare(priv, cipher);
+	if (sess_private_data->dek == NULL) {
+		rte_mempool_put(mp, sess_private_data);
+		DRV_LOG(ERR, "Failed to prepare dek.");
+		return -ENOMEM;
+	}
+	if (cipher->op == RTE_CRYPTO_CIPHER_OP_ENCRYPT)
+		encryption_order = MLX5_ENCRYPTION_ORDER_ENCRYPTED_RAW_MEMORY;
+	else
+		encryption_order = MLX5_ENCRYPTION_ORDER_ENCRYPTED_RAW_WIRE;
+	sess_private_data->bs_bpt_eo_es = rte_cpu_to_be_32
+			(MLX5_BSF_SIZE_64B << MLX5_BSF_SIZE_OFFSET |
+			 MLX5_BSF_P_TYPE_CRYPTO << MLX5_BSF_P_TYPE_OFFSET |
+			 encryption_order << MLX5_ENCRYPTION_ORDER_OFFSET |
+			 MLX5_ENCRYPTION_STANDARD_AES_XTS);
+	sess_private_data->iv_offset = cipher->iv.offset;
+	sess_private_data->dek_id =
+			rte_cpu_to_be_32(sess_private_data->dek->obj->id &
+					 0xffffff);
+	set_sym_session_private_data(session, dev->driver_id,
+				     sess_private_data);
+	DRV_LOG(DEBUG, "Session %p was configured.", sess_private_data);
+	return 0;
+}
+
+static void
+mlx5_crypto_sym_session_clear(struct rte_cryptodev *dev,
+			      struct rte_cryptodev_sym_session *sess)
+{
+	struct mlx5_crypto_priv *priv = dev->data->dev_private;
+	struct mlx5_crypto_session *sess_private_data =
+			get_sym_session_private_data(sess, dev->driver_id);
+
+	if (unlikely(sess_private_data == NULL)) {
+		DRV_LOG(ERR, "Failed to get session %p private data.",
+				sess_private_data);
+		return;
+	}
+	mlx5_crypto_dek_destroy(priv, sess_private_data->dek);
+	DRV_LOG(DEBUG, "Session %p was cleared.", sess_private_data);
+}
+
 static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_configure			= mlx5_crypto_dev_configure,
 	.dev_start			= NULL,
@@ -68,9 +158,9 @@ static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.stats_reset			= NULL,
 	.queue_pair_setup		= NULL,
 	.queue_pair_release		= NULL,
-	.sym_session_get_size		= NULL,
-	.sym_session_configure		= NULL,
-	.sym_session_clear		= NULL,
+	.sym_session_get_size		= mlx5_crypto_sym_session_get_size,
+	.sym_session_configure		= mlx5_crypto_sym_session_configure,
+	.sym_session_clear		= mlx5_crypto_sym_session_clear,
 	.sym_get_raw_dp_ctx_size	= NULL,
 	.sym_configure_raw_dp_ctx	= NULL,
 };
-- 
2.25.1

[dpdk-dev] [PATCH v2 04/15] crypto/mlx5: add basic operations

[Matan Azrad]

From: Shiri Kuzin <shirik@nvidia.com>

The basic dev control operations are configure, close and get info.

Extended the existing support of configure and close:
	-mlx5_crypto_dev_configure- function used to configure device.
	-mlx5_crypto_dev_close-  function used to close a configured
	 device.

Added support of get info function:
	-mlx5_crypto_dev_infos_get- function used to get specific
	 information of a device.

Added config struct to user private data with the fields socket id,
number of queue pairs and feature flags to be disabled.

Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/crypto/mlx5/mlx5_crypto.c | 46 +++++++++++++++++++++++++++----
 drivers/crypto/mlx5/mlx5_crypto.h |  1 +
 2 files changed, 42 insertions(+), 5 deletions(-)

diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index c494ed00d1..cc11cabbea 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -19,6 +19,7 @@
 
 #define MLX5_CRYPTO_DRIVER_NAME mlx5_crypto
 #define MLX5_CRYPTO_LOG_NAME pmd.crypto.mlx5
+#define MLX5_CRYPTO_MAX_QPS 1024
 
 TAILQ_HEAD(mlx5_crypto_privs, mlx5_crypto_priv) mlx5_crypto_priv_list =
 				TAILQ_HEAD_INITIALIZER(mlx5_crypto_priv_list);
@@ -28,6 +29,9 @@ int mlx5_crypto_logtype;
 
 uint8_t mlx5_crypto_driver_id;
 
+const struct rte_cryptodev_capabilities
+		mlx5_crypto_caps[RTE_CRYPTO_OP_TYPE_UNDEFINED];
+
 static const char mlx5_crypto_drv_name[] = RTE_STR(MLX5_CRYPTO_DRIVER_NAME);
 
 static const struct rte_driver mlx5_drv = {
@@ -49,22 +53,47 @@ struct mlx5_crypto_session {
 	uint32_t dek_id; /* DEK ID */
 } __rte_packed;
 
-static unsigned int
-mlx5_crypto_sym_session_get_size(struct rte_cryptodev *dev __rte_unused)
+static void
+mlx5_crypto_dev_infos_get(struct rte_cryptodev *dev,
+			  struct rte_cryptodev_info *dev_info)
 {
-	return sizeof(struct mlx5_crypto_session);
+	RTE_SET_USED(dev);
+	if (dev_info != NULL) {
+		dev_info->driver_id = mlx5_crypto_driver_id;
+		dev_info->feature_flags = 0;
+		dev_info->capabilities = mlx5_crypto_caps;
+		dev_info->max_nb_queue_pairs = MLX5_CRYPTO_MAX_QPS;
+		dev_info->min_mbuf_headroom_req = 0;
+		dev_info->min_mbuf_tailroom_req = 0;
+		dev_info->sym.max_nb_sessions = 0;
+		/*
+		 * If 0, the device does not have any limitation in number of
+		 * sessions that can be used.
+		 */
+	}
 }
 
 static int
 mlx5_crypto_dev_configure(struct rte_cryptodev *dev,
-		struct rte_cryptodev_config *config __rte_unused)
+			  struct rte_cryptodev_config *config)
 {
 	struct mlx5_crypto_priv *priv = dev->data->dev_private;
 
+	if (config == NULL) {
+		DRV_LOG(ERR, "Invalid crypto dev configure parameters.");
+		return -EINVAL;
+	}
+	if ((config->ff_disable & RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO) != 0) {
+		DRV_LOG(ERR,
+			"Disabled symmetric crypto feature is not supported.");
+		return -ENOTSUP;
+	}
 	if (mlx5_crypto_dek_setup(priv) != 0) {
 		DRV_LOG(ERR, "Dek hash list creation has failed.");
 		return -ENOMEM;
 	}
+	priv->dev_config = *config;
+	DRV_LOG(DEBUG, "Device %u was configured.", dev->driver_id);
 	return 0;
 }
 
@@ -74,9 +103,16 @@ mlx5_crypto_dev_close(struct rte_cryptodev *dev)
 	struct mlx5_crypto_priv *priv = dev->data->dev_private;
 
 	mlx5_crypto_dek_unset(priv);
+	DRV_LOG(DEBUG, "Device %u was closed.", dev->driver_id);
 	return 0;
 }
 
+static unsigned int
+mlx5_crypto_sym_session_get_size(struct rte_cryptodev *dev __rte_unused)
+{
+	return sizeof(struct mlx5_crypto_session);
+}
+
 static int
 mlx5_crypto_sym_session_configure(struct rte_cryptodev *dev,
 				  struct rte_crypto_sym_xform *xform,
@@ -153,7 +189,7 @@ static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_start			= NULL,
 	.dev_stop			= NULL,
 	.dev_close			= mlx5_crypto_dev_close,
-	.dev_infos_get			= NULL,
+	.dev_infos_get			= mlx5_crypto_dev_infos_get,
 	.stats_get			= NULL,
 	.stats_reset			= NULL,
 	.queue_pair_setup		= NULL,
diff --git a/drivers/crypto/mlx5/mlx5_crypto.h b/drivers/crypto/mlx5/mlx5_crypto.h
index 4ec67a7e0f..5e270d3d5a 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.h
+++ b/drivers/crypto/mlx5/mlx5_crypto.h
@@ -24,6 +24,7 @@ struct mlx5_crypto_priv {
 	uint32_t pdn; /* Protection Domain number. */
 	struct ibv_pd *pd;
 	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
+	struct rte_cryptodev_config dev_config;
 };
 
 struct mlx5_crypto_dek {
-- 
2.25.1

[dpdk-dev] [PATCH v2 05/15] crypto/mlx5: support queue pairs operations

[Matan Azrad]

From: Shiri Kuzin <shirik@nvidia.com>

The HW queue pairs are a pair of send queue and receive queue of
independent work queues packed together in one object for the purpose
of transferring data between nodes of a network.

Completion Queue is a FIFO queue of completed work requests.

In crypto driver we use one QP in loopback in order to encrypt and
decrypt data locally without sending it to the wire.
In the configured QP we only use the SQ to perform the encryption and
decryption operations.

Added implementation for the QP setup function which creates the CQ,
creates the QP and changes its state to RTS (ready to send).

Added implementation for the release QP function to release all the QP
resources.

Added the ops structure that contains any operation which is supported
by the cryptodev.

Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/crypto/mlx5/mlx5_crypto.c | 124 +++++++++++++++++++++++++++++-
 drivers/crypto/mlx5/mlx5_crypto.h |  11 +++
 2 files changed, 133 insertions(+), 2 deletions(-)

diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index cc11cabbea..5a14e05d55 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -7,6 +7,7 @@
 #include <rte_errno.h>
 #include <rte_log.h>
 #include <rte_pci.h>
+#include <rte_memory.h>
 
 #include <mlx5_glue.h>
 #include <mlx5_common.h>
@@ -184,6 +185,125 @@ mlx5_crypto_sym_session_clear(struct rte_cryptodev *dev,
 	DRV_LOG(DEBUG, "Session %p was cleared.", sess_private_data);
 }
 
+static int
+mlx5_crypto_queue_pair_release(struct rte_cryptodev *dev, uint16_t qp_id)
+{
+	struct mlx5_crypto_qp *qp = dev->data->queue_pairs[qp_id];
+
+	if (qp->qp_obj != NULL)
+		claim_zero(mlx5_devx_cmd_destroy(qp->qp_obj));
+	if (qp->umem_obj != NULL)
+		claim_zero(mlx5_glue->devx_umem_dereg(qp->umem_obj));
+	if (qp->umem_buf != NULL)
+		rte_free(qp->umem_buf);
+	mlx5_devx_cq_destroy(&qp->cq_obj);
+	rte_free(qp);
+	dev->data->queue_pairs[qp_id] = NULL;
+	return 0;
+}
+
+static int
+mlx5_crypto_qp2rts(struct mlx5_crypto_qp *qp)
+{
+	/*
+	 * In Order to configure self loopback, when calling these functions the
+	 * remote QP id that is used is the id of the same QP.
+	 */
+	if (mlx5_devx_cmd_modify_qp_state(qp->qp_obj, MLX5_CMD_OP_RST2INIT_QP,
+					  qp->qp_obj->id)) {
+		DRV_LOG(ERR, "Failed to modify QP to INIT state(%u).",
+			rte_errno);
+		return -1;
+	}
+	if (mlx5_devx_cmd_modify_qp_state(qp->qp_obj, MLX5_CMD_OP_INIT2RTR_QP,
+					  qp->qp_obj->id)) {
+		DRV_LOG(ERR, "Failed to modify QP to RTR state(%u).",
+			rte_errno);
+		return -1;
+	}
+	if (mlx5_devx_cmd_modify_qp_state(qp->qp_obj, MLX5_CMD_OP_RTR2RTS_QP,
+					  qp->qp_obj->id)) {
+		DRV_LOG(ERR, "Failed to modify QP to RTS state(%u).",
+			rte_errno);
+		return -1;
+	}
+	return 0;
+}
+
+static int
+mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
+			     const struct rte_cryptodev_qp_conf *qp_conf,
+			     int socket_id)
+{
+	struct mlx5_crypto_priv *priv = dev->data->dev_private;
+	struct mlx5_devx_qp_attr attr = {0};
+	struct mlx5_crypto_qp *qp;
+	uint16_t log_nb_desc = rte_log2_u32(qp_conf->nb_descriptors);
+	uint32_t umem_size = RTE_BIT32(log_nb_desc) *
+			      MLX5_CRYPTO_WQE_SET_SIZE +
+			      sizeof(*qp->db_rec) * 2;
+	uint32_t alloc_size = sizeof(*qp);
+	struct mlx5_devx_cq_attr cq_attr = {
+		.uar_page_id = mlx5_os_get_devx_uar_page_id(priv->uar),
+	};
+
+	alloc_size = RTE_ALIGN(alloc_size, RTE_CACHE_LINE_SIZE);
+	alloc_size += sizeof(struct rte_crypto_op *) * RTE_BIT32(log_nb_desc);
+	qp = rte_zmalloc_socket(__func__, alloc_size, RTE_CACHE_LINE_SIZE,
+				socket_id);
+	if (qp == NULL) {
+		DRV_LOG(ERR, "Failed to allocate QP memory.");
+		rte_errno = ENOMEM;
+		return -rte_errno;
+	}
+	if (mlx5_devx_cq_create(priv->ctx, &qp->cq_obj, log_nb_desc,
+				&cq_attr, socket_id) != 0) {
+		DRV_LOG(ERR, "Failed to create CQ.");
+		goto error;
+	}
+	qp->umem_buf = rte_zmalloc_socket(__func__, umem_size, 4096, socket_id);
+	if (qp->umem_buf == NULL) {
+		DRV_LOG(ERR, "Failed to allocate QP umem.");
+		rte_errno = ENOMEM;
+		goto error;
+	}
+	qp->umem_obj = mlx5_glue->devx_umem_reg(priv->ctx,
+					       (void *)(uintptr_t)qp->umem_buf,
+					       umem_size,
+					       IBV_ACCESS_LOCAL_WRITE);
+	if (qp->umem_obj == NULL) {
+		DRV_LOG(ERR, "Failed to register QP umem.");
+		goto error;
+	}
+	attr.pd = priv->pdn;
+	attr.uar_index = mlx5_os_get_devx_uar_page_id(priv->uar);
+	attr.cqn = qp->cq_obj.cq->id;
+	attr.log_page_size = rte_log2_u32(sysconf(_SC_PAGESIZE));
+	attr.rq_size =  0;
+	attr.sq_size = RTE_BIT32(log_nb_desc);
+	attr.dbr_umem_valid = 1;
+	attr.wq_umem_id = qp->umem_obj->umem_id;
+	attr.wq_umem_offset = 0;
+	attr.dbr_umem_id = qp->umem_obj->umem_id;
+	attr.dbr_address = RTE_BIT64(log_nb_desc) *
+			   MLX5_CRYPTO_WQE_SET_SIZE;
+	qp->qp_obj = mlx5_devx_cmd_create_qp(priv->ctx, &attr);
+	if (qp->qp_obj == NULL) {
+		DRV_LOG(ERR, "Failed to create QP(%u).", rte_errno);
+		goto error;
+	}
+	qp->db_rec = RTE_PTR_ADD(qp->umem_buf, (uintptr_t)attr.dbr_address);
+	if (mlx5_crypto_qp2rts(qp))
+		goto error;
+	qp->ops = (struct rte_crypto_op **)RTE_ALIGN((uintptr_t)(qp + 1),
+							   RTE_CACHE_LINE_SIZE);
+	dev->data->queue_pairs[qp_id] = qp;
+	return 0;
+error:
+	mlx5_crypto_queue_pair_release(dev, qp_id);
+	return -1;
+}
+
 static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_configure			= mlx5_crypto_dev_configure,
 	.dev_start			= NULL,
@@ -192,8 +312,8 @@ static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_infos_get			= mlx5_crypto_dev_infos_get,
 	.stats_get			= NULL,
 	.stats_reset			= NULL,
-	.queue_pair_setup		= NULL,
-	.queue_pair_release		= NULL,
+	.queue_pair_setup		= mlx5_crypto_queue_pair_setup,
+	.queue_pair_release		= mlx5_crypto_queue_pair_release,
 	.sym_session_get_size		= mlx5_crypto_sym_session_get_size,
 	.sym_session_configure		= mlx5_crypto_sym_session_configure,
 	.sym_session_clear		= mlx5_crypto_sym_session_clear,
diff --git a/drivers/crypto/mlx5/mlx5_crypto.h b/drivers/crypto/mlx5/mlx5_crypto.h
index 5e270d3d5a..f5313b89f2 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.h
+++ b/drivers/crypto/mlx5/mlx5_crypto.h
@@ -11,9 +11,11 @@
 #include <rte_cryptodev_pmd.h>
 
 #include <mlx5_common_utils.h>
+#include <mlx5_common_devx.h>
 
 #define MLX5_CRYPTO_DEK_HTABLE_SZ (1 << 11)
 #define MLX5_CRYPTO_KEY_LENGTH 80
+#define MLX5_CRYPTO_WQE_SET_SIZE 1024
 
 struct mlx5_crypto_priv {
 	TAILQ_ENTRY(mlx5_crypto_priv) next;
@@ -27,6 +29,15 @@ struct mlx5_crypto_priv {
 	struct rte_cryptodev_config dev_config;
 };
 
+struct mlx5_crypto_qp {
+	struct mlx5_devx_cq cq_obj;
+	struct mlx5_devx_obj *qp_obj;
+	struct mlx5dv_devx_umem *umem_obj;
+	void *umem_buf;
+	volatile uint32_t *db_rec;
+	struct rte_crypto_op **ops;
+};
+
 struct mlx5_crypto_dek {
 	struct mlx5_hlist_entry entry; /* Pointer to DEK hash list entry. */
 	struct mlx5_devx_obj *obj; /* Pointer to DEK DevX object. */
-- 
2.25.1

[dpdk-dev] [PATCH v2 06/15] crypto/mlx5: add dev stop and start operations

[Matan Azrad]

From: Shiri Kuzin <shirik@nvidia.com>

Add the dev_start function that is used to start a configured device.
Add the dev_stop function that is used to stop a configured device.

Both functions set the dev parameter as used and return 0.

Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/crypto/mlx5/mlx5_crypto.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index 5a14e05d55..495d5448b2 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -98,6 +98,19 @@ mlx5_crypto_dev_configure(struct rte_cryptodev *dev,
 	return 0;
 }
 
+static void
+mlx5_crypto_dev_stop(struct rte_cryptodev *dev)
+{
+	RTE_SET_USED(dev);
+}
+
+static int
+mlx5_crypto_dev_start(struct rte_cryptodev *dev)
+{
+	RTE_SET_USED(dev);
+	return 0;
+}
+
 static int
 mlx5_crypto_dev_close(struct rte_cryptodev *dev)
 {
@@ -306,8 +319,8 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 
 static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_configure			= mlx5_crypto_dev_configure,
-	.dev_start			= NULL,
-	.dev_stop			= NULL,
+	.dev_start			= mlx5_crypto_dev_start,
+	.dev_stop			= mlx5_crypto_dev_stop,
 	.dev_close			= mlx5_crypto_dev_close,
 	.dev_infos_get			= mlx5_crypto_dev_infos_get,
 	.stats_get			= NULL,
-- 
2.25.1

[dpdk-dev] [PATCH v2 07/15] crypto/mlx5: add memory region management

[Matan Azrad]

From: Shiri Kuzin <shirik@nvidia.com>

Mellanox user space drivers don't deal with physical addresses as part
of a memory protection mechanism.
The device translates the given virtual address to a physical address
using the given memory key as an address space identifier.
That's why any mbuf virtual address is moved directly to the HW
descriptor(WQE).

The mapping between the virtual address to the physical address is saved
in MR configured by the kernel to the HW.

Each MR has a key that should also be moved to the WQE by the SW.

When the SW sees an unmapped address, it extends the address range and
creates a MR using a system call.

Add memory region cache management:
	- 2 level cache per queue-pair - no locks.
	- 1 shared cache between all the queues using a lock.

Using this way, the MR key search per data-path address is optimized.

Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/crypto/mlx5/mlx5_crypto.c | 20 ++++++++++++++++++++
 drivers/crypto/mlx5/mlx5_crypto.h |  3 +++
 2 files changed, 23 insertions(+)

diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index 495d5448b2..79e3d3ee45 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -209,6 +209,7 @@ mlx5_crypto_queue_pair_release(struct rte_cryptodev *dev, uint16_t qp_id)
 		claim_zero(mlx5_glue->devx_umem_dereg(qp->umem_obj));
 	if (qp->umem_buf != NULL)
 		rte_free(qp->umem_buf);
+	mlx5_mr_btree_free(&qp->mr_ctrl.cache_bh);
 	mlx5_devx_cq_destroy(&qp->cq_obj);
 	rte_free(qp);
 	dev->data->queue_pairs[qp_id] = NULL;
@@ -288,6 +289,13 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 		DRV_LOG(ERR, "Failed to register QP umem.");
 		goto error;
 	}
+	if (mlx5_mr_btree_init(&qp->mr_ctrl.cache_bh, MLX5_MR_BTREE_CACHE_N,
+			       priv->dev_config.socket_id) != 0) {
+		DRV_LOG(ERR, "Cannot allocate MR Btree for qp %u.",
+			(uint32_t)qp_id);
+		rte_errno = ENOMEM;
+		goto error;
+	}
 	attr.pd = priv->pdn;
 	attr.uar_index = mlx5_os_get_devx_uar_page_id(priv->uar);
 	attr.cqn = qp->cq_obj.cq->id;
@@ -476,6 +484,17 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 		claim_zero(mlx5_glue->close_device(priv->ctx));
 		return -1;
 	}
+	if (mlx5_mr_btree_init(&priv->mr_scache.cache,
+			     MLX5_MR_BTREE_CACHE_N * 2, rte_socket_id()) != 0) {
+		DRV_LOG(ERR, "Failed to allocate shared cache MR memory.");
+		mlx5_crypto_hw_global_release(priv);
+		rte_cryptodev_pmd_destroy(priv->crypto_dev);
+		claim_zero(mlx5_glue->close_device(priv->ctx));
+		rte_errno = ENOMEM;
+		return -rte_errno;
+	}
+	priv->mr_scache.reg_mr_cb = mlx5_common_verbs_reg_mr;
+	priv->mr_scache.dereg_mr_cb = mlx5_common_verbs_dereg_mr;
 	pthread_mutex_lock(&priv_list_lock);
 	TAILQ_INSERT_TAIL(&mlx5_crypto_priv_list, priv, next);
 	pthread_mutex_unlock(&priv_list_lock);
@@ -495,6 +514,7 @@ mlx5_crypto_pci_remove(struct rte_pci_device *pdev)
 		TAILQ_REMOVE(&mlx5_crypto_priv_list, priv, next);
 	pthread_mutex_unlock(&priv_list_lock);
 	if (priv) {
+		mlx5_mr_release_cache(&priv->mr_scache);
 		mlx5_crypto_hw_global_release(priv);
 		rte_cryptodev_pmd_destroy(priv->crypto_dev);
 		claim_zero(mlx5_glue->close_device(priv->ctx));
diff --git a/drivers/crypto/mlx5/mlx5_crypto.h b/drivers/crypto/mlx5/mlx5_crypto.h
index f5313b89f2..397267d249 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.h
+++ b/drivers/crypto/mlx5/mlx5_crypto.h
@@ -12,6 +12,7 @@
 
 #include <mlx5_common_utils.h>
 #include <mlx5_common_devx.h>
+#include <mlx5_common_mr.h>
 
 #define MLX5_CRYPTO_DEK_HTABLE_SZ (1 << 11)
 #define MLX5_CRYPTO_KEY_LENGTH 80
@@ -27,6 +28,7 @@ struct mlx5_crypto_priv {
 	struct ibv_pd *pd;
 	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
 	struct rte_cryptodev_config dev_config;
+	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
 };
 
 struct mlx5_crypto_qp {
@@ -36,6 +38,7 @@ struct mlx5_crypto_qp {
 	void *umem_buf;
 	volatile uint32_t *db_rec;
 	struct rte_crypto_op **ops;
+	struct mlx5_mr_ctrl mr_ctrl;
 };
 
 struct mlx5_crypto_dek {
-- 
2.25.1

[dpdk-dev] [PATCH v2 08/15] crypto/mlx5: create login object using DevX

[Matan Azrad]

From: Shiri Kuzin <shirik@nvidia.com>

To work with crypto engines that are marked with wrapped_import_method,
a login session is required.
A crypto login object needs to be created using DevX.

The crypto login object contains:
	- The credential pointer.
	- The import_KEK pointer to be used for all secured information
	  communicated in crypto commands (key fields), including the
	  provided credential in this command.
	- The credential secret, wrapped by the import_KEK indicated in
	  this command. Size includes 8 bytes IV for wrapping.

Added devargs for the required login values:
	- wcs_file - path to the file containing the credential.
	- import_kek_id - the import KEK pointer.
	- credential_id - the credential pointer.

Create the login DevX object in pci_probe function and destroy it in
pci_remove.
Destroying the crypto login object means logout.

Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/crypto/mlx5/mlx5_crypto.c | 103 ++++++++++++++++++++++++++++++
 drivers/crypto/mlx5/mlx5_crypto.h |   7 ++
 2 files changed, 110 insertions(+)

diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index 79e3d3ee45..44038f0e05 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -402,6 +402,101 @@ mlx5_crypto_hw_global_prepare(struct mlx5_crypto_priv *priv)
 	return 0;
 }
 
+
+static int
+mlx5_crypto_args_check_handler(const char *key, const char *val, void *opaque)
+{
+	struct mlx5_crypto_devarg_params *devarg_prms = opaque;
+	struct mlx5_devx_crypto_login_attr *attr = &devarg_prms->login_attr;
+	unsigned long tmp;
+	FILE *file;
+	int ret;
+	int i;
+
+	if (strcmp(key, "class") == 0)
+		return 0;
+	if (strcmp(key, "wcs_file") == 0) {
+		file = fopen(val, "rb");
+		if (file == NULL) {
+			rte_errno = ENOTSUP;
+			return -rte_errno;
+		}
+		for (i = 0 ; i < MLX5_CRYPTO_CREDENTIAL_SIZE ; i++) {
+			ret = fscanf(file, "%02hhX", &attr->credential[i]);
+			if (ret <= 0) {
+				fclose(file);
+				DRV_LOG(ERR,
+					"Failed to read credential from file.");
+				rte_errno = EINVAL;
+				return -rte_errno;
+			}
+		}
+		fclose(file);
+		devarg_prms->login_devarg = true;
+		return 0;
+	}
+	errno = 0;
+	tmp = strtoul(val, NULL, 0);
+	if (errno) {
+		DRV_LOG(WARNING, "%s: \"%s\" is an invalid integer.", key, val);
+		return -errno;
+	}
+	if (strcmp(key, "import_kek_id") == 0)
+		attr->session_import_kek_ptr = (uint32_t)tmp;
+	else if (strcmp(key, "credential_id") == 0)
+		attr->credential_pointer = (uint32_t)tmp;
+	else
+		DRV_LOG(WARNING, "Invalid key %s.", key);
+	return 0;
+}
+
+static struct mlx5_devx_obj *
+mlx5_crypto_config_login(struct rte_devargs *devargs,
+			 struct ibv_context *ctx)
+{
+	/*
+	 * Set credential pointer and session import KEK pointer to a default
+	 * value of 0.
+	 */
+	struct mlx5_crypto_devarg_params login = {
+			.login_devarg = false,
+			.login_attr = {
+					.credential_pointer = 0,
+					.session_import_kek_ptr = 0,
+			}
+	};
+	struct rte_kvargs *kvlist;
+
+	if (devargs == NULL) {
+		DRV_LOG(ERR,
+	"No login devargs in order to enable crypto operations in the device.");
+		rte_errno = EINVAL;
+		return NULL;
+	}
+	kvlist = rte_kvargs_parse(devargs->args, NULL);
+	if (kvlist == NULL) {
+		DRV_LOG(ERR, "Failed to parse devargs.");
+		rte_errno = EINVAL;
+		return NULL;
+	}
+	if (rte_kvargs_process(kvlist, NULL, mlx5_crypto_args_check_handler,
+			   &login) != 0) {
+		DRV_LOG(ERR, "Devargs handler function Failed.");
+		rte_kvargs_free(kvlist);
+		rte_errno = EINVAL;
+		return NULL;
+	}
+	rte_kvargs_free(kvlist);
+	if (login.login_devarg == false) {
+		DRV_LOG(ERR,
+	"No login credential devarg in order to enable crypto operations "
+	"in the device.");
+		rte_errno = EINVAL;
+		return NULL;
+	}
+	return mlx5_devx_cmd_create_crypto_login_obj(ctx, &login.login_attr);
+}
+
 /**
  * DPDK callback to register a PCI device.
  *
@@ -423,6 +518,7 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 	struct ibv_device *ibv;
 	struct rte_cryptodev *crypto_dev;
 	struct ibv_context *ctx;
+	struct mlx5_devx_obj *login;
 	struct mlx5_crypto_priv *priv;
 	struct mlx5_hca_attr attr = { 0 };
 	struct rte_cryptodev_pmd_init_params init_params = {
@@ -461,6 +557,11 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 		rte_errno = ENOTSUP;
 		return -ENOTSUP;
 	}
+	login = mlx5_crypto_config_login(pci_dev->device.devargs, ctx);
+	if (login == NULL) {
+		DRV_LOG(ERR, "Failed to configure login.");
+		return -rte_errno;
+	}
 	crypto_dev = rte_cryptodev_pmd_create(ibv->name, &pci_dev->device,
 					&init_params);
 	if (crypto_dev == NULL) {
@@ -477,6 +578,7 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 	crypto_dev->driver_id = mlx5_crypto_driver_id;
 	priv = crypto_dev->data->dev_private;
 	priv->ctx = ctx;
+	priv->login_obj = login;
 	priv->pci_dev = pci_dev;
 	priv->crypto_dev = crypto_dev;
 	if (mlx5_crypto_hw_global_prepare(priv) != 0) {
@@ -517,6 +619,7 @@ mlx5_crypto_pci_remove(struct rte_pci_device *pdev)
 		mlx5_mr_release_cache(&priv->mr_scache);
 		mlx5_crypto_hw_global_release(priv);
 		rte_cryptodev_pmd_destroy(priv->crypto_dev);
+		claim_zero(mlx5_devx_cmd_destroy(priv->login_obj));
 		claim_zero(mlx5_glue->close_device(priv->ctx));
 	}
 	return 0;
diff --git a/drivers/crypto/mlx5/mlx5_crypto.h b/drivers/crypto/mlx5/mlx5_crypto.h
index 397267d249..0aef804b92 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.h
+++ b/drivers/crypto/mlx5/mlx5_crypto.h
@@ -29,6 +29,7 @@ struct mlx5_crypto_priv {
 	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
 	struct rte_cryptodev_config dev_config;
 	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
+	struct mlx5_devx_obj *login_obj;
 };
 
 struct mlx5_crypto_qp {
@@ -48,6 +49,12 @@ struct mlx5_crypto_dek {
 	bool size_is_48; /* Whether the key\data size is 48 bytes or not. */
 };
 
+
+struct mlx5_crypto_devarg_params {
+	bool login_devarg;
+	struct mlx5_devx_crypto_login_attr login_attr;
+};
+
 int
 mlx5_crypto_dek_destroy(struct mlx5_crypto_priv *priv,
 			struct mlx5_crypto_dek *dek);
-- 
2.25.1

[dpdk-dev] [PATCH v2 09/15] crypto/mlx5: adjust to the multiple data unit API

[Matan Azrad]

From: Shiri Kuzin <shirik@nvidia.com>

In AES-XTS the data to be encrypted\decrypted does not have to be
in multiples of 16B size, the unit of data is called data-unit.

As a result of patch [1] a new field is added to the cipher capability,
called dataunit_set, where the devices can report the range of
supported data-unit sizes.

The new field enables saving the data-unit size in the session
structure to the block size pointer variable in order to support
several data-unit sizes.

[1] https://www.mail-archive.com/dev@dpdk.org/msg205337.html

Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
---
 drivers/crypto/mlx5/mlx5_crypto.c | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index 44038f0e05..1dcebce04c 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -48,6 +48,11 @@ struct mlx5_crypto_session {
 	 * bsf_size, bsf_p_type, encryption_order and encryption standard,
 	 * saved in big endian format.
 	 */
+	uint32_t bsp_res;
+	/*
+	 * crypto_block_size_pointer and reserved 24 bits saved in big endian
+	 * format.
+	 */
 	uint32_t iv_offset:16;
 	/* Starting point for Initialisation Vector. */
 	struct mlx5_crypto_dek *dek; /* Pointer to dek struct. */
@@ -171,6 +176,24 @@ mlx5_crypto_sym_session_configure(struct rte_cryptodev *dev,
 			 MLX5_BSF_P_TYPE_CRYPTO << MLX5_BSF_P_TYPE_OFFSET |
 			 encryption_order << MLX5_ENCRYPTION_ORDER_OFFSET |
 			 MLX5_ENCRYPTION_STANDARD_AES_XTS);
+	switch (xform->cipher.dataunit_len) {
+	case 0:
+		sess_private_data->bsp_res = 0;
+		break;
+	case 512:
+		sess_private_data->bsp_res = rte_cpu_to_be_32
+					     ((uint32_t)MLX5_BLOCK_SIZE_512B <<
+					     MLX5_BLOCK_SIZE_OFFSET);
+		break;
+	case 4096:
+		sess_private_data->bsp_res = rte_cpu_to_be_32
+					     ((uint32_t)MLX5_BLOCK_SIZE_4096B <<
+					     MLX5_BLOCK_SIZE_OFFSET);
+		break;
+	default:
+		DRV_LOG(ERR, "Cipher data unit length is not supported.");
+		return -ENOTSUP;
+	}
 	sess_private_data->iv_offset = cipher->iv.offset;
 	sess_private_data->dek_id =
 			rte_cpu_to_be_32(sess_private_data->dek->obj->id &
-- 
2.25.1

[dpdk-dev] [PATCH v2 10/15] crypto/mlx5: add keytag device argument

[Matan Azrad]

From: Suanming Mou <suanmingm@nvidia.com>

A keytag is a piece of data encrypted together with a DEK.

When a DEK is referenced by an MKEY.bsf through its index, the keytag is
also supplied in the BSF as plaintext. The HW will decrypt the DEK (and
the attached keytag) and will fail the operation if the keytags don't
match.

This commit adds the configuration of the keytag with devargs.

Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Signed-off-by: Matan Azrad <matan@nvidia.com>
---
 drivers/crypto/mlx5/mlx5_crypto.c | 50 +++++++++++++++++--------------
 drivers/crypto/mlx5/mlx5_crypto.h |  3 +-
 2 files changed, 30 insertions(+), 23 deletions(-)

diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index 1dcebce04c..9f270e4382 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -468,56 +468,52 @@ mlx5_crypto_args_check_handler(const char *key, const char *val, void *opaque)
 		attr->session_import_kek_ptr = (uint32_t)tmp;
 	else if (strcmp(key, "credential_id") == 0)
 		attr->credential_pointer = (uint32_t)tmp;
+	else if (strcmp(key, "keytag") == 0)
+		devarg_prms->keytag = tmp;
 	else
 		DRV_LOG(WARNING, "Invalid key %s.", key);
 	return 0;
 }
 
-static struct mlx5_devx_obj *
-mlx5_crypto_config_login(struct rte_devargs *devargs,
-			 struct ibv_context *ctx)
+static int
+mlx5_crypto_parse_devargs(struct rte_devargs *devargs,
+			  struct mlx5_crypto_devarg_params *devarg_prms)
 {
-	/*
-	 * Set credential pointer and session import KEK pointer to a default
-	 * value of 0.
-	 */
-	struct mlx5_crypto_devarg_params login = {
-			.login_devarg = false,
-			.login_attr = {
-					.credential_pointer = 0,
-					.session_import_kek_ptr = 0,
-			}
-	};
+	struct mlx5_devx_crypto_login_attr *attr = &devarg_prms->login_attr;
 	struct rte_kvargs *kvlist;
 
+	/* Default values. */
+	attr->credential_pointer = 0;
+	attr->session_import_kek_ptr = 0;
+	devarg_prms->keytag = 0;
 	if (devargs == NULL) {
 		DRV_LOG(ERR,
 	"No login devargs in order to enable crypto operations in the device.");
 		rte_errno = EINVAL;
-		return NULL;
+		return -1;
 	}
 	kvlist = rte_kvargs_parse(devargs->args, NULL);
 	if (kvlist == NULL) {
 		DRV_LOG(ERR, "Failed to parse devargs.");
 		rte_errno = EINVAL;
-		return NULL;
+		return -1;
 	}
 	if (rte_kvargs_process(kvlist, NULL, mlx5_crypto_args_check_handler,
-			   &login) != 0) {
+			   devarg_prms) != 0) {
 		DRV_LOG(ERR, "Devargs handler function Failed.");
 		rte_kvargs_free(kvlist);
 		rte_errno = EINVAL;
-		return NULL;
+		return -1;
 	}
 	rte_kvargs_free(kvlist);
-	if (login.login_devarg == false) {
+	if (devarg_prms->login_devarg == false) {
 		DRV_LOG(ERR,
 	"No login credential devarg in order to enable crypto operations "
 	"in the device.");
 		rte_errno = EINVAL;
-		return NULL;
+		return -1;
 	}
-	return mlx5_devx_cmd_create_crypto_login_obj(ctx, &login.login_attr);
+	return 0;
 }
 
 /**
@@ -543,6 +539,7 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 	struct ibv_context *ctx;
 	struct mlx5_devx_obj *login;
 	struct mlx5_crypto_priv *priv;
+	struct mlx5_crypto_devarg_params devarg_prms = { 0 };
 	struct mlx5_hca_attr attr = { 0 };
 	struct rte_cryptodev_pmd_init_params init_params = {
 		.name = "",
@@ -551,6 +548,8 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 		.max_nb_queue_pairs =
 				RTE_CRYPTODEV_PMD_DEFAULT_MAX_NB_QUEUE_PAIRS,
 	};
+	int ret;
+
 	RTE_SET_USED(pci_drv);
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
 		DRV_LOG(ERR, "Non-primary process type is not supported.");
@@ -580,7 +579,13 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 		rte_errno = ENOTSUP;
 		return -ENOTSUP;
 	}
-	login = mlx5_crypto_config_login(pci_dev->device.devargs, ctx);
+	ret = mlx5_crypto_parse_devargs(pci_dev->device.devargs, &devarg_prms);
+	if (ret) {
+		DRV_LOG(ERR, "Failed to parse devargs.");
+		return -rte_errno;
+	}
+	login = mlx5_devx_cmd_create_crypto_login_obj(ctx,
+						      &devarg_prms.login_attr);
 	if (login == NULL) {
 		DRV_LOG(ERR, "Failed to configure login.");
 		return -rte_errno;
@@ -620,6 +625,7 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 	}
 	priv->mr_scache.reg_mr_cb = mlx5_common_verbs_reg_mr;
 	priv->mr_scache.dereg_mr_cb = mlx5_common_verbs_dereg_mr;
+	priv->keytag = rte_cpu_to_be_64(devarg_prms.keytag);
 	pthread_mutex_lock(&priv_list_lock);
 	TAILQ_INSERT_TAIL(&mlx5_crypto_priv_list, priv, next);
 	pthread_mutex_unlock(&priv_list_lock);
diff --git a/drivers/crypto/mlx5/mlx5_crypto.h b/drivers/crypto/mlx5/mlx5_crypto.h
index 0aef804b92..34c65f9a24 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.h
+++ b/drivers/crypto/mlx5/mlx5_crypto.h
@@ -30,6 +30,7 @@ struct mlx5_crypto_priv {
 	struct rte_cryptodev_config dev_config;
 	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
 	struct mlx5_devx_obj *login_obj;
+	uint64_t keytag;
 };
 
 struct mlx5_crypto_qp {
@@ -49,10 +50,10 @@ struct mlx5_crypto_dek {
 	bool size_is_48; /* Whether the key\data size is 48 bytes or not. */
 };
 
-
 struct mlx5_crypto_devarg_params {
 	bool login_devarg;
 	struct mlx5_devx_crypto_login_attr login_attr;
+	uint64_t keytag;
 };
 
 int
-- 
2.25.1

[dpdk-dev] [PATCH v2 11/15] crypto/mlx5: add maximum segments device argument

[Matan Azrad]

From: Suanming Mou <suanmingm@nvidia.com>

The mlx5 HW crypto operations are done by attaching crypto property
to a memory region. Once done, every access to the memory via the
crypto-enabled memory region will result with in-line encryption or
decryption of the data.

As a result, the design choice is to provide two types of WQEs. One
is UMR WQE which sets the crypto property and the other is rdma write
WQE which sends DMA command to copy data from local MR to remote MR.

The size of the WQEs will be defined by a new devarg called
max_segs_num.

This devarg also defines the maximum segments in mbuf chain that will be
supported for crypto operations.

Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Signed-off-by: Matan Azrad <matan@nvidia.com>
---
 drivers/crypto/mlx5/mlx5_crypto.c | 35 +++++++++++++++++++++++++++----
 drivers/crypto/mlx5/mlx5_crypto.h |  7 +++++++
 2 files changed, 38 insertions(+), 4 deletions(-)

diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index 9f270e4382..5e8e7b63d0 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -21,6 +21,7 @@
 #define MLX5_CRYPTO_DRIVER_NAME mlx5_crypto
 #define MLX5_CRYPTO_LOG_NAME pmd.crypto.mlx5
 #define MLX5_CRYPTO_MAX_QPS 1024
+#define MLX5_CRYPTO_MAX_SEGS 56
 
 TAILQ_HEAD(mlx5_crypto_privs, mlx5_crypto_priv) mlx5_crypto_priv_list =
 				TAILQ_HEAD_INITIALIZER(mlx5_crypto_priv_list);
@@ -464,14 +465,24 @@ mlx5_crypto_args_check_handler(const char *key, const char *val, void *opaque)
 		DRV_LOG(WARNING, "%s: \"%s\" is an invalid integer.", key, val);
 		return -errno;
 	}
-	if (strcmp(key, "import_kek_id") == 0)
+	if (strcmp(key, "max_segs_num") == 0) {
+		if (!tmp || tmp > MLX5_CRYPTO_MAX_SEGS) {
+			DRV_LOG(WARNING, "Invalid max_segs_num: %d, should"
+				" be less than %d.",
+				(uint32_t)tmp, MLX5_CRYPTO_MAX_SEGS);
+			rte_errno = EINVAL;
+			return -rte_errno;
+		}
+		devarg_prms->max_segs_num = (uint32_t)tmp;
+	} else if (strcmp(key, "import_kek_id") == 0) {
 		attr->session_import_kek_ptr = (uint32_t)tmp;
-	else if (strcmp(key, "credential_id") == 0)
+	} else if (strcmp(key, "credential_id") == 0) {
 		attr->credential_pointer = (uint32_t)tmp;
-	else if (strcmp(key, "keytag") == 0)
+	} else if (strcmp(key, "keytag") == 0) {
 		devarg_prms->keytag = tmp;
-	else
+	} else {
 		DRV_LOG(WARNING, "Invalid key %s.", key);
+	}
 	return 0;
 }
 
@@ -486,6 +497,7 @@ mlx5_crypto_parse_devargs(struct rte_devargs *devargs,
 	attr->credential_pointer = 0;
 	attr->session_import_kek_ptr = 0;
 	devarg_prms->keytag = 0;
+	devarg_prms->max_segs_num = 8;
 	if (devargs == NULL) {
 		DRV_LOG(ERR,
 	"No login devargs in order to enable crypto operations in the device.");
@@ -626,6 +638,21 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 	priv->mr_scache.reg_mr_cb = mlx5_common_verbs_reg_mr;
 	priv->mr_scache.dereg_mr_cb = mlx5_common_verbs_dereg_mr;
 	priv->keytag = rte_cpu_to_be_64(devarg_prms.keytag);
+	priv->max_segs_num = devarg_prms.max_segs_num;
+	priv->umr_wqe_size = sizeof(struct mlx5_wqe_umr_bsf_seg) +
+			     sizeof(struct mlx5_umr_wqe) +
+			     RTE_ALIGN(priv->max_segs_num, 4) *
+			     sizeof(struct mlx5_wqe_dseg);
+	priv->rdmw_wqe_size = sizeof(struct mlx5_rdma_write_wqe) +
+			      sizeof(struct mlx5_wqe_dseg) *
+			      (priv->max_segs_num <= 2 ? 2 : 2 +
+			       RTE_ALIGN(priv->max_segs_num - 2, 4));
+	priv->wqe_set_size = priv->umr_wqe_size + priv->rdmw_wqe_size;
+	priv->wqe_stride = (priv->umr_wqe_size + priv->rdmw_wqe_size) /
+							       MLX5_SEND_WQE_BB;
+	priv->max_rdmaw_klm_n = (priv->rdmw_wqe_size -
+				 sizeof(struct mlx5_rdma_write_wqe)) /
+				 sizeof(struct mlx5_wqe_dseg);
 	pthread_mutex_lock(&priv_list_lock);
 	TAILQ_INSERT_TAIL(&mlx5_crypto_priv_list, priv, next);
 	pthread_mutex_unlock(&priv_list_lock);
diff --git a/drivers/crypto/mlx5/mlx5_crypto.h b/drivers/crypto/mlx5/mlx5_crypto.h
index 34c65f9a24..81452bd700 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.h
+++ b/drivers/crypto/mlx5/mlx5_crypto.h
@@ -25,12 +25,18 @@ struct mlx5_crypto_priv {
 	struct rte_cryptodev *crypto_dev;
 	void *uar; /* User Access Region. */
 	uint32_t pdn; /* Protection Domain number. */
+	uint32_t max_segs_num; /* Maximum supported data segs. */
 	struct ibv_pd *pd;
 	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
 	struct rte_cryptodev_config dev_config;
 	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
 	struct mlx5_devx_obj *login_obj;
 	uint64_t keytag;
+	uint16_t wqe_set_size;
+	uint16_t umr_wqe_size;
+	uint16_t rdmw_wqe_size;
+	uint16_t wqe_stride;
+	uint16_t max_rdmaw_klm_n;
 };
 
 struct mlx5_crypto_qp {
@@ -54,6 +60,7 @@ struct mlx5_crypto_devarg_params {
 	bool login_devarg;
 	struct mlx5_devx_crypto_login_attr login_attr;
 	uint64_t keytag;
+	uint32_t max_segs_num;
 };
 
 int
-- 
2.25.1

[dpdk-dev] [PATCH v2 12/15] crypto/mlx5: add WQE set initialization

[Matan Azrad]

From: Suanming Mou <suanmingm@nvidia.com>

Currently, HW handles the WQEs much faster than the software,
Using the constant WQE set layout can initialize most of the WQE
segments in advanced, and software only needs to configure very
limited segments in datapath. This accelerates the software WQE
organize in datapath.

This commit initializes the fixed WQE set segments.

Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Signed-off-by: Matan Azrad <matan@nvidia.com>
---
 drivers/crypto/mlx5/mlx5_crypto.c | 83 +++++++++++++++++++++++++++++--
 drivers/crypto/mlx5/mlx5_crypto.h | 10 +++-
 2 files changed, 87 insertions(+), 6 deletions(-)

diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index 5e8e7b63d0..e8f68eb115 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -268,6 +268,69 @@ mlx5_crypto_qp2rts(struct mlx5_crypto_qp *qp)
 	return 0;
 }
 
+static void
+mlx5_crypto_qp_init(struct mlx5_crypto_priv *priv, struct mlx5_crypto_qp *qp)
+{
+	uint32_t i;
+
+	for (i = 0 ; i < qp->entries_n; i++) {
+		struct mlx5_wqe_cseg *cseg = RTE_PTR_ADD(qp->umem_buf, i *
+							 priv->wqe_set_size);
+		struct mlx5_wqe_umr_cseg *ucseg = (struct mlx5_wqe_umr_cseg *)
+								     (cseg + 1);
+		struct mlx5_wqe_umr_bsf_seg *bsf =
+			(struct mlx5_wqe_umr_bsf_seg *)(RTE_PTR_ADD(cseg,
+						       priv->umr_wqe_size)) - 1;
+		struct mlx5_wqe_rseg *rseg;
+
+		/* Init UMR WQE. */
+		cseg->sq_ds = rte_cpu_to_be_32((qp->qp_obj->id << 8) |
+					 (priv->umr_wqe_size / MLX5_WSEG_SIZE));
+		cseg->flags = RTE_BE32(MLX5_COMP_ONLY_FIRST_ERR <<
+				       MLX5_COMP_MODE_OFFSET);
+		cseg->misc = rte_cpu_to_be_32(qp->mkey[i]->id);
+		ucseg->if_cf_toe_cq_res = RTE_BE32(1u << MLX5_UMRC_IF_OFFSET);
+		ucseg->mkey_mask = RTE_BE64(1u << 0); /* Mkey length bit. */
+		ucseg->ko_to_bs = rte_cpu_to_be_32
+			((RTE_ALIGN(priv->max_segs_num, 4u) <<
+			 MLX5_UMRC_KO_OFFSET) | (4 << MLX5_UMRC_TO_BS_OFFSET));
+		bsf->keytag = priv->keytag;
+		/* Init RDMA WRITE WQE. */
+		cseg = RTE_PTR_ADD(cseg, priv->umr_wqe_size);
+		cseg->flags = RTE_BE32((MLX5_COMP_ALWAYS <<
+				      MLX5_COMP_MODE_OFFSET) |
+				      MLX5_WQE_CTRL_INITIATOR_SMALL_FENCE);
+		rseg = (struct mlx5_wqe_rseg *)(cseg + 1);
+		rseg->rkey = rte_cpu_to_be_32(qp->mkey[i]->id);
+	}
+}
+
+static int
+mlx5_crypto_indirect_mkeys_prepare(struct mlx5_crypto_priv *priv,
+				  struct mlx5_crypto_qp *qp)
+{
+	struct mlx5_umr_wqe *umr;
+	uint32_t i;
+	struct mlx5_devx_mkey_attr attr = {
+		.pd = priv->pdn,
+		.umr_en = 1,
+		.crypto_en = 1,
+		.set_remote_rw = 1,
+		.klm_num = RTE_ALIGN(priv->max_segs_num, 4),
+	};
+
+	for (umr = (struct mlx5_umr_wqe *)qp->umem_buf, i = 0;
+	   i < qp->entries_n; i++, umr = RTE_PTR_ADD(umr, priv->wqe_set_size)) {
+		attr.klm_array = (struct mlx5_klm *)&umr->kseg[0];
+		qp->mkey[i] = mlx5_devx_cmd_mkey_create(priv->ctx, &attr);
+		if (!qp->mkey[i]) {
+			DRV_LOG(ERR, "Failed to allocate indirect mkey.");
+			return -1;
+		}
+	}
+	return 0;
+}
+
 static int
 mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 			     const struct rte_cryptodev_qp_conf *qp_conf,
@@ -278,7 +341,7 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	struct mlx5_crypto_qp *qp;
 	uint16_t log_nb_desc = rte_log2_u32(qp_conf->nb_descriptors);
 	uint32_t umem_size = RTE_BIT32(log_nb_desc) *
-			      MLX5_CRYPTO_WQE_SET_SIZE +
+			      priv->wqe_set_size +
 			      sizeof(*qp->db_rec) * 2;
 	uint32_t alloc_size = sizeof(*qp);
 	struct mlx5_devx_cq_attr cq_attr = {
@@ -286,7 +349,9 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	};
 
 	alloc_size = RTE_ALIGN(alloc_size, RTE_CACHE_LINE_SIZE);
-	alloc_size += sizeof(struct rte_crypto_op *) * RTE_BIT32(log_nb_desc);
+	alloc_size += (sizeof(struct rte_crypto_op *) +
+		       sizeof(struct mlx5_devx_obj *)) *
+		       RTE_BIT32(log_nb_desc);
 	qp = rte_zmalloc_socket(__func__, alloc_size, RTE_CACHE_LINE_SIZE,
 				socket_id);
 	if (qp == NULL) {
@@ -330,8 +395,7 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	attr.wq_umem_id = qp->umem_obj->umem_id;
 	attr.wq_umem_offset = 0;
 	attr.dbr_umem_id = qp->umem_obj->umem_id;
-	attr.dbr_address = RTE_BIT64(log_nb_desc) *
-			   MLX5_CRYPTO_WQE_SET_SIZE;
+	attr.dbr_address = RTE_BIT64(log_nb_desc) * priv->wqe_set_size;
 	qp->qp_obj = mlx5_devx_cmd_create_qp(priv->ctx, &attr);
 	if (qp->qp_obj == NULL) {
 		DRV_LOG(ERR, "Failed to create QP(%u).", rte_errno);
@@ -340,8 +404,17 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	qp->db_rec = RTE_PTR_ADD(qp->umem_buf, (uintptr_t)attr.dbr_address);
 	if (mlx5_crypto_qp2rts(qp))
 		goto error;
-	qp->ops = (struct rte_crypto_op **)RTE_ALIGN((uintptr_t)(qp + 1),
+	qp->mkey = (struct mlx5_devx_obj **)RTE_ALIGN((uintptr_t)(qp + 1),
 							   RTE_CACHE_LINE_SIZE);
+	qp->ops = (struct rte_crypto_op **)(qp->mkey + RTE_BIT32(log_nb_desc));
+	qp->entries_n = 1 << log_nb_desc;
+	if (mlx5_crypto_indirect_mkeys_prepare(priv, qp)) {
+		DRV_LOG(ERR, "Cannot allocate indirect memory regions.");
+		rte_errno = ENOMEM;
+		goto error;
+	}
+	mlx5_crypto_qp_init(priv, qp);
+	qp->priv = priv;
 	dev->data->queue_pairs[qp_id] = qp;
 	return 0;
 error:
diff --git a/drivers/crypto/mlx5/mlx5_crypto.h b/drivers/crypto/mlx5/mlx5_crypto.h
index 81452bd700..52fcf5217f 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.h
+++ b/drivers/crypto/mlx5/mlx5_crypto.h
@@ -16,7 +16,6 @@
 
 #define MLX5_CRYPTO_DEK_HTABLE_SZ (1 << 11)
 #define MLX5_CRYPTO_KEY_LENGTH 80
-#define MLX5_CRYPTO_WQE_SET_SIZE 1024
 
 struct mlx5_crypto_priv {
 	TAILQ_ENTRY(mlx5_crypto_priv) next;
@@ -24,6 +23,7 @@ struct mlx5_crypto_priv {
 	struct rte_pci_device *pci_dev;
 	struct rte_cryptodev *crypto_dev;
 	void *uar; /* User Access Region. */
+	volatile uint64_t *uar_addr;
 	uint32_t pdn; /* Protection Domain number. */
 	uint32_t max_segs_num; /* Maximum supported data segs. */
 	struct ibv_pd *pd;
@@ -40,13 +40,21 @@ struct mlx5_crypto_priv {
 };
 
 struct mlx5_crypto_qp {
+	struct mlx5_crypto_priv *priv;
 	struct mlx5_devx_cq cq_obj;
 	struct mlx5_devx_obj *qp_obj;
+	struct rte_cryptodev_stats stats;
 	struct mlx5dv_devx_umem *umem_obj;
 	void *umem_buf;
 	volatile uint32_t *db_rec;
 	struct rte_crypto_op **ops;
+	struct mlx5_devx_obj **mkey; /* WQE's indirect mekys. */
 	struct mlx5_mr_ctrl mr_ctrl;
+	uint8_t *wqe;
+	uint16_t entries_n;
+	uint16_t pi;
+	uint16_t ci;
+	uint16_t db_pi;
 };
 
 struct mlx5_crypto_dek {
-- 
2.25.1

[dpdk-dev] [PATCH v2 13/15] crypto/mlx5: add enqueue and dequeue operations

[Matan Azrad]

From: Suanming Mou <suanmingm@nvidia.com>

The crypto operations are done with the WQE set which contains
one UMR WQE and one rdma write WQE. Most segments of the WQE
set are initialized properly during queue setup, only limited
segments are initialized according to the crypto detail in the
datapath process.

This commit adds the enquue and dequeue operations and updates
the WQE set segments accordingly.

Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Signed-off-by: Matan Azrad <matan@nvidia.com>
---
 drivers/crypto/mlx5/mlx5_crypto.c | 243 +++++++++++++++++++++++++++++-
 drivers/crypto/mlx5/mlx5_crypto.h |   3 +
 2 files changed, 241 insertions(+), 5 deletions(-)

diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index e8f68eb115..08a8c1e925 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -268,6 +268,239 @@ mlx5_crypto_qp2rts(struct mlx5_crypto_qp *qp)
 	return 0;
 }
 
+static __rte_noinline uint32_t
+mlx5_crypto_get_block_size(struct rte_crypto_op *op)
+{
+	uint32_t bl = op->sym->cipher.data.length;
+
+	switch (bl) {
+	case (1 << 20):
+		return RTE_BE32(MLX5_BLOCK_SIZE_1MB << MLX5_BLOCK_SIZE_OFFSET);
+	case (1 << 12):
+		return RTE_BE32(MLX5_BLOCK_SIZE_4096B <<
+				MLX5_BLOCK_SIZE_OFFSET);
+	case (1 << 9):
+		return RTE_BE32(MLX5_BLOCK_SIZE_512B << MLX5_BLOCK_SIZE_OFFSET);
+	default:
+		DRV_LOG(ERR, "Unknown block size: %u.", bl);
+		return UINT32_MAX;
+	}
+}
+
+static __rte_always_inline uint32_t
+mlx5_crypto_klm_set(struct mlx5_crypto_priv *priv, struct mlx5_crypto_qp *qp,
+		      struct rte_mbuf *mbuf, struct mlx5_wqe_dseg *klm,
+		      uint32_t offset, uint32_t *remain)
+{
+	uint32_t data_len = (rte_pktmbuf_data_len(mbuf) - offset);
+	uintptr_t addr = rte_pktmbuf_mtod_offset(mbuf, uintptr_t, offset);
+
+	if (data_len > *remain)
+		data_len = *remain;
+	*remain -= data_len;
+	klm->bcount = rte_cpu_to_be_32(data_len);
+	klm->pbuf = rte_cpu_to_be_64(addr);
+	klm->lkey = mlx5_mr_addr2mr_bh(priv->pd, 0,
+		&priv->mr_scache, &qp->mr_ctrl, addr,
+		!!(mbuf->ol_flags & EXT_ATTACHED_MBUF));
+	return klm->lkey;
+
+}
+
+static __rte_always_inline uint32_t
+mlx5_crypto_klms_set(struct mlx5_crypto_priv *priv, struct mlx5_crypto_qp *qp,
+		     struct rte_crypto_op *op, struct rte_mbuf *mbuf,
+		     struct mlx5_wqe_dseg *klm)
+{
+	uint32_t remain_len = op->sym->cipher.data.length;
+	uint32_t nb_segs = mbuf->nb_segs;
+	uint32_t klm_n = 1;
+
+	/* First mbuf needs to take the cipher offset. */
+	if (unlikely(mlx5_crypto_klm_set(priv, qp, mbuf, klm,
+		     op->sym->cipher.data.offset, &remain_len) == UINT32_MAX)) {
+		op->status = RTE_CRYPTO_OP_STATUS_ERROR;
+		return 0;
+	}
+	while (remain_len) {
+		nb_segs--;
+		mbuf = mbuf->next;
+		if (unlikely(mbuf == NULL || nb_segs == 0)) {
+			op->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS;
+			return 0;
+		}
+		if (unlikely(mlx5_crypto_klm_set(priv, qp, mbuf, klm, 0,
+						 &remain_len) == UINT32_MAX)) {
+			op->status = RTE_CRYPTO_OP_STATUS_ERROR;
+			return 0;
+		}
+		klm_n++;
+	}
+	return klm_n;
+}
+
+static __rte_always_inline int
+mlx5_crypto_wqe_set(struct mlx5_crypto_priv *priv,
+			 struct mlx5_crypto_qp *qp,
+			 struct rte_crypto_op *op,
+			 struct mlx5_umr_wqe *umr)
+{
+	struct mlx5_crypto_session *sess = get_sym_session_private_data
+				(op->sym->session, mlx5_crypto_driver_id);
+	struct mlx5_wqe_cseg *cseg = &umr->ctr;
+	struct mlx5_wqe_mkey_cseg *mkc = &umr->mkc;
+	struct mlx5_wqe_dseg *klms = &umr->kseg[0];
+	struct mlx5_wqe_umr_bsf_seg *bsf = ((struct mlx5_wqe_umr_bsf_seg *)
+				      RTE_PTR_ADD(umr, priv->umr_wqe_size)) - 1;
+	uint16_t nop_ds;
+	/* Set UMR WQE. */
+	uint32_t klm_n = mlx5_crypto_klms_set(priv, qp, op,
+			op->sym->m_dst ? op->sym->m_dst : op->sym->m_src, klms);
+
+	if (unlikely(klm_n == 0))
+		return 0;
+	bsf->bs_bpt_eo_es = sess->bs_bpt_eo_es;
+	if (unlikely(!sess->bsp_res)) {
+		bsf->bsp_res = mlx5_crypto_get_block_size(op);
+		if (unlikely(bsf->bsp_res == UINT32_MAX)) {
+			op->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS;
+			return 0;
+		}
+	} else {
+		bsf->bsp_res = sess->bsp_res;
+	}
+	bsf->raw_data_size = rte_cpu_to_be_32(op->sym->cipher.data.length);
+	memcpy(bsf->xts_initial_tweak,
+	       rte_crypto_op_ctod_offset(op, uint8_t *, sess->iv_offset), 16);
+	bsf->res_dp = sess->dek_id;
+	cseg->opcode = rte_cpu_to_be_32((qp->db_pi << 8) | MLX5_OPCODE_UMR);
+	mkc->len = rte_cpu_to_be_64(op->sym->cipher.data.length);
+	/* Set RDMA_WRITE WQE. */
+	cseg = RTE_PTR_ADD(cseg, priv->umr_wqe_size);
+	klms = RTE_PTR_ADD(cseg, sizeof(struct mlx5_rdma_write_wqe));
+	cseg->opcode = rte_cpu_to_be_32((qp->db_pi << 8) |
+					MLX5_OPCODE_RDMA_WRITE);
+	if (op->sym->m_dst != op->sym->m_src) {
+		klm_n = mlx5_crypto_klms_set(priv, qp, op, op->sym->m_src,
+					     klms);
+		if (unlikely(klm_n == 0))
+			return 0;
+	} else {
+		memcpy(klms, &umr->kseg[0], sizeof(*klms) * klm_n);
+	}
+	cseg->sq_ds = rte_cpu_to_be_32((qp->qp_obj->id << 8) | (2 + klm_n));
+	qp->db_pi += priv->wqe_stride;
+	/* Set NOP WQE if needed. */
+	klm_n = RTE_ALIGN(klm_n + 2, 4) - 2;
+	nop_ds = priv->max_rdmaw_klm_n - klm_n;
+	if (nop_ds) {
+		cseg = (struct mlx5_wqe_cseg *)(klms + klm_n);
+		cseg->opcode = rte_cpu_to_be_32(((qp->db_pi - (nop_ds >> 2)) <<
+						 8) | MLX5_OPCODE_NOP);
+		cseg->sq_ds = rte_cpu_to_be_32((qp->qp_obj->id << 8) | nop_ds);
+	}
+	qp->wqe = (uint8_t *)cseg;
+	return 1;
+}
+
+static __rte_always_inline void
+mlx5_crypto_uar_write(uint64_t val, struct mlx5_crypto_priv *priv)
+{
+#ifdef RTE_ARCH_64
+	*priv->uar_addr = val;
+#else /* !RTE_ARCH_64 */
+	rte_spinlock_lock(&priv->uar32_sl);
+	*(volatile uint32_t *)priv->uar_addr = val;
+	rte_io_wmb();
+	*((volatile uint32_t *)priv->uar_addr + 1) = val >> 32;
+	rte_spinlock_unlock(&priv->uar32_sl);
+#endif
+}
+
+static uint16_t
+mlx5_crypto_enqueue_burst(void *queue_pair, struct rte_crypto_op **ops,
+			  uint16_t nb_ops)
+{
+	struct mlx5_crypto_qp *qp = queue_pair;
+	struct mlx5_crypto_priv *priv = qp->priv;
+	struct mlx5_umr_wqe *umr;
+	struct rte_crypto_op *op;
+	uint16_t mask = qp->entries_n - 1;
+	uint16_t remain = qp->entries_n - (qp->pi - qp->ci);
+
+	if (remain < nb_ops)
+		nb_ops = remain;
+	else
+		remain = nb_ops;
+	if (unlikely(remain == 0))
+		return 0;
+	do {
+		op = *ops++;
+		umr = RTE_PTR_ADD(qp->umem_buf, priv->wqe_set_size * qp->pi);
+		if (unlikely(mlx5_crypto_wqe_set(priv, qp, op, umr) == 0))
+			break;
+		qp->ops[qp->pi] = op;
+		qp->pi = (qp->pi + 1) & mask;
+	} while (--remain);
+	rte_io_wmb();
+	qp->db_rec[MLX5_SND_DBR] = rte_cpu_to_be_32(qp->db_pi);
+	rte_wmb();
+	mlx5_crypto_uar_write(*(volatile uint64_t *)qp->wqe, qp->priv);
+	rte_wmb();
+	return nb_ops;
+}
+
+static __rte_noinline void
+mlx5_crypto_cqe_err_handle(struct mlx5_crypto_qp *qp, struct rte_crypto_op *op)
+{
+	const uint32_t idx = qp->ci & (qp->entries_n - 1);
+	volatile struct mlx5_err_cqe *cqe = (volatile struct mlx5_err_cqe *)
+							&qp->cq_obj.cqes[idx];
+
+	op->status = RTE_CRYPTO_OP_STATUS_ERROR;
+	DRV_LOG(ERR, "CQE ERR:%x.\n", rte_be_to_cpu_32(cqe->syndrome));
+}
+
+static uint16_t
+mlx5_crypto_dequeue_burst(void *queue_pair, struct rte_crypto_op **ops,
+			  uint16_t nb_ops)
+{
+	struct mlx5_crypto_qp *qp = queue_pair;
+	volatile struct mlx5_cqe *restrict cqe;
+	struct rte_crypto_op *restrict op;
+	const unsigned int cq_size = qp->entries_n;
+	const unsigned int mask = cq_size - 1;
+	uint32_t idx;
+	uint32_t next_idx = qp->ci & mask;
+	const uint16_t max = RTE_MIN((uint16_t)(qp->pi - qp->ci), nb_ops);
+	uint16_t i = 0;
+	int ret;
+
+	if (unlikely(max == 0))
+		return 0;
+	do {
+		idx = next_idx;
+		next_idx = (qp->ci + 1) & mask;
+		op = qp->ops[idx];
+		cqe = &qp->cq_obj.cqes[idx];
+		ret = check_cqe(cqe, cq_size, qp->ci);
+		rte_io_rmb();
+		if (unlikely(ret != MLX5_CQE_STATUS_SW_OWN)) {
+			if (unlikely(ret != MLX5_CQE_STATUS_HW_OWN))
+				mlx5_crypto_cqe_err_handle(qp, op);
+			break;
+		}
+		op->status = RTE_CRYPTO_OP_STATUS_SUCCESS;
+		ops[i++] = op;
+		qp->ci++;
+	} while (i < max);
+	if (likely(i != 0)) {
+		rte_io_wmb();
+		qp->cq_obj.db_rec[0] = rte_cpu_to_be_32(qp->ci);
+	}
+	return i;
+}
+
 static void
 mlx5_crypto_qp_init(struct mlx5_crypto_priv *priv, struct mlx5_crypto_qp *qp)
 {
@@ -489,8 +722,9 @@ mlx5_crypto_hw_global_prepare(struct mlx5_crypto_priv *priv)
 	if (mlx5_crypto_pd_create(priv) != 0)
 		return -1;
 	priv->uar = mlx5_devx_alloc_uar(priv->ctx, -1);
-	if (priv->uar == NULL || mlx5_os_get_devx_uar_reg_addr(priv->uar) ==
-	    NULL) {
+	if (priv->uar)
+		priv->uar_addr = mlx5_os_get_devx_uar_reg_addr(priv->uar);
+	if (priv->uar == NULL || priv->uar_addr == NULL) {
 		rte_errno = errno;
 		claim_zero(mlx5_glue->dealloc_pd(priv->pd));
 		DRV_LOG(ERR, "Failed to allocate UAR.");
@@ -685,9 +919,8 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 	DRV_LOG(INFO,
 		"Crypto device %s was created successfully.", ibv->name);
 	crypto_dev->dev_ops = &mlx5_crypto_ops;
-	crypto_dev->dequeue_burst = NULL;
-	crypto_dev->enqueue_burst = NULL;
-	crypto_dev->feature_flags = RTE_CRYPTODEV_FF_HW_ACCELERATED;
+	crypto_dev->dequeue_burst = mlx5_crypto_dequeue_burst;
+	crypto_dev->enqueue_burst = mlx5_crypto_enqueue_burst;
 	crypto_dev->driver_id = mlx5_crypto_driver_id;
 	priv = crypto_dev->data->dev_private;
 	priv->ctx = ctx;
diff --git a/drivers/crypto/mlx5/mlx5_crypto.h b/drivers/crypto/mlx5/mlx5_crypto.h
index 52fcf5217f..ac4ad1834f 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.h
+++ b/drivers/crypto/mlx5/mlx5_crypto.h
@@ -37,6 +37,9 @@ struct mlx5_crypto_priv {
 	uint16_t rdmw_wqe_size;
 	uint16_t wqe_stride;
 	uint16_t max_rdmaw_klm_n;
+#ifndef RTE_ARCH_64
+	rte_spinlock_t uar32_sl;
+#endif /* RTE_ARCH_64 */
 };
 
 struct mlx5_crypto_qp {
-- 
2.25.1

[dpdk-dev] [PATCH v2 14/15] crypto/mlx5: add statistic get and reset operations

[Matan Azrad]

From: Suanming Mou <suanmingm@nvidia.com>

This commit adds mlx5 crypto statistic get and reset operations.

Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Signed-off-by: Matan Azrad <matan@nvidia.com>
---
 drivers/crypto/mlx5/mlx5_crypto.c | 39 ++++++++++++++++++++++++++++---
 1 file changed, 36 insertions(+), 3 deletions(-)

diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index 08a8c1e925..af8985939e 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -437,11 +437,14 @@ mlx5_crypto_enqueue_burst(void *queue_pair, struct rte_crypto_op **ops,
 	do {
 		op = *ops++;
 		umr = RTE_PTR_ADD(qp->umem_buf, priv->wqe_set_size * qp->pi);
-		if (unlikely(mlx5_crypto_wqe_set(priv, qp, op, umr) == 0))
+		if (unlikely(mlx5_crypto_wqe_set(priv, qp, op, umr) == 0)) {
+			qp->stats.enqueue_err_count++;
 			break;
+		}
 		qp->ops[qp->pi] = op;
 		qp->pi = (qp->pi + 1) & mask;
 	} while (--remain);
+	qp->stats.enqueued_count += nb_ops;
 	rte_io_wmb();
 	qp->db_rec[MLX5_SND_DBR] = rte_cpu_to_be_32(qp->db_pi);
 	rte_wmb();
@@ -458,6 +461,7 @@ mlx5_crypto_cqe_err_handle(struct mlx5_crypto_qp *qp, struct rte_crypto_op *op)
 							&qp->cq_obj.cqes[idx];
 
 	op->status = RTE_CRYPTO_OP_STATUS_ERROR;
+	qp->stats.dequeue_err_count++;
 	DRV_LOG(ERR, "CQE ERR:%x.\n", rte_be_to_cpu_32(cqe->syndrome));
 }
 
@@ -497,6 +501,7 @@ mlx5_crypto_dequeue_burst(void *queue_pair, struct rte_crypto_op **ops,
 	if (likely(i != 0)) {
 		rte_io_wmb();
 		qp->cq_obj.db_rec[0] = rte_cpu_to_be_32(qp->ci);
+		qp->stats.dequeued_count += i;
 	}
 	return i;
 }
@@ -655,14 +660,42 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	return -1;
 }
 
+static void
+mlx5_crypto_stats_get(struct rte_cryptodev *dev,
+		      struct rte_cryptodev_stats *stats)
+{
+	int qp_id;
+
+	for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) {
+		struct mlx5_crypto_qp *qp = dev->data->queue_pairs[qp_id];
+
+		stats->enqueued_count += qp->stats.enqueued_count;
+		stats->dequeued_count += qp->stats.dequeued_count;
+		stats->enqueue_err_count += qp->stats.enqueue_err_count;
+		stats->dequeue_err_count += qp->stats.dequeue_err_count;
+	}
+}
+
+static void
+mlx5_crypto_stats_reset(struct rte_cryptodev *dev)
+{
+	int qp_id;
+
+	for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) {
+		struct mlx5_crypto_qp *qp = dev->data->queue_pairs[qp_id];
+
+		memset(&qp->stats, 0, sizeof(qp->stats));
+	}
+}
+
 static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_configure			= mlx5_crypto_dev_configure,
 	.dev_start			= mlx5_crypto_dev_start,
 	.dev_stop			= mlx5_crypto_dev_stop,
 	.dev_close			= mlx5_crypto_dev_close,
 	.dev_infos_get			= mlx5_crypto_dev_infos_get,
-	.stats_get			= NULL,
-	.stats_reset			= NULL,
+	.stats_get			= mlx5_crypto_stats_get,
+	.stats_reset			= mlx5_crypto_stats_reset,
 	.queue_pair_setup		= mlx5_crypto_queue_pair_setup,
 	.queue_pair_release		= mlx5_crypto_queue_pair_release,
 	.sym_session_get_size		= mlx5_crypto_sym_session_get_size,
-- 
2.25.1

[dpdk-dev] [PATCH v2 15/15] crypto/mlx5: set feature flags and capabilities

[Matan Azrad]

From: Shiri Kuzin <shirik@nvidia.com>

Add the supported capabilities to the crypto driver.

Add supported feature flags.

Add crypto driver documentation.

Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
Signed-off-by: Matan Azrad <matan@nvidia.com>
---
 doc/guides/cryptodevs/features/mlx5.ini |  37 ++++++
 doc/guides/cryptodevs/index.rst         |   1 +
 doc/guides/cryptodevs/mlx5.rst          | 152 ++++++++++++++++++++++++
 doc/guides/rel_notes/release_21_05.rst  |   5 +
 drivers/crypto/mlx5/mlx5_crypto.c       |  39 +++++-
 5 files changed, 231 insertions(+), 3 deletions(-)
 create mode 100644 doc/guides/cryptodevs/features/mlx5.ini
 create mode 100644 doc/guides/cryptodevs/mlx5.rst

diff --git a/doc/guides/cryptodevs/features/mlx5.ini b/doc/guides/cryptodevs/features/mlx5.ini
new file mode 100644
index 0000000000..a89526add0
--- /dev/null
+++ b/doc/guides/cryptodevs/features/mlx5.ini
@@ -0,0 +1,37 @@
+;
+; Features of a mlx5 crypto driver.
+;
+; Refer to default.ini for the full list of available PMD features.
+;
+[Features]
+Symmetric crypto       = Y
+HW Accelerated         = Y
+In Place SGL           = Y
+OOP SGL In SGL Out     = Y
+OOP SGL In LB  Out     = Y
+OOP LB  In SGL Out     = Y
+OOP LB  In LB  Out     = Y
+Cipher multiple data units = Y
+Cipher wrapped key     = Y
+
+;
+; Supported crypto algorithms of a mlx5 crypto driver.
+;
+[Cipher]
+AES XTS (128)  = Y
+AES XTS (256)  = Y
+
+;
+; Supported authentication algorithms of a mlx5 crypto driver.
+;
+[Auth]
+
+;
+; Supported AEAD algorithms of a mlx5 crypto driver.
+;
+[AEAD]
+
+;
+; Supported Asymmetric algorithms of a mlx5 crypto driver.
+;
+[Asymmetric]
diff --git a/doc/guides/cryptodevs/index.rst b/doc/guides/cryptodevs/index.rst
index 279f56a002..747409c441 100644
--- a/doc/guides/cryptodevs/index.rst
+++ b/doc/guides/cryptodevs/index.rst
@@ -22,6 +22,7 @@ Crypto Device Drivers
     octeontx
     octeontx2
     openssl
+    mlx5
     mvsam
     nitrox
     null
diff --git a/doc/guides/cryptodevs/mlx5.rst b/doc/guides/cryptodevs/mlx5.rst
new file mode 100644
index 0000000000..4ccec78be8
--- /dev/null
+++ b/doc/guides/cryptodevs/mlx5.rst
@@ -0,0 +1,152 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+   Copyright 2021 Mellanox Technologies, Ltd
+
+.. include:: <isonum.txt>
+
+MLX5 Crypto Driver
+==================
+
+The MLX5 crypto driver library
+(**librte_crypto_mlx5**) provides support for **Mellanox ConnectX-6**
+family adapters.
+
+Overview
+--------
+
+The device can provide disk encryption services, allowing data encryption
+and decryption towards a disk. Having all encryption/decryption
+operations done in a single device can reduce cost and overheads of the related
+FIPS certification, as ConnectX-6 is FIPS 140-2 level-2 ready.
+The encryption cipher is AES-XTS of 256/512 key size.
+
+MKEY is a memory region object in the hardware, that holds address translation information and
+attributes per memory area. Its ID must be tied to addresses provided to the hardware.
+The encryption operations are performed with MKEY read\write transactions, when
+the MKEY is configured to perform crypto operations.
+
+The encryption does not require text to be aligned to the AES block size (128b).
+
+In order to move the device to crypto operational mode, credential and KEK
+(Key Encrypting Key) should be set as the first step.
+The credential will be used by the software in order to perform crypto login, and the KEK is
+the AES Key Wrap Algorithm (rfc3394) key that will be used for sensitive data
+wrapping.
+The credential and the AES-XTS keys should be provided to the hardware, as ciphertext
+encrypted by the KEK.
+
+A keytag (64 bits) should be appended to the AES-XTS keys (before wrapping),
+and will be validated when the hardware attempts to access it.
+
+For security reasons and to increase robustness, this driver only deals with virtual
+memory addresses. The way resources allocations are handled by the kernel,
+combined with hardware specifications that allow handling virtual memory
+addresses directly, ensure that DPDK applications cannot access random
+physical memory (or memory that does not belong to the current process).
+
+The PMD uses libibverbs and libmlx5 to access the device firmware or to
+access the hardware components directly.
+There are different levels of objects and bypassing abilities.
+To get the best performances:
+
+- Verbs is a complete high-level generic API.
+- Direct Verbs is a device-specific API.
+- DevX allows to access firmware objects.
+
+Enabling librte_crypto_mlx5 causes DPDK applications to be linked against
+libibverbs.
+
+Mellanox mlx5 PCI device can be probed by a number of different PCI devices, such as
+net / vDPA / RegEx. To select the crypto PMD, ``class=crypto``
+should be specified as a device parameter. The crypto device can be probed and
+used with other Mellanox classes by adding more options in the class.
+For example: ``class=net:crypto`` will probe both the net PMD and the crypto
+PMD.
+
+When crypto engines are defined to work in wrapped import method, they come out
+of the factory in Commissioning mode, and thus, cannot be used for crypto operations
+yet. A dedicated tool is used for changing the mode from Commissioning to
+Operational, while setting the first import_KEK and credential in plaintext.
+The mlxreg dedicated tool should be used as follows:
+
+- Set CRYPTO_OPERATIONAL register to set the device in crypto operational mode.
+
+  The input to this tool is:
+    The first credential in plaintext, 40B.
+    The first import_KEK in plaintext: kek size 0 for 16B or 1 for 32B, kek data.
+
+  Example:
+  mlxreg -d /dev/mst/mt4123_pciconf0 --reg_name CRYPTO_OPERATIONAL --get
+
+  The "wrapped_crypto_operational" value will be "0x00000000".
+  The command to set the register should be executed only once, and all the
+  values mentioned above should be specified in the same command.
+
+  Example:
+  mlxreg -d /dev/mst/mt4123_pciconf0 --reg_name CRYPTO_OPERATIONAL
+  --set "credential[0]=0x10000000, credential[1]=0x10000000, kek[0]=0x00000000"
+
+  All values not specified will remain 0.
+  "wrapped_crypto_going_to_commissioning" and  "wrapped_crypto_operational"
+  should not be specified.
+
+  All the device ports should set it in order to move to operational mode.
+
+- Query CRYPTO_OPERATIONAL register to make sure the device is in Operational
+  mode.
+
+  Example:
+  mlxreg -d /dev/mst/mt4123_pciconf0 --reg_name CRYPTO_OPERATIONAL --get
+  The "wrapped_crypto_operational" value will be "0x00000001" if the mode was
+  successfully changed to operational mode.
+
+
+Driver options
+--------------
+
+- ``class`` parameter [string]
+
+  Select the class of the driver that should probe the device.
+  `crypto` for the mlx5 crypto driver.
+
+- ``wcs_file`` parameter [string] - mandatory
+
+  File path including only the wrapped credential in string format of hexadecimal
+  numbers, represent 48 bytes (8 bytes IV added by the AES key wrap algorithm).
+
+- ``import_kek_id`` parameter [int]
+
+  The identifier of the KEK, default value is 0 represents the operational
+  register import_kek..
+
+- ``credential_id`` parameter [int]
+
+  The identifier of the credential, default value is 0 represents the operational
+  register credential.
+
+- ``max_segs_num`` parameter [int]
+
+  Maximum number of mbuf chain segments(src or dest), default value is 8.
+
+- ``keytag`` parameter [int]
+
+  The plaintext of the keytag appanded to the AES-XTS keys, default value is 0.
+
+
+Limitations
+-----------
+
+- AES-XTS keys provided in Xform must include keytag and should be wrappend.
+- The supported data-unit lengths are: 512B, 1KB, 1MB. In case the `dataunit_len`
+  is not provided in the cipher Xform, the OP length is limitted to the above values.
+
+
+Supported NICs
+--------------
+
+* Mellanox\ |reg| ConnectX\ |reg|-6 200G MCX654106A-HCAT (2x200G)
+
+Prerequisites
+-------------
+
+- Mellanox OFED version: **5.3**
+  see :doc:`../../nics/mlx5` guide for more Mellanox OFED details.
diff --git a/doc/guides/rel_notes/release_21_05.rst b/doc/guides/rel_notes/release_21_05.rst
index 133a0dbbf2..dc508fc63c 100644
--- a/doc/guides/rel_notes/release_21_05.rst
+++ b/doc/guides/rel_notes/release_21_05.rst
@@ -273,6 +273,11 @@ New Features
   * Added support for crypto adapter forward mode in octeontx2 event and crypto
     device driver.
 
+* **Added support for Nvidia crypto device driver.**
+
+  * Added mlx5 crypto driver to support AES-XTS cipher operations.
+    the first device to support it is ConnectX-6.
+
 
 Removed Items
 -------------
diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index af8985939e..c2e75905fd 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -22,6 +22,14 @@
 #define MLX5_CRYPTO_LOG_NAME pmd.crypto.mlx5
 #define MLX5_CRYPTO_MAX_QPS 1024
 #define MLX5_CRYPTO_MAX_SEGS 56
+#define MLX5_CRYPTO_FEATURE_FLAGS \
+	(RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO | RTE_CRYPTODEV_FF_HW_ACCELERATED | \
+	RTE_CRYPTODEV_FF_IN_PLACE_SGL | RTE_CRYPTODEV_FF_OOP_SGL_IN_SGL_OUT | \
+	RTE_CRYPTODEV_FF_OOP_SGL_IN_LB_OUT | \
+	RTE_CRYPTODEV_FF_OOP_LB_IN_SGL_OUT | \
+	RTE_CRYPTODEV_FF_OOP_LB_IN_LB_OUT | \
+	RTE_CRYPTODEV_FF_CIPHER_WRAPPED_KEY | \
+	RTE_CRYPTODEV_FF_CIPHER_MULTIPLE_DATA_UNITS)
 
 TAILQ_HEAD(mlx5_crypto_privs, mlx5_crypto_priv) mlx5_crypto_priv_list =
 				TAILQ_HEAD_INITIALIZER(mlx5_crypto_priv_list);
@@ -31,8 +39,32 @@ int mlx5_crypto_logtype;
 
 uint8_t mlx5_crypto_driver_id;
 
-const struct rte_cryptodev_capabilities
-		mlx5_crypto_caps[RTE_CRYPTO_OP_TYPE_UNDEFINED];
+const struct rte_cryptodev_capabilities mlx5_crypto_caps[] = {
+	{		/* AES XTS */
+		.op = RTE_CRYPTO_OP_TYPE_SYMMETRIC,
+		{.sym = {
+			.xform_type = RTE_CRYPTO_SYM_XFORM_CIPHER,
+			{.cipher = {
+				.algo = RTE_CRYPTO_CIPHER_AES_XTS,
+				.block_size = 16,
+				.key_size = {
+					.min = 32,
+					.max = 64,
+					.increment = 32
+				},
+				.iv_size = {
+					.min = 16,
+					.max = 16,
+					.increment = 0
+				},
+				.dataunit_set =
+				RTE_CRYPTO_CIPHER_DATA_UNIT_LEN_512_BYTES |
+				RTE_CRYPTO_CIPHER_DATA_UNIT_LEN_4096_BYTES,
+			}, }
+		}, }
+	},
+};
+
 
 static const char mlx5_crypto_drv_name[] = RTE_STR(MLX5_CRYPTO_DRIVER_NAME);
 
@@ -67,7 +99,7 @@ mlx5_crypto_dev_infos_get(struct rte_cryptodev *dev,
 	RTE_SET_USED(dev);
 	if (dev_info != NULL) {
 		dev_info->driver_id = mlx5_crypto_driver_id;
-		dev_info->feature_flags = 0;
+		dev_info->feature_flags = MLX5_CRYPTO_FEATURE_FLAGS;
 		dev_info->capabilities = mlx5_crypto_caps;
 		dev_info->max_nb_queue_pairs = MLX5_CRYPTO_MAX_QPS;
 		dev_info->min_mbuf_headroom_req = 0;
@@ -954,6 +986,7 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 	crypto_dev->dev_ops = &mlx5_crypto_ops;
 	crypto_dev->dequeue_burst = mlx5_crypto_dequeue_burst;
 	crypto_dev->enqueue_burst = mlx5_crypto_enqueue_burst;
+	crypto_dev->feature_flags = MLX5_CRYPTO_FEATURE_FLAGS;
 	crypto_dev->driver_id = mlx5_crypto_driver_id;
 	priv = crypto_dev->data->dev_private;
 	priv->ctx = ctx;
-- 
2.25.1

Re: [dpdk-dev] [PATCH v2 07/16] common/mlx5: fix cypto bsf attr

[Matan Azrad]

Hi Raslan

See below , need to squash this to
" common/mlx5: adjust DevX MKEY fields for crypto"

The one below it...missed that, sorry.

From: Matan Azrad <matan@nvidia.com>
> From: Suanming Mou <suanmingm@nvidia.com>
> 
> This commit should be squash to crypto en commit.
> 
> Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
> ---
>  drivers/common/mlx5/mlx5_devx_cmds.c | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c
> b/drivers/common/mlx5/mlx5_devx_cmds.c
> index 68a10b149a..7e3c8b55fa 100644
> --- a/drivers/common/mlx5/mlx5_devx_cmds.c
> +++ b/drivers/common/mlx5/mlx5_devx_cmds.c
> @@ -278,6 +278,10 @@ mlx5_devx_cmd_mkey_create(void *ctx,
>  	MLX5_SET64(mkc, mkc, start_addr, attr->addr);
>  	MLX5_SET64(mkc, mkc, len, attr->size);
>  	MLX5_SET(mkc, mkc, crypto_en, attr->crypto_en);
> +	if (attr->crypto_en) {
> +		MLX5_SET(mkc, mkc, bsf_en, attr->crypto_en);
> +		MLX5_SET(mkc, mkc, bsf_octword_size, 4);
> +	}
>  	mkey->obj = mlx5_glue->devx_obj_create(ctx, in, in_size_dw * 4,
> out,
>  					       sizeof(out));
>  	if (!mkey->obj) {
> --
> 2.25.1

Re: [dpdk-dev] [PATCH v2 00/15] drivers: introduce mlx5 crypto PMD

[Matan Azrad]

From: Matan Azrad <matan@nvidia.com>
 
> Add a new PMD for Nvidia devices- crypto PMD.
> The crypto PMD will be supported on Nvidia ConnectX6 The crypto PMD will
> add the support of encryption and decryption using the AES-XTS symmetric
> algorithm.
> The crypto PMD requires rdma-core and uses mlx5 DevX.
> 
> v2:
> Add data-path part.
> 
> Shiri Kuzin (10):
>   drivers: introduce mlx5 crypto PMD
>   crypto/mlx5: add DEK object management
>   crypto/mlx5: support session operations
>   crypto/mlx5: add basic operations
>   crypto/mlx5: support queue pairs operations
>   crypto/mlx5: add dev stop and start operations
>   crypto/mlx5: add memory region management
>   crypto/mlx5: create login object using DevX
>   crypto/mlx5: adjust to the multiple data unit API
>   crypto/mlx5: set feature flags and capabilities
> 
> Suanming Mou (5):
>   crypto/mlx5: add keytag device argument
>   crypto/mlx5: add maximum segments device argument
>   crypto/mlx5: add WQE set initialization
>   crypto/mlx5: add enqueue and dequeue operations
>   crypto/mlx5: add statistic get and reset operations


Series-acked-by: Matan Azrad <matan@nvidia.com>
 
>  MAINTAINERS                             |    4 +
>  doc/guides/cryptodevs/features/mlx5.ini |   37 +
>  doc/guides/cryptodevs/index.rst         |    1 +
>  doc/guides/cryptodevs/mlx5.rst          |  152 ++++
>  doc/guides/rel_notes/release_21_05.rst  |    5 +
>  drivers/common/mlx5/mlx5_common_pci.c   |   14 +
>  drivers/common/mlx5/mlx5_common_pci.h   |   21 +-
>  drivers/crypto/meson.build              |    1 +
>  drivers/crypto/mlx5/meson.build         |   27 +
>  drivers/crypto/mlx5/mlx5_crypto.c       | 1092 +++++++++++++++++++++++
>  drivers/crypto/mlx5/mlx5_crypto.h       |   92 ++
>  drivers/crypto/mlx5/mlx5_crypto_dek.c   |  136 +++
>  drivers/crypto/mlx5/mlx5_crypto_utils.h |   19 +
>  drivers/crypto/mlx5/version.map         |    3 +
>  14 files changed, 1594 insertions(+), 10 deletions(-)  create mode 100644
> doc/guides/cryptodevs/features/mlx5.ini
>  create mode 100644 doc/guides/cryptodevs/mlx5.rst  create mode 100644
> drivers/crypto/mlx5/meson.build  create mode 100644
> drivers/crypto/mlx5/mlx5_crypto.c  create mode 100644
> drivers/crypto/mlx5/mlx5_crypto.h  create mode 100644
> drivers/crypto/mlx5/mlx5_crypto_dek.c
>  create mode 100644 drivers/crypto/mlx5/mlx5_crypto_utils.h
>  create mode 100644 drivers/crypto/mlx5/version.map
> 
> --
> 2.25.1

Re: [dpdk-dev] [PATCH v2 00/16] mlx5 common part for crypto driver

[Matan Azrad]

From: Matan Azrad <matan@nvidia.com>
> The crypto PMD will be supported on Nvidia ConnectX6 The crypto PMD will
> add the support of encryption and decryption using the AES-XTS symmetric
> algorithm.
> The crypto PMD requires rdma-core and uses mlx5 DevX.
> This is the mlx5 common part that added support for DevX commands
> needed for crypto driver.
> Also includes PRM definitions.
> 
> v2:
> Add data-path part.
> 
> Dekel Peled (12):
>   common/mlx5: remove redundant spaces in header file
>   common/mlx5: update GENEVE TLV OPT obj name
>   common/mlx5: optimize read of general obj type caps
>   common/mlx5: add HCA cap for AES-XTS crypto
>   common/mlx5: support general object DEK create op
>   common/mlx5: adjust DevX MKEY fields for crypto
>   common/mlx5: support general obj IMPORT KEK create
>   common/mlx5: support general obj CRYPTO LOGIN create
>   common/mlx5: add crypto BSF struct and defines
>   common/mlx5: support general obj CREDENTIAL create
>   common/mlx5: add crypto register structs and defs
>   common/mlx5: support register write access
> 
> Shiri Kuzin (2):
>   common/mlx5: share hash list tool
>   common/mlx5: share get ib device match function
> 
> Suanming Mou (2):
>   common/mlx5: fix cypto bsf attr
>   common/mlx5: add UMR and RDMA write WQE defines

Series-acked-by: Matan Azrad <matan@nvidia.com>

>  drivers/common/mlx5/linux/mlx5_common_os.c   |  30 +-
>  drivers/common/mlx5/linux/mlx5_common_os.h   |   5 +
>  drivers/common/mlx5/linux/mlx5_nl.c          |   2 +-
>  drivers/common/mlx5/meson.build              |  15 +-
>  drivers/common/mlx5/mlx5_common.c            |   2 +-
>  drivers/common/mlx5/mlx5_common.h            |   2 +
>  drivers/common/mlx5/mlx5_common_devx.c       |   2 +-
>  drivers/common/mlx5/mlx5_common_log.h        |  21 +
>  drivers/common/mlx5/mlx5_common_mp.c         |   2 +-
>  drivers/common/mlx5/mlx5_common_mr.c         |   2 +-
>  drivers/common/mlx5/mlx5_common_pci.c        |   4 +-
>  drivers/common/mlx5/mlx5_common_utils.c      | 221 +++++++
>  drivers/common/mlx5/mlx5_common_utils.h      | 202 +++++-
>  drivers/common/mlx5/mlx5_devx_cmds.c         | 312 +++++++++-
>  drivers/common/mlx5/mlx5_devx_cmds.h         |  66 ++
>  drivers/common/mlx5/mlx5_malloc.c            |   2 +-
>  drivers/common/mlx5/mlx5_prm.h               | 610 +++++++++++++------
>  drivers/common/mlx5/version.map              |  24 +-
>  drivers/common/mlx5/windows/mlx5_common_os.c |   2 +-
>  drivers/common/mlx5/windows/mlx5_glue.c      |   2 +-
>  drivers/compress/mlx5/mlx5_compress.c        |  30 +-
>  drivers/net/mlx5/mlx5_utils.c                | 209 -------
>  drivers/net/mlx5/mlx5_utils.h                | 194 +-----
>  drivers/regex/mlx5/mlx5_regex.c              |  30 +-
>  drivers/vdpa/mlx5/mlx5_vdpa.c                |  32 +-
>  25 files changed, 1317 insertions(+), 706 deletions(-)  create mode 100644
> drivers/common/mlx5/mlx5_common_log.h
>  create mode 100644 drivers/common/mlx5/mlx5_common_utils.c
> 
> --
> 2.25.1

Re: [dpdk-dev] [PATCH v2 12/16] common/mlx5: share get ib device match function

[Thomas Monjalon]

29/04/2021 17:43, Matan Azrad:
> From: Shiri Kuzin <shirik@nvidia.com>
> 
> The get_ib_device_match function iterates over the list of ib devices
> returned by the get_device_list glue function and returns the ib device
> matching the provided address.
> 
> Since this function is in use by several drivers, in this patch we
> share the function in common part.
> 
> Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
> Acked-by: Matan Azrad <matan@nvidia.com>
> ---
> --- a/drivers/common/mlx5/linux/mlx5_common_os.c
> +++ b/drivers/common/mlx5/linux/mlx5_common_os.c
> +struct ibv_device *
> +mlx5_os_get_ib_device_match(struct rte_pci_addr *addr)

It would be better as mlx5_get_pci_ibv_device()
in drivers/common/mlx5/mlx5_common_pci.c
Just need to forward declare struct ibv_device
in drivers/common/mlx5/mlx5_common.h
This is what is done internally for sub-functions.

> --- a/drivers/common/mlx5/mlx5_common.h
> +++ b/drivers/common/mlx5/mlx5_common.h
> @@ -18,6 +18,7 @@
>  
>  #include "mlx5_prm.h"
>  #include "mlx5_devx_cmds.h"
> +#include "mlx5_common_os.h"
>  
>  /* Reported driver name. */
>  #define MLX5_PCI_DRIVER_NAME "mlx5_pci"
> @@ -215,6 +216,7 @@ enum mlx5_class {
>  	MLX5_CLASS_VDPA = RTE_BIT64(1),
>  	MLX5_CLASS_REGEX = RTE_BIT64(2),
>  	MLX5_CLASS_COMPRESS = RTE_BIT64(3),
> +	MLX5_CLASS_CRYPTO = RTE_BIT64(4),
>  };

This change has nothing to do in this patch I think.

> --- a/drivers/common/mlx5/version.map
> +++ b/drivers/common/mlx5/version.map
> @@ -111,6 +111,7 @@ INTERNAL {
>  	mlx5_os_reg_mr;
>  	mlx5_os_umem_reg;
>  	mlx5_os_umem_dereg;
> +	mlx5_os_get_ib_device_match; # WINDOWS_NO_EXPORT

Please keep alphabetical order.

Re: [dpdk-dev] [PATCH v2 15/16] common/mlx5: support register write access

[Thomas Monjalon]

29/04/2021 17:43, Matan Azrad:
> From: Dekel Peled <dekelp@nvidia.com>
> 
> This patch adds support of write operation to NIC registers.
> 
> Signed-off-by: Dekel Peled <dekelp@nvidia.com>
> Acked-by: Matan Azrad <matan@nvidia.com>
> ---
> --- a/drivers/common/mlx5/version.map
> +++ b/drivers/common/mlx5/version.map
> @@ -13,8 +13,17 @@ INTERNAL {
>  	mlx5_dev_to_pci_addr; # WINDOWS_NO_EXPORT
>  
>  	mlx5_devx_cmd_alloc_pd;
> +	mlx5_devx_alloc_uar;

Please keep alphabetical order.

> +
>  	mlx5_devx_cmd_create_cq;
> +	mlx5_devx_cmd_create_credential_obj;
> +	mlx5_devx_cmd_create_crypto_login_obj;
> +	mlx5_devx_cmd_create_dek_obj;
>  	mlx5_devx_cmd_create_flex_parser;
> +	mlx5_devx_cmd_create_flow_hit_aso_obj;
> +	mlx5_devx_cmd_create_flow_meter_aso_obj;
> +	mlx5_devx_cmd_create_geneve_tlv_option;
> +	mlx5_devx_cmd_create_import_kek_obj;
>  	mlx5_devx_cmd_create_qp;
>  	mlx5_devx_cmd_create_rq;
>  	mlx5_devx_cmd_create_rqt;
> @@ -24,13 +33,6 @@ INTERNAL {
>  	mlx5_devx_cmd_create_tis;
>  	mlx5_devx_cmd_create_virtio_q_counters; # WINDOWS_NO_EXPORT
>  	mlx5_devx_cmd_create_virtq;
> -	mlx5_devx_cmd_create_flow_hit_aso_obj;
> -	mlx5_devx_cmd_create_flow_meter_aso_obj;
> -	mlx5_devx_cmd_create_geneve_tlv_option;
> -        mlx5_devx_cmd_create_dek_obj;
> -        mlx5_devx_cmd_create_import_kek_obj;
> -        mlx5_devx_cmd_create_credential_obj;
> -        mlx5_devx_cmd_create_crypto_login_obj;

Are they changes of previous patches?
It seems some cleaning is required in previous patches.

>  	mlx5_devx_cmd_destroy;
>  	mlx5_devx_cmd_flow_counter_alloc;
>  	mlx5_devx_cmd_flow_counter_query;
> @@ -51,12 +53,17 @@ INTERNAL {
>  	mlx5_devx_cmd_queue_counter_alloc; # WINDOWS_NO_EXPORT
>  	mlx5_devx_cmd_queue_counter_query; # WINDOWS_NO_EXPORT
>  	mlx5_devx_cmd_register_read;
> +	mlx5_devx_cmd_register_write;
>  	mlx5_devx_cmd_wq_query; # WINDOWS_NO_EXPORT
>  	mlx5_devx_get_out_command_status;
>  	mlx5_devx_alloc_uar; # WINDOWS_NO_EXPORT
>  
> +
>  	mlx5_devx_cq_create;

There is an extra blank line here.

Re: [dpdk-dev] [PATCH v2 12/16] common/mlx5: share get ib device match function

[Matan Azrad]

Hi Thomas

From: Thomas Monjalon
> 29/04/2021 17:43, Matan Azrad:
> > From: Shiri Kuzin <shirik@nvidia.com>
> >
> > The get_ib_device_match function iterates over the list of ib devices
> > returned by the get_device_list glue function and returns the ib
> > device matching the provided address.
> >
> > Since this function is in use by several drivers, in this patch we
> > share the function in common part.
> >
> > Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
> > Acked-by: Matan Azrad <matan@nvidia.com>
> > ---
> > --- a/drivers/common/mlx5/linux/mlx5_common_os.c
> > +++ b/drivers/common/mlx5/linux/mlx5_common_os.c
> > +struct ibv_device *
> > +mlx5_os_get_ib_device_match(struct rte_pci_addr *addr)
> 
> It would be better as mlx5_get_pci_ibv_device() in
> drivers/common/mlx5/mlx5_common_pci.c
> Just need to forward declare struct ibv_device in
> drivers/common/mlx5/mlx5_common.h This is what is done internally for
> sub-functions.

struct ibv_device Is linux only, defined in Verbs library.
So, no need it in OS common place.


> > --- a/drivers/common/mlx5/mlx5_common.h
> > +++ b/drivers/common/mlx5/mlx5_common.h
> > @@ -18,6 +18,7 @@
> >
> >  #include "mlx5_prm.h"
> >  #include "mlx5_devx_cmds.h"
> > +#include "mlx5_common_os.h"
> >
> >  /* Reported driver name. */
> >  #define MLX5_PCI_DRIVER_NAME "mlx5_pci"
> > @@ -215,6 +216,7 @@ enum mlx5_class {
> >       MLX5_CLASS_VDPA = RTE_BIT64(1),
> >       MLX5_CLASS_REGEX = RTE_BIT64(2),
> >       MLX5_CLASS_COMPRESS = RTE_BIT64(3),
> > +     MLX5_CLASS_CRYPTO = RTE_BIT64(4),
> >  };
> 
> This change has nothing to do in this patch I think.
> 

Yes, will handle in v3.

> > --- a/drivers/common/mlx5/version.map
> > +++ b/drivers/common/mlx5/version.map
> > @@ -111,6 +111,7 @@ INTERNAL {
> >       mlx5_os_reg_mr;
> >       mlx5_os_umem_reg;
> >       mlx5_os_umem_dereg;
> > +     mlx5_os_get_ib_device_match; # WINDOWS_NO_EXPORT
> 
> Please keep alphabetical order.

Sure.

Re: [dpdk-dev] [PATCH v2 15/16] common/mlx5: support register write access

[Matan Azrad]

From: Thomas Monjalon
> 29/04/2021 17:43, Matan Azrad:
> > From: Dekel Peled <dekelp@nvidia.com>
> >
> > This patch adds support of write operation to NIC registers.
> >
> > Signed-off-by: Dekel Peled <dekelp@nvidia.com>
> > Acked-by: Matan Azrad <matan@nvidia.com>
> > ---
> > --- a/drivers/common/mlx5/version.map
> > +++ b/drivers/common/mlx5/version.map
> > @@ -13,8 +13,17 @@ INTERNAL {
> >       mlx5_dev_to_pci_addr; # WINDOWS_NO_EXPORT
> >
> >       mlx5_devx_cmd_alloc_pd;
> > +     mlx5_devx_alloc_uar;
> 
> Please keep alphabetical order.
> 
> > +
> >       mlx5_devx_cmd_create_cq;
> > +     mlx5_devx_cmd_create_credential_obj;
> > +     mlx5_devx_cmd_create_crypto_login_obj;
> > +     mlx5_devx_cmd_create_dek_obj;
> >       mlx5_devx_cmd_create_flex_parser;
> > +     mlx5_devx_cmd_create_flow_hit_aso_obj;
> > +     mlx5_devx_cmd_create_flow_meter_aso_obj;
> > +     mlx5_devx_cmd_create_geneve_tlv_option;
> > +     mlx5_devx_cmd_create_import_kek_obj;
> >       mlx5_devx_cmd_create_qp;
> >       mlx5_devx_cmd_create_rq;
> >       mlx5_devx_cmd_create_rqt;
> > @@ -24,13 +33,6 @@ INTERNAL {
> >       mlx5_devx_cmd_create_tis;
> >       mlx5_devx_cmd_create_virtio_q_counters; # WINDOWS_NO_EXPORT
> >       mlx5_devx_cmd_create_virtq;
> > -     mlx5_devx_cmd_create_flow_hit_aso_obj;
> > -     mlx5_devx_cmd_create_flow_meter_aso_obj;
> > -     mlx5_devx_cmd_create_geneve_tlv_option;
> > -        mlx5_devx_cmd_create_dek_obj;
> > -        mlx5_devx_cmd_create_import_kek_obj;
> > -        mlx5_devx_cmd_create_credential_obj;
> > -        mlx5_devx_cmd_create_crypto_login_obj;
> 
> Are they changes of previous patches?
> It seems some cleaning is required in previous patches.

Yes, this one tried to do the cleaning, I will handle it in the first patch (of this series) accessing this file. 

> >       mlx5_devx_cmd_destroy;
> >       mlx5_devx_cmd_flow_counter_alloc;
> >       mlx5_devx_cmd_flow_counter_query; @@ -51,12 +53,17 @@
> INTERNAL {
> >       mlx5_devx_cmd_queue_counter_alloc; # WINDOWS_NO_EXPORT
> >       mlx5_devx_cmd_queue_counter_query; # WINDOWS_NO_EXPORT
> >       mlx5_devx_cmd_register_read;
> > +     mlx5_devx_cmd_register_write;
> >       mlx5_devx_cmd_wq_query; # WINDOWS_NO_EXPORT
> >       mlx5_devx_get_out_command_status;
> >       mlx5_devx_alloc_uar; # WINDOWS_NO_EXPORT
> >
> > +
> >       mlx5_devx_cq_create;
> 
> There is an extra blank line here.
> 
> 

[dpdk-dev] [PATCH v3 00/15] mlx5 common part for crypto driver

[Matan Azrad]

The crypto PMD will be supported on Nvidia ConnectX6.
The crypto PMD will add the support of encryption and decryption using the AES-XTS symmetric algorithm.
The crypto PMD requires rdma-core and uses mlx5 DevX.
This is the mlx5 common part that added support for DevX commands needed for crypto driver.
Also includes PRM definitions.

v2:
Add data-path part.

v3:
Squash fix commit to the one before it.
Change get IB device function name.
Rearrange common version.map file.
Moved class definition to the driver seires.

Dekel Peled (12):
  common/mlx5: remove redundant spaces in header file
  common/mlx5: update GENEVE TLV OPT obj name
  common/mlx5: optimize read of general obj type caps
  common/mlx5: add HCA cap for AES-XTS crypto
  common/mlx5: support general object DEK create op
  common/mlx5: adjust DevX MKEY fields for crypto
  common/mlx5: support general obj IMPORT KEK create
  common/mlx5: support general obj CRYPTO LOGIN create
  common/mlx5: add crypto BSF struct and defines
  common/mlx5: support general obj CREDENTIAL create
  common/mlx5: add crypto register structs and defs
  common/mlx5: support register write access

Shiri Kuzin (2):
  common/mlx5: share hash list tool
  common/mlx5: share get ib device match function

Suanming Mou (1):
  common/mlx5: add UMR and RDMA write WQE defines

 drivers/common/mlx5/linux/mlx5_common_os.c   |  30 +-
 drivers/common/mlx5/linux/mlx5_common_os.h   |   5 +
 drivers/common/mlx5/linux/mlx5_nl.c          |   2 +-
 drivers/common/mlx5/meson.build              |  15 +-
 drivers/common/mlx5/mlx5_common.c            |   2 +-
 drivers/common/mlx5/mlx5_common.h            |   1 +
 drivers/common/mlx5/mlx5_common_devx.c       |   2 +-
 drivers/common/mlx5/mlx5_common_log.h        |  21 +
 drivers/common/mlx5/mlx5_common_mp.c         |   2 +-
 drivers/common/mlx5/mlx5_common_mr.c         |   2 +-
 drivers/common/mlx5/mlx5_common_pci.c        |   4 +-
 drivers/common/mlx5/mlx5_common_utils.c      | 221 +++++++
 drivers/common/mlx5/mlx5_common_utils.h      | 202 +++++-
 drivers/common/mlx5/mlx5_devx_cmds.c         | 312 +++++++++-
 drivers/common/mlx5/mlx5_devx_cmds.h         |  66 ++
 drivers/common/mlx5/mlx5_malloc.c            |   2 +-
 drivers/common/mlx5/mlx5_prm.h               | 610 +++++++++++++------
 drivers/common/mlx5/version.map              |  66 +-
 drivers/common/mlx5/windows/mlx5_common_os.c |   2 +-
 drivers/common/mlx5/windows/mlx5_glue.c      |   2 +-
 drivers/compress/mlx5/mlx5_compress.c        |  30 +-
 drivers/net/mlx5/mlx5_utils.c                | 209 -------
 drivers/net/mlx5/mlx5_utils.h                | 194 +-----
 drivers/regex/mlx5/mlx5_regex.c              |  30 +-
 drivers/vdpa/mlx5/mlx5_vdpa.c                |  32 +-
 25 files changed, 1337 insertions(+), 727 deletions(-)
 create mode 100644 drivers/common/mlx5/mlx5_common_log.h
 create mode 100644 drivers/common/mlx5/mlx5_common_utils.c

-- 
2.25.1

[dpdk-dev] [PATCH v3 01/15] common/mlx5: remove redundant spaces in header file

[Matan Azrad]

From: Dekel Peled <dekelp@nvidia.com>

File drivers/common/mlx5/mlx5_prm.h includes structs representing
data items as defined in PRM document.
Some of these structs were copied as-is from kernel file mlx5_ifc.h.
As result the structs are not all aligned with the same spacing.

This patch removes redundant spaces and new lines from several structs,
to align all structs in mlx5_prm.h to the same format.

Signed-off-by: Dekel Peled <dekelp@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/mlx5_prm.h | 334 +++++++++++++++------------------
 1 file changed, 155 insertions(+), 179 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h
index efa5ae67bf..da1510ac1e 100644
--- a/drivers/common/mlx5/mlx5_prm.h
+++ b/drivers/common/mlx5/mlx5_prm.h
@@ -953,163 +953,139 @@ enum {
 
 /* Flow counters. */
 struct mlx5_ifc_alloc_flow_counter_out_bits {
-	u8         status[0x8];
-	u8         reserved_at_8[0x18];
-	u8         syndrome[0x20];
-	u8         flow_counter_id[0x20];
-	u8         reserved_at_60[0x20];
+	u8 status[0x8];
+	u8 reserved_at_8[0x18];
+	u8 syndrome[0x20];
+	u8 flow_counter_id[0x20];
+	u8 reserved_at_60[0x20];
 };
 
 struct mlx5_ifc_alloc_flow_counter_in_bits {
-	u8         opcode[0x10];
-	u8         reserved_at_10[0x10];
-	u8         reserved_at_20[0x10];
-	u8         op_mod[0x10];
-	u8         flow_counter_id[0x20];
-	u8         reserved_at_40[0x18];
-	u8         flow_counter_bulk[0x8];
+	u8 opcode[0x10];
+	u8 reserved_at_10[0x10];
+	u8 reserved_at_20[0x10];
+	u8 op_mod[0x10];
+	u8 flow_counter_id[0x20];
+	u8 reserved_at_40[0x18];
+	u8 flow_counter_bulk[0x8];
 };
 
 struct mlx5_ifc_dealloc_flow_counter_out_bits {
-	u8         status[0x8];
-	u8         reserved_at_8[0x18];
-	u8         syndrome[0x20];
-	u8         reserved_at_40[0x40];
+	u8 status[0x8];
+	u8 reserved_at_8[0x18];
+	u8 syndrome[0x20];
+	u8 reserved_at_40[0x40];
 };
 
 struct mlx5_ifc_dealloc_flow_counter_in_bits {
-	u8         opcode[0x10];
-	u8         reserved_at_10[0x10];
-	u8         reserved_at_20[0x10];
-	u8         op_mod[0x10];
-	u8         flow_counter_id[0x20];
-	u8         reserved_at_60[0x20];
+	u8 opcode[0x10];
+	u8 reserved_at_10[0x10];
+	u8 reserved_at_20[0x10];
+	u8 op_mod[0x10];
+	u8 flow_counter_id[0x20];
+	u8 reserved_at_60[0x20];
 };
 
 struct mlx5_ifc_traffic_counter_bits {
-	u8         packets[0x40];
-	u8         octets[0x40];
+	u8 packets[0x40];
+	u8 octets[0x40];
 };
 
 struct mlx5_ifc_query_flow_counter_out_bits {
-	u8         status[0x8];
-	u8         reserved_at_8[0x18];
-	u8         syndrome[0x20];
-	u8         reserved_at_40[0x40];
+	u8 status[0x8];
+	u8 reserved_at_8[0x18];
+	u8 syndrome[0x20];
+	u8 reserved_at_40[0x40];
 	struct mlx5_ifc_traffic_counter_bits flow_statistics[];
 };
 
 struct mlx5_ifc_query_flow_counter_in_bits {
-	u8         opcode[0x10];
-	u8         reserved_at_10[0x10];
-	u8         reserved_at_20[0x10];
-	u8         op_mod[0x10];
-	u8         reserved_at_40[0x20];
-	u8         mkey[0x20];
-	u8         address[0x40];
-	u8         clear[0x1];
-	u8         dump_to_memory[0x1];
-	u8         num_of_counters[0x1e];
-	u8         flow_counter_id[0x20];
+	u8 opcode[0x10];
+	u8 reserved_at_10[0x10];
+	u8 reserved_at_20[0x10];
+	u8 op_mod[0x10];
+	u8 reserved_at_40[0x20];
+	u8 mkey[0x20];
+	u8 address[0x40];
+	u8 clear[0x1];
+	u8 dump_to_memory[0x1];
+	u8 num_of_counters[0x1e];
+	u8 flow_counter_id[0x20];
 };
 
 #define MLX5_MAX_KLM_BYTE_COUNT 0x80000000u
 #define MLX5_MIN_KLM_FIXED_BUFFER_SIZE 0x1000u
 
-
 struct mlx5_ifc_klm_bits {
-	u8         byte_count[0x20];
-	u8         mkey[0x20];
-	u8         address[0x40];
+	u8 byte_count[0x20];
+	u8 mkey[0x20];
+	u8 address[0x40];
 };
 
 struct mlx5_ifc_mkc_bits {
-	u8         reserved_at_0[0x1];
-	u8         free[0x1];
-	u8         reserved_at_2[0x1];
-	u8         access_mode_4_2[0x3];
-	u8         reserved_at_6[0x7];
-	u8         relaxed_ordering_write[0x1];
-	u8         reserved_at_e[0x1];
-	u8         small_fence_on_rdma_read_response[0x1];
-	u8         umr_en[0x1];
-	u8         a[0x1];
-	u8         rw[0x1];
-	u8         rr[0x1];
-	u8         lw[0x1];
-	u8         lr[0x1];
-	u8         access_mode_1_0[0x2];
-	u8         reserved_at_18[0x8];
-
-	u8         qpn[0x18];
-	u8         mkey_7_0[0x8];
-
-	u8         reserved_at_40[0x20];
-
-	u8         length64[0x1];
-	u8         bsf_en[0x1];
-	u8         sync_umr[0x1];
-	u8         reserved_at_63[0x2];
-	u8         expected_sigerr_count[0x1];
-	u8         reserved_at_66[0x1];
-	u8         en_rinval[0x1];
-	u8         pd[0x18];
-
-	u8         start_addr[0x40];
-
-	u8         len[0x40];
-
-	u8         bsf_octword_size[0x20];
-
-	u8         reserved_at_120[0x80];
-
-	u8         translations_octword_size[0x20];
-
-	u8         reserved_at_1c0[0x19];
-	u8		   relaxed_ordering_read[0x1];
-	u8		   reserved_at_1da[0x1];
-	u8         log_page_size[0x5];
-
-	u8         reserved_at_1e0[0x20];
+	u8 reserved_at_0[0x1];
+	u8 free[0x1];
+	u8 reserved_at_2[0x1];
+	u8 access_mode_4_2[0x3];
+	u8 reserved_at_6[0x7];
+	u8 relaxed_ordering_write[0x1];
+	u8 reserved_at_e[0x1];
+	u8 small_fence_on_rdma_read_response[0x1];
+	u8 umr_en[0x1];
+	u8 a[0x1];
+	u8 rw[0x1];
+	u8 rr[0x1];
+	u8 lw[0x1];
+	u8 lr[0x1];
+	u8 access_mode_1_0[0x2];
+	u8 reserved_at_18[0x8];
+	u8 qpn[0x18];
+	u8 mkey_7_0[0x8];
+	u8 reserved_at_40[0x20];
+	u8 length64[0x1];
+	u8 bsf_en[0x1];
+	u8 sync_umr[0x1];
+	u8 reserved_at_63[0x2];
+	u8 expected_sigerr_count[0x1];
+	u8 reserved_at_66[0x1];
+	u8 en_rinval[0x1];
+	u8 pd[0x18];
+	u8 start_addr[0x40];
+	u8 len[0x40];
+	u8 bsf_octword_size[0x20];
+	u8 reserved_at_120[0x80];
+	u8 translations_octword_size[0x20];
+	u8 reserved_at_1c0[0x19];
+	u8 relaxed_ordering_read[0x1];
+	u8 reserved_at_1da[0x1];
+	u8 log_page_size[0x5];
+	u8 reserved_at_1e0[0x20];
 };
 
 struct mlx5_ifc_create_mkey_out_bits {
-	u8         status[0x8];
-	u8         reserved_at_8[0x18];
-
-	u8         syndrome[0x20];
-
-	u8         reserved_at_40[0x8];
-	u8         mkey_index[0x18];
-
-	u8         reserved_at_60[0x20];
+	u8 status[0x8];
+	u8 reserved_at_8[0x18];
+	u8 syndrome[0x20];
+	u8 reserved_at_40[0x8];
+	u8 mkey_index[0x18];
+	u8 reserved_at_60[0x20];
 };
 
 struct mlx5_ifc_create_mkey_in_bits {
-	u8         opcode[0x10];
-	u8         reserved_at_10[0x10];
-
-	u8         reserved_at_20[0x10];
-	u8         op_mod[0x10];
-
-	u8         reserved_at_40[0x20];
-
-	u8         pg_access[0x1];
-	u8         reserved_at_61[0x1f];
-
+	u8 opcode[0x10];
+	u8 reserved_at_10[0x10];
+	u8 reserved_at_20[0x10];
+	u8 op_mod[0x10];
+	u8 reserved_at_40[0x20];
+	u8 pg_access[0x1];
+	u8 reserved_at_61[0x1f];
 	struct mlx5_ifc_mkc_bits memory_key_mkey_entry;
-
-	u8         reserved_at_280[0x80];
-
-	u8         translations_octword_actual_size[0x20];
-
-	u8         mkey_umem_id[0x20];
-
-	u8         mkey_umem_offset[0x40];
-
-	u8         reserved_at_380[0x500];
-
-	u8         klm_pas_mtt[][0x20];
+	u8 reserved_at_280[0x80];
+	u8 translations_octword_actual_size[0x20];
+	u8 mkey_umem_id[0x20];
+	u8 mkey_umem_offset[0x40];
+	u8 reserved_at_380[0x500];
+	u8 klm_pas_mtt[][0x20];
 };
 
 enum {
@@ -2272,27 +2248,27 @@ enum {
 };
 
 struct mlx5_ifc_flow_meter_parameters_bits {
-	u8         valid[0x1];			// 00h
-	u8         bucket_overflow[0x1];
-	u8         start_color[0x2];
-	u8         both_buckets_on_green[0x1];
-	u8         meter_mode[0x2];
-	u8         reserved_at_1[0x19];
-	u8         reserved_at_2[0x20]; //04h
-	u8         reserved_at_3[0x3];
-	u8         cbs_exponent[0x5];		// 08h
-	u8         cbs_mantissa[0x8];
-	u8         reserved_at_4[0x3];
-	u8         cir_exponent[0x5];
-	u8         cir_mantissa[0x8];
-	u8         reserved_at_5[0x20];		// 0Ch
-	u8         reserved_at_6[0x3];
-	u8         ebs_exponent[0x5];		// 10h
-	u8         ebs_mantissa[0x8];
-	u8         reserved_at_7[0x3];
-	u8         eir_exponent[0x5];
-	u8         eir_mantissa[0x8];
-	u8         reserved_at_8[0x60];		// 14h-1Ch
+	u8 valid[0x1];
+	u8 bucket_overflow[0x1];
+	u8 start_color[0x2];
+	u8 both_buckets_on_green[0x1];
+	u8 meter_mode[0x2];
+	u8 reserved_at_1[0x19];
+	u8 reserved_at_2[0x20];
+	u8 reserved_at_3[0x3];
+	u8 cbs_exponent[0x5];
+	u8 cbs_mantissa[0x8];
+	u8 reserved_at_4[0x3];
+	u8 cir_exponent[0x5];
+	u8 cir_mantissa[0x8];
+	u8 reserved_at_5[0x20];
+	u8 reserved_at_6[0x3];
+	u8 ebs_exponent[0x5];
+	u8 ebs_mantissa[0x8];
+	u8 reserved_at_7[0x3];
+	u8 eir_exponent[0x5];
+	u8 eir_mantissa[0x8];
+	u8 reserved_at_8[0x60];
 };
 #define MLX5_IFC_FLOW_METER_PARAM_MASK UINT64_C(0x80FFFFFF)
 #define MLX5_IFC_FLOW_METER_DISABLE_CBS_CIR_VAL 0x14BF00C8
@@ -2359,46 +2335,46 @@ struct mlx5_ifc_cqc_bits {
 };
 
 struct mlx5_ifc_health_buffer_bits {
-	u8         reserved_0[0x100];
-	u8         assert_existptr[0x20];
-	u8         assert_callra[0x20];
-	u8         reserved_1[0x40];
-	u8         fw_version[0x20];
-	u8         hw_id[0x20];
-	u8         reserved_2[0x20];
-	u8         irisc_index[0x8];
-	u8         synd[0x8];
-	u8         ext_synd[0x10];
+	u8 reserved_0[0x100];
+	u8 assert_existptr[0x20];
+	u8 assert_callra[0x20];
+	u8 reserved_1[0x40];
+	u8 fw_version[0x20];
+	u8 hw_id[0x20];
+	u8 reserved_2[0x20];
+	u8 irisc_index[0x8];
+	u8 synd[0x8];
+	u8 ext_synd[0x10];
 };
 
 struct mlx5_ifc_initial_seg_bits {
-	u8         fw_rev_minor[0x10];
-	u8         fw_rev_major[0x10];
-	u8         cmd_interface_rev[0x10];
-	u8         fw_rev_subminor[0x10];
-	u8         reserved_0[0x40];
-	u8         cmdq_phy_addr_63_32[0x20];
-	u8         cmdq_phy_addr_31_12[0x14];
-	u8         reserved_1[0x2];
-	u8         nic_interface[0x2];
-	u8         log_cmdq_size[0x4];
-	u8         log_cmdq_stride[0x4];
-	u8         command_doorbell_vector[0x20];
-	u8         reserved_2[0xf00];
-	u8         initializing[0x1];
-	u8         nic_interface_supported[0x7];
-	u8         reserved_4[0x18];
+	u8 fw_rev_minor[0x10];
+	u8 fw_rev_major[0x10];
+	u8 cmd_interface_rev[0x10];
+	u8 fw_rev_subminor[0x10];
+	u8 reserved_0[0x40];
+	u8 cmdq_phy_addr_63_32[0x20];
+	u8 cmdq_phy_addr_31_12[0x14];
+	u8 reserved_1[0x2];
+	u8 nic_interface[0x2];
+	u8 log_cmdq_size[0x4];
+	u8 log_cmdq_stride[0x4];
+	u8 command_doorbell_vector[0x20];
+	u8 reserved_2[0xf00];
+	u8 initializing[0x1];
+	u8 nic_interface_supported[0x7];
+	u8 reserved_4[0x18];
 	struct mlx5_ifc_health_buffer_bits health_buffer;
-	u8         no_dram_nic_offset[0x20];
-	u8         reserved_5[0x6de0];
-	u8         internal_timer_h[0x20];
-	u8         internal_timer_l[0x20];
-	u8         reserved_6[0x20];
-	u8         reserved_7[0x1f];
-	u8         clear_int[0x1];
-	u8         health_syndrome[0x8];
-	u8         health_counter[0x18];
-	u8         reserved_8[0x17fc0];
+	u8 no_dram_nic_offset[0x20];
+	u8 reserved_5[0x6de0];
+	u8 internal_timer_h[0x20];
+	u8 internal_timer_l[0x20];
+	u8 reserved_6[0x20];
+	u8 reserved_7[0x1f];
+	u8 clear_int[0x1];
+	u8 health_syndrome[0x8];
+	u8 health_counter[0x18];
+	u8 reserved_8[0x17fc0];
 };
 
 struct mlx5_ifc_create_cq_out_bits {
-- 
2.25.1

[dpdk-dev] [PATCH v3 02/15] common/mlx5: update GENEVE TLV OPT obj name

[Matan Azrad]

From: Dekel Peled <dekelp@nvidia.com>

Rename MLX5_OBJ_TYPE_GENEVE_TLV_OPT as
MLX5_GENERAL_OBJ_TYPE_GENEVE_TLV_OPT, to align with other general
objects names.

Signed-off-by: Dekel Peled <dekelp@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/mlx5_devx_cmds.c | 2 +-
 drivers/common/mlx5/mlx5_prm.h       | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c
index 79fff6457c..831175efc5 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.c
+++ b/drivers/common/mlx5/mlx5_devx_cmds.c
@@ -2265,7 +2265,7 @@ mlx5_devx_cmd_create_geneve_tlv_option(void *ctx,
 	MLX5_SET(general_obj_in_cmd_hdr, hdr, opcode,
 			MLX5_CMD_OP_CREATE_GENERAL_OBJECT);
 	MLX5_SET(general_obj_in_cmd_hdr, hdr, obj_type,
-			MLX5_OBJ_TYPE_GENEVE_TLV_OPT);
+		 MLX5_GENERAL_OBJ_TYPE_GENEVE_TLV_OPT);
 	MLX5_SET(geneve_tlv_option, opt, option_class,
 			rte_be_to_cpu_16(class));
 	MLX5_SET(geneve_tlv_option, opt, option_type, type);
diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h
index da1510ac1e..2e5e42f6e9 100644
--- a/drivers/common/mlx5/mlx5_prm.h
+++ b/drivers/common/mlx5/mlx5_prm.h
@@ -1108,7 +1108,7 @@ enum {
 #define MLX5_GENERAL_OBJ_TYPES_CAP_FLOW_METER_ASO \
 			(1ULL << MLX5_GENERAL_OBJ_TYPE_FLOW_METER_ASO)
 #define MLX5_GENERAL_OBJ_TYPES_CAP_GENEVE_TLV_OPT \
-			(1ULL << MLX5_OBJ_TYPE_GENEVE_TLV_OPT)
+			(1ULL << MLX5_GENERAL_OBJ_TYPE_GENEVE_TLV_OPT)
 
 enum {
 	MLX5_HCA_CAP_OPMOD_GET_MAX   = 0,
@@ -2402,7 +2402,7 @@ struct mlx5_ifc_create_cq_in_bits {
 };
 
 enum {
-	MLX5_OBJ_TYPE_GENEVE_TLV_OPT = 0x000b,
+	MLX5_GENERAL_OBJ_TYPE_GENEVE_TLV_OPT = 0x000b,
 	MLX5_GENERAL_OBJ_TYPE_VIRTQ = 0x000d,
 	MLX5_GENERAL_OBJ_TYPE_VIRTIO_Q_COUNTERS = 0x001c,
 	MLX5_GENERAL_OBJ_TYPE_FLEX_PARSE_GRAPH = 0x0022,
-- 
2.25.1

[dpdk-dev] [PATCH v3 03/15] common/mlx5: optimize read of general obj type caps

[Matan Azrad]

From: Dekel Peled <dekelp@nvidia.com>

General object types support is indicated in bitmap general_obj_types,
which is part of HCA capabilities list.
Currently this bitmap is read multiple times, and each time a different
bit is extracted.

This patch optimizes the code, reading the bitmap once into a local
variable, and then extracting the required bits.

Signed-off-by: Dekel Peled <dekelp@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/mlx5_devx_cmds.c | 21 ++++++++++++++++-----
 1 file changed, 16 insertions(+), 5 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c
index 831175efc5..a0bf0d3009 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.c
+++ b/drivers/common/mlx5/mlx5_devx_cmds.c
@@ -647,6 +647,7 @@ mlx5_devx_cmd_query_hca_attr(void *ctx,
 	uint32_t out[MLX5_ST_SZ_DW(query_hca_cap_out)] = {0};
 	void *hcattr;
 	int status, syndrome, rc, i;
+	uint64_t general_obj_types_supported = 0;
 
 	MLX5_SET(query_hca_cap_in, in, opcode, MLX5_CMD_OP_QUERY_HCA_CAP);
 	MLX5_SET(query_hca_cap_in, in, op_mod,
@@ -725,12 +726,22 @@ mlx5_devx_cmd_query_hca_attr(void *ctx,
 	attr->regex = MLX5_GET(cmd_hca_cap, hcattr, regexp);
 	attr->regexp_num_of_engines = MLX5_GET(cmd_hca_cap, hcattr,
 					       regexp_num_of_engines);
-	attr->flow_hit_aso = !!(MLX5_GET64(cmd_hca_cap, hcattr,
-					   general_obj_types) &
+	/* Read the general_obj_types bitmap and extract the relevant bits. */
+	general_obj_types_supported = MLX5_GET64(cmd_hca_cap, hcattr,
+						 general_obj_types);
+	attr->vdpa.valid = !!(general_obj_types_supported &
+			      MLX5_GENERAL_OBJ_TYPES_CAP_VIRTQ_NET_Q);
+	attr->vdpa.queue_counters_valid =
+			!!(general_obj_types_supported &
+			   MLX5_GENERAL_OBJ_TYPES_CAP_VIRTIO_Q_COUNTERS);
+	attr->parse_graph_flex_node =
+			!!(general_obj_types_supported &
+			   MLX5_GENERAL_OBJ_TYPES_CAP_PARSE_GRAPH_FLEX_NODE);
+	attr->flow_hit_aso = !!(general_obj_types_supported &
 				MLX5_GENERAL_OBJ_TYPES_CAP_FLOW_HIT_ASO);
-	attr->geneve_tlv_opt = !!(MLX5_GET64(cmd_hca_cap, hcattr,
-					   general_obj_types) &
-				MLX5_GENERAL_OBJ_TYPES_CAP_GENEVE_TLV_OPT);
+	attr->geneve_tlv_opt = !!(general_obj_types_supported &
+				  MLX5_GENERAL_OBJ_TYPES_CAP_GENEVE_TLV_OPT);
+	/* Add reading of other GENERAL_OBJ_TYPES_CAP bits above this line. */
 	attr->log_max_cq = MLX5_GET(cmd_hca_cap, hcattr, log_max_cq);
 	attr->log_max_qp = MLX5_GET(cmd_hca_cap, hcattr, log_max_qp);
 	attr->log_max_cq_sz = MLX5_GET(cmd_hca_cap, hcattr, log_max_cq_sz);
-- 
2.25.1

[dpdk-dev] [PATCH v3 04/15] common/mlx5: add HCA cap for AES-XTS crypto

[Matan Azrad]

From: Dekel Peled <dekelp@nvidia.com>

Update the PRM structure and HCA capabilities reading, to include
relevant capabilities for AES-XTS crypto.

Signed-off-by: Dekel Peled <dekelp@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/mlx5_devx_cmds.c | 3 +++
 drivers/common/mlx5/mlx5_devx_cmds.h | 2 ++
 drivers/common/mlx5/mlx5_prm.h       | 5 ++++-
 3 files changed, 9 insertions(+), 1 deletion(-)

diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c
index a0bf0d3009..7ca767944e 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.c
+++ b/drivers/common/mlx5/mlx5_devx_cmds.c
@@ -771,6 +771,9 @@ mlx5_devx_cmd_query_hca_attr(void *ctx,
 		MLX5_GET(cmd_hca_cap, hcattr, umr_indirect_mkey_disabled);
 	attr->umr_modify_entity_size_disabled =
 		MLX5_GET(cmd_hca_cap, hcattr, umr_modify_entity_size_disabled);
+	attr->crypto = MLX5_GET(cmd_hca_cap, hcattr, crypto);
+	if (attr->crypto)
+		attr->aes_xts = MLX5_GET(cmd_hca_cap, hcattr, aes_xts);
 	if (attr->qos.sup) {
 		MLX5_SET(query_hca_cap_in, in, op_mod,
 			 MLX5_GET_HCA_CAP_OP_MOD_QOS_CAP |
diff --git a/drivers/common/mlx5/mlx5_devx_cmds.h b/drivers/common/mlx5/mlx5_devx_cmds.h
index 870bdb6b30..28ade5bbc4 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.h
+++ b/drivers/common/mlx5/mlx5_devx_cmds.h
@@ -137,6 +137,8 @@ struct mlx5_hca_attr {
 	uint32_t qp_ts_format:2;
 	uint32_t regex:1;
 	uint32_t reg_c_preserve:1;
+	uint32_t crypto:1; /* Crypto engine is supported. */
+	uint32_t aes_xts:1; /* AES-XTS crypto is supported. */
 	uint32_t regexp_num_of_engines;
 	uint32_t log_max_ft_sampler_num:8;
 	uint32_t geneve_tlv_opt;
diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h
index 2e5e42f6e9..a8fbfbb0f5 100644
--- a/drivers/common/mlx5/mlx5_prm.h
+++ b/drivers/common/mlx5/mlx5_prm.h
@@ -1427,7 +1427,10 @@ struct mlx5_ifc_cmd_hca_cap_bits {
 	u8 sq_ts_format[0x2];
 	u8 rq_ts_format[0x2];
 	u8 reserved_at_444[0x1C];
-	u8 reserved_at_460[0x10];
+	u8 reserved_at_460[0x8];
+	u8 aes_xts[0x1];
+	u8 crypto[0x1];
+	u8 reserved_at_46a[0x6];
 	u8 max_num_eqs[0x10];
 	u8 reserved_at_480[0x3];
 	u8 log_max_l2_table[0x5];
-- 
2.25.1

[dpdk-dev] [PATCH v3 05/15] common/mlx5: support general object DEK create op

[Matan Azrad]

From: Dekel Peled <dekelp@nvidia.com>

Data Encryption Keys (DEKs) are the keys used for data
encryption/decryption operations.

Add reading of DEK support capability.
Add function to create general object type DEK, using DevX API.

Arrange common version.map file in alphabetical order.

Signed-off-by: Dekel Peled <dekelp@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/mlx5_devx_cmds.c | 53 +++++++++++++++++++++++++++
 drivers/common/mlx5/mlx5_devx_cmds.h | 17 +++++++++
 drivers/common/mlx5/mlx5_prm.h       | 39 ++++++++++++++++++++
 drivers/common/mlx5/version.map      | 55 ++++++++++++++++------------
 4 files changed, 140 insertions(+), 24 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c
index 7ca767944e..742c82cca4 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.c
+++ b/drivers/common/mlx5/mlx5_devx_cmds.c
@@ -741,6 +741,8 @@ mlx5_devx_cmd_query_hca_attr(void *ctx,
 				MLX5_GENERAL_OBJ_TYPES_CAP_FLOW_HIT_ASO);
 	attr->geneve_tlv_opt = !!(general_obj_types_supported &
 				  MLX5_GENERAL_OBJ_TYPES_CAP_GENEVE_TLV_OPT);
+	attr->dek = !!(general_obj_types_supported &
+		       MLX5_GENERAL_OBJ_TYPES_CAP_DEK);
 	/* Add reading of other GENERAL_OBJ_TYPES_CAP bits above this line. */
 	attr->log_max_cq = MLX5_GET(cmd_hca_cap, hcattr, log_max_cq);
 	attr->log_max_qp = MLX5_GET(cmd_hca_cap, hcattr, log_max_qp);
@@ -2397,3 +2399,54 @@ mlx5_devx_cmd_queue_counter_query(struct mlx5_devx_obj *dcs, int clear,
 	*out_of_buffers = MLX5_GET(query_q_counter_out, out, out_of_buffer);
 	return 0;
 }
+
+/**
+ * Create general object of type DEK using DevX API.
+ *
+ * @param[in] ctx
+ *   Context returned from mlx5 open_device() glue function.
+ * @param [in] attr
+ *   Pointer to DEK attributes structure.
+ *
+ * @return
+ *   The DevX object created, NULL otherwise and rte_errno is set.
+ */
+struct mlx5_devx_obj *
+mlx5_devx_cmd_create_dek_obj(void *ctx, struct mlx5_devx_dek_attr *attr)
+{
+	uint32_t in[MLX5_ST_SZ_DW(create_dek_in)] = {0};
+	uint32_t out[MLX5_ST_SZ_DW(general_obj_out_cmd_hdr)] = {0};
+	struct mlx5_devx_obj *dek_obj = NULL;
+	void *ptr = NULL, *key_addr = NULL;
+
+	dek_obj = mlx5_malloc(MLX5_MEM_ZERO, sizeof(*dek_obj),
+			      0, SOCKET_ID_ANY);
+	if (dek_obj == NULL) {
+		DRV_LOG(ERR, "Failed to allocate DEK object data");
+		rte_errno = ENOMEM;
+		return NULL;
+	}
+	ptr = MLX5_ADDR_OF(create_dek_in, in, hdr);
+	MLX5_SET(general_obj_in_cmd_hdr, ptr, opcode,
+		 MLX5_CMD_OP_CREATE_GENERAL_OBJECT);
+	MLX5_SET(general_obj_in_cmd_hdr, ptr, obj_type,
+		 MLX5_GENERAL_OBJ_TYPE_DEK);
+	ptr = MLX5_ADDR_OF(create_dek_in, in, dek);
+	MLX5_SET(dek, ptr, key_size, attr->key_size);
+	MLX5_SET(dek, ptr, has_keytag, attr->has_keytag);
+	MLX5_SET(dek, ptr, key_purpose, attr->key_purpose);
+	MLX5_SET(dek, ptr, pd, attr->pd);
+	MLX5_SET64(dek, ptr, opaque, attr->opaque);
+	key_addr = MLX5_ADDR_OF(dek, ptr, key);
+	memcpy(key_addr, (void *)(attr->key), MLX5_CRYPTO_KEY_MAX_SIZE);
+	dek_obj->obj = mlx5_glue->devx_obj_create(ctx, in, sizeof(in),
+						  out, sizeof(out));
+	if (dek_obj->obj == NULL) {
+		rte_errno = errno;
+		DRV_LOG(ERR, "Failed to create DEK obj using DevX.");
+		mlx5_free(dek_obj);
+		return NULL;
+	}
+	dek_obj->id = MLX5_GET(general_obj_out_cmd_hdr, out, obj_id);
+	return dek_obj;
+}
diff --git a/drivers/common/mlx5/mlx5_devx_cmds.h b/drivers/common/mlx5/mlx5_devx_cmds.h
index 28ade5bbc4..b9ff7ab87d 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.h
+++ b/drivers/common/mlx5/mlx5_devx_cmds.h
@@ -139,6 +139,7 @@ struct mlx5_hca_attr {
 	uint32_t reg_c_preserve:1;
 	uint32_t crypto:1; /* Crypto engine is supported. */
 	uint32_t aes_xts:1; /* AES-XTS crypto is supported. */
+	uint32_t dek:1; /* General obj type DEK is supported. */
 	uint32_t regexp_num_of_engines;
 	uint32_t log_max_ft_sampler_num:8;
 	uint32_t geneve_tlv_opt;
@@ -435,6 +436,18 @@ struct mlx5_devx_graph_node_attr {
 	struct mlx5_devx_graph_arc_attr out[MLX5_GRAPH_NODE_ARC_NUM];
 };
 
+/* Encryption key size is up to 1024 bit, 128 bytes. */
+#define MLX5_CRYPTO_KEY_MAX_SIZE	128
+
+struct mlx5_devx_dek_attr {
+	uint32_t key_size:4;
+	uint32_t has_keytag:1;
+	uint32_t key_purpose:4;
+	uint32_t pd:24;
+	uint64_t opaque;
+	uint8_t key[MLX5_CRYPTO_KEY_MAX_SIZE];
+};
+
 /* mlx5_devx_cmds.c */
 
 __rte_internal
@@ -587,4 +600,8 @@ int mlx5_devx_cmd_queue_counter_query(struct mlx5_devx_obj *dcs, int clear,
 __rte_internal
 struct mlx5_devx_obj *mlx5_devx_cmd_create_flow_meter_aso_obj(void *ctx,
 					uint32_t pd, uint32_t log_obj_size);
+__rte_internal
+struct mlx5_devx_obj *
+mlx5_devx_cmd_create_dek_obj(void *ctx, struct mlx5_devx_dek_attr *attr);
+
 #endif /* RTE_PMD_MLX5_DEVX_CMDS_H_ */
diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h
index a8fbfbb0f5..bc9f58ad03 100644
--- a/drivers/common/mlx5/mlx5_prm.h
+++ b/drivers/common/mlx5/mlx5_prm.h
@@ -1109,6 +1109,8 @@ enum {
 			(1ULL << MLX5_GENERAL_OBJ_TYPE_FLOW_METER_ASO)
 #define MLX5_GENERAL_OBJ_TYPES_CAP_GENEVE_TLV_OPT \
 			(1ULL << MLX5_GENERAL_OBJ_TYPE_GENEVE_TLV_OPT)
+#define MLX5_GENERAL_OBJ_TYPES_CAP_DEK \
+			(1ULL << MLX5_GENERAL_OBJ_TYPE_DEK)
 
 enum {
 	MLX5_HCA_CAP_OPMOD_GET_MAX   = 0,
@@ -2406,6 +2408,7 @@ struct mlx5_ifc_create_cq_in_bits {
 
 enum {
 	MLX5_GENERAL_OBJ_TYPE_GENEVE_TLV_OPT = 0x000b,
+	MLX5_GENERAL_OBJ_TYPE_DEK = 0x000c,
 	MLX5_GENERAL_OBJ_TYPE_VIRTQ = 0x000d,
 	MLX5_GENERAL_OBJ_TYPE_VIRTIO_Q_COUNTERS = 0x001c,
 	MLX5_GENERAL_OBJ_TYPE_FLEX_PARSE_GRAPH = 0x0022,
@@ -2469,6 +2472,42 @@ struct mlx5_ifc_create_geneve_tlv_option_in_bits {
 	struct mlx5_ifc_geneve_tlv_option_bits geneve_tlv_opt;
 };
 
+enum {
+	MLX5_CRYPTO_KEY_SIZE_128b = 0x0,
+	MLX5_CRYPTO_KEY_SIZE_256b = 0x1,
+};
+
+enum {
+	MLX5_CRYPTO_KEY_PURPOSE_TLS	= 0x1,
+	MLX5_CRYPTO_KEY_PURPOSE_IPSEC	= 0x2,
+	MLX5_CRYPTO_KEY_PURPOSE_AES_XTS	= 0x3,
+	MLX5_CRYPTO_KEY_PURPOSE_MACSEC	= 0x4,
+	MLX5_CRYPTO_KEY_PURPOSE_GCM	= 0x5,
+	MLX5_CRYPTO_KEY_PURPOSE_PSP	= 0x6,
+};
+
+struct mlx5_ifc_dek_bits {
+	u8 modify_field_select[0x40];
+	u8 state[0x8];
+	u8 reserved_at_48[0xc];
+	u8 key_size[0x4];
+	u8 has_keytag[0x1];
+	u8 reserved_at_59[0x3];
+	u8 key_purpose[0x4];
+	u8 reserved_at_60[0x8];
+	u8 pd[0x18];
+	u8 reserved_at_80[0x100];
+	u8 opaque[0x40];
+	u8 reserved_at_1c0[0x40];
+	u8 key[0x400];
+	u8 reserved_at_600[0x200];
+};
+
+struct mlx5_ifc_create_dek_in_bits {
+	struct mlx5_ifc_general_obj_in_cmd_hdr_bits hdr;
+	struct mlx5_ifc_dek_bits dek;
+};
+
 enum {
 	MLX5_VIRTQ_STATE_INIT = 0,
 	MLX5_VIRTQ_STATE_RDY = 1,
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index 18dc96276d..42bb985fb1 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -12,21 +12,24 @@ INTERNAL {
 
 	mlx5_dev_to_pci_addr; # WINDOWS_NO_EXPORT
 
+	mlx5_devx_alloc_uar; # WINDOWS_NO_EXPORT
+
 	mlx5_devx_cmd_alloc_pd;
 	mlx5_devx_cmd_create_cq;
+	mlx5_devx_cmd_create_dek_obj;
 	mlx5_devx_cmd_create_flex_parser;
+	mlx5_devx_cmd_create_flow_hit_aso_obj;
+	mlx5_devx_cmd_create_flow_meter_aso_obj;
+	mlx5_devx_cmd_create_geneve_tlv_option;
 	mlx5_devx_cmd_create_qp;
 	mlx5_devx_cmd_create_rq;
 	mlx5_devx_cmd_create_rqt;
 	mlx5_devx_cmd_create_sq;
-	mlx5_devx_cmd_create_tir;
 	mlx5_devx_cmd_create_td;
+	mlx5_devx_cmd_create_tir;
 	mlx5_devx_cmd_create_tis;
 	mlx5_devx_cmd_create_virtio_q_counters; # WINDOWS_NO_EXPORT
 	mlx5_devx_cmd_create_virtq;
-	mlx5_devx_cmd_create_flow_hit_aso_obj;
-	mlx5_devx_cmd_create_flow_meter_aso_obj;
-	mlx5_devx_cmd_create_geneve_tlv_option;
 	mlx5_devx_cmd_destroy;
 	mlx5_devx_cmd_flow_counter_alloc;
 	mlx5_devx_cmd_flow_counter_query;
@@ -48,41 +51,49 @@ INTERNAL {
 	mlx5_devx_cmd_queue_counter_query; # WINDOWS_NO_EXPORT
 	mlx5_devx_cmd_register_read;
 	mlx5_devx_cmd_wq_query; # WINDOWS_NO_EXPORT
-	mlx5_devx_get_out_command_status;
-	mlx5_devx_alloc_uar; # WINDOWS_NO_EXPORT
 
 	mlx5_devx_cq_create;
 	mlx5_devx_cq_destroy;
+
+	mlx5_devx_get_out_command_status;
+
 	mlx5_devx_rq_create;
 	mlx5_devx_rq_destroy;
 	mlx5_devx_sq_create;
 	mlx5_devx_sq_destroy;
 
+	mlx5_free;
+
 	mlx5_get_ifname_sysfs; # WINDOWS_NO_EXPORT
 
 	mlx5_glue;
 
+	mlx5_malloc;
+	mlx5_malloc_mem_select;
+
+	mlx5_memory_stat_dump; # WINDOWS_NO_EXPORT
+
 	mlx5_mp_init_primary; # WINDOWS_NO_EXPORT
-	mlx5_mp_uninit_primary; # WINDOWS_NO_EXPORT
 	mlx5_mp_init_secondary; # WINDOWS_NO_EXPORT
-	mlx5_mp_uninit_secondary; # WINDOWS_NO_EXPORT
 	mlx5_mp_req_mr_create; # WINDOWS_NO_EXPORT
 	mlx5_mp_req_queue_state_modify;
 	mlx5_mp_req_verbs_cmd_fd; # WINDOWS_NO_EXPORT
+	mlx5_mp_uninit_primary; # WINDOWS_NO_EXPORT
+	mlx5_mp_uninit_secondary; # WINDOWS_NO_EXPORT
 
-	mlx5_mr_btree_init;
-	mlx5_mr_btree_free;
-	mlx5_mr_btree_dump;
 	mlx5_mr_addr2mr_bh;
-	mlx5_mr_release_cache;
+	mlx5_mr_btree_dump;
+	mlx5_mr_btree_free;
+	mlx5_mr_btree_init;
+	mlx5_mr_create_primary;
 	mlx5_mr_dump_cache;
-	mlx5_mr_rebuild_cache;
+	mlx5_mr_flush_local_cache;
+	mlx5_mr_free;
 	mlx5_mr_insert_cache;
 	mlx5_mr_lookup_cache;
 	mlx5_mr_lookup_list;
-	mlx5_mr_create_primary;
-	mlx5_mr_flush_local_cache;
-	mlx5_mr_free;
+	mlx5_mr_rebuild_cache;
+	mlx5_mr_release_cache;
 
 	mlx5_nl_allmulti; # WINDOWS_NO_EXPORT
 	mlx5_nl_devlink_family_id_get; # WINDOWS_NO_EXPORT
@@ -102,20 +113,16 @@ INTERNAL {
 	mlx5_nl_vlan_vmwa_create; # WINDOWS_NO_EXPORT
 	mlx5_nl_vlan_vmwa_delete; # WINDOWS_NO_EXPORT
 
+	mlx5_pci_driver_register;
+
 	mlx5_os_alloc_pd;
 	mlx5_os_dealloc_pd;
 	mlx5_os_dereg_mr;
 	mlx5_os_reg_mr;
-	mlx5_os_umem_reg;
 	mlx5_os_umem_dereg;
+	mlx5_os_umem_reg;
 
-	mlx5_translate_port_name; # WINDOWS_NO_EXPORT
-
-	mlx5_malloc_mem_select;
-	mlx5_memory_stat_dump; # WINDOWS_NO_EXPORT
-	mlx5_malloc;
 	mlx5_realloc;
-	mlx5_free;
 
-	mlx5_pci_driver_register;
+	mlx5_translate_port_name; # WINDOWS_NO_EXPORT
 };
-- 
2.25.1

[dpdk-dev] [PATCH v3 06/15] common/mlx5: adjust DevX MKEY fields for crypto

[Matan Azrad]

From: Dekel Peled <dekelp@nvidia.com>

MKEY that will be used for crypto purposes must be created with
crypto_en and remote access attributes.
This patch adds support for them in the DevX MKEY context.

Signed-off-by: Dekel Peled <dekelp@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/mlx5_devx_cmds.c |  9 +++++++++
 drivers/common/mlx5/mlx5_devx_cmds.h |  2 ++
 drivers/common/mlx5/mlx5_prm.h       | 10 +++++++++-
 3 files changed, 20 insertions(+), 1 deletion(-)

diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c
index 742c82cca4..7e3c8b55fa 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.c
+++ b/drivers/common/mlx5/mlx5_devx_cmds.c
@@ -263,6 +263,10 @@ mlx5_devx_cmd_mkey_create(void *ctx,
 	MLX5_SET(create_mkey_in, in, pg_access, attr->pg_access);
 	MLX5_SET(mkc, mkc, lw, 0x1);
 	MLX5_SET(mkc, mkc, lr, 0x1);
+	if (attr->set_remote_rw) {
+		MLX5_SET(mkc, mkc, rw, 0x1);
+		MLX5_SET(mkc, mkc, rr, 0x1);
+	}
 	MLX5_SET(mkc, mkc, qpn, 0xffffff);
 	MLX5_SET(mkc, mkc, pd, attr->pd);
 	MLX5_SET(mkc, mkc, mkey_7_0, attr->umem_id & 0xFF);
@@ -273,6 +277,11 @@ mlx5_devx_cmd_mkey_create(void *ctx,
 	MLX5_SET(mkc, mkc, relaxed_ordering_read, attr->relaxed_ordering_read);
 	MLX5_SET64(mkc, mkc, start_addr, attr->addr);
 	MLX5_SET64(mkc, mkc, len, attr->size);
+	MLX5_SET(mkc, mkc, crypto_en, attr->crypto_en);
+	if (attr->crypto_en) {
+		MLX5_SET(mkc, mkc, bsf_en, attr->crypto_en);
+		MLX5_SET(mkc, mkc, bsf_octword_size, 4);
+	}
 	mkey->obj = mlx5_glue->devx_obj_create(ctx, in, in_size_dw * 4, out,
 					       sizeof(out));
 	if (!mkey->obj) {
diff --git a/drivers/common/mlx5/mlx5_devx_cmds.h b/drivers/common/mlx5/mlx5_devx_cmds.h
index b9ff7ab87d..600577f18a 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.h
+++ b/drivers/common/mlx5/mlx5_devx_cmds.h
@@ -32,6 +32,8 @@ struct mlx5_devx_mkey_attr {
 	uint32_t relaxed_ordering_write:1;
 	uint32_t relaxed_ordering_read:1;
 	uint32_t umr_en:1;
+	uint32_t crypto_en:2;
+	uint32_t set_remote_rw:1;
 	struct mlx5_klm *klm_array;
 	int klm_num;
 };
diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h
index bc9f58ad03..25f6f8ff00 100644
--- a/drivers/common/mlx5/mlx5_prm.h
+++ b/drivers/common/mlx5/mlx5_prm.h
@@ -1059,7 +1059,15 @@ struct mlx5_ifc_mkc_bits {
 	u8 relaxed_ordering_read[0x1];
 	u8 reserved_at_1da[0x1];
 	u8 log_page_size[0x5];
-	u8 reserved_at_1e0[0x20];
+	u8 reserved_at_1e0[0x3];
+	u8 crypto_en[0x2];
+	u8 reserved_at_1e5[0x1b];
+};
+
+/* Range of values for MKEY context crypto_en field. */
+enum {
+	MLX5_MKEY_CRYPTO_DISABLED = 0x0,
+	MLX5_MKEY_CRYPTO_ENABLED = 0x1,
 };
 
 struct mlx5_ifc_create_mkey_out_bits {
-- 
2.25.1

[dpdk-dev] [PATCH v3 07/15] common/mlx5: support general obj IMPORT KEK create

[Matan Azrad]

From: Dekel Peled <dekelp@nvidia.com>

IMPORT_KEK object is used to wrap (encrypt) critical security
parameters, such as other keys and credentials, when those need
to be passed between the device and the software.

This patch add support of IMPORT_KEK object create operation.
Add reading of IMPORT_KEK support capability.
Add function to create general object type IMPORT_KEK, using DevX API.

Signed-off-by: Dekel Peled <dekelp@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/mlx5_devx_cmds.c | 50 ++++++++++++++++++++++++++++
 drivers/common/mlx5/mlx5_devx_cmds.h | 13 ++++++++
 drivers/common/mlx5/mlx5_prm.h       | 18 ++++++++++
 drivers/common/mlx5/version.map      |  1 +
 4 files changed, 82 insertions(+)

diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c
index 7e3c8b55fa..afef7a5f63 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.c
+++ b/drivers/common/mlx5/mlx5_devx_cmds.c
@@ -752,6 +752,8 @@ mlx5_devx_cmd_query_hca_attr(void *ctx,
 				  MLX5_GENERAL_OBJ_TYPES_CAP_GENEVE_TLV_OPT);
 	attr->dek = !!(general_obj_types_supported &
 		       MLX5_GENERAL_OBJ_TYPES_CAP_DEK);
+	attr->import_kek = !!(general_obj_types_supported &
+			      MLX5_GENERAL_OBJ_TYPES_CAP_IMPORT_KEK);
 	/* Add reading of other GENERAL_OBJ_TYPES_CAP bits above this line. */
 	attr->log_max_cq = MLX5_GET(cmd_hca_cap, hcattr, log_max_cq);
 	attr->log_max_qp = MLX5_GET(cmd_hca_cap, hcattr, log_max_qp);
@@ -2459,3 +2461,51 @@ mlx5_devx_cmd_create_dek_obj(void *ctx, struct mlx5_devx_dek_attr *attr)
 	dek_obj->id = MLX5_GET(general_obj_out_cmd_hdr, out, obj_id);
 	return dek_obj;
 }
+
+/**
+ * Create general object of type IMPORT_KEK using DevX API.
+ *
+ * @param[in] ctx
+ *   Context returned from mlx5 open_device() glue function.
+ * @param [in] attr
+ *   Pointer to IMPORT_KEK attributes structure.
+ *
+ * @return
+ *   The DevX object created, NULL otherwise and rte_errno is set.
+ */
+struct mlx5_devx_obj *
+mlx5_devx_cmd_create_import_kek_obj(void *ctx,
+				    struct mlx5_devx_import_kek_attr *attr)
+{
+	uint32_t in[MLX5_ST_SZ_DW(create_import_kek_in)] = {0};
+	uint32_t out[MLX5_ST_SZ_DW(general_obj_out_cmd_hdr)] = {0};
+	struct mlx5_devx_obj *import_kek_obj = NULL;
+	void *ptr = NULL, *key_addr = NULL;
+
+	import_kek_obj = mlx5_malloc(MLX5_MEM_ZERO, sizeof(*import_kek_obj),
+				     0, SOCKET_ID_ANY);
+	if (import_kek_obj == NULL) {
+		DRV_LOG(ERR, "Failed to allocate IMPORT_KEK object data");
+		rte_errno = ENOMEM;
+		return NULL;
+	}
+	ptr = MLX5_ADDR_OF(create_import_kek_in, in, hdr);
+	MLX5_SET(general_obj_in_cmd_hdr, ptr, opcode,
+		 MLX5_CMD_OP_CREATE_GENERAL_OBJECT);
+	MLX5_SET(general_obj_in_cmd_hdr, ptr, obj_type,
+		 MLX5_GENERAL_OBJ_TYPE_IMPORT_KEK);
+	ptr = MLX5_ADDR_OF(create_import_kek_in, in, import_kek);
+	MLX5_SET(import_kek, ptr, key_size, attr->key_size);
+	key_addr = MLX5_ADDR_OF(import_kek, ptr, key);
+	memcpy(key_addr, (void *)(attr->key), MLX5_CRYPTO_KEY_MAX_SIZE);
+	import_kek_obj->obj = mlx5_glue->devx_obj_create(ctx, in, sizeof(in),
+							 out, sizeof(out));
+	if (import_kek_obj->obj == NULL) {
+		rte_errno = errno;
+		DRV_LOG(ERR, "Failed to create IMPORT_KEK object using DevX.");
+		mlx5_free(import_kek_obj);
+		return NULL;
+	}
+	import_kek_obj->id = MLX5_GET(general_obj_out_cmd_hdr, out, obj_id);
+	return import_kek_obj;
+}
diff --git a/drivers/common/mlx5/mlx5_devx_cmds.h b/drivers/common/mlx5/mlx5_devx_cmds.h
index 600577f18a..6423610dae 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.h
+++ b/drivers/common/mlx5/mlx5_devx_cmds.h
@@ -142,6 +142,7 @@ struct mlx5_hca_attr {
 	uint32_t crypto:1; /* Crypto engine is supported. */
 	uint32_t aes_xts:1; /* AES-XTS crypto is supported. */
 	uint32_t dek:1; /* General obj type DEK is supported. */
+	uint32_t import_kek:1; /* General obj type IMPORT_KEK supported. */
 	uint32_t regexp_num_of_engines;
 	uint32_t log_max_ft_sampler_num:8;
 	uint32_t geneve_tlv_opt;
@@ -450,6 +451,13 @@ struct mlx5_devx_dek_attr {
 	uint8_t key[MLX5_CRYPTO_KEY_MAX_SIZE];
 };
 
+struct mlx5_devx_import_kek_attr {
+	uint64_t modify_field_select;
+	uint32_t state:8;
+	uint32_t key_size:4;
+	uint8_t key[MLX5_CRYPTO_KEY_MAX_SIZE];
+};
+
 /* mlx5_devx_cmds.c */
 
 __rte_internal
@@ -606,4 +614,9 @@ __rte_internal
 struct mlx5_devx_obj *
 mlx5_devx_cmd_create_dek_obj(void *ctx, struct mlx5_devx_dek_attr *attr);
 
+__rte_internal
+struct mlx5_devx_obj *
+mlx5_devx_cmd_create_import_kek_obj(void *ctx,
+				    struct mlx5_devx_import_kek_attr *attr);
+
 #endif /* RTE_PMD_MLX5_DEVX_CMDS_H_ */
diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h
index 25f6f8ff00..bc339566a6 100644
--- a/drivers/common/mlx5/mlx5_prm.h
+++ b/drivers/common/mlx5/mlx5_prm.h
@@ -1119,6 +1119,8 @@ enum {
 			(1ULL << MLX5_GENERAL_OBJ_TYPE_GENEVE_TLV_OPT)
 #define MLX5_GENERAL_OBJ_TYPES_CAP_DEK \
 			(1ULL << MLX5_GENERAL_OBJ_TYPE_DEK)
+#define MLX5_GENERAL_OBJ_TYPES_CAP_IMPORT_KEK \
+			(1ULL << MLX5_GENERAL_OBJ_TYPE_IMPORT_KEK)
 
 enum {
 	MLX5_HCA_CAP_OPMOD_GET_MAX   = 0,
@@ -2419,6 +2421,7 @@ enum {
 	MLX5_GENERAL_OBJ_TYPE_DEK = 0x000c,
 	MLX5_GENERAL_OBJ_TYPE_VIRTQ = 0x000d,
 	MLX5_GENERAL_OBJ_TYPE_VIRTIO_Q_COUNTERS = 0x001c,
+	MLX5_GENERAL_OBJ_TYPE_IMPORT_KEK = 0x001d,
 	MLX5_GENERAL_OBJ_TYPE_FLEX_PARSE_GRAPH = 0x0022,
 	MLX5_GENERAL_OBJ_TYPE_FLOW_METER_ASO = 0x0024,
 	MLX5_GENERAL_OBJ_TYPE_FLOW_HIT_ASO = 0x0025,
@@ -2516,6 +2519,21 @@ struct mlx5_ifc_create_dek_in_bits {
 	struct mlx5_ifc_dek_bits dek;
 };
 
+struct mlx5_ifc_import_kek_bits {
+	u8 modify_field_select[0x40];
+	u8 state[0x8];
+	u8 reserved_at_48[0xc];
+	u8 key_size[0x4];
+	u8 reserved_at_58[0x1a8];
+	u8 key[0x400];
+	u8 reserved_at_600[0x200];
+};
+
+struct mlx5_ifc_create_import_kek_in_bits {
+	struct mlx5_ifc_general_obj_in_cmd_hdr_bits hdr;
+	struct mlx5_ifc_import_kek_bits import_kek;
+};
+
 enum {
 	MLX5_VIRTQ_STATE_INIT = 0,
 	MLX5_VIRTQ_STATE_RDY = 1,
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index 42bb985fb1..60bff5f799 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -21,6 +21,7 @@ INTERNAL {
 	mlx5_devx_cmd_create_flow_hit_aso_obj;
 	mlx5_devx_cmd_create_flow_meter_aso_obj;
 	mlx5_devx_cmd_create_geneve_tlv_option;
+	mlx5_devx_cmd_create_import_kek_obj;
 	mlx5_devx_cmd_create_qp;
 	mlx5_devx_cmd_create_rq;
 	mlx5_devx_cmd_create_rqt;
-- 
2.25.1

[dpdk-dev] [PATCH v3 08/15] common/mlx5: support general obj CRYPTO LOGIN create

[Matan Azrad]

From: Dekel Peled <dekelp@nvidia.com>

CRYPTO_LOGIN Object is used to login to the device as crypto user
or crypto officer.
Required in order to perform any crypto related control operations.

This patch adds support of CRYPTO_LOGIN object create operation.
Add reading of CRYPTO_LOGIN support capability.
Add function to create general object type CRYPTO_LOGIN, using DevX API.

Signed-off-by: Dekel Peled <dekelp@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/mlx5_devx_cmds.c | 54 ++++++++++++++++++++++++++++
 drivers/common/mlx5/mlx5_devx_cmds.h | 15 ++++++++
 drivers/common/mlx5/mlx5_prm.h       | 19 ++++++++++
 drivers/common/mlx5/version.map      |  1 +
 4 files changed, 89 insertions(+)

diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c
index afef7a5f63..5e082ebb78 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.c
+++ b/drivers/common/mlx5/mlx5_devx_cmds.c
@@ -754,6 +754,8 @@ mlx5_devx_cmd_query_hca_attr(void *ctx,
 		       MLX5_GENERAL_OBJ_TYPES_CAP_DEK);
 	attr->import_kek = !!(general_obj_types_supported &
 			      MLX5_GENERAL_OBJ_TYPES_CAP_IMPORT_KEK);
+	attr->crypto_login = !!(general_obj_types_supported &
+				MLX5_GENERAL_OBJ_TYPES_CAP_CRYPTO_LOGIN);
 	/* Add reading of other GENERAL_OBJ_TYPES_CAP bits above this line. */
 	attr->log_max_cq = MLX5_GET(cmd_hca_cap, hcattr, log_max_cq);
 	attr->log_max_qp = MLX5_GET(cmd_hca_cap, hcattr, log_max_qp);
@@ -2509,3 +2511,55 @@ mlx5_devx_cmd_create_import_kek_obj(void *ctx,
 	import_kek_obj->id = MLX5_GET(general_obj_out_cmd_hdr, out, obj_id);
 	return import_kek_obj;
 }
+
+/**
+ * Create general object of type CRYPTO_LOGIN using DevX API.
+ *
+ * @param[in] ctx
+ *   Context returned from mlx5 open_device() glue function.
+ * @param [in] attr
+ *   Pointer to CRYPTO_LOGIN attributes structure.
+ *
+ * @return
+ *   The DevX object created, NULL otherwise and rte_errno is set.
+ */
+struct mlx5_devx_obj *
+mlx5_devx_cmd_create_crypto_login_obj(void *ctx,
+				      struct mlx5_devx_crypto_login_attr *attr)
+{
+	uint32_t in[MLX5_ST_SZ_DW(create_crypto_login_in)] = {0};
+	uint32_t out[MLX5_ST_SZ_DW(general_obj_out_cmd_hdr)] = {0};
+	struct mlx5_devx_obj *crypto_login_obj = NULL;
+	void *ptr = NULL, *credential_addr = NULL;
+
+	crypto_login_obj = mlx5_malloc(MLX5_MEM_ZERO, sizeof(*crypto_login_obj),
+				       0, SOCKET_ID_ANY);
+	if (crypto_login_obj == NULL) {
+		DRV_LOG(ERR, "Failed to allocate CRYPTO_LOGIN object data");
+		rte_errno = ENOMEM;
+		return NULL;
+	}
+	ptr = MLX5_ADDR_OF(create_crypto_login_in, in, hdr);
+	MLX5_SET(general_obj_in_cmd_hdr, ptr, opcode,
+		 MLX5_CMD_OP_CREATE_GENERAL_OBJECT);
+	MLX5_SET(general_obj_in_cmd_hdr, ptr, obj_type,
+		 MLX5_GENERAL_OBJ_TYPE_CRYPTO_LOGIN);
+	ptr = MLX5_ADDR_OF(create_crypto_login_in, in, crypto_login);
+	MLX5_SET(crypto_login, ptr, credential_pointer,
+		 attr->credential_pointer);
+	MLX5_SET(crypto_login, ptr, session_import_kek_ptr,
+		 attr->session_import_kek_ptr);
+	credential_addr = MLX5_ADDR_OF(crypto_login, ptr, credential);
+	memcpy(credential_addr, (void *)(attr->credential),
+	       MLX5_CRYPTO_LOGIN_CREDENTIAL_SIZE);
+	crypto_login_obj->obj = mlx5_glue->devx_obj_create(ctx, in, sizeof(in),
+							   out, sizeof(out));
+	if (crypto_login_obj->obj == NULL) {
+		rte_errno = errno;
+		DRV_LOG(ERR, "Failed to create CRYPTO_LOGIN obj using DevX.");
+		mlx5_free(crypto_login_obj);
+		return NULL;
+	}
+	crypto_login_obj->id = MLX5_GET(general_obj_out_cmd_hdr, out, obj_id);
+	return crypto_login_obj;
+}
diff --git a/drivers/common/mlx5/mlx5_devx_cmds.h b/drivers/common/mlx5/mlx5_devx_cmds.h
index 6423610dae..709e28bfba 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.h
+++ b/drivers/common/mlx5/mlx5_devx_cmds.h
@@ -143,6 +143,7 @@ struct mlx5_hca_attr {
 	uint32_t aes_xts:1; /* AES-XTS crypto is supported. */
 	uint32_t dek:1; /* General obj type DEK is supported. */
 	uint32_t import_kek:1; /* General obj type IMPORT_KEK supported. */
+	uint32_t crypto_login:1; /* General obj type CRYPTO_LOGIN supported. */
 	uint32_t regexp_num_of_engines;
 	uint32_t log_max_ft_sampler_num:8;
 	uint32_t geneve_tlv_opt;
@@ -458,6 +459,15 @@ struct mlx5_devx_import_kek_attr {
 	uint8_t key[MLX5_CRYPTO_KEY_MAX_SIZE];
 };
 
+#define MLX5_CRYPTO_LOGIN_CREDENTIAL_SIZE	48
+
+struct mlx5_devx_crypto_login_attr {
+	uint64_t modify_field_select;
+	uint32_t credential_pointer:24;
+	uint32_t session_import_kek_ptr:24;
+	uint8_t credential[MLX5_CRYPTO_LOGIN_CREDENTIAL_SIZE];
+};
+
 /* mlx5_devx_cmds.c */
 
 __rte_internal
@@ -619,4 +629,9 @@ struct mlx5_devx_obj *
 mlx5_devx_cmd_create_import_kek_obj(void *ctx,
 				    struct mlx5_devx_import_kek_attr *attr);
 
+__rte_internal
+struct mlx5_devx_obj *
+mlx5_devx_cmd_create_crypto_login_obj(void *ctx,
+				      struct mlx5_devx_crypto_login_attr *attr);
+
 #endif /* RTE_PMD_MLX5_DEVX_CMDS_H_ */
diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h
index bc339566a6..a2437faec0 100644
--- a/drivers/common/mlx5/mlx5_prm.h
+++ b/drivers/common/mlx5/mlx5_prm.h
@@ -1121,6 +1121,8 @@ enum {
 			(1ULL << MLX5_GENERAL_OBJ_TYPE_DEK)
 #define MLX5_GENERAL_OBJ_TYPES_CAP_IMPORT_KEK \
 			(1ULL << MLX5_GENERAL_OBJ_TYPE_IMPORT_KEK)
+#define MLX5_GENERAL_OBJ_TYPES_CAP_CRYPTO_LOGIN \
+			(1ULL << MLX5_GENERAL_OBJ_TYPE_CRYPTO_LOGIN)
 
 enum {
 	MLX5_HCA_CAP_OPMOD_GET_MAX   = 0,
@@ -2422,6 +2424,7 @@ enum {
 	MLX5_GENERAL_OBJ_TYPE_VIRTQ = 0x000d,
 	MLX5_GENERAL_OBJ_TYPE_VIRTIO_Q_COUNTERS = 0x001c,
 	MLX5_GENERAL_OBJ_TYPE_IMPORT_KEK = 0x001d,
+	MLX5_GENERAL_OBJ_TYPE_CRYPTO_LOGIN = 0x001f,
 	MLX5_GENERAL_OBJ_TYPE_FLEX_PARSE_GRAPH = 0x0022,
 	MLX5_GENERAL_OBJ_TYPE_FLOW_METER_ASO = 0x0024,
 	MLX5_GENERAL_OBJ_TYPE_FLOW_HIT_ASO = 0x0025,
@@ -2534,6 +2537,22 @@ struct mlx5_ifc_create_import_kek_in_bits {
 	struct mlx5_ifc_import_kek_bits import_kek;
 };
 
+struct mlx5_ifc_crypto_login_bits {
+	u8 modify_field_select[0x40];
+	u8 reserved_at_40[0x48];
+	u8 credential_pointer[0x18];
+	u8 reserved_at_a0[0x8];
+	u8 session_import_kek_ptr[0x18];
+	u8 reserved_at_c0[0x140];
+	u8 credential[0x180];
+	u8 reserved_at_380[0x480];
+};
+
+struct mlx5_ifc_create_crypto_login_in_bits {
+	struct mlx5_ifc_general_obj_in_cmd_hdr_bits hdr;
+	struct mlx5_ifc_crypto_login_bits crypto_login;
+};
+
 enum {
 	MLX5_VIRTQ_STATE_INIT = 0,
 	MLX5_VIRTQ_STATE_RDY = 1,
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index 60bff5f799..89f0ee04cb 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -16,6 +16,7 @@ INTERNAL {
 
 	mlx5_devx_cmd_alloc_pd;
 	mlx5_devx_cmd_create_cq;
+	mlx5_devx_cmd_create_crypto_login_obj;
 	mlx5_devx_cmd_create_dek_obj;
 	mlx5_devx_cmd_create_flex_parser;
 	mlx5_devx_cmd_create_flow_hit_aso_obj;
-- 
2.25.1

[dpdk-dev] [PATCH v3 09/15] common/mlx5: add crypto BSF struct and defines

[Matan Azrad]

From: Dekel Peled <dekelp@nvidia.com>

This patch adds the struct defining crypto BSF segment of UMR WQE,
and the related value definitions and offsets.

Signed-off-by: Dekel Peled <dekelp@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/mlx5_prm.h | 66 ++++++++++++++++++++++++++++++++++
 1 file changed, 66 insertions(+)

diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h
index a2437faec0..a9dcbfa63c 100644
--- a/drivers/common/mlx5/mlx5_prm.h
+++ b/drivers/common/mlx5/mlx5_prm.h
@@ -1096,6 +1096,72 @@ struct mlx5_ifc_create_mkey_in_bits {
 	u8 klm_pas_mtt[][0x20];
 };
 
+enum {
+	MLX5_BSF_SIZE_16B = 0x0,
+	MLX5_BSF_SIZE_32B = 0x1,
+	MLX5_BSF_SIZE_64B = 0x2,
+	MLX5_BSF_SIZE_128B = 0x3,
+};
+
+enum {
+	MLX5_BSF_P_TYPE_SIGNATURE = 0x0,
+	MLX5_BSF_P_TYPE_CRYPTO = 0x1,
+};
+
+enum {
+	MLX5_ENCRYPTION_ORDER_ENCRYPTED_WIRE_SIGNATURE = 0x0,
+	MLX5_ENCRYPTION_ORDER_ENCRYPTED_MEMORY_SIGNATURE = 0x1,
+	MLX5_ENCRYPTION_ORDER_ENCRYPTED_RAW_WIRE = 0x2,
+	MLX5_ENCRYPTION_ORDER_ENCRYPTED_RAW_MEMORY = 0x3,
+};
+
+enum {
+	MLX5_ENCRYPTION_STANDARD_AES_XTS = 0x0,
+};
+
+enum {
+	MLX5_BLOCK_SIZE_512B	= 0x1,
+	MLX5_BLOCK_SIZE_520B	= 0x2,
+	MLX5_BLOCK_SIZE_4096B	= 0x3,
+	MLX5_BLOCK_SIZE_4160B	= 0x4,
+	MLX5_BLOCK_SIZE_1MB	= 0x5,
+	MLX5_BLOCK_SIZE_4048B	= 0x6,
+};
+
+#define MLX5_BSF_SIZE_OFFSET		30
+#define MLX5_BSF_P_TYPE_OFFSET		24
+#define MLX5_ENCRYPTION_ORDER_OFFSET	16
+#define MLX5_BLOCK_SIZE_OFFSET		24
+
+struct mlx5_wqe_umr_bsf_seg {
+	/*
+	 * bs_bpt_eo_es contains:
+	 * bs	bsf_size		2 bits at MLX5_BSF_SIZE_OFFSET
+	 * bpt	bsf_p_type		2 bits at MLX5_BSF_P_TYPE_OFFSET
+	 * eo	encryption_order	4 bits at MLX5_ENCRYPTION_ORDER_OFFSET
+	 * es	encryption_standard	4 bits at offset 0
+	 */
+	uint32_t bs_bpt_eo_es;
+	uint32_t raw_data_size;
+	/*
+	 * bsp_res contains:
+	 * bsp	crypto_block_size_pointer	8 bits at MLX5_BLOCK_SIZE_OFFSET
+	 * res	reserved 24 bits
+	 */
+	uint32_t bsp_res;
+	uint32_t reserved0;
+	uint8_t xts_initial_tweak[16];
+	/*
+	 * res_dp contains:
+	 * res	reserved 8 bits
+	 * dp	dek_pointer		24 bits at offset 0
+	 */
+	uint32_t res_dp;
+	uint32_t reserved1;
+	uint64_t keytag;
+	uint32_t reserved2[4];
+} __rte_packed;
+
 enum {
 	MLX5_GET_HCA_CAP_OP_MOD_GENERAL_DEVICE = 0x0 << 1,
 	MLX5_GET_HCA_CAP_OP_MOD_ETHERNET_OFFLOAD_CAPS = 0x1 << 1,
-- 
2.25.1

[dpdk-dev] [PATCH v3 10/15] common/mlx5: share hash list tool

[Matan Azrad]

From: Shiri Kuzin <shirik@nvidia.com>

In order to use the hash list defined in net in other drivers, the
hash list is moved to common utilities.

In addition, the log definition was moved from the common utilities to
a dedicated new log file in common in order to prevent a conflict.

Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/linux/mlx5_common_os.c   |   2 +-
 drivers/common/mlx5/linux/mlx5_nl.c          |   2 +-
 drivers/common/mlx5/meson.build              |  15 +-
 drivers/common/mlx5/mlx5_common.c            |   2 +-
 drivers/common/mlx5/mlx5_common_devx.c       |   2 +-
 drivers/common/mlx5/mlx5_common_log.h        |  21 ++
 drivers/common/mlx5/mlx5_common_mp.c         |   2 +-
 drivers/common/mlx5/mlx5_common_mr.c         |   2 +-
 drivers/common/mlx5/mlx5_common_pci.c        |   4 +-
 drivers/common/mlx5/mlx5_common_utils.c      | 221 +++++++++++++++++++
 drivers/common/mlx5/mlx5_common_utils.h      | 202 ++++++++++++++++-
 drivers/common/mlx5/mlx5_devx_cmds.c         |   2 +-
 drivers/common/mlx5/mlx5_malloc.c            |   2 +-
 drivers/common/mlx5/version.map              |   6 +
 drivers/common/mlx5/windows/mlx5_common_os.c |   2 +-
 drivers/common/mlx5/windows/mlx5_glue.c      |   2 +-
 drivers/net/mlx5/mlx5_utils.c                | 209 ------------------
 drivers/net/mlx5/mlx5_utils.h                | 194 +---------------
 18 files changed, 465 insertions(+), 427 deletions(-)
 create mode 100644 drivers/common/mlx5/mlx5_common_log.h
 create mode 100644 drivers/common/mlx5/mlx5_common_utils.c

diff --git a/drivers/common/mlx5/linux/mlx5_common_os.c b/drivers/common/mlx5/linux/mlx5_common_os.c
index 5cf9576921..fba8245b8b 100644
--- a/drivers/common/mlx5/linux/mlx5_common_os.c
+++ b/drivers/common/mlx5/linux/mlx5_common_os.c
@@ -15,7 +15,7 @@
 #include <rte_string_fns.h>
 
 #include "mlx5_common.h"
-#include "mlx5_common_utils.h"
+#include "mlx5_common_log.h"
 #include "mlx5_glue.h"
 
 #ifdef MLX5_GLUE
diff --git a/drivers/common/mlx5/linux/mlx5_nl.c b/drivers/common/mlx5/linux/mlx5_nl.c
index 752c57b33d..f0d04f9473 100644
--- a/drivers/common/mlx5/linux/mlx5_nl.c
+++ b/drivers/common/mlx5/linux/mlx5_nl.c
@@ -20,7 +20,7 @@
 #include <rte_errno.h>
 
 #include "mlx5_nl.h"
-#include "mlx5_common_utils.h"
+#include "../mlx5_common_log.h"
 #include "mlx5_malloc.h"
 #ifdef HAVE_DEVLINK
 #include <linux/devlink.h>
diff --git a/drivers/common/mlx5/meson.build b/drivers/common/mlx5/meson.build
index e78b4f47bc..c7ce4c7045 100644
--- a/drivers/common/mlx5/meson.build
+++ b/drivers/common/mlx5/meson.build
@@ -9,13 +9,14 @@ endif
 
 deps += ['hash', 'pci', 'bus_pci', 'net', 'eal', 'kvargs']
 sources += files(
-        'mlx5_devx_cmds.c',
-        'mlx5_common.c',
-        'mlx5_common_mp.c',
-        'mlx5_common_mr.c',
-        'mlx5_malloc.c',
-        'mlx5_common_pci.c',
-        'mlx5_common_devx.c',
+	'mlx5_devx_cmds.c',
+	'mlx5_common.c',
+	'mlx5_common_mp.c',
+	'mlx5_common_mr.c',
+	'mlx5_malloc.c',
+	'mlx5_common_pci.c',
+	'mlx5_common_devx.c',
+	'mlx5_common_utils.c',
 )
 
 cflags_options = [
diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index f92f05bda5..d397459a3d 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -11,7 +11,7 @@
 
 #include "mlx5_common.h"
 #include "mlx5_common_os.h"
-#include "mlx5_common_utils.h"
+#include "mlx5_common_log.h"
 #include "mlx5_common_pci.h"
 
 uint8_t haswell_broadwell_cpu;
diff --git a/drivers/common/mlx5/mlx5_common_devx.c b/drivers/common/mlx5/mlx5_common_devx.c
index d19be122bd..22c8d356c4 100644
--- a/drivers/common/mlx5/mlx5_common_devx.c
+++ b/drivers/common/mlx5/mlx5_common_devx.c
@@ -12,7 +12,7 @@
 
 #include "mlx5_prm.h"
 #include "mlx5_devx_cmds.h"
-#include "mlx5_common_utils.h"
+#include "mlx5_common_log.h"
 #include "mlx5_malloc.h"
 #include "mlx5_common.h"
 #include "mlx5_common_devx.h"
diff --git a/drivers/common/mlx5/mlx5_common_log.h b/drivers/common/mlx5/mlx5_common_log.h
new file mode 100644
index 0000000000..26b13fedaf
--- /dev/null
+++ b/drivers/common/mlx5/mlx5_common_log.h
@@ -0,0 +1,21 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2019 Mellanox Technologies, Ltd
+ */
+
+#ifndef RTE_PMD_MLX5_COMMON_LOG_H_
+#define RTE_PMD_MLX5_COMMON_LOG_H_
+
+#include "mlx5_common.h"
+
+
+extern int mlx5_common_logtype;
+
+#define MLX5_COMMON_LOG_PREFIX "mlx5_common"
+/* Generic printf()-like logging macro with automatic line feed. */
+#define DRV_LOG(level, ...) \
+	PMD_DRV_LOG_(level, mlx5_common_logtype, MLX5_COMMON_LOG_PREFIX, \
+		__VA_ARGS__ PMD_DRV_LOG_STRIP PMD_DRV_LOG_OPAREN, \
+		PMD_DRV_LOG_CPAREN)
+
+#endif /* RTE_PMD_MLX5_COMMON_LOG_H_ */
+
diff --git a/drivers/common/mlx5/mlx5_common_mp.c b/drivers/common/mlx5/mlx5_common_mp.c
index 40e3956e45..673a7c31de 100644
--- a/drivers/common/mlx5/mlx5_common_mp.c
+++ b/drivers/common/mlx5/mlx5_common_mp.c
@@ -10,7 +10,7 @@
 #include <rte_errno.h>
 
 #include "mlx5_common_mp.h"
-#include "mlx5_common_utils.h"
+#include "mlx5_common_log.h"
 #include "mlx5_malloc.h"
 
 /**
diff --git a/drivers/common/mlx5/mlx5_common_mr.c b/drivers/common/mlx5/mlx5_common_mr.c
index e1ed0caf3a..afb5b3d0a7 100644
--- a/drivers/common/mlx5/mlx5_common_mr.c
+++ b/drivers/common/mlx5/mlx5_common_mr.c
@@ -11,7 +11,7 @@
 #include "mlx5_glue.h"
 #include "mlx5_common_mp.h"
 #include "mlx5_common_mr.h"
-#include "mlx5_common_utils.h"
+#include "mlx5_common_log.h"
 #include "mlx5_malloc.h"
 
 struct mr_find_contig_memsegs_data {
diff --git a/drivers/common/mlx5/mlx5_common_pci.c b/drivers/common/mlx5/mlx5_common_pci.c
index 5a73ffa60a..3f16cd21cf 100644
--- a/drivers/common/mlx5/mlx5_common_pci.c
+++ b/drivers/common/mlx5/mlx5_common_pci.c
@@ -4,7 +4,9 @@
 
 #include <stdlib.h>
 #include <rte_malloc.h>
-#include "mlx5_common_utils.h"
+#include <rte_class.h>
+
+#include "mlx5_common_log.h"
 #include "mlx5_common_pci.h"
 
 struct mlx5_pci_device {
diff --git a/drivers/common/mlx5/mlx5_common_utils.c b/drivers/common/mlx5/mlx5_common_utils.c
new file mode 100644
index 0000000000..ad2011e858
--- /dev/null
+++ b/drivers/common/mlx5/mlx5_common_utils.c
@@ -0,0 +1,221 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2019 Mellanox Technologies, Ltd
+ */
+
+#include <rte_malloc.h>
+#include <rte_hash_crc.h>
+#include <rte_errno.h>
+
+#include <mlx5_malloc.h>
+
+#include "mlx5_common_utils.h"
+#include "mlx5_common_log.h"
+
+/********************* Hash List **********************/
+
+static struct mlx5_hlist_entry *
+mlx5_hlist_default_create_cb(struct mlx5_hlist *h, uint64_t key __rte_unused,
+			     void *ctx __rte_unused)
+{
+	return mlx5_malloc(MLX5_MEM_ZERO, h->entry_sz, 0, SOCKET_ID_ANY);
+}
+
+static void
+mlx5_hlist_default_remove_cb(struct mlx5_hlist *h __rte_unused,
+			     struct mlx5_hlist_entry *entry)
+{
+	mlx5_free(entry);
+}
+
+struct mlx5_hlist *
+mlx5_hlist_create(const char *name, uint32_t size, uint32_t entry_size,
+		  uint32_t flags, mlx5_hlist_create_cb cb_create,
+		  mlx5_hlist_match_cb cb_match, mlx5_hlist_remove_cb cb_remove)
+{
+	struct mlx5_hlist *h;
+	uint32_t act_size;
+	uint32_t alloc_size;
+	uint32_t i;
+
+	if (!size || !cb_match || (!cb_create ^ !cb_remove))
+		return NULL;
+	/* Align to the next power of 2, 32bits integer is enough now. */
+	if (!rte_is_power_of_2(size)) {
+		act_size = rte_align32pow2(size);
+		DRV_LOG(DEBUG, "Size 0x%" PRIX32 " is not power of 2, "
+			"will be aligned to 0x%" PRIX32 ".", size, act_size);
+	} else {
+		act_size = size;
+	}
+	alloc_size = sizeof(struct mlx5_hlist) +
+		     sizeof(struct mlx5_hlist_bucket) * act_size;
+	/* Using zmalloc, then no need to initialize the heads. */
+	h = mlx5_malloc(MLX5_MEM_ZERO, alloc_size, RTE_CACHE_LINE_SIZE,
+			SOCKET_ID_ANY);
+	if (!h) {
+		DRV_LOG(ERR, "No memory for hash list %s creation",
+			name ? name : "None");
+		return NULL;
+	}
+	if (name)
+		snprintf(h->name, MLX5_HLIST_NAMESIZE, "%s", name);
+	h->table_sz = act_size;
+	h->mask = act_size - 1;
+	h->entry_sz = entry_size;
+	h->direct_key = !!(flags & MLX5_HLIST_DIRECT_KEY);
+	h->write_most = !!(flags & MLX5_HLIST_WRITE_MOST);
+	h->cb_create = cb_create ? cb_create : mlx5_hlist_default_create_cb;
+	h->cb_match = cb_match;
+	h->cb_remove = cb_remove ? cb_remove : mlx5_hlist_default_remove_cb;
+	for (i = 0; i < act_size; i++)
+		rte_rwlock_init(&h->buckets[i].lock);
+	DRV_LOG(DEBUG, "Hash list with %s size 0x%" PRIX32 " is created.",
+		h->name, act_size);
+	return h;
+}
+
+static struct mlx5_hlist_entry *
+__hlist_lookup(struct mlx5_hlist *h, uint64_t key, uint32_t idx,
+	       void *ctx, bool reuse)
+{
+	struct mlx5_hlist_head *first;
+	struct mlx5_hlist_entry *node;
+
+	MLX5_ASSERT(h);
+	first = &h->buckets[idx].head;
+	LIST_FOREACH(node, first, next) {
+		if (!h->cb_match(h, node, key, ctx)) {
+			if (reuse) {
+				__atomic_add_fetch(&node->ref_cnt, 1,
+						   __ATOMIC_RELAXED);
+				DRV_LOG(DEBUG, "Hash list %s entry %p "
+					"reuse: %u.",
+					h->name, (void *)node, node->ref_cnt);
+			}
+			break;
+		}
+	}
+	return node;
+}
+
+static struct mlx5_hlist_entry *
+hlist_lookup(struct mlx5_hlist *h, uint64_t key, uint32_t idx,
+	     void *ctx, bool reuse)
+{
+	struct mlx5_hlist_entry *node;
+
+	MLX5_ASSERT(h);
+	rte_rwlock_read_lock(&h->buckets[idx].lock);
+	node = __hlist_lookup(h, key, idx, ctx, reuse);
+	rte_rwlock_read_unlock(&h->buckets[idx].lock);
+	return node;
+}
+
+struct mlx5_hlist_entry *
+mlx5_hlist_lookup(struct mlx5_hlist *h, uint64_t key, void *ctx)
+{
+	uint32_t idx;
+
+	if (h->direct_key)
+		idx = (uint32_t)(key & h->mask);
+	else
+		idx = rte_hash_crc_8byte(key, 0) & h->mask;
+	return hlist_lookup(h, key, idx, ctx, false);
+}
+
+struct mlx5_hlist_entry*
+mlx5_hlist_register(struct mlx5_hlist *h, uint64_t key, void *ctx)
+{
+	uint32_t idx;
+	struct mlx5_hlist_head *first;
+	struct mlx5_hlist_bucket *b;
+	struct mlx5_hlist_entry *entry;
+	uint32_t prev_gen_cnt = 0;
+
+	if (h->direct_key)
+		idx = (uint32_t)(key & h->mask);
+	else
+		idx = rte_hash_crc_8byte(key, 0) & h->mask;
+	MLX5_ASSERT(h);
+	b = &h->buckets[idx];
+	/* Use write lock directly for write-most list. */
+	if (!h->write_most) {
+		prev_gen_cnt = __atomic_load_n(&b->gen_cnt, __ATOMIC_ACQUIRE);
+		entry = hlist_lookup(h, key, idx, ctx, true);
+		if (entry)
+			return entry;
+	}
+	rte_rwlock_write_lock(&b->lock);
+	/* Check if the list changed by other threads. */
+	if (h->write_most ||
+	    prev_gen_cnt != __atomic_load_n(&b->gen_cnt, __ATOMIC_ACQUIRE)) {
+		entry = __hlist_lookup(h, key, idx, ctx, true);
+		if (entry)
+			goto done;
+	}
+	first = &b->head;
+	entry = h->cb_create(h, key, ctx);
+	if (!entry) {
+		rte_errno = ENOMEM;
+		DRV_LOG(DEBUG, "Can't allocate hash list %s entry.", h->name);
+		goto done;
+	}
+	entry->idx = idx;
+	entry->ref_cnt = 1;
+	LIST_INSERT_HEAD(first, entry, next);
+	__atomic_add_fetch(&b->gen_cnt, 1, __ATOMIC_ACQ_REL);
+	DRV_LOG(DEBUG, "Hash list %s entry %p new: %u.",
+		h->name, (void *)entry, entry->ref_cnt);
+done:
+	rte_rwlock_write_unlock(&b->lock);
+	return entry;
+}
+
+int
+mlx5_hlist_unregister(struct mlx5_hlist *h, struct mlx5_hlist_entry *entry)
+{
+	uint32_t idx = entry->idx;
+
+	rte_rwlock_write_lock(&h->buckets[idx].lock);
+	MLX5_ASSERT(entry && entry->ref_cnt && entry->next.le_prev);
+	DRV_LOG(DEBUG, "Hash list %s entry %p deref: %u.",
+		h->name, (void *)entry, entry->ref_cnt);
+	if (--entry->ref_cnt) {
+		rte_rwlock_write_unlock(&h->buckets[idx].lock);
+		return 1;
+	}
+	LIST_REMOVE(entry, next);
+	/* Set to NULL to get rid of removing action for more than once. */
+	entry->next.le_prev = NULL;
+	h->cb_remove(h, entry);
+	rte_rwlock_write_unlock(&h->buckets[idx].lock);
+	DRV_LOG(DEBUG, "Hash list %s entry %p removed.",
+		h->name, (void *)entry);
+	return 0;
+}
+
+void
+mlx5_hlist_destroy(struct mlx5_hlist *h)
+{
+	uint32_t idx;
+	struct mlx5_hlist_entry *entry;
+
+	MLX5_ASSERT(h);
+	for (idx = 0; idx < h->table_sz; ++idx) {
+		/* No LIST_FOREACH_SAFE, using while instead. */
+		while (!LIST_EMPTY(&h->buckets[idx].head)) {
+			entry = LIST_FIRST(&h->buckets[idx].head);
+			LIST_REMOVE(entry, next);
+			/*
+			 * The owner of whole element which contains data entry
+			 * is the user, so it's the user's duty to do the clean
+			 * up and the free work because someone may not put the
+			 * hlist entry at the beginning(suggested to locate at
+			 * the beginning). Or else the default free function
+			 * will be used.
+			 */
+			h->cb_remove(h, entry);
+		}
+	}
+	mlx5_free(h);
+}
diff --git a/drivers/common/mlx5/mlx5_common_utils.h b/drivers/common/mlx5/mlx5_common_utils.h
index 6cba39c8cc..ed378ce9bd 100644
--- a/drivers/common/mlx5/mlx5_common_utils.h
+++ b/drivers/common/mlx5/mlx5_common_utils.h
@@ -7,14 +7,202 @@
 
 #include "mlx5_common.h"
 
+#define MLX5_HLIST_DIRECT_KEY 0x0001 /* Use the key directly as hash index. */
+#define MLX5_HLIST_WRITE_MOST 0x0002 /* List mostly used for append new. */
 
-extern int mlx5_common_logtype;
+/** Maximum size of string for naming the hlist table. */
+#define MLX5_HLIST_NAMESIZE			32
 
-#define MLX5_COMMON_LOG_PREFIX "mlx5_common"
-/* Generic printf()-like logging macro with automatic line feed. */
-#define DRV_LOG(level, ...) \
-	PMD_DRV_LOG_(level, mlx5_common_logtype, MLX5_COMMON_LOG_PREFIX, \
-		__VA_ARGS__ PMD_DRV_LOG_STRIP PMD_DRV_LOG_OPAREN, \
-		PMD_DRV_LOG_CPAREN)
+struct mlx5_hlist;
+
+/**
+ * Structure of the entry in the hash list, user should define its own struct
+ * that contains this in order to store the data. The 'key' is 64-bits right
+ * now and its user's responsibility to guarantee there is no collision.
+ */
+struct mlx5_hlist_entry {
+	LIST_ENTRY(mlx5_hlist_entry) next; /* entry pointers in the list. */
+	uint32_t idx; /* Bucket index the entry belongs to. */
+	uint32_t ref_cnt; /* Reference count. */
+};
+
+/** Structure for hash head. */
+LIST_HEAD(mlx5_hlist_head, mlx5_hlist_entry);
+
+/**
+ * Type of callback function for entry removal.
+ *
+ * @param list
+ *   The hash list.
+ * @param entry
+ *   The entry in the list.
+ */
+typedef void (*mlx5_hlist_remove_cb)(struct mlx5_hlist *list,
+				     struct mlx5_hlist_entry *entry);
+
+/**
+ * Type of function for user defined matching.
+ *
+ * @param list
+ *   The hash list.
+ * @param entry
+ *   The entry in the list.
+ * @param key
+ *   The new entry key.
+ * @param ctx
+ *   The pointer to new entry context.
+ *
+ * @return
+ *   0 if matching, non-zero number otherwise.
+ */
+typedef int (*mlx5_hlist_match_cb)(struct mlx5_hlist *list,
+				   struct mlx5_hlist_entry *entry,
+				   uint64_t key, void *ctx);
+
+/**
+ * Type of function for user defined hash list entry creation.
+ *
+ * @param list
+ *   The hash list.
+ * @param key
+ *   The key of the new entry.
+ * @param ctx
+ *   The pointer to new entry context.
+ *
+ * @return
+ *   Pointer to allocated entry on success, NULL otherwise.
+ */
+typedef struct mlx5_hlist_entry *(*mlx5_hlist_create_cb)
+				  (struct mlx5_hlist *list,
+				   uint64_t key, void *ctx);
+
+/* Hash list bucket head. */
+struct mlx5_hlist_bucket {
+	struct mlx5_hlist_head head; /* List head. */
+	rte_rwlock_t lock; /* Bucket lock. */
+	uint32_t gen_cnt; /* List modification will update generation count. */
+} __rte_cache_aligned;
+
+/**
+ * Hash list table structure
+ *
+ * Entry in hash list could be reused if entry already exists, reference
+ * count will increase and the existing entry returns.
+ *
+ * When destroy an entry from list, decrease reference count and only
+ * destroy when no further reference.
+ */
+struct mlx5_hlist {
+	char name[MLX5_HLIST_NAMESIZE]; /**< Name of the hash list. */
+	/**< number of heads, need to be power of 2. */
+	uint32_t table_sz;
+	uint32_t entry_sz; /**< Size of entry, used to allocate entry. */
+	/**< mask to get the index of the list heads. */
+	uint32_t mask;
+	bool direct_key; /* Use the new entry key directly as hash index. */
+	bool write_most; /* List mostly used for append new or destroy. */
+	void *ctx;
+	mlx5_hlist_create_cb cb_create; /**< entry create callback. */
+	mlx5_hlist_match_cb cb_match; /**< entry match callback. */
+	mlx5_hlist_remove_cb cb_remove; /**< entry remove callback. */
+	struct mlx5_hlist_bucket buckets[] __rte_cache_aligned;
+	/**< list bucket arrays. */
+};
+
+/**
+ * Create a hash list table, the user can specify the list heads array size
+ * of the table, now the size should be a power of 2 in order to get better
+ * distribution for the entries. Each entry is a part of the whole data element
+ * and the caller should be responsible for the data element's allocation and
+ * cleanup / free. Key of each entry will be calculated with CRC in order to
+ * generate a little fairer distribution.
+ *
+ * @param name
+ *   Name of the hash list(optional).
+ * @param size
+ *   Heads array size of the hash list.
+ * @param entry_size
+ *   Entry size to allocate if cb_create not specified.
+ * @param flags
+ *   The hash list attribute flags.
+ * @param cb_create
+ *   Callback function for entry create.
+ * @param cb_match
+ *   Callback function for entry match.
+ * @param cb_destroy
+ *   Callback function for entry destroy.
+ * @return
+ *   Pointer of the hash list table created, NULL on failure.
+ */
+__rte_internal
+struct mlx5_hlist *mlx5_hlist_create(const char *name, uint32_t size,
+				     uint32_t entry_size, uint32_t flags,
+				     mlx5_hlist_create_cb cb_create,
+				     mlx5_hlist_match_cb cb_match,
+				     mlx5_hlist_remove_cb cb_destroy);
+
+/**
+ * Search an entry matching the key.
+ *
+ * Result returned might be destroyed by other thread, must use
+ * this function only in main thread.
+ *
+ * @param h
+ *   Pointer to the hast list table.
+ * @param key
+ *   Key for the searching entry.
+ * @param ctx
+ *   Common context parameter used by entry callback function.
+ *
+ * @return
+ *   Pointer of the hlist entry if found, NULL otherwise.
+ */
+__rte_internal
+struct mlx5_hlist_entry *mlx5_hlist_lookup(struct mlx5_hlist *h, uint64_t key,
+					   void *ctx);
+
+/**
+ * Insert an entry to the hash list table, the entry is only part of whole data
+ * element and a 64B key is used for matching. User should construct the key or
+ * give a calculated hash signature and guarantee there is no collision.
+ *
+ * @param h
+ *   Pointer to the hast list table.
+ * @param entry
+ *   Entry to be inserted into the hash list table.
+ * @param ctx
+ *   Common context parameter used by callback function.
+ *
+ * @return
+ *   registered entry on success, NULL otherwise
+ */
+__rte_internal
+struct mlx5_hlist_entry *mlx5_hlist_register(struct mlx5_hlist *h, uint64_t key,
+					     void *ctx);
+
+/**
+ * Remove an entry from the hash list table. User should guarantee the validity
+ * of the entry.
+ *
+ * @param h
+ *   Pointer to the hast list table. (not used)
+ * @param entry
+ *   Entry to be removed from the hash list table.
+ * @return
+ *   0 on entry removed, 1 on entry still referenced.
+ */
+__rte_internal
+int mlx5_hlist_unregister(struct mlx5_hlist *h, struct mlx5_hlist_entry *entry);
+
+/**
+ * Destroy the hash list table, all the entries already inserted into the lists
+ * will be handled by the callback function provided by the user (including
+ * free if needed) before the table is freed.
+ *
+ * @param h
+ *   Pointer to the hast list table.
+ */
+__rte_internal
+void mlx5_hlist_destroy(struct mlx5_hlist *h);
 
 #endif /* RTE_PMD_MLX5_COMMON_UTILS_H_ */
diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c
index 5e082ebb78..c0061741e8 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.c
+++ b/drivers/common/mlx5/mlx5_devx_cmds.c
@@ -9,7 +9,7 @@
 
 #include "mlx5_prm.h"
 #include "mlx5_devx_cmds.h"
-#include "mlx5_common_utils.h"
+#include "mlx5_common_log.h"
 #include "mlx5_malloc.h"
 
 
diff --git a/drivers/common/mlx5/mlx5_malloc.c b/drivers/common/mlx5/mlx5_malloc.c
index 9d30cedbaa..b19501e1bc 100644
--- a/drivers/common/mlx5/mlx5_malloc.c
+++ b/drivers/common/mlx5/mlx5_malloc.c
@@ -8,7 +8,7 @@
 #include <stdbool.h>
 #include <string.h>
 
-#include "mlx5_common_utils.h"
+#include "mlx5_common_log.h"
 #include "mlx5_common_os.h"
 #include "mlx5_malloc.h"
 
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index 89f0ee04cb..40b0d713ac 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -70,6 +70,12 @@ INTERNAL {
 
 	mlx5_glue;
 
+	mlx5_hlist_create;
+	mlx5_hlist_lookup;
+	mlx5_hlist_register;
+	mlx5_hlist_unregister;
+	mlx5_hlist_destroy;
+
 	mlx5_malloc;
 	mlx5_malloc_mem_select;
 
diff --git a/drivers/common/mlx5/windows/mlx5_common_os.c b/drivers/common/mlx5/windows/mlx5_common_os.c
index cebf42d53c..78975d9d4d 100644
--- a/drivers/common/mlx5/windows/mlx5_common_os.c
+++ b/drivers/common/mlx5/windows/mlx5_common_os.c
@@ -11,7 +11,7 @@
 #include <rte_errno.h>
 
 #include "mlx5_devx_cmds.h"
-#include "mlx5_common_utils.h"
+#include "../mlx5_common_log.h"
 #include "mlx5_common.h"
 #include "mlx5_common_os.h"
 #include "mlx5_malloc.h"
diff --git a/drivers/common/mlx5/windows/mlx5_glue.c b/drivers/common/mlx5/windows/mlx5_glue.c
index aef6d3b5f4..535487a8d4 100644
--- a/drivers/common/mlx5/windows/mlx5_glue.c
+++ b/drivers/common/mlx5/windows/mlx5_glue.c
@@ -12,7 +12,7 @@
 #include <rte_malloc.h>
 
 #include "mlx5_glue.h"
-#include "mlx5_common_utils.h"
+#include "../mlx5_common_log.h"
 #include "mlx5_win_ext.h"
 
 /*
diff --git a/drivers/net/mlx5/mlx5_utils.c b/drivers/net/mlx5/mlx5_utils.c
index a39b5edddc..18fe23e4fb 100644
--- a/drivers/net/mlx5/mlx5_utils.c
+++ b/drivers/net/mlx5/mlx5_utils.c
@@ -3,220 +3,11 @@
  */
 
 #include <rte_malloc.h>
-#include <rte_hash_crc.h>
 
 #include <mlx5_malloc.h>
 
 #include "mlx5_utils.h"
 
-/********************* Hash List **********************/
-
-static struct mlx5_hlist_entry *
-mlx5_hlist_default_create_cb(struct mlx5_hlist *h, uint64_t key __rte_unused,
-			     void *ctx __rte_unused)
-{
-	return mlx5_malloc(MLX5_MEM_ZERO, h->entry_sz, 0, SOCKET_ID_ANY);
-}
-
-static void
-mlx5_hlist_default_remove_cb(struct mlx5_hlist *h __rte_unused,
-			     struct mlx5_hlist_entry *entry)
-{
-	mlx5_free(entry);
-}
-
-struct mlx5_hlist *
-mlx5_hlist_create(const char *name, uint32_t size, uint32_t entry_size,
-		  uint32_t flags, mlx5_hlist_create_cb cb_create,
-		  mlx5_hlist_match_cb cb_match, mlx5_hlist_remove_cb cb_remove)
-{
-	struct mlx5_hlist *h;
-	uint32_t act_size;
-	uint32_t alloc_size;
-	uint32_t i;
-
-	if (!size || !cb_match || (!cb_create ^ !cb_remove))
-		return NULL;
-	/* Align to the next power of 2, 32bits integer is enough now. */
-	if (!rte_is_power_of_2(size)) {
-		act_size = rte_align32pow2(size);
-		DRV_LOG(DEBUG, "Size 0x%" PRIX32 " is not power of 2, "
-			"will be aligned to 0x%" PRIX32 ".", size, act_size);
-	} else {
-		act_size = size;
-	}
-	alloc_size = sizeof(struct mlx5_hlist) +
-		     sizeof(struct mlx5_hlist_bucket) * act_size;
-	/* Using zmalloc, then no need to initialize the heads. */
-	h = mlx5_malloc(MLX5_MEM_ZERO, alloc_size, RTE_CACHE_LINE_SIZE,
-			SOCKET_ID_ANY);
-	if (!h) {
-		DRV_LOG(ERR, "No memory for hash list %s creation",
-			name ? name : "None");
-		return NULL;
-	}
-	if (name)
-		snprintf(h->name, MLX5_HLIST_NAMESIZE, "%s", name);
-	h->table_sz = act_size;
-	h->mask = act_size - 1;
-	h->entry_sz = entry_size;
-	h->direct_key = !!(flags & MLX5_HLIST_DIRECT_KEY);
-	h->write_most = !!(flags & MLX5_HLIST_WRITE_MOST);
-	h->cb_create = cb_create ? cb_create : mlx5_hlist_default_create_cb;
-	h->cb_match = cb_match;
-	h->cb_remove = cb_remove ? cb_remove : mlx5_hlist_default_remove_cb;
-	for (i = 0; i < act_size; i++)
-		rte_rwlock_init(&h->buckets[i].lock);
-	DRV_LOG(DEBUG, "Hash list with %s size 0x%" PRIX32 " is created.",
-		h->name, act_size);
-	return h;
-}
-
-static struct mlx5_hlist_entry *
-__hlist_lookup(struct mlx5_hlist *h, uint64_t key, uint32_t idx,
-	       void *ctx, bool reuse)
-{
-	struct mlx5_hlist_head *first;
-	struct mlx5_hlist_entry *node;
-
-	MLX5_ASSERT(h);
-	first = &h->buckets[idx].head;
-	LIST_FOREACH(node, first, next) {
-		if (!h->cb_match(h, node, key, ctx)) {
-			if (reuse) {
-				__atomic_add_fetch(&node->ref_cnt, 1,
-						   __ATOMIC_RELAXED);
-				DRV_LOG(DEBUG, "Hash list %s entry %p "
-					"reuse: %u.",
-					h->name, (void *)node, node->ref_cnt);
-			}
-			break;
-		}
-	}
-	return node;
-}
-
-static struct mlx5_hlist_entry *
-hlist_lookup(struct mlx5_hlist *h, uint64_t key, uint32_t idx,
-	     void *ctx, bool reuse)
-{
-	struct mlx5_hlist_entry *node;
-
-	MLX5_ASSERT(h);
-	rte_rwlock_read_lock(&h->buckets[idx].lock);
-	node = __hlist_lookup(h, key, idx, ctx, reuse);
-	rte_rwlock_read_unlock(&h->buckets[idx].lock);
-	return node;
-}
-
-struct mlx5_hlist_entry *
-mlx5_hlist_lookup(struct mlx5_hlist *h, uint64_t key, void *ctx)
-{
-	uint32_t idx;
-
-	if (h->direct_key)
-		idx = (uint32_t)(key & h->mask);
-	else
-		idx = rte_hash_crc_8byte(key, 0) & h->mask;
-	return hlist_lookup(h, key, idx, ctx, false);
-}
-
-struct mlx5_hlist_entry*
-mlx5_hlist_register(struct mlx5_hlist *h, uint64_t key, void *ctx)
-{
-	uint32_t idx;
-	struct mlx5_hlist_head *first;
-	struct mlx5_hlist_bucket *b;
-	struct mlx5_hlist_entry *entry;
-	uint32_t prev_gen_cnt = 0;
-
-	if (h->direct_key)
-		idx = (uint32_t)(key & h->mask);
-	else
-		idx = rte_hash_crc_8byte(key, 0) & h->mask;
-	MLX5_ASSERT(h);
-	b = &h->buckets[idx];
-	/* Use write lock directly for write-most list. */
-	if (!h->write_most) {
-		prev_gen_cnt = __atomic_load_n(&b->gen_cnt, __ATOMIC_ACQUIRE);
-		entry = hlist_lookup(h, key, idx, ctx, true);
-		if (entry)
-			return entry;
-	}
-	rte_rwlock_write_lock(&b->lock);
-	/* Check if the list changed by other threads. */
-	if (h->write_most ||
-	    prev_gen_cnt != __atomic_load_n(&b->gen_cnt, __ATOMIC_ACQUIRE)) {
-		entry = __hlist_lookup(h, key, idx, ctx, true);
-		if (entry)
-			goto done;
-	}
-	first = &b->head;
-	entry = h->cb_create(h, key, ctx);
-	if (!entry) {
-		rte_errno = ENOMEM;
-		DRV_LOG(DEBUG, "Can't allocate hash list %s entry.", h->name);
-		goto done;
-	}
-	entry->idx = idx;
-	entry->ref_cnt = 1;
-	LIST_INSERT_HEAD(first, entry, next);
-	__atomic_add_fetch(&b->gen_cnt, 1, __ATOMIC_ACQ_REL);
-	DRV_LOG(DEBUG, "Hash list %s entry %p new: %u.",
-		h->name, (void *)entry, entry->ref_cnt);
-done:
-	rte_rwlock_write_unlock(&b->lock);
-	return entry;
-}
-
-int
-mlx5_hlist_unregister(struct mlx5_hlist *h, struct mlx5_hlist_entry *entry)
-{
-	uint32_t idx = entry->idx;
-
-	rte_rwlock_write_lock(&h->buckets[idx].lock);
-	MLX5_ASSERT(entry && entry->ref_cnt && entry->next.le_prev);
-	DRV_LOG(DEBUG, "Hash list %s entry %p deref: %u.",
-		h->name, (void *)entry, entry->ref_cnt);
-	if (--entry->ref_cnt) {
-		rte_rwlock_write_unlock(&h->buckets[idx].lock);
-		return 1;
-	}
-	LIST_REMOVE(entry, next);
-	/* Set to NULL to get rid of removing action for more than once. */
-	entry->next.le_prev = NULL;
-	h->cb_remove(h, entry);
-	rte_rwlock_write_unlock(&h->buckets[idx].lock);
-	DRV_LOG(DEBUG, "Hash list %s entry %p removed.",
-		h->name, (void *)entry);
-	return 0;
-}
-
-void
-mlx5_hlist_destroy(struct mlx5_hlist *h)
-{
-	uint32_t idx;
-	struct mlx5_hlist_entry *entry;
-
-	MLX5_ASSERT(h);
-	for (idx = 0; idx < h->table_sz; ++idx) {
-		/* No LIST_FOREACH_SAFE, using while instead. */
-		while (!LIST_EMPTY(&h->buckets[idx].head)) {
-			entry = LIST_FIRST(&h->buckets[idx].head);
-			LIST_REMOVE(entry, next);
-			/*
-			 * The owner of whole element which contains data entry
-			 * is the user, so it's the user's duty to do the clean
-			 * up and the free work because someone may not put the
-			 * hlist entry at the beginning(suggested to locate at
-			 * the beginning). Or else the default free function
-			 * will be used.
-			 */
-			h->cb_remove(h, entry);
-		}
-	}
-	mlx5_free(h);
-}
 
 /********************* Cache list ************************/
 
diff --git a/drivers/net/mlx5/mlx5_utils.h b/drivers/net/mlx5/mlx5_utils.h
index 289941cebc..b54517c6df 100644
--- a/drivers/net/mlx5/mlx5_utils.h
+++ b/drivers/net/mlx5/mlx5_utils.h
@@ -18,6 +18,7 @@
 #include <rte_bitmap.h>
 
 #include <mlx5_common.h>
+#include <mlx5_common_utils.h>
 
 #include "mlx5_defs.h"
 
@@ -261,199 +262,6 @@ log2above(unsigned int v)
 	return l + r;
 }
 
-#define MLX5_HLIST_DIRECT_KEY 0x0001 /* Use the key directly as hash index. */
-#define MLX5_HLIST_WRITE_MOST 0x0002 /* List mostly used for append new. */
-
-/** Maximum size of string for naming the hlist table. */
-#define MLX5_HLIST_NAMESIZE			32
-
-struct mlx5_hlist;
-
-/**
- * Structure of the entry in the hash list, user should define its own struct
- * that contains this in order to store the data. The 'key' is 64-bits right
- * now and its user's responsibility to guarantee there is no collision.
- */
-struct mlx5_hlist_entry {
-	LIST_ENTRY(mlx5_hlist_entry) next; /* entry pointers in the list. */
-	uint32_t idx; /* Bucket index the entry belongs to. */
-	uint32_t ref_cnt; /* Reference count. */
-};
-
-/** Structure for hash head. */
-LIST_HEAD(mlx5_hlist_head, mlx5_hlist_entry);
-
-/**
- * Type of callback function for entry removal.
- *
- * @param list
- *   The hash list.
- * @param entry
- *   The entry in the list.
- */
-typedef void (*mlx5_hlist_remove_cb)(struct mlx5_hlist *list,
-				     struct mlx5_hlist_entry *entry);
-
-/**
- * Type of function for user defined matching.
- *
- * @param list
- *   The hash list.
- * @param entry
- *   The entry in the list.
- * @param key
- *   The new entry key.
- * @param ctx
- *   The pointer to new entry context.
- *
- * @return
- *   0 if matching, non-zero number otherwise.
- */
-typedef int (*mlx5_hlist_match_cb)(struct mlx5_hlist *list,
-				   struct mlx5_hlist_entry *entry,
-				   uint64_t key, void *ctx);
-
-/**
- * Type of function for user defined hash list entry creation.
- *
- * @param list
- *   The hash list.
- * @param key
- *   The key of the new entry.
- * @param ctx
- *   The pointer to new entry context.
- *
- * @return
- *   Pointer to allocated entry on success, NULL otherwise.
- */
-typedef struct mlx5_hlist_entry *(*mlx5_hlist_create_cb)
-				  (struct mlx5_hlist *list,
-				   uint64_t key, void *ctx);
-
-/* Hash list bucket head. */
-struct mlx5_hlist_bucket {
-	struct mlx5_hlist_head head; /* List head. */
-	rte_rwlock_t lock; /* Bucket lock. */
-	uint32_t gen_cnt; /* List modification will update generation count. */
-} __rte_cache_aligned;
-
-/**
- * Hash list table structure
- *
- * Entry in hash list could be reused if entry already exists, reference
- * count will increase and the existing entry returns.
- *
- * When destroy an entry from list, decrease reference count and only
- * destroy when no further reference.
- */
-struct mlx5_hlist {
-	char name[MLX5_HLIST_NAMESIZE]; /**< Name of the hash list. */
-	/**< number of heads, need to be power of 2. */
-	uint32_t table_sz;
-	uint32_t entry_sz; /**< Size of entry, used to allocate entry. */
-	/**< mask to get the index of the list heads. */
-	uint32_t mask;
-	bool direct_key; /* Use the new entry key directly as hash index. */
-	bool write_most; /* List mostly used for append new or destroy. */
-	void *ctx;
-	mlx5_hlist_create_cb cb_create; /**< entry create callback. */
-	mlx5_hlist_match_cb cb_match; /**< entry match callback. */
-	mlx5_hlist_remove_cb cb_remove; /**< entry remove callback. */
-	struct mlx5_hlist_bucket buckets[] __rte_cache_aligned;
-	/**< list bucket arrays. */
-};
-
-/**
- * Create a hash list table, the user can specify the list heads array size
- * of the table, now the size should be a power of 2 in order to get better
- * distribution for the entries. Each entry is a part of the whole data element
- * and the caller should be responsible for the data element's allocation and
- * cleanup / free. Key of each entry will be calculated with CRC in order to
- * generate a little fairer distribution.
- *
- * @param name
- *   Name of the hash list(optional).
- * @param size
- *   Heads array size of the hash list.
- * @param entry_size
- *   Entry size to allocate if cb_create not specified.
- * @param flags
- *   The hash list attribute flags.
- * @param cb_create
- *   Callback function for entry create.
- * @param cb_match
- *   Callback function for entry match.
- * @param cb_destroy
- *   Callback function for entry destroy.
- * @return
- *   Pointer of the hash list table created, NULL on failure.
- */
-struct mlx5_hlist *mlx5_hlist_create(const char *name, uint32_t size,
-				     uint32_t entry_size, uint32_t flags,
-				     mlx5_hlist_create_cb cb_create,
-				     mlx5_hlist_match_cb cb_match,
-				     mlx5_hlist_remove_cb cb_destroy);
-
-/**
- * Search an entry matching the key.
- *
- * Result returned might be destroyed by other thread, must use
- * this function only in main thread.
- *
- * @param h
- *   Pointer to the hast list table.
- * @param key
- *   Key for the searching entry.
- * @param ctx
- *   Common context parameter used by entry callback function.
- *
- * @return
- *   Pointer of the hlist entry if found, NULL otherwise.
- */
-struct mlx5_hlist_entry *mlx5_hlist_lookup(struct mlx5_hlist *h, uint64_t key,
-					   void *ctx);
-
-/**
- * Insert an entry to the hash list table, the entry is only part of whole data
- * element and a 64B key is used for matching. User should construct the key or
- * give a calculated hash signature and guarantee there is no collision.
- *
- * @param h
- *   Pointer to the hast list table.
- * @param entry
- *   Entry to be inserted into the hash list table.
- * @param ctx
- *   Common context parameter used by callback function.
- *
- * @return
- *   registered entry on success, NULL otherwise
- */
-struct mlx5_hlist_entry *mlx5_hlist_register(struct mlx5_hlist *h, uint64_t key,
-					     void *ctx);
-
-/**
- * Remove an entry from the hash list table. User should guarantee the validity
- * of the entry.
- *
- * @param h
- *   Pointer to the hast list table. (not used)
- * @param entry
- *   Entry to be removed from the hash list table.
- * @return
- *   0 on entry removed, 1 on entry still referenced.
- */
-int mlx5_hlist_unregister(struct mlx5_hlist *h, struct mlx5_hlist_entry *entry);
-
-/**
- * Destroy the hash list table, all the entries already inserted into the lists
- * will be handled by the callback function provided by the user (including
- * free if needed) before the table is freed.
- *
- * @param h
- *   Pointer to the hast list table.
- */
-void mlx5_hlist_destroy(struct mlx5_hlist *h);
-
 /************************ cache list *****************************/
 
 /** Maximum size of string for naming. */
-- 
2.25.1

[dpdk-dev] [PATCH v3 11/15] common/mlx5: share get ib device match function

[Matan Azrad]

From: Shiri Kuzin <shirik@nvidia.com>

The get_ib_device_match function iterates over the list of ib devices
returned by the get_device_list glue function and returns the ib device
matching the provided address.

Since this function is in use by several drivers, in this patch we
share the function in common part.

Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/linux/mlx5_common_os.c | 28 +++++++++++++++++++
 drivers/common/mlx5/linux/mlx5_common_os.h |  5 ++++
 drivers/common/mlx5/mlx5_common.h          |  1 +
 drivers/common/mlx5/version.map            |  1 +
 drivers/compress/mlx5/mlx5_compress.c      | 30 +-------------------
 drivers/regex/mlx5/mlx5_regex.c            | 30 ++------------------
 drivers/vdpa/mlx5/mlx5_vdpa.c              | 32 ++--------------------
 7 files changed, 40 insertions(+), 87 deletions(-)

diff --git a/drivers/common/mlx5/linux/mlx5_common_os.c b/drivers/common/mlx5/linux/mlx5_common_os.c
index fba8245b8b..ea0b71e425 100644
--- a/drivers/common/mlx5/linux/mlx5_common_os.c
+++ b/drivers/common/mlx5/linux/mlx5_common_os.c
@@ -16,6 +16,7 @@
 
 #include "mlx5_common.h"
 #include "mlx5_common_log.h"
+#include "mlx5_common_os.h"
 #include "mlx5_glue.h"
 
 #ifdef MLX5_GLUE
@@ -423,3 +424,30 @@ mlx5_glue_constructor(void)
 	mlx5_glue = NULL;
 }
 
+struct ibv_device *
+mlx5_os_get_ibv_device(struct rte_pci_addr *addr)
+{
+	int n;
+	struct ibv_device **ibv_list = mlx5_glue->get_device_list(&n);
+	struct ibv_device *ibv_match = NULL;
+
+	if (ibv_list == NULL) {
+		rte_errno = ENOSYS;
+		return NULL;
+	}
+	while (n-- > 0) {
+		struct rte_pci_addr paddr;
+
+		DRV_LOG(DEBUG, "Checking device \"%s\"..", ibv_list[n]->name);
+		if (mlx5_dev_to_pci_addr(ibv_list[n]->ibdev_path, &paddr) != 0)
+			continue;
+		if (rte_pci_addr_cmp(addr, &paddr) != 0)
+			continue;
+		ibv_match = ibv_list[n];
+		break;
+	}
+	if (ibv_match == NULL)
+		rte_errno = ENOENT;
+	mlx5_glue->free_device_list(ibv_list);
+	return ibv_match;
+}
diff --git a/drivers/common/mlx5/linux/mlx5_common_os.h b/drivers/common/mlx5/linux/mlx5_common_os.h
index d1c7e3dce0..72d6bf828b 100644
--- a/drivers/common/mlx5/linux/mlx5_common_os.h
+++ b/drivers/common/mlx5/linux/mlx5_common_os.h
@@ -288,4 +288,9 @@ mlx5_os_free(void *addr)
 {
 	free(addr);
 }
+
+__rte_internal
+struct ibv_device *
+mlx5_os_get_ibv_device(struct rte_pci_addr *addr);
+
 #endif /* RTE_PMD_MLX5_COMMON_OS_H_ */
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index f3c6beb23b..1fbefe0fa6 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -18,6 +18,7 @@
 
 #include "mlx5_prm.h"
 #include "mlx5_devx_cmds.h"
+#include "mlx5_common_os.h"
 
 /* Reported driver name. */
 #define MLX5_PCI_DRIVER_NAME "mlx5_pci"
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index 40b0d713ac..5706c6bbef 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -126,6 +126,7 @@ INTERNAL {
 	mlx5_os_alloc_pd;
 	mlx5_os_dealloc_pd;
 	mlx5_os_dereg_mr;
+	mlx5_os_get_ibv_device; # WINDOWS_NO_EXPORT
 	mlx5_os_reg_mr;
 	mlx5_os_umem_dereg;
 	mlx5_os_umem_reg;
diff --git a/drivers/compress/mlx5/mlx5_compress.c b/drivers/compress/mlx5/mlx5_compress.c
index ec3c237512..0581dee505 100644
--- a/drivers/compress/mlx5/mlx5_compress.c
+++ b/drivers/compress/mlx5/mlx5_compress.c
@@ -647,34 +647,6 @@ mlx5_compress_dequeue_burst(void *queue_pair, struct rte_comp_op **ops,
 	return i;
 }
 
-static struct ibv_device *
-mlx5_compress_get_ib_device_match(struct rte_pci_addr *addr)
-{
-	int n;
-	struct ibv_device **ibv_list = mlx5_glue->get_device_list(&n);
-	struct ibv_device *ibv_match = NULL;
-
-	if (ibv_list == NULL) {
-		rte_errno = ENOSYS;
-		return NULL;
-	}
-	while (n-- > 0) {
-		struct rte_pci_addr paddr;
-
-		DRV_LOG(DEBUG, "Checking device \"%s\"..", ibv_list[n]->name);
-		if (mlx5_dev_to_pci_addr(ibv_list[n]->ibdev_path, &paddr) != 0)
-			continue;
-		if (rte_pci_addr_cmp(addr, &paddr) != 0)
-			continue;
-		ibv_match = ibv_list[n];
-		break;
-	}
-	if (ibv_match == NULL)
-		rte_errno = ENOENT;
-	mlx5_glue->free_device_list(ibv_list);
-	return ibv_match;
-}
-
 static void
 mlx5_compress_hw_global_release(struct mlx5_compress_priv *priv)
 {
@@ -774,7 +746,7 @@ mlx5_compress_pci_probe(struct rte_pci_driver *pci_drv,
 		rte_errno = ENOTSUP;
 		return -rte_errno;
 	}
-	ibv = mlx5_compress_get_ib_device_match(&pci_dev->addr);
+	ibv = mlx5_os_get_ibv_device(&pci_dev->addr);
 	if (ibv == NULL) {
 		DRV_LOG(ERR, "No matching IB device for PCI slot "
 			PCI_PRI_FMT ".", pci_dev->addr.domain,
diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
index 82c485e50c..dd96436d42 100644
--- a/drivers/regex/mlx5/mlx5_regex.c
+++ b/drivers/regex/mlx5/mlx5_regex.c
@@ -11,6 +11,7 @@
 #include <rte_regexdev_driver.h>
 
 #include <mlx5_common_pci.h>
+#include <mlx5_common.h>
 #include <mlx5_glue.h>
 #include <mlx5_devx_cmds.h>
 #include <mlx5_prm.h>
@@ -52,33 +53,6 @@ mlx5_regex_close(struct rte_regexdev *dev __rte_unused)
 	return 0;
 }
 
-static struct ibv_device *
-mlx5_regex_get_ib_device_match(struct rte_pci_addr *addr)
-{
-	int n;
-	struct ibv_device **ibv_list = mlx5_glue->get_device_list(&n);
-	struct ibv_device *ibv_match = NULL;
-
-	if (!ibv_list) {
-		rte_errno = ENOSYS;
-		return NULL;
-	}
-	while (n-- > 0) {
-		struct rte_pci_addr pci_addr;
-
-		DRV_LOG(DEBUG, "Checking device \"%s\"..", ibv_list[n]->name);
-		if (mlx5_dev_to_pci_addr(ibv_list[n]->ibdev_path, &pci_addr))
-			continue;
-		if (rte_pci_addr_cmp(addr, &pci_addr))
-			continue;
-		ibv_match = ibv_list[n];
-		break;
-	}
-	if (!ibv_match)
-		rte_errno = ENOENT;
-	mlx5_glue->free_device_list(ibv_list);
-	return ibv_match;
-}
 static int
 mlx5_regex_engines_status(struct ibv_context *ctx, int num_engines)
 {
@@ -121,7 +95,7 @@ mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	int ret;
 	uint32_t val;
 
-	ibv = mlx5_regex_get_ib_device_match(&pci_dev->addr);
+	ibv = mlx5_os_get_ibv_device(&pci_dev->addr);
 	if (!ibv) {
 		DRV_LOG(ERR, "No matching IB device for PCI slot "
 			PCI_PRI_FMT ".", pci_dev->addr.domain,
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 898e50f807..17753ae63f 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -472,34 +472,6 @@ static struct rte_vdpa_dev_ops mlx5_vdpa_ops = {
 	.reset_stats = mlx5_vdpa_reset_stats,
 };
 
-static struct ibv_device *
-mlx5_vdpa_get_ib_device_match(struct rte_pci_addr *addr)
-{
-	int n;
-	struct ibv_device **ibv_list = mlx5_glue->get_device_list(&n);
-	struct ibv_device *ibv_match = NULL;
-
-	if (!ibv_list) {
-		rte_errno = ENOSYS;
-		return NULL;
-	}
-	while (n-- > 0) {
-		struct rte_pci_addr pci_addr;
-
-		DRV_LOG(DEBUG, "Checking device \"%s\"..", ibv_list[n]->name);
-		if (mlx5_dev_to_pci_addr(ibv_list[n]->ibdev_path, &pci_addr))
-			continue;
-		if (rte_pci_addr_cmp(addr, &pci_addr))
-			continue;
-		ibv_match = ibv_list[n];
-		break;
-	}
-	if (!ibv_match)
-		rte_errno = ENOENT;
-	mlx5_glue->free_device_list(ibv_list);
-	return ibv_match;
-}
-
 /* Try to disable ROCE by Netlink\Devlink. */
 static int
 mlx5_vdpa_nl_roce_disable(const char *addr)
@@ -595,7 +567,7 @@ mlx5_vdpa_roce_disable(struct rte_pci_addr *addr, struct ibv_device **ibv)
 		struct ibv_device *ibv_new;
 
 		for (r = MLX5_VDPA_MAX_RETRIES; r; r--) {
-			ibv_new = mlx5_vdpa_get_ib_device_match(addr);
+			ibv_new = mlx5_os_get_ibv_device(addr);
 			if (ibv_new) {
 				*ibv = ibv_new;
 				return 0;
@@ -698,7 +670,7 @@ mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	struct mlx5_hca_attr attr;
 	int ret;
 
-	ibv = mlx5_vdpa_get_ib_device_match(&pci_dev->addr);
+	ibv = mlx5_os_get_ibv_device(&pci_dev->addr);
 	if (!ibv) {
 		DRV_LOG(ERR, "No matching IB device for PCI slot "
 			PCI_PRI_FMT ".", pci_dev->addr.domain,
-- 
2.25.1

[dpdk-dev] [PATCH v3 12/15] common/mlx5: support general obj CREDENTIAL create

[Matan Azrad]

From: Dekel Peled <dekelp@nvidia.com>

CREDENTIAL object is used for any crypto operation in wrapped mode.

This patch add support of CREDENTIAL object create operation.
Add reading of CREDENTIAL support capability.
Add function to create general object type CREDENTIAL, using DevX API.

Signed-off-by: Dekel Peled <dekelp@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/mlx5_devx_cmds.c | 53 +++++++++++++++++++++++++++-
 drivers/common/mlx5/mlx5_devx_cmds.h | 17 +++++++--
 drivers/common/mlx5/mlx5_prm.h       | 23 ++++++++++++
 drivers/common/mlx5/version.map      |  1 +
 4 files changed, 91 insertions(+), 3 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c
index c0061741e8..c0a0853c3a 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.c
+++ b/drivers/common/mlx5/mlx5_devx_cmds.c
@@ -754,6 +754,8 @@ mlx5_devx_cmd_query_hca_attr(void *ctx,
 		       MLX5_GENERAL_OBJ_TYPES_CAP_DEK);
 	attr->import_kek = !!(general_obj_types_supported &
 			      MLX5_GENERAL_OBJ_TYPES_CAP_IMPORT_KEK);
+	attr->credential = !!(general_obj_types_supported &
+			      MLX5_GENERAL_OBJ_TYPES_CAP_CREDENTIAL);
 	attr->crypto_login = !!(general_obj_types_supported &
 				MLX5_GENERAL_OBJ_TYPES_CAP_CRYPTO_LOGIN);
 	/* Add reading of other GENERAL_OBJ_TYPES_CAP bits above this line. */
@@ -2512,6 +2514,55 @@ mlx5_devx_cmd_create_import_kek_obj(void *ctx,
 	return import_kek_obj;
 }
 
+/**
+ * Create general object of type CREDENTIAL using DevX API.
+ *
+ * @param[in] ctx
+ *   Context returned from mlx5 open_device() glue function.
+ * @param [in] attr
+ *   Pointer to CREDENTIAL attributes structure.
+ *
+ * @return
+ *   The DevX object created, NULL otherwise and rte_errno is set.
+ */
+struct mlx5_devx_obj *
+mlx5_devx_cmd_create_credential_obj(void *ctx,
+				    struct mlx5_devx_credential_attr *attr)
+{
+	uint32_t in[MLX5_ST_SZ_DW(create_credential_in)] = {0};
+	uint32_t out[MLX5_ST_SZ_DW(general_obj_out_cmd_hdr)] = {0};
+	struct mlx5_devx_obj *credential_obj = NULL;
+	void *ptr = NULL, *credential_addr = NULL;
+
+	credential_obj = mlx5_malloc(MLX5_MEM_ZERO, sizeof(*credential_obj),
+				     0, SOCKET_ID_ANY);
+	if (credential_obj == NULL) {
+		DRV_LOG(ERR, "Failed to allocate CREDENTIAL object data");
+		rte_errno = ENOMEM;
+		return NULL;
+	}
+	ptr = MLX5_ADDR_OF(create_credential_in, in, hdr);
+	MLX5_SET(general_obj_in_cmd_hdr, ptr, opcode,
+		 MLX5_CMD_OP_CREATE_GENERAL_OBJECT);
+	MLX5_SET(general_obj_in_cmd_hdr, ptr, obj_type,
+		 MLX5_GENERAL_OBJ_TYPE_CREDENTIAL);
+	ptr = MLX5_ADDR_OF(create_credential_in, in, credential);
+	MLX5_SET(credential, ptr, credential_role, attr->credential_role);
+	credential_addr = MLX5_ADDR_OF(credential, ptr, credential);
+	memcpy(credential_addr, (void *)(attr->credential),
+	       MLX5_CRYPTO_CREDENTIAL_SIZE);
+	credential_obj->obj = mlx5_glue->devx_obj_create(ctx, in, sizeof(in),
+							 out, sizeof(out));
+	if (credential_obj->obj == NULL) {
+		rte_errno = errno;
+		DRV_LOG(ERR, "Failed to create CREDENTIAL object using DevX.");
+		mlx5_free(credential_obj);
+		return NULL;
+	}
+	credential_obj->id = MLX5_GET(general_obj_out_cmd_hdr, out, obj_id);
+	return credential_obj;
+}
+
 /**
  * Create general object of type CRYPTO_LOGIN using DevX API.
  *
@@ -2551,7 +2602,7 @@ mlx5_devx_cmd_create_crypto_login_obj(void *ctx,
 		 attr->session_import_kek_ptr);
 	credential_addr = MLX5_ADDR_OF(crypto_login, ptr, credential);
 	memcpy(credential_addr, (void *)(attr->credential),
-	       MLX5_CRYPTO_LOGIN_CREDENTIAL_SIZE);
+	       MLX5_CRYPTO_CREDENTIAL_SIZE);
 	crypto_login_obj->obj = mlx5_glue->devx_obj_create(ctx, in, sizeof(in),
 							   out, sizeof(out));
 	if (crypto_login_obj->obj == NULL) {
diff --git a/drivers/common/mlx5/mlx5_devx_cmds.h b/drivers/common/mlx5/mlx5_devx_cmds.h
index 709e28bfba..811e7a1462 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.h
+++ b/drivers/common/mlx5/mlx5_devx_cmds.h
@@ -143,6 +143,7 @@ struct mlx5_hca_attr {
 	uint32_t aes_xts:1; /* AES-XTS crypto is supported. */
 	uint32_t dek:1; /* General obj type DEK is supported. */
 	uint32_t import_kek:1; /* General obj type IMPORT_KEK supported. */
+	uint32_t credential:1; /* General obj type CREDENTIAL supported. */
 	uint32_t crypto_login:1; /* General obj type CRYPTO_LOGIN supported. */
 	uint32_t regexp_num_of_engines;
 	uint32_t log_max_ft_sampler_num:8;
@@ -459,13 +460,20 @@ struct mlx5_devx_import_kek_attr {
 	uint8_t key[MLX5_CRYPTO_KEY_MAX_SIZE];
 };
 
-#define MLX5_CRYPTO_LOGIN_CREDENTIAL_SIZE	48
+#define MLX5_CRYPTO_CREDENTIAL_SIZE	48
+
+struct mlx5_devx_credential_attr {
+	uint64_t modify_field_select;
+	uint32_t state:8;
+	uint32_t credential_role:8;
+	uint8_t credential[MLX5_CRYPTO_CREDENTIAL_SIZE];
+};
 
 struct mlx5_devx_crypto_login_attr {
 	uint64_t modify_field_select;
 	uint32_t credential_pointer:24;
 	uint32_t session_import_kek_ptr:24;
-	uint8_t credential[MLX5_CRYPTO_LOGIN_CREDENTIAL_SIZE];
+	uint8_t credential[MLX5_CRYPTO_CREDENTIAL_SIZE];
 };
 
 /* mlx5_devx_cmds.c */
@@ -629,6 +637,11 @@ struct mlx5_devx_obj *
 mlx5_devx_cmd_create_import_kek_obj(void *ctx,
 				    struct mlx5_devx_import_kek_attr *attr);
 
+__rte_internal
+struct mlx5_devx_obj *
+mlx5_devx_cmd_create_credential_obj(void *ctx,
+				    struct mlx5_devx_credential_attr *attr);
+
 __rte_internal
 struct mlx5_devx_obj *
 mlx5_devx_cmd_create_crypto_login_obj(void *ctx,
diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h
index a9dcbfa63c..432c8fdb63 100644
--- a/drivers/common/mlx5/mlx5_prm.h
+++ b/drivers/common/mlx5/mlx5_prm.h
@@ -1187,6 +1187,8 @@ enum {
 			(1ULL << MLX5_GENERAL_OBJ_TYPE_DEK)
 #define MLX5_GENERAL_OBJ_TYPES_CAP_IMPORT_KEK \
 			(1ULL << MLX5_GENERAL_OBJ_TYPE_IMPORT_KEK)
+#define MLX5_GENERAL_OBJ_TYPES_CAP_CREDENTIAL \
+			(1ULL << MLX5_GENERAL_OBJ_TYPE_CREDENTIAL)
 #define MLX5_GENERAL_OBJ_TYPES_CAP_CRYPTO_LOGIN \
 			(1ULL << MLX5_GENERAL_OBJ_TYPE_CRYPTO_LOGIN)
 
@@ -2490,6 +2492,7 @@ enum {
 	MLX5_GENERAL_OBJ_TYPE_VIRTQ = 0x000d,
 	MLX5_GENERAL_OBJ_TYPE_VIRTIO_Q_COUNTERS = 0x001c,
 	MLX5_GENERAL_OBJ_TYPE_IMPORT_KEK = 0x001d,
+	MLX5_GENERAL_OBJ_TYPE_CREDENTIAL = 0x001e,
 	MLX5_GENERAL_OBJ_TYPE_CRYPTO_LOGIN = 0x001f,
 	MLX5_GENERAL_OBJ_TYPE_FLEX_PARSE_GRAPH = 0x0022,
 	MLX5_GENERAL_OBJ_TYPE_FLOW_METER_ASO = 0x0024,
@@ -2603,6 +2606,26 @@ struct mlx5_ifc_create_import_kek_in_bits {
 	struct mlx5_ifc_import_kek_bits import_kek;
 };
 
+enum {
+	MLX5_CREDENTIAL_ROLE_OFFICER = 0x0,
+	MLX5_CREDENTIAL_ROLE_USER = 0x1,
+};
+
+struct mlx5_ifc_credential_bits {
+	u8 modify_field_select[0x40];
+	u8 state[0x8];
+	u8 reserved_at_48[0x10];
+	u8 credential_role[0x8];
+	u8 reserved_at_60[0x1a0];
+	u8 credential[0x180];
+	u8 reserved_at_380[0x480];
+};
+
+struct mlx5_ifc_create_credential_in_bits {
+	struct mlx5_ifc_general_obj_in_cmd_hdr_bits hdr;
+	struct mlx5_ifc_credential_bits credential;
+};
+
 struct mlx5_ifc_crypto_login_bits {
 	u8 modify_field_select[0x40];
 	u8 reserved_at_40[0x48];
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index 5706c6bbef..04b2179b2c 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -16,6 +16,7 @@ INTERNAL {
 
 	mlx5_devx_cmd_alloc_pd;
 	mlx5_devx_cmd_create_cq;
+	mlx5_devx_cmd_create_credential_obj;
 	mlx5_devx_cmd_create_crypto_login_obj;
 	mlx5_devx_cmd_create_dek_obj;
 	mlx5_devx_cmd_create_flex_parser;
-- 
2.25.1

[dpdk-dev] [PATCH v3 13/15] common/mlx5: add crypto register structs and defs

[Matan Azrad]

From: Dekel Peled <dekelp@nvidia.com>

Encryption key management requires use of several related registers.
This patch adds the relevant structs and values, according to PRM
definitions.

Signed-off-by: Dekel Peled <dekelp@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/mlx5_prm.h | 41 ++++++++++++++++++++++++++++++++++
 1 file changed, 41 insertions(+)

diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h
index 432c8fdb63..c2cd2d9f70 100644
--- a/drivers/common/mlx5/mlx5_prm.h
+++ b/drivers/common/mlx5/mlx5_prm.h
@@ -3307,6 +3307,10 @@ enum {
 
 enum {
 	MLX5_REGISTER_ID_MTUTC  = 0x9055,
+	MLX5_CRYPTO_OPERATIONAL_REGISTER_ID = 0xC002,
+	MLX5_CRYPTO_COMMISSIONING_REGISTER_ID = 0xC003,
+	MLX5_IMPORT_KEK_HANDLE_REGISTER_ID = 0xC004,
+	MLX5_CREDENTIAL_HANDLE_REGISTER_ID = 0xC005,
 };
 
 struct mlx5_ifc_register_mtutc_bits {
@@ -3324,6 +3328,43 @@ struct mlx5_ifc_register_mtutc_bits {
 #define MLX5_MTUTC_TIMESTAMP_MODE_INTERNAL_TIMER 0
 #define MLX5_MTUTC_TIMESTAMP_MODE_REAL_TIME 1
 
+struct mlx5_ifc_crypto_operational_register_bits {
+	u8 wrapped_crypto_operational[0x1];
+	u8 reserved_at_1[0x1b];
+	u8 kek_size[0x4];
+	u8 reserved_at_20[0x20];
+	u8 credential[0x140];
+	u8 kek[0x100];
+	u8 reserved_at_280[0x180];
+};
+
+struct mlx5_ifc_crypto_commissioning_register_bits {
+	u8 token[0x1]; /* TODO: add size after PRM update */
+};
+
+struct mlx5_ifc_import_kek_handle_register_bits {
+	struct mlx5_ifc_crypto_login_bits crypto_login_object;
+	struct mlx5_ifc_import_kek_bits import_kek_object;
+	u8 reserved_at_200[0x4];
+	u8 write_operation[0x4];
+	u8 import_kek_id[0x18];
+	u8 reserved_at_220[0xe0];
+};
+
+struct mlx5_ifc_credential_handle_register_bits {
+	struct mlx5_ifc_crypto_login_bits crypto_login_object;
+	struct mlx5_ifc_credential_bits credential_object;
+	u8 reserved_at_200[0x4];
+	u8 write_operation[0x4];
+	u8 credential_id[0x18];
+	u8 reserved_at_220[0xe0];
+};
+
+enum {
+	MLX5_REGISTER_ADD_OPERATION = 0x1,
+	MLX5_REGISTER_DELETE_OPERATION = 0x2,
+};
+
 struct mlx5_ifc_parse_graph_arc_bits {
 	u8 start_inner_tunnel[0x1];
 	u8 reserved_at_1[0x7];
-- 
2.25.1

[dpdk-dev] [PATCH v3 14/15] common/mlx5: support register write access

[Matan Azrad]

From: Dekel Peled <dekelp@nvidia.com>

This patch adds support of write operation to NIC registers.

Signed-off-by: Dekel Peled <dekelp@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/mlx5_devx_cmds.c | 67 +++++++++++++++++++++++++++-
 drivers/common/mlx5/mlx5_devx_cmds.h |  4 ++
 drivers/common/mlx5/version.map      |  1 +
 3 files changed, 70 insertions(+), 2 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c
index c0a0853c3a..0b421933ce 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.c
+++ b/drivers/common/mlx5/mlx5_devx_cmds.c
@@ -12,7 +12,6 @@
 #include "mlx5_common_log.h"
 #include "mlx5_malloc.h"
 
-
 /**
  * Perform read access to the registers. Reads data from register
  * and writes ones to the specified buffer.
@@ -61,7 +60,7 @@ mlx5_devx_cmd_register_read(void *ctx, uint16_t reg_id, uint32_t arg,
 	if (status) {
 		int syndrome = MLX5_GET(access_register_out, out, syndrome);
 
-		DRV_LOG(DEBUG, "Failed to access NIC register 0x%X, "
+		DRV_LOG(DEBUG, "Failed to read access NIC register 0x%X, "
 			       "status %x, syndrome = %x",
 			       reg_id, status, syndrome);
 		return -1;
@@ -74,6 +73,70 @@ mlx5_devx_cmd_register_read(void *ctx, uint16_t reg_id, uint32_t arg,
 	return rc;
 }
 
+/**
+ * Perform write access to the registers.
+ *
+ * @param[in] ctx
+ *   Context returned from mlx5 open_device() glue function.
+ * @param[in] reg_id
+ *   Register identifier according to the PRM.
+ * @param[in] arg
+ *   Register access auxiliary parameter according to the PRM.
+ * @param[out] data
+ *   Pointer to the buffer containing data to write.
+ * @param[in] dw_cnt
+ *   Buffer size in double words (32bit units).
+ *
+ * @return
+ *   0 on success, a negative value otherwise.
+ */
+int
+mlx5_devx_cmd_register_write(void *ctx, uint16_t reg_id, uint32_t arg,
+			     uint32_t *data, uint32_t dw_cnt)
+{
+	uint32_t in[MLX5_ST_SZ_DW(access_register_in) +
+		    MLX5_ACCESS_REGISTER_DATA_DWORD_MAX] = {0};
+	uint32_t out[MLX5_ST_SZ_DW(access_register_out)] = {0};
+	int status, rc;
+	void *ptr;
+
+	MLX5_ASSERT(data && dw_cnt);
+	MLX5_ASSERT(dw_cnt <= MLX5_ACCESS_REGISTER_DATA_DWORD_MAX);
+	if (dw_cnt > MLX5_ACCESS_REGISTER_DATA_DWORD_MAX) {
+		DRV_LOG(ERR, "Data to write exceeds max size");
+		return -1;
+	}
+	MLX5_SET(access_register_in, in, opcode,
+		 MLX5_CMD_OP_ACCESS_REGISTER_USER);
+	MLX5_SET(access_register_in, in, op_mod,
+		 MLX5_ACCESS_REGISTER_IN_OP_MOD_WRITE);
+	MLX5_SET(access_register_in, in, register_id, reg_id);
+	MLX5_SET(access_register_in, in, argument, arg);
+	ptr = MLX5_ADDR_OF(access_register_in, in, register_data);
+	memcpy(ptr, data, dw_cnt * sizeof(uint32_t));
+	rc = mlx5_glue->devx_general_cmd(ctx, in, sizeof(in), out, sizeof(out));
+
+	rc = mlx5_glue->devx_general_cmd(ctx, in,
+					 MLX5_ST_SZ_BYTES(access_register_in) +
+					 dw_cnt * sizeof(uint32_t),
+					 out, sizeof(out));
+	if (rc)
+		goto error;
+	status = MLX5_GET(access_register_out, out, status);
+	if (status) {
+		int syndrome = MLX5_GET(access_register_out, out, syndrome);
+
+		DRV_LOG(DEBUG, "Failed to write access NIC register 0x%X, "
+			       "status %x, syndrome = %x",
+			       reg_id, status, syndrome);
+		return -1;
+	}
+	return 0;
+error:
+	rc = (rc > 0) ? -rc : rc;
+	return rc;
+}
+
 /**
  * Allocate flow counters via devx interface.
  *
diff --git a/drivers/common/mlx5/mlx5_devx_cmds.h b/drivers/common/mlx5/mlx5_devx_cmds.h
index 811e7a1462..ce570ad28a 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.h
+++ b/drivers/common/mlx5/mlx5_devx_cmds.h
@@ -566,6 +566,10 @@ __rte_internal
 int mlx5_devx_cmd_register_read(void *ctx, uint16_t reg_id,
 				uint32_t arg, uint32_t *data, uint32_t dw_cnt);
 
+__rte_internal
+int mlx5_devx_cmd_register_write(void *ctx, uint16_t reg_id,
+				 uint32_t arg, uint32_t *data, uint32_t dw_cnt);
+
 __rte_internal
 struct mlx5_devx_obj *
 mlx5_devx_cmd_create_geneve_tlv_option(void *ctx,
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index 04b2179b2c..c630696213 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -53,6 +53,7 @@ INTERNAL {
 	mlx5_devx_cmd_queue_counter_alloc; # WINDOWS_NO_EXPORT
 	mlx5_devx_cmd_queue_counter_query; # WINDOWS_NO_EXPORT
 	mlx5_devx_cmd_register_read;
+	mlx5_devx_cmd_register_write;
 	mlx5_devx_cmd_wq_query; # WINDOWS_NO_EXPORT
 
 	mlx5_devx_cq_create;
-- 
2.25.1

[dpdk-dev] [PATCH v3 15/15] common/mlx5: add UMR and RDMA write WQE defines

[Matan Azrad]

From: Suanming Mou <suanmingm@nvidia.com>

This patch adds the struct defining UMR and RDMA write WQEs.

Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Signed-off-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/mlx5_prm.h | 187 +++++++++++++++++++++------------
 1 file changed, 121 insertions(+), 66 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h
index c2cd2d9f70..1ffee5fd56 100644
--- a/drivers/common/mlx5/mlx5_prm.h
+++ b/drivers/common/mlx5/mlx5_prm.h
@@ -412,6 +412,127 @@ struct mlx5_cqe_ts {
 	uint8_t op_own;
 };
 
+struct mlx5_wqe_rseg {
+	uint64_t raddr;
+	uint32_t rkey;
+	uint32_t reserved;
+} __rte_packed;
+
+#define MLX5_UMRC_IF_OFFSET 31u
+#define MLX5_UMRC_KO_OFFSET 16u
+#define MLX5_UMRC_TO_BS_OFFSET 0u
+
+struct mlx5_wqe_umr_cseg {
+	uint32_t if_cf_toe_cq_res;
+	uint32_t ko_to_bs;
+	uint64_t mkey_mask;
+	uint32_t rsvd1[8];
+} __rte_packed;
+
+struct mlx5_wqe_mkey_cseg {
+	uint32_t fr_res_af_sf;
+	uint32_t qpn_mkey;
+	uint32_t reserved2;
+	uint32_t flags_pd;
+	uint64_t start_addr;
+	uint64_t len;
+	uint32_t bsf_octword_size;
+	uint32_t reserved3[4];
+	uint32_t translations_octword_size;
+	uint32_t res4_lps;
+	uint32_t reserved;
+} __rte_packed;
+
+enum {
+	MLX5_BSF_SIZE_16B = 0x0,
+	MLX5_BSF_SIZE_32B = 0x1,
+	MLX5_BSF_SIZE_64B = 0x2,
+	MLX5_BSF_SIZE_128B = 0x3,
+};
+
+enum {
+	MLX5_BSF_P_TYPE_SIGNATURE = 0x0,
+	MLX5_BSF_P_TYPE_CRYPTO = 0x1,
+};
+
+enum {
+	MLX5_ENCRYPTION_ORDER_ENCRYPTED_WIRE_SIGNATURE = 0x0,
+	MLX5_ENCRYPTION_ORDER_ENCRYPTED_MEMORY_SIGNATURE = 0x1,
+	MLX5_ENCRYPTION_ORDER_ENCRYPTED_RAW_WIRE = 0x2,
+	MLX5_ENCRYPTION_ORDER_ENCRYPTED_RAW_MEMORY = 0x3,
+};
+
+enum {
+	MLX5_ENCRYPTION_STANDARD_AES_XTS = 0x0,
+};
+
+enum {
+	MLX5_BLOCK_SIZE_512B	= 0x1,
+	MLX5_BLOCK_SIZE_520B	= 0x2,
+	MLX5_BLOCK_SIZE_4096B	= 0x3,
+	MLX5_BLOCK_SIZE_4160B	= 0x4,
+	MLX5_BLOCK_SIZE_1MB	= 0x5,
+	MLX5_BLOCK_SIZE_4048B	= 0x6,
+};
+
+#define MLX5_BSF_SIZE_OFFSET		30
+#define MLX5_BSF_P_TYPE_OFFSET		24
+#define MLX5_ENCRYPTION_ORDER_OFFSET	16
+#define MLX5_BLOCK_SIZE_OFFSET		24
+
+struct mlx5_wqe_umr_bsf_seg {
+	/*
+	 * bs_bpt_eo_es contains:
+	 * bs	bsf_size		2 bits at MLX5_BSF_SIZE_OFFSET
+	 * bpt	bsf_p_type		2 bits at MLX5_BSF_P_TYPE_OFFSET
+	 * eo	encryption_order	4 bits at MLX5_ENCRYPTION_ORDER_OFFSET
+	 * es	encryption_standard	4 bits at offset 0
+	 */
+	uint32_t bs_bpt_eo_es;
+	uint32_t raw_data_size;
+	/*
+	 * bsp_res contains:
+	 * bsp	crypto_block_size_pointer	8 bits at MLX5_BLOCK_SIZE_OFFSET
+	 * res	reserved 24 bits
+	 */
+	uint32_t bsp_res;
+	uint32_t reserved0;
+	uint8_t xts_initial_tweak[16];
+	/*
+	 * res_dp contains:
+	 * res	reserved 8 bits
+	 * dp	dek_pointer		24 bits at offset 0
+	 */
+	uint32_t res_dp;
+	uint32_t reserved1;
+	uint64_t keytag;
+	uint32_t reserved2[4];
+} __rte_packed;
+
+#ifdef PEDANTIC
+#pragma GCC diagnostic ignored "-Wpedantic"
+#endif
+
+struct mlx5_umr_wqe {
+	struct mlx5_wqe_cseg ctr;
+	struct mlx5_wqe_umr_cseg ucseg;
+	struct mlx5_wqe_mkey_cseg mkc;
+	union {
+		struct mlx5_wqe_dseg kseg[0];
+		struct mlx5_wqe_umr_bsf_seg bsf[0];
+	};
+} __rte_packed;
+
+struct mlx5_rdma_write_wqe {
+	struct mlx5_wqe_cseg ctr;
+	struct mlx5_wqe_rseg rseg;
+	struct mlx5_wqe_dseg dseg[0];
+} __rte_packed;
+
+#ifdef PEDANTIC
+#pragma GCC diagnostic error "-Wpedantic"
+#endif
+
 /* GGA */
 /* MMO metadata segment */
 
@@ -1096,72 +1217,6 @@ struct mlx5_ifc_create_mkey_in_bits {
 	u8 klm_pas_mtt[][0x20];
 };
 
-enum {
-	MLX5_BSF_SIZE_16B = 0x0,
-	MLX5_BSF_SIZE_32B = 0x1,
-	MLX5_BSF_SIZE_64B = 0x2,
-	MLX5_BSF_SIZE_128B = 0x3,
-};
-
-enum {
-	MLX5_BSF_P_TYPE_SIGNATURE = 0x0,
-	MLX5_BSF_P_TYPE_CRYPTO = 0x1,
-};
-
-enum {
-	MLX5_ENCRYPTION_ORDER_ENCRYPTED_WIRE_SIGNATURE = 0x0,
-	MLX5_ENCRYPTION_ORDER_ENCRYPTED_MEMORY_SIGNATURE = 0x1,
-	MLX5_ENCRYPTION_ORDER_ENCRYPTED_RAW_WIRE = 0x2,
-	MLX5_ENCRYPTION_ORDER_ENCRYPTED_RAW_MEMORY = 0x3,
-};
-
-enum {
-	MLX5_ENCRYPTION_STANDARD_AES_XTS = 0x0,
-};
-
-enum {
-	MLX5_BLOCK_SIZE_512B	= 0x1,
-	MLX5_BLOCK_SIZE_520B	= 0x2,
-	MLX5_BLOCK_SIZE_4096B	= 0x3,
-	MLX5_BLOCK_SIZE_4160B	= 0x4,
-	MLX5_BLOCK_SIZE_1MB	= 0x5,
-	MLX5_BLOCK_SIZE_4048B	= 0x6,
-};
-
-#define MLX5_BSF_SIZE_OFFSET		30
-#define MLX5_BSF_P_TYPE_OFFSET		24
-#define MLX5_ENCRYPTION_ORDER_OFFSET	16
-#define MLX5_BLOCK_SIZE_OFFSET		24
-
-struct mlx5_wqe_umr_bsf_seg {
-	/*
-	 * bs_bpt_eo_es contains:
-	 * bs	bsf_size		2 bits at MLX5_BSF_SIZE_OFFSET
-	 * bpt	bsf_p_type		2 bits at MLX5_BSF_P_TYPE_OFFSET
-	 * eo	encryption_order	4 bits at MLX5_ENCRYPTION_ORDER_OFFSET
-	 * es	encryption_standard	4 bits at offset 0
-	 */
-	uint32_t bs_bpt_eo_es;
-	uint32_t raw_data_size;
-	/*
-	 * bsp_res contains:
-	 * bsp	crypto_block_size_pointer	8 bits at MLX5_BLOCK_SIZE_OFFSET
-	 * res	reserved 24 bits
-	 */
-	uint32_t bsp_res;
-	uint32_t reserved0;
-	uint8_t xts_initial_tweak[16];
-	/*
-	 * res_dp contains:
-	 * res	reserved 8 bits
-	 * dp	dek_pointer		24 bits at offset 0
-	 */
-	uint32_t res_dp;
-	uint32_t reserved1;
-	uint64_t keytag;
-	uint32_t reserved2[4];
-} __rte_packed;
-
 enum {
 	MLX5_GET_HCA_CAP_OP_MOD_GENERAL_DEVICE = 0x0 << 1,
 	MLX5_GET_HCA_CAP_OP_MOD_ETHERNET_OFFLOAD_CAPS = 0x1 << 1,
-- 
2.25.1

[dpdk-dev] [PATCH v3 00/15] drivers: introduce mlx5 crypto PMD

[Matan Azrad]

Add a new PMD for Nvidia devices- crypto PMD.
The crypto PMD will be supported on Nvidia ConnectX6 The crypto PMD will add the support of encryption and decryption using the AES-XTS symmetric algorithm.
The crypto PMD requires rdma-core and uses mlx5 DevX.

v2:
Add data-path part.

v3:
Rebase.

Shiri Kuzin (10):
  drivers: introduce mlx5 crypto PMD
  crypto/mlx5: add DEK object management
  crypto/mlx5: support session operations
  crypto/mlx5: add basic operations
  crypto/mlx5: support queue pairs operations
  crypto/mlx5: add dev stop and start operations
  crypto/mlx5: add memory region management
  crypto/mlx5: create login object using DevX
  crypto/mlx5: adjust to the multiple data unit API
  crypto/mlx5: set feature flags and capabilities

Suanming Mou (5):
  crypto/mlx5: add keytag device argument
  crypto/mlx5: add maximum segments device argument
  crypto/mlx5: add WQE set initialization
  crypto/mlx5: add enqueue and dequeue operations
  crypto/mlx5: add statistic get and reset operations

 MAINTAINERS                             |    4 +
 doc/guides/cryptodevs/features/mlx5.ini |   37 +
 doc/guides/cryptodevs/index.rst         |    1 +
 doc/guides/cryptodevs/mlx5.rst          |  152 ++++
 doc/guides/rel_notes/release_21_05.rst  |    5 +
 drivers/common/mlx5/mlx5_common.h       |    1 +
 drivers/common/mlx5/mlx5_common_pci.c   |   14 +
 drivers/common/mlx5/mlx5_common_pci.h   |   21 +-
 drivers/crypto/meson.build              |    1 +
 drivers/crypto/mlx5/meson.build         |   27 +
 drivers/crypto/mlx5/mlx5_crypto.c       | 1092 +++++++++++++++++++++++
 drivers/crypto/mlx5/mlx5_crypto.h       |   92 ++
 drivers/crypto/mlx5/mlx5_crypto_dek.c   |  136 +++
 drivers/crypto/mlx5/mlx5_crypto_utils.h |   19 +
 drivers/crypto/mlx5/version.map         |    3 +
 15 files changed, 1595 insertions(+), 10 deletions(-)
 create mode 100644 doc/guides/cryptodevs/features/mlx5.ini
 create mode 100644 doc/guides/cryptodevs/mlx5.rst
 create mode 100644 drivers/crypto/mlx5/meson.build
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto.c
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto.h
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto_dek.c
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto_utils.h
 create mode 100644 drivers/crypto/mlx5/version.map

-- 
2.25.1

[dpdk-dev] [PATCH v3 01/15] drivers: introduce mlx5 crypto PMD

[Matan Azrad]

From: Shiri Kuzin <shirik@nvidia.com>

Add a new PMD for Nvidia devices- crypto PMD.

The crypto PMD will be supported starting Nvidia ConnectX6 and
BlueField2.

The crypto PMD will add the support of encryption and decryption using
the AES-XTS symmetric algorithm.

The crypto PMD requires rdma-core and uses mlx5 DevX.

This patch adds the PCI probing, basic functions, build files and
log utility.

Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 MAINTAINERS                             |   4 +
 drivers/common/mlx5/mlx5_common.h       |   1 +
 drivers/common/mlx5/mlx5_common_pci.c   |  14 ++
 drivers/common/mlx5/mlx5_common_pci.h   |  21 +-
 drivers/crypto/meson.build              |   1 +
 drivers/crypto/mlx5/meson.build         |  26 +++
 drivers/crypto/mlx5/mlx5_crypto.c       | 272 ++++++++++++++++++++++++
 drivers/crypto/mlx5/mlx5_crypto_utils.h |  19 ++
 drivers/crypto/mlx5/version.map         |   3 +
 9 files changed, 351 insertions(+), 10 deletions(-)
 create mode 100644 drivers/crypto/mlx5/meson.build
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto.c
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto_utils.h
 create mode 100644 drivers/crypto/mlx5/version.map

diff --git a/MAINTAINERS b/MAINTAINERS
index b40d8ae266..165474c91f 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1078,6 +1078,10 @@ F: drivers/crypto/octeontx2/
 F: doc/guides/cryptodevs/octeontx2.rst
 F: doc/guides/cryptodevs/features/octeontx2.ini
 
+Mellanox mlx5
+M: Matan Azrad <matan@nvidia.com>
+F: drivers/crypto/mlx5/
+
 Null Crypto
 M: Declan Doherty <declan.doherty@intel.com>
 F: drivers/crypto/null/
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index 1fbefe0fa6..89aca32305 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -216,6 +216,7 @@ enum mlx5_class {
 	MLX5_CLASS_VDPA = RTE_BIT64(1),
 	MLX5_CLASS_REGEX = RTE_BIT64(2),
 	MLX5_CLASS_COMPRESS = RTE_BIT64(3),
+	MLX5_CLASS_CRYPTO = RTE_BIT64(4),
 };
 
 #define MLX5_DBR_SIZE RTE_CACHE_LINE_SIZE
diff --git a/drivers/common/mlx5/mlx5_common_pci.c b/drivers/common/mlx5/mlx5_common_pci.c
index 3f16cd21cf..8a47afee20 100644
--- a/drivers/common/mlx5/mlx5_common_pci.c
+++ b/drivers/common/mlx5/mlx5_common_pci.c
@@ -31,6 +31,7 @@ static const struct {
 	{ .name = "net", .driver_class = MLX5_CLASS_NET },
 	{ .name = "regex", .driver_class = MLX5_CLASS_REGEX },
 	{ .name = "compress", .driver_class = MLX5_CLASS_COMPRESS },
+	{ .name = "crypto", .driver_class = MLX5_CLASS_CRYPTO },
 };
 
 static const unsigned int mlx5_class_combinations[] = {
@@ -38,13 +39,26 @@ static const unsigned int mlx5_class_combinations[] = {
 	MLX5_CLASS_VDPA,
 	MLX5_CLASS_REGEX,
 	MLX5_CLASS_COMPRESS,
+	MLX5_CLASS_CRYPTO,
 	MLX5_CLASS_NET | MLX5_CLASS_REGEX,
 	MLX5_CLASS_VDPA | MLX5_CLASS_REGEX,
 	MLX5_CLASS_NET | MLX5_CLASS_COMPRESS,
 	MLX5_CLASS_VDPA | MLX5_CLASS_COMPRESS,
 	MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS,
+	MLX5_CLASS_NET | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_VDPA | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_REGEX | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_COMPRESS | MLX5_CLASS_CRYPTO,
 	MLX5_CLASS_NET | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS,
 	MLX5_CLASS_VDPA | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS,
+	MLX5_CLASS_NET | MLX5_CLASS_REGEX | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_VDPA | MLX5_CLASS_REGEX | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_NET | MLX5_CLASS_COMPRESS | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_VDPA | MLX5_CLASS_COMPRESS | MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_NET | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS |
+	MLX5_CLASS_CRYPTO,
+	MLX5_CLASS_VDPA | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS |
+	MLX5_CLASS_CRYPTO,
 	/* New class combination should be added here. */
 };
 
diff --git a/drivers/common/mlx5/mlx5_common_pci.h b/drivers/common/mlx5/mlx5_common_pci.h
index de89bb98bc..cb8d2f5f87 100644
--- a/drivers/common/mlx5/mlx5_common_pci.h
+++ b/drivers/common/mlx5/mlx5_common_pci.h
@@ -9,17 +9,18 @@
  * @file
  *
  * RTE Mellanox PCI Driver Interface
- * Mellanox ConnectX PCI device supports multiple class: net,vdpa,regex and
- * compress devices. This layer enables creating such multiple class of devices
- * on a single PCI device by allowing to bind multiple class specific device
- * driver to attach to mlx5_pci driver.
+ * Mellanox ConnectX PCI device supports multiple class: net,vdpa,regex,compress
+ * and crypto devices. This layer enables creating such multiple class of
+ * devices on a single PCI device by allowing to bind multiple class specific
+ * device driver to attach to mlx5_pci driver.
  *
- * -----------    ------------    -------------    ----------------
- * |   mlx5  |    |   mlx5   |    |   mlx5    |    |     mlx5     |
- * | net pmd |    | vdpa pmd |    | regex pmd |    | compress pmd |
- * -----------    ------------    -------------    ----------------
- *      \              \                    /              /
- *       \              \                  /              /
+ * --------    --------    ---------    ------------    ----------
+ * | mlx5 |    | mlx5 |    | mlx5  |    |   mlx5   |    |  mlx5  |
+ * | net  |    | vdpa |    | regex |    | compress |    | crypto |
+ * | pmd  |    | pmd  |    |  pmd  |    |   pmd    |    |  pmd   |
+ * --------    --------    ---------    ------------    ----------
+ *      \              \         |          /              /
+ *       \              \        |         /              /
  *        \              \_--------------_/              /
  *         \_______________|   mlx5     |_______________/
  *                         | pci common |
diff --git a/drivers/crypto/meson.build b/drivers/crypto/meson.build
index b9fdf9392f..6951607def 100644
--- a/drivers/crypto/meson.build
+++ b/drivers/crypto/meson.build
@@ -15,6 +15,7 @@ drivers = [
         'dpaa_sec',
         'dpaa2_sec',
         'kasumi',
+        'mlx5',
         'mvsam',
         'nitrox',
         'null',
diff --git a/drivers/crypto/mlx5/meson.build b/drivers/crypto/mlx5/meson.build
new file mode 100644
index 0000000000..fd00283665
--- /dev/null
+++ b/drivers/crypto/mlx5/meson.build
@@ -0,0 +1,26 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright 2021 Mellanox Technologies, Ltd
+
+if not is_linux
+	build = false
+	reason = 'only supported on Linux'
+	subdir_done()
+endif
+
+fmt_name = 'mlx5_crypto'
+deps += ['common_mlx5', 'eal', 'cryptodev']
+sources = files(
+	'mlx5_crypto.c',
+)
+cflags_options = [
+	'-std=c11',
+	'-Wno-strict-prototypes',
+	'-D_BSD_SOURCE',
+	'-D_DEFAULT_SOURCE',
+	'-D_XOPEN_SOURCE=600',
+]
+foreach option:cflags_options
+	if cc.has_argument(option)
+		cflags += option
+	endif
+endforeach
diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
new file mode 100644
index 0000000000..ffbce5d68a
--- /dev/null
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -0,0 +1,272 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2021 Mellanox Technologies, Ltd
+ */
+
+#include <rte_malloc.h>
+#include <rte_log.h>
+#include <rte_errno.h>
+#include <rte_pci.h>
+#include <rte_crypto.h>
+#include <rte_cryptodev.h>
+#include <rte_cryptodev_pmd.h>
+
+#include <mlx5_glue.h>
+#include <mlx5_common.h>
+#include <mlx5_common_pci.h>
+#include <mlx5_devx_cmds.h>
+#include <mlx5_common_os.h>
+
+#include "mlx5_crypto_utils.h"
+
+#define MLX5_CRYPTO_DRIVER_NAME mlx5_crypto
+#define MLX5_CRYPTO_LOG_NAME    pmd.crypto.mlx5
+
+struct mlx5_crypto_priv {
+	TAILQ_ENTRY(mlx5_crypto_priv) next;
+	struct ibv_context *ctx; /* Device context. */
+	struct rte_pci_device *pci_dev;
+	struct rte_cryptodev *crypto_dev;
+	void *uar; /* User Access Region. */
+	uint32_t pdn; /* Protection Domain number. */
+	struct ibv_pd *pd;
+};
+
+TAILQ_HEAD(mlx5_crypto_privs, mlx5_crypto_priv) mlx5_crypto_priv_list =
+				TAILQ_HEAD_INITIALIZER(mlx5_crypto_priv_list);
+static pthread_mutex_t priv_list_lock = PTHREAD_MUTEX_INITIALIZER;
+
+int mlx5_crypto_logtype;
+
+uint8_t mlx5_crypto_driver_id;
+
+static const char mlx5_crypto_drv_name[] = RTE_STR(MLX5_CRYPTO_DRIVER_NAME);
+
+static const struct rte_driver mlx5_drv = {
+	.name = mlx5_crypto_drv_name,
+	.alias = mlx5_crypto_drv_name
+};
+
+static struct cryptodev_driver mlx5_cryptodev_driver;
+
+static struct rte_cryptodev_ops mlx5_crypto_ops = {
+	.dev_configure			= NULL,
+	.dev_start			= NULL,
+	.dev_stop			= NULL,
+	.dev_close			= NULL,
+	.dev_infos_get			= NULL,
+	.stats_get			= NULL,
+	.stats_reset			= NULL,
+	.queue_pair_setup		= NULL,
+	.queue_pair_release		= NULL,
+	.sym_session_get_size		= NULL,
+	.sym_session_configure		= NULL,
+	.sym_session_clear		= NULL,
+	.sym_get_raw_dp_ctx_size	= NULL,
+	.sym_configure_raw_dp_ctx	= NULL,
+};
+
+static void
+mlx5_crypto_hw_global_release(struct mlx5_crypto_priv *priv)
+{
+	if (priv->pd != NULL) {
+		claim_zero(mlx5_glue->dealloc_pd(priv->pd));
+		priv->pd = NULL;
+	}
+	if (priv->uar != NULL) {
+		mlx5_glue->devx_free_uar(priv->uar);
+		priv->uar = NULL;
+	}
+}
+
+static int
+mlx5_crypto_pd_create(struct mlx5_crypto_priv *priv)
+{
+#ifdef HAVE_IBV_FLOW_DV_SUPPORT
+	struct mlx5dv_obj obj;
+	struct mlx5dv_pd pd_info;
+	int ret;
+
+	priv->pd = mlx5_glue->alloc_pd(priv->ctx);
+	if (priv->pd == NULL) {
+		DRV_LOG(ERR, "Failed to allocate PD.");
+		return errno ? -errno : -ENOMEM;
+	}
+	obj.pd.in = priv->pd;
+	obj.pd.out = &pd_info;
+	ret = mlx5_glue->dv_init_obj(&obj, MLX5DV_OBJ_PD);
+	if (ret != 0) {
+		DRV_LOG(ERR, "Fail to get PD object info.");
+		mlx5_glue->dealloc_pd(priv->pd);
+		priv->pd = NULL;
+		return -errno;
+	}
+	priv->pdn = pd_info.pdn;
+	return 0;
+#else
+	(void)priv;
+	DRV_LOG(ERR, "Cannot get pdn - no DV support.");
+	return -ENOTSUP;
+#endif /* HAVE_IBV_FLOW_DV_SUPPORT */
+}
+
+static int
+mlx5_crypto_hw_global_prepare(struct mlx5_crypto_priv *priv)
+{
+	if (mlx5_crypto_pd_create(priv) != 0)
+		return -1;
+	priv->uar = mlx5_devx_alloc_uar(priv->ctx, -1);
+	if (priv->uar == NULL || mlx5_os_get_devx_uar_reg_addr(priv->uar) ==
+	    NULL) {
+		rte_errno = errno;
+		claim_zero(mlx5_glue->dealloc_pd(priv->pd));
+		DRV_LOG(ERR, "Failed to allocate UAR.");
+		return -1;
+	}
+	return 0;
+}
+
+/**
+ * DPDK callback to register a PCI device.
+ *
+ * This function spawns crypto device out of a given PCI device.
+ *
+ * @param[in] pci_drv
+ *   PCI driver structure (mlx5_crypto_driver).
+ * @param[in] pci_dev
+ *   PCI device information.
+ *
+ * @return
+ *   0 on success, 1 to skip this driver, a negative errno value otherwise
+ *   and rte_errno is set.
+ */
+static int
+mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
+			struct rte_pci_device *pci_dev)
+{
+	struct ibv_device *ibv;
+	struct rte_cryptodev *crypto_dev;
+	struct ibv_context *ctx;
+	struct mlx5_crypto_priv *priv;
+	struct mlx5_hca_attr attr = { 0 };
+	struct rte_cryptodev_pmd_init_params init_params = {
+		.name = "",
+		.private_data_size = sizeof(struct mlx5_crypto_priv),
+		.socket_id = pci_dev->device.numa_node,
+		.max_nb_queue_pairs =
+				RTE_CRYPTODEV_PMD_DEFAULT_MAX_NB_QUEUE_PAIRS,
+	};
+	RTE_SET_USED(pci_drv);
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		DRV_LOG(ERR, "Non-primary process type is not supported.");
+		rte_errno = ENOTSUP;
+		return -rte_errno;
+	}
+	ibv = mlx5_os_get_ibv_device(&pci_dev->addr);
+	if (ibv == NULL) {
+		DRV_LOG(ERR, "No matching IB device for PCI slot "
+			PCI_PRI_FMT ".", pci_dev->addr.domain,
+			pci_dev->addr.bus, pci_dev->addr.devid,
+			pci_dev->addr.function);
+		return -rte_errno;
+	}
+	DRV_LOG(INFO, "PCI information matches for device \"%s\".", ibv->name);
+	ctx = mlx5_glue->dv_open_device(ibv);
+	if (ctx == NULL) {
+		DRV_LOG(ERR, "Failed to open IB device \"%s\".", ibv->name);
+		rte_errno = ENODEV;
+		return -rte_errno;
+	}
+	if (mlx5_devx_cmd_query_hca_attr(ctx, &attr) != 0 ||
+	    attr.crypto == 0 || attr.aes_xts == 0) {
+		DRV_LOG(ERR, "Not enough capabilities to support crypto "
+			"operations, maybe old FW/OFED version?");
+		claim_zero(mlx5_glue->close_device(ctx));
+		rte_errno = ENOTSUP;
+		return -ENOTSUP;
+	}
+	crypto_dev = rte_cryptodev_pmd_create(ibv->name, &pci_dev->device,
+					&init_params);
+	if (crypto_dev == NULL) {
+		DRV_LOG(ERR, "Failed to create device \"%s\".", ibv->name);
+		claim_zero(mlx5_glue->close_device(ctx));
+		return -ENODEV;
+	}
+	DRV_LOG(INFO,
+		"Crypto device %s was created successfully.", ibv->name);
+	crypto_dev->dev_ops = &mlx5_crypto_ops;
+	crypto_dev->dequeue_burst = NULL;
+	crypto_dev->enqueue_burst = NULL;
+	crypto_dev->feature_flags = RTE_CRYPTODEV_FF_HW_ACCELERATED;
+	crypto_dev->driver_id = mlx5_crypto_driver_id;
+	priv = crypto_dev->data->dev_private;
+	priv->ctx = ctx;
+	priv->pci_dev = pci_dev;
+	priv->crypto_dev = crypto_dev;
+	if (mlx5_crypto_hw_global_prepare(priv) != 0) {
+		rte_cryptodev_pmd_destroy(priv->crypto_dev);
+		claim_zero(mlx5_glue->close_device(priv->ctx));
+		return -1;
+	}
+	pthread_mutex_lock(&priv_list_lock);
+	TAILQ_INSERT_TAIL(&mlx5_crypto_priv_list, priv, next);
+	pthread_mutex_unlock(&priv_list_lock);
+	return 0;
+}
+
+static int
+mlx5_crypto_pci_remove(struct rte_pci_device *pdev)
+{
+	struct mlx5_crypto_priv *priv = NULL;
+
+	pthread_mutex_lock(&priv_list_lock);
+	TAILQ_FOREACH(priv, &mlx5_crypto_priv_list, next)
+		if (rte_pci_addr_cmp(&priv->pci_dev->addr, &pdev->addr) != 0)
+			break;
+	if (priv)
+		TAILQ_REMOVE(&mlx5_crypto_priv_list, priv, next);
+	pthread_mutex_unlock(&priv_list_lock);
+	if (priv) {
+		mlx5_crypto_hw_global_release(priv);
+		rte_cryptodev_pmd_destroy(priv->crypto_dev);
+		claim_zero(mlx5_glue->close_device(priv->ctx));
+	}
+	return 0;
+}
+
+static const struct rte_pci_id mlx5_crypto_pci_id_map[] = {
+		{
+			RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
+					PCI_DEVICE_ID_MELLANOX_CONNECTX6)
+		},
+		{
+			.vendor_id = 0
+		}
+	};
+
+static struct mlx5_pci_driver mlx5_crypto_driver = {
+	.driver_class = MLX5_CLASS_CRYPTO,
+	.pci_driver = {
+		.driver = {
+			.name = RTE_STR(MLX5_CRYPTO_DRIVER_NAME),
+		},
+		.id_table = mlx5_crypto_pci_id_map,
+		.probe = mlx5_crypto_pci_probe,
+		.remove = mlx5_crypto_pci_remove,
+		.drv_flags = 0,
+	},
+};
+
+RTE_INIT(rte_mlx5_crypto_init)
+{
+	mlx5_common_init();
+	if (mlx5_glue != NULL)
+		mlx5_pci_driver_register(&mlx5_crypto_driver);
+}
+
+RTE_PMD_REGISTER_CRYPTO_DRIVER(mlx5_cryptodev_driver, mlx5_drv,
+			       mlx5_crypto_driver_id);
+
+RTE_LOG_REGISTER(mlx5_crypto_logtype, MLX5_CRYPTO_LOG_NAME, NOTICE)
+RTE_PMD_EXPORT_NAME(MLX5_CRYPTO_DRIVER_NAME, __COUNTER__);
+RTE_PMD_REGISTER_PCI_TABLE(MLX5_CRYPTO_DRIVER_NAME, mlx5_crypto_pci_id_map);
+RTE_PMD_REGISTER_KMOD_DEP(MLX5_CRYPTO_DRIVER_NAME, "* ib_uverbs & mlx5_core & mlx5_ib");
diff --git a/drivers/crypto/mlx5/mlx5_crypto_utils.h b/drivers/crypto/mlx5/mlx5_crypto_utils.h
new file mode 100644
index 0000000000..cef4b07a36
--- /dev/null
+++ b/drivers/crypto/mlx5/mlx5_crypto_utils.h
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2021 Mellanox Technologies, Ltd
+ */
+
+#ifndef RTE_PMD_MLX5_CRYPTO_UTILS_H_
+#define RTE_PMD_MLX5_CRYPTO_UTILS_H_
+
+#include <mlx5_common.h>
+
+extern int mlx5_crypto_logtype;
+
+#define MLX5_CRYPTO_LOG_PREFIX "mlx5_crypto"
+/* Generic printf()-like logging macro with automatic line feed. */
+#define DRV_LOG(level, ...) \
+	PMD_DRV_LOG_(level, mlx5_crypto_logtype, MLX5_CRYPTO_LOG_PREFIX, \
+		__VA_ARGS__ PMD_DRV_LOG_STRIP PMD_DRV_LOG_OPAREN, \
+		PMD_DRV_LOG_CPAREN)
+
+#endif /* RTE_PMD_MLX5_CRYPTO_UTILS_H_ */
diff --git a/drivers/crypto/mlx5/version.map b/drivers/crypto/mlx5/version.map
new file mode 100644
index 0000000000..4a76d1d52d
--- /dev/null
+++ b/drivers/crypto/mlx5/version.map
@@ -0,0 +1,3 @@
+DPDK_21 {
+	local: *;
+};
-- 
2.25.1

[dpdk-dev] [PATCH v3 02/15] crypto/mlx5: add DEK object management

[Matan Azrad]

From: Shiri Kuzin <shirik@nvidia.com>

A DEK(Data encryption Key) is an mlx5 HW object which represents the
cipher algorithm key.
The DEKs are used during data encryption/decryption operations.

In symmetric algorithms like AES-STS, we use the same DEK for both
encryption and decryption.

Use the mlx5 hash-list tool to manage the DEK objects in the PMD.

Provide the compare, create and destroy functions to manage DEKs in
hash-list and introduce an internal API to setup and unset the DEK
management and to prepare and destroy specific DEK object.

The DEK hash-list will be created in dev_configure routine and
destroyed in dev_close routine.

Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/crypto/mlx5/meson.build       |   1 +
 drivers/crypto/mlx5/mlx5_crypto.c     |  44 +++++----
 drivers/crypto/mlx5/mlx5_crypto.h     |  51 ++++++++++
 drivers/crypto/mlx5/mlx5_crypto_dek.c | 136 ++++++++++++++++++++++++++
 4 files changed, 215 insertions(+), 17 deletions(-)
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto.h
 create mode 100644 drivers/crypto/mlx5/mlx5_crypto_dek.c

diff --git a/drivers/crypto/mlx5/meson.build b/drivers/crypto/mlx5/meson.build
index fd00283665..9cb3bd214f 100644
--- a/drivers/crypto/mlx5/meson.build
+++ b/drivers/crypto/mlx5/meson.build
@@ -11,6 +11,7 @@ fmt_name = 'mlx5_crypto'
 deps += ['common_mlx5', 'eal', 'cryptodev']
 sources = files(
 	'mlx5_crypto.c',
+	'mlx5_crypto_dek.c',
 )
 cflags_options = [
 	'-std=c11',
diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index ffbce5d68a..2bdfb1a10f 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -3,12 +3,9 @@
  */
 
 #include <rte_malloc.h>
-#include <rte_log.h>
 #include <rte_errno.h>
+#include <rte_log.h>
 #include <rte_pci.h>
-#include <rte_crypto.h>
-#include <rte_cryptodev.h>
-#include <rte_cryptodev_pmd.h>
 
 #include <mlx5_glue.h>
 #include <mlx5_common.h>
@@ -17,19 +14,10 @@
 #include <mlx5_common_os.h>
 
 #include "mlx5_crypto_utils.h"
+#include "mlx5_crypto.h"
 
 #define MLX5_CRYPTO_DRIVER_NAME mlx5_crypto
-#define MLX5_CRYPTO_LOG_NAME    pmd.crypto.mlx5
-
-struct mlx5_crypto_priv {
-	TAILQ_ENTRY(mlx5_crypto_priv) next;
-	struct ibv_context *ctx; /* Device context. */
-	struct rte_pci_device *pci_dev;
-	struct rte_cryptodev *crypto_dev;
-	void *uar; /* User Access Region. */
-	uint32_t pdn; /* Protection Domain number. */
-	struct ibv_pd *pd;
-};
+#define MLX5_CRYPTO_LOG_NAME pmd.crypto.mlx5
 
 TAILQ_HEAD(mlx5_crypto_privs, mlx5_crypto_priv) mlx5_crypto_priv_list =
 				TAILQ_HEAD_INITIALIZER(mlx5_crypto_priv_list);
@@ -48,11 +36,33 @@ static const struct rte_driver mlx5_drv = {
 
 static struct cryptodev_driver mlx5_cryptodev_driver;
 
+static int
+mlx5_crypto_dev_configure(struct rte_cryptodev *dev,
+		struct rte_cryptodev_config *config __rte_unused)
+{
+	struct mlx5_crypto_priv *priv = dev->data->dev_private;
+
+	if (mlx5_crypto_dek_setup(priv) != 0) {
+		DRV_LOG(ERR, "Dek hash list creation has failed.");
+		return -ENOMEM;
+	}
+	return 0;
+}
+
+static int
+mlx5_crypto_dev_close(struct rte_cryptodev *dev)
+{
+	struct mlx5_crypto_priv *priv = dev->data->dev_private;
+
+	mlx5_crypto_dek_unset(priv);
+	return 0;
+}
+
 static struct rte_cryptodev_ops mlx5_crypto_ops = {
-	.dev_configure			= NULL,
+	.dev_configure			= mlx5_crypto_dev_configure,
 	.dev_start			= NULL,
 	.dev_stop			= NULL,
-	.dev_close			= NULL,
+	.dev_close			= mlx5_crypto_dev_close,
 	.dev_infos_get			= NULL,
 	.stats_get			= NULL,
 	.stats_reset			= NULL,
diff --git a/drivers/crypto/mlx5/mlx5_crypto.h b/drivers/crypto/mlx5/mlx5_crypto.h
new file mode 100644
index 0000000000..4ec67a7e0f
--- /dev/null
+++ b/drivers/crypto/mlx5/mlx5_crypto.h
@@ -0,0 +1,51 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2021 Mellanox Technologies, Ltd
+ */
+
+#ifndef MLX5_CRYPTO_H_
+#define MLX5_CRYPTO_H_
+
+#include <stdbool.h>
+
+#include <rte_cryptodev.h>
+#include <rte_cryptodev_pmd.h>
+
+#include <mlx5_common_utils.h>
+
+#define MLX5_CRYPTO_DEK_HTABLE_SZ (1 << 11)
+#define MLX5_CRYPTO_KEY_LENGTH 80
+
+struct mlx5_crypto_priv {
+	TAILQ_ENTRY(mlx5_crypto_priv) next;
+	struct ibv_context *ctx; /* Device context. */
+	struct rte_pci_device *pci_dev;
+	struct rte_cryptodev *crypto_dev;
+	void *uar; /* User Access Region. */
+	uint32_t pdn; /* Protection Domain number. */
+	struct ibv_pd *pd;
+	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
+};
+
+struct mlx5_crypto_dek {
+	struct mlx5_hlist_entry entry; /* Pointer to DEK hash list entry. */
+	struct mlx5_devx_obj *obj; /* Pointer to DEK DevX object. */
+	uint8_t data[MLX5_CRYPTO_KEY_LENGTH]; /* DEK key data. */
+	bool size_is_48; /* Whether the key\data size is 48 bytes or not. */
+};
+
+int
+mlx5_crypto_dek_destroy(struct mlx5_crypto_priv *priv,
+			struct mlx5_crypto_dek *dek);
+
+struct mlx5_crypto_dek *
+mlx5_crypto_dek_prepare(struct mlx5_crypto_priv *priv,
+			struct rte_crypto_cipher_xform *cipher);
+
+int
+mlx5_crypto_dek_setup(struct mlx5_crypto_priv *priv);
+
+void
+mlx5_crypto_dek_unset(struct mlx5_crypto_priv *priv);
+
+#endif /* MLX5_CRYPTO_H_ */
+
diff --git a/drivers/crypto/mlx5/mlx5_crypto_dek.c b/drivers/crypto/mlx5/mlx5_crypto_dek.c
new file mode 100644
index 0000000000..c76e208845
--- /dev/null
+++ b/drivers/crypto/mlx5/mlx5_crypto_dek.c
@@ -0,0 +1,136 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2018 Mellanox Technologies, Ltd
+ */
+
+#include <rte_ip.h>
+#include <rte_common.h>
+#include <rte_errno.h>
+#include <rte_log.h>
+
+#include <mlx5_prm.h>
+#include <mlx5_devx_cmds.h>
+
+#include "mlx5_crypto_utils.h"
+#include "mlx5_crypto.h"
+
+struct mlx5_crypto_dek_ctx {
+	struct rte_crypto_cipher_xform *cipher;
+	struct mlx5_crypto_priv *priv;
+};
+
+int
+mlx5_crypto_dek_destroy(struct mlx5_crypto_priv *priv,
+			struct mlx5_crypto_dek *dek)
+{
+	return mlx5_hlist_unregister(priv->dek_hlist, &dek->entry);
+}
+
+struct mlx5_crypto_dek *
+mlx5_crypto_dek_prepare(struct mlx5_crypto_priv *priv,
+			struct rte_crypto_cipher_xform *cipher)
+{
+	struct mlx5_hlist *dek_hlist = priv->dek_hlist;
+	struct mlx5_crypto_dek_ctx dek_ctx = {
+		.cipher = cipher,
+		.priv = priv,
+	};
+	struct rte_crypto_cipher_xform *cipher_ctx = cipher;
+	uint64_t key64 = __rte_raw_cksum(cipher_ctx->key.data,
+					 cipher_ctx->key.length, 0);
+	struct mlx5_hlist_entry *entry = mlx5_hlist_register(dek_hlist,
+							     key64, &dek_ctx);
+
+	return entry == NULL ? NULL :
+			     container_of(entry, struct mlx5_crypto_dek, entry);
+}
+
+static int
+mlx5_crypto_dek_match_cb(struct mlx5_hlist *list __rte_unused,
+			 struct mlx5_hlist_entry *entry,
+			 uint64_t key __rte_unused, void *cb_ctx)
+{
+	struct mlx5_crypto_dek_ctx *ctx = cb_ctx;
+	struct rte_crypto_cipher_xform *cipher_ctx = ctx->cipher;
+	struct mlx5_crypto_dek *dek =
+			container_of(entry, typeof(*dek), entry);
+	uint32_t key_len = dek->size_is_48 ? 48 : 80;
+
+	if (key_len != cipher_ctx->key.length)
+		return -1;
+	return memcmp(cipher_ctx->key.data, dek->data, key_len);
+}
+
+static struct mlx5_hlist_entry *
+mlx5_crypto_dek_create_cb(struct mlx5_hlist *list __rte_unused,
+			  uint64_t key __rte_unused, void *cb_ctx)
+{
+	struct mlx5_crypto_dek_ctx *ctx = cb_ctx;
+	struct rte_crypto_cipher_xform *cipher_ctx = ctx->cipher;
+	struct mlx5_crypto_dek *dek = rte_zmalloc(__func__, sizeof(*dek),
+						  RTE_CACHE_LINE_SIZE);
+	struct mlx5_devx_dek_attr dek_attr = {
+		.pd = ctx->priv->pdn,
+		.key_purpose = MLX5_CRYPTO_KEY_PURPOSE_AES_XTS,
+		.has_keytag = 1,
+	};
+
+	if (dek == NULL) {
+		DRV_LOG(ERR, "Failed to allocate dek memory.");
+		return NULL;
+	}
+	switch (cipher_ctx->key.length) {
+	case 48:
+		dek->size_is_48 = true;
+		dek_attr.key_size = MLX5_CRYPTO_KEY_SIZE_128b;
+		break;
+	case 80:
+		dek->size_is_48 = false;
+		dek_attr.key_size = MLX5_CRYPTO_KEY_SIZE_256b;
+		break;
+	default:
+		DRV_LOG(ERR, "Key size not supported.");
+		return NULL;
+	}
+	rte_memcpy(&dek_attr.key, cipher_ctx->key.data, cipher_ctx->key.length);
+	dek->obj = mlx5_devx_cmd_create_dek_obj(ctx->priv->ctx, &dek_attr);
+	if (dek->obj == NULL) {
+		rte_free(dek);
+		return NULL;
+	}
+	rte_memcpy(&dek->data, cipher_ctx->key.data, cipher_ctx->key.length);
+	return &dek->entry;
+}
+
+static void
+mlx5_crypto_dek_remove_cb(struct mlx5_hlist *list __rte_unused,
+			  struct mlx5_hlist_entry *entry)
+{
+	struct mlx5_crypto_dek *dek =
+		container_of(entry, typeof(*dek), entry);
+
+	claim_zero(mlx5_devx_cmd_destroy(dek->obj));
+	rte_free(dek);
+}
+
+
+int
+mlx5_crypto_dek_setup(struct mlx5_crypto_priv *priv)
+{
+	priv->dek_hlist = mlx5_hlist_create("dek_hlist",
+				 MLX5_CRYPTO_DEK_HTABLE_SZ,
+				 0, MLX5_HLIST_WRITE_MOST |
+				 MLX5_HLIST_DIRECT_KEY,
+				 mlx5_crypto_dek_create_cb,
+				 mlx5_crypto_dek_match_cb,
+				 mlx5_crypto_dek_remove_cb);
+	if (priv->dek_hlist == NULL)
+		return -1;
+	return 0;
+}
+
+void
+mlx5_crypto_dek_unset(struct mlx5_crypto_priv *priv)
+{
+	mlx5_hlist_destroy(priv->dek_hlist);
+	priv->dek_hlist = NULL;
+}
-- 
2.25.1

[dpdk-dev] [PATCH v3 03/15] crypto/mlx5: support session operations

[Matan Azrad]

From: Shiri Kuzin <shirik@nvidia.com>

Sessions are used in symmetric transformations in order to prepare
objects and data for packet processing stage.

A mlx5 session includes iv_offset, pointer to mlx5_crypto_dek struct,
bsf_size, bsf_p_type, encryption_order and encryption standard.

Implement the next session operations:
        mlx5_crypto_sym_session_get_size- returns the size of the mlx5
	session struct.
	mlx5_crypto_sym_session_configure- prepares the DEK hash-list
	and saves all the session data.
	mlx5_crypto_sym_session_clear - destroys the DEK hash-list.

Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/crypto/mlx5/mlx5_crypto.c | 96 ++++++++++++++++++++++++++++++-
 1 file changed, 93 insertions(+), 3 deletions(-)

diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index 2bdfb1a10f..32f5077066 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -3,6 +3,7 @@
  */
 
 #include <rte_malloc.h>
+#include <rte_mempool.h>
 #include <rte_errno.h>
 #include <rte_log.h>
 #include <rte_pci.h>
@@ -36,6 +37,24 @@ static const struct rte_driver mlx5_drv = {
 
 static struct cryptodev_driver mlx5_cryptodev_driver;
 
+struct mlx5_crypto_session {
+	uint32_t bs_bpt_eo_es;
+	/*
+	 * bsf_size, bsf_p_type, encryption_order and encryption standard,
+	 * saved in big endian format.
+	 */
+	uint32_t iv_offset:16;
+	/* Starting point for Initialisation Vector. */
+	struct mlx5_crypto_dek *dek; /* Pointer to dek struct. */
+	uint32_t dek_id; /* DEK ID */
+} __rte_packed;
+
+static unsigned int
+mlx5_crypto_sym_session_get_size(struct rte_cryptodev *dev __rte_unused)
+{
+	return sizeof(struct mlx5_crypto_session);
+}
+
 static int
 mlx5_crypto_dev_configure(struct rte_cryptodev *dev,
 		struct rte_cryptodev_config *config __rte_unused)
@@ -58,6 +77,77 @@ mlx5_crypto_dev_close(struct rte_cryptodev *dev)
 	return 0;
 }
 
+static int
+mlx5_crypto_sym_session_configure(struct rte_cryptodev *dev,
+				  struct rte_crypto_sym_xform *xform,
+				  struct rte_cryptodev_sym_session *session,
+				  struct rte_mempool *mp)
+{
+	struct mlx5_crypto_priv *priv = dev->data->dev_private;
+	struct mlx5_crypto_session *sess_private_data;
+	struct rte_crypto_cipher_xform *cipher;
+	uint8_t encryption_order;
+	int ret;
+
+	if (unlikely(xform->next != NULL)) {
+		DRV_LOG(ERR, "Xform next is not supported.");
+		return -ENOTSUP;
+	}
+	if (unlikely((xform->type != RTE_CRYPTO_SYM_XFORM_CIPHER) ||
+		     (xform->cipher.algo != RTE_CRYPTO_CIPHER_AES_XTS))) {
+		DRV_LOG(ERR, "Only AES-XTS algorithm is supported.");
+		return -ENOTSUP;
+	}
+	ret = rte_mempool_get(mp, (void *)&sess_private_data);
+	if (ret != 0) {
+		DRV_LOG(ERR,
+			"Failed to get session %p private data from mempool.",
+			sess_private_data);
+		return -ENOMEM;
+	}
+	cipher = &xform->cipher;
+	sess_private_data->dek = mlx5_crypto_dek_prepare(priv, cipher);
+	if (sess_private_data->dek == NULL) {
+		rte_mempool_put(mp, sess_private_data);
+		DRV_LOG(ERR, "Failed to prepare dek.");
+		return -ENOMEM;
+	}
+	if (cipher->op == RTE_CRYPTO_CIPHER_OP_ENCRYPT)
+		encryption_order = MLX5_ENCRYPTION_ORDER_ENCRYPTED_RAW_MEMORY;
+	else
+		encryption_order = MLX5_ENCRYPTION_ORDER_ENCRYPTED_RAW_WIRE;
+	sess_private_data->bs_bpt_eo_es = rte_cpu_to_be_32
+			(MLX5_BSF_SIZE_64B << MLX5_BSF_SIZE_OFFSET |
+			 MLX5_BSF_P_TYPE_CRYPTO << MLX5_BSF_P_TYPE_OFFSET |
+			 encryption_order << MLX5_ENCRYPTION_ORDER_OFFSET |
+			 MLX5_ENCRYPTION_STANDARD_AES_XTS);
+	sess_private_data->iv_offset = cipher->iv.offset;
+	sess_private_data->dek_id =
+			rte_cpu_to_be_32(sess_private_data->dek->obj->id &
+					 0xffffff);
+	set_sym_session_private_data(session, dev->driver_id,
+				     sess_private_data);
+	DRV_LOG(DEBUG, "Session %p was configured.", sess_private_data);
+	return 0;
+}
+
+static void
+mlx5_crypto_sym_session_clear(struct rte_cryptodev *dev,
+			      struct rte_cryptodev_sym_session *sess)
+{
+	struct mlx5_crypto_priv *priv = dev->data->dev_private;
+	struct mlx5_crypto_session *sess_private_data =
+			get_sym_session_private_data(sess, dev->driver_id);
+
+	if (unlikely(sess_private_data == NULL)) {
+		DRV_LOG(ERR, "Failed to get session %p private data.",
+				sess_private_data);
+		return;
+	}
+	mlx5_crypto_dek_destroy(priv, sess_private_data->dek);
+	DRV_LOG(DEBUG, "Session %p was cleared.", sess_private_data);
+}
+
 static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_configure			= mlx5_crypto_dev_configure,
 	.dev_start			= NULL,
@@ -68,9 +158,9 @@ static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.stats_reset			= NULL,
 	.queue_pair_setup		= NULL,
 	.queue_pair_release		= NULL,
-	.sym_session_get_size		= NULL,
-	.sym_session_configure		= NULL,
-	.sym_session_clear		= NULL,
+	.sym_session_get_size		= mlx5_crypto_sym_session_get_size,
+	.sym_session_configure		= mlx5_crypto_sym_session_configure,
+	.sym_session_clear		= mlx5_crypto_sym_session_clear,
 	.sym_get_raw_dp_ctx_size	= NULL,
 	.sym_configure_raw_dp_ctx	= NULL,
 };
-- 
2.25.1

[dpdk-dev] [PATCH v3 04/15] crypto/mlx5: add basic operations

[Matan Azrad]

From: Shiri Kuzin <shirik@nvidia.com>

The basic dev control operations are configure, close and get info.

Extended the existing support of configure and close:
	-mlx5_crypto_dev_configure- function used to configure device.
	-mlx5_crypto_dev_close-  function used to close a configured
	 device.

Added support of get info function:
	-mlx5_crypto_dev_infos_get- function used to get specific
	 information of a device.

Added config struct to user private data with the fields socket id,
number of queue pairs and feature flags to be disabled.

Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/crypto/mlx5/mlx5_crypto.c | 46 +++++++++++++++++++++++++++----
 drivers/crypto/mlx5/mlx5_crypto.h |  1 +
 2 files changed, 42 insertions(+), 5 deletions(-)

diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index 32f5077066..cec21dbea7 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -19,6 +19,7 @@
 
 #define MLX5_CRYPTO_DRIVER_NAME mlx5_crypto
 #define MLX5_CRYPTO_LOG_NAME pmd.crypto.mlx5
+#define MLX5_CRYPTO_MAX_QPS 1024
 
 TAILQ_HEAD(mlx5_crypto_privs, mlx5_crypto_priv) mlx5_crypto_priv_list =
 				TAILQ_HEAD_INITIALIZER(mlx5_crypto_priv_list);
@@ -28,6 +29,9 @@ int mlx5_crypto_logtype;
 
 uint8_t mlx5_crypto_driver_id;
 
+const struct rte_cryptodev_capabilities
+		mlx5_crypto_caps[RTE_CRYPTO_OP_TYPE_UNDEFINED];
+
 static const char mlx5_crypto_drv_name[] = RTE_STR(MLX5_CRYPTO_DRIVER_NAME);
 
 static const struct rte_driver mlx5_drv = {
@@ -49,22 +53,47 @@ struct mlx5_crypto_session {
 	uint32_t dek_id; /* DEK ID */
 } __rte_packed;
 
-static unsigned int
-mlx5_crypto_sym_session_get_size(struct rte_cryptodev *dev __rte_unused)
+static void
+mlx5_crypto_dev_infos_get(struct rte_cryptodev *dev,
+			  struct rte_cryptodev_info *dev_info)
 {
-	return sizeof(struct mlx5_crypto_session);
+	RTE_SET_USED(dev);
+	if (dev_info != NULL) {
+		dev_info->driver_id = mlx5_crypto_driver_id;
+		dev_info->feature_flags = 0;
+		dev_info->capabilities = mlx5_crypto_caps;
+		dev_info->max_nb_queue_pairs = MLX5_CRYPTO_MAX_QPS;
+		dev_info->min_mbuf_headroom_req = 0;
+		dev_info->min_mbuf_tailroom_req = 0;
+		dev_info->sym.max_nb_sessions = 0;
+		/*
+		 * If 0, the device does not have any limitation in number of
+		 * sessions that can be used.
+		 */
+	}
 }
 
 static int
 mlx5_crypto_dev_configure(struct rte_cryptodev *dev,
-		struct rte_cryptodev_config *config __rte_unused)
+			  struct rte_cryptodev_config *config)
 {
 	struct mlx5_crypto_priv *priv = dev->data->dev_private;
 
+	if (config == NULL) {
+		DRV_LOG(ERR, "Invalid crypto dev configure parameters.");
+		return -EINVAL;
+	}
+	if ((config->ff_disable & RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO) != 0) {
+		DRV_LOG(ERR,
+			"Disabled symmetric crypto feature is not supported.");
+		return -ENOTSUP;
+	}
 	if (mlx5_crypto_dek_setup(priv) != 0) {
 		DRV_LOG(ERR, "Dek hash list creation has failed.");
 		return -ENOMEM;
 	}
+	priv->dev_config = *config;
+	DRV_LOG(DEBUG, "Device %u was configured.", dev->driver_id);
 	return 0;
 }
 
@@ -74,9 +103,16 @@ mlx5_crypto_dev_close(struct rte_cryptodev *dev)
 	struct mlx5_crypto_priv *priv = dev->data->dev_private;
 
 	mlx5_crypto_dek_unset(priv);
+	DRV_LOG(DEBUG, "Device %u was closed.", dev->driver_id);
 	return 0;
 }
 
+static unsigned int
+mlx5_crypto_sym_session_get_size(struct rte_cryptodev *dev __rte_unused)
+{
+	return sizeof(struct mlx5_crypto_session);
+}
+
 static int
 mlx5_crypto_sym_session_configure(struct rte_cryptodev *dev,
 				  struct rte_crypto_sym_xform *xform,
@@ -153,7 +189,7 @@ static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_start			= NULL,
 	.dev_stop			= NULL,
 	.dev_close			= mlx5_crypto_dev_close,
-	.dev_infos_get			= NULL,
+	.dev_infos_get			= mlx5_crypto_dev_infos_get,
 	.stats_get			= NULL,
 	.stats_reset			= NULL,
 	.queue_pair_setup		= NULL,
diff --git a/drivers/crypto/mlx5/mlx5_crypto.h b/drivers/crypto/mlx5/mlx5_crypto.h
index 4ec67a7e0f..5e270d3d5a 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.h
+++ b/drivers/crypto/mlx5/mlx5_crypto.h
@@ -24,6 +24,7 @@ struct mlx5_crypto_priv {
 	uint32_t pdn; /* Protection Domain number. */
 	struct ibv_pd *pd;
 	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
+	struct rte_cryptodev_config dev_config;
 };
 
 struct mlx5_crypto_dek {
-- 
2.25.1

[dpdk-dev] [PATCH v3 05/15] crypto/mlx5: support queue pairs operations

[Matan Azrad]

From: Shiri Kuzin <shirik@nvidia.com>

The HW queue pairs are a pair of send queue and receive queue of
independent work queues packed together in one object for the purpose
of transferring data between nodes of a network.

Completion Queue is a FIFO queue of completed work requests.

In crypto driver we use one QP in loopback in order to encrypt and
decrypt data locally without sending it to the wire.
In the configured QP we only use the SQ to perform the encryption and
decryption operations.

Added implementation for the QP setup function which creates the CQ,
creates the QP and changes its state to RTS (ready to send).

Added implementation for the release QP function to release all the QP
resources.

Added the ops structure that contains any operation which is supported
by the cryptodev.

Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/crypto/mlx5/mlx5_crypto.c | 124 +++++++++++++++++++++++++++++-
 drivers/crypto/mlx5/mlx5_crypto.h |  11 +++
 2 files changed, 133 insertions(+), 2 deletions(-)

diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index cec21dbea7..8c3417ee96 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -7,6 +7,7 @@
 #include <rte_errno.h>
 #include <rte_log.h>
 #include <rte_pci.h>
+#include <rte_memory.h>
 
 #include <mlx5_glue.h>
 #include <mlx5_common.h>
@@ -184,6 +185,125 @@ mlx5_crypto_sym_session_clear(struct rte_cryptodev *dev,
 	DRV_LOG(DEBUG, "Session %p was cleared.", sess_private_data);
 }
 
+static int
+mlx5_crypto_queue_pair_release(struct rte_cryptodev *dev, uint16_t qp_id)
+{
+	struct mlx5_crypto_qp *qp = dev->data->queue_pairs[qp_id];
+
+	if (qp->qp_obj != NULL)
+		claim_zero(mlx5_devx_cmd_destroy(qp->qp_obj));
+	if (qp->umem_obj != NULL)
+		claim_zero(mlx5_glue->devx_umem_dereg(qp->umem_obj));
+	if (qp->umem_buf != NULL)
+		rte_free(qp->umem_buf);
+	mlx5_devx_cq_destroy(&qp->cq_obj);
+	rte_free(qp);
+	dev->data->queue_pairs[qp_id] = NULL;
+	return 0;
+}
+
+static int
+mlx5_crypto_qp2rts(struct mlx5_crypto_qp *qp)
+{
+	/*
+	 * In Order to configure self loopback, when calling these functions the
+	 * remote QP id that is used is the id of the same QP.
+	 */
+	if (mlx5_devx_cmd_modify_qp_state(qp->qp_obj, MLX5_CMD_OP_RST2INIT_QP,
+					  qp->qp_obj->id)) {
+		DRV_LOG(ERR, "Failed to modify QP to INIT state(%u).",
+			rte_errno);
+		return -1;
+	}
+	if (mlx5_devx_cmd_modify_qp_state(qp->qp_obj, MLX5_CMD_OP_INIT2RTR_QP,
+					  qp->qp_obj->id)) {
+		DRV_LOG(ERR, "Failed to modify QP to RTR state(%u).",
+			rte_errno);
+		return -1;
+	}
+	if (mlx5_devx_cmd_modify_qp_state(qp->qp_obj, MLX5_CMD_OP_RTR2RTS_QP,
+					  qp->qp_obj->id)) {
+		DRV_LOG(ERR, "Failed to modify QP to RTS state(%u).",
+			rte_errno);
+		return -1;
+	}
+	return 0;
+}
+
+static int
+mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
+			     const struct rte_cryptodev_qp_conf *qp_conf,
+			     int socket_id)
+{
+	struct mlx5_crypto_priv *priv = dev->data->dev_private;
+	struct mlx5_devx_qp_attr attr = {0};
+	struct mlx5_crypto_qp *qp;
+	uint16_t log_nb_desc = rte_log2_u32(qp_conf->nb_descriptors);
+	uint32_t umem_size = RTE_BIT32(log_nb_desc) *
+			      MLX5_CRYPTO_WQE_SET_SIZE +
+			      sizeof(*qp->db_rec) * 2;
+	uint32_t alloc_size = sizeof(*qp);
+	struct mlx5_devx_cq_attr cq_attr = {
+		.uar_page_id = mlx5_os_get_devx_uar_page_id(priv->uar),
+	};
+
+	alloc_size = RTE_ALIGN(alloc_size, RTE_CACHE_LINE_SIZE);
+	alloc_size += sizeof(struct rte_crypto_op *) * RTE_BIT32(log_nb_desc);
+	qp = rte_zmalloc_socket(__func__, alloc_size, RTE_CACHE_LINE_SIZE,
+				socket_id);
+	if (qp == NULL) {
+		DRV_LOG(ERR, "Failed to allocate QP memory.");
+		rte_errno = ENOMEM;
+		return -rte_errno;
+	}
+	if (mlx5_devx_cq_create(priv->ctx, &qp->cq_obj, log_nb_desc,
+				&cq_attr, socket_id) != 0) {
+		DRV_LOG(ERR, "Failed to create CQ.");
+		goto error;
+	}
+	qp->umem_buf = rte_zmalloc_socket(__func__, umem_size, 4096, socket_id);
+	if (qp->umem_buf == NULL) {
+		DRV_LOG(ERR, "Failed to allocate QP umem.");
+		rte_errno = ENOMEM;
+		goto error;
+	}
+	qp->umem_obj = mlx5_glue->devx_umem_reg(priv->ctx,
+					       (void *)(uintptr_t)qp->umem_buf,
+					       umem_size,
+					       IBV_ACCESS_LOCAL_WRITE);
+	if (qp->umem_obj == NULL) {
+		DRV_LOG(ERR, "Failed to register QP umem.");
+		goto error;
+	}
+	attr.pd = priv->pdn;
+	attr.uar_index = mlx5_os_get_devx_uar_page_id(priv->uar);
+	attr.cqn = qp->cq_obj.cq->id;
+	attr.log_page_size = rte_log2_u32(sysconf(_SC_PAGESIZE));
+	attr.rq_size =  0;
+	attr.sq_size = RTE_BIT32(log_nb_desc);
+	attr.dbr_umem_valid = 1;
+	attr.wq_umem_id = qp->umem_obj->umem_id;
+	attr.wq_umem_offset = 0;
+	attr.dbr_umem_id = qp->umem_obj->umem_id;
+	attr.dbr_address = RTE_BIT64(log_nb_desc) *
+			   MLX5_CRYPTO_WQE_SET_SIZE;
+	qp->qp_obj = mlx5_devx_cmd_create_qp(priv->ctx, &attr);
+	if (qp->qp_obj == NULL) {
+		DRV_LOG(ERR, "Failed to create QP(%u).", rte_errno);
+		goto error;
+	}
+	qp->db_rec = RTE_PTR_ADD(qp->umem_buf, (uintptr_t)attr.dbr_address);
+	if (mlx5_crypto_qp2rts(qp))
+		goto error;
+	qp->ops = (struct rte_crypto_op **)RTE_ALIGN((uintptr_t)(qp + 1),
+							   RTE_CACHE_LINE_SIZE);
+	dev->data->queue_pairs[qp_id] = qp;
+	return 0;
+error:
+	mlx5_crypto_queue_pair_release(dev, qp_id);
+	return -1;
+}
+
 static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_configure			= mlx5_crypto_dev_configure,
 	.dev_start			= NULL,
@@ -192,8 +312,8 @@ static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_infos_get			= mlx5_crypto_dev_infos_get,
 	.stats_get			= NULL,
 	.stats_reset			= NULL,
-	.queue_pair_setup		= NULL,
-	.queue_pair_release		= NULL,
+	.queue_pair_setup		= mlx5_crypto_queue_pair_setup,
+	.queue_pair_release		= mlx5_crypto_queue_pair_release,
 	.sym_session_get_size		= mlx5_crypto_sym_session_get_size,
 	.sym_session_configure		= mlx5_crypto_sym_session_configure,
 	.sym_session_clear		= mlx5_crypto_sym_session_clear,
diff --git a/drivers/crypto/mlx5/mlx5_crypto.h b/drivers/crypto/mlx5/mlx5_crypto.h
index 5e270d3d5a..f5313b89f2 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.h
+++ b/drivers/crypto/mlx5/mlx5_crypto.h
@@ -11,9 +11,11 @@
 #include <rte_cryptodev_pmd.h>
 
 #include <mlx5_common_utils.h>
+#include <mlx5_common_devx.h>
 
 #define MLX5_CRYPTO_DEK_HTABLE_SZ (1 << 11)
 #define MLX5_CRYPTO_KEY_LENGTH 80
+#define MLX5_CRYPTO_WQE_SET_SIZE 1024
 
 struct mlx5_crypto_priv {
 	TAILQ_ENTRY(mlx5_crypto_priv) next;
@@ -27,6 +29,15 @@ struct mlx5_crypto_priv {
 	struct rte_cryptodev_config dev_config;
 };
 
+struct mlx5_crypto_qp {
+	struct mlx5_devx_cq cq_obj;
+	struct mlx5_devx_obj *qp_obj;
+	struct mlx5dv_devx_umem *umem_obj;
+	void *umem_buf;
+	volatile uint32_t *db_rec;
+	struct rte_crypto_op **ops;
+};
+
 struct mlx5_crypto_dek {
 	struct mlx5_hlist_entry entry; /* Pointer to DEK hash list entry. */
 	struct mlx5_devx_obj *obj; /* Pointer to DEK DevX object. */
-- 
2.25.1

[dpdk-dev] [PATCH v3 06/15] crypto/mlx5: add dev stop and start operations

[Matan Azrad]

From: Shiri Kuzin <shirik@nvidia.com>

Add the dev_start function that is used to start a configured device.
Add the dev_stop function that is used to stop a configured device.

Both functions set the dev parameter as used and return 0.

Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/crypto/mlx5/mlx5_crypto.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index 8c3417ee96..538fe5ce56 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -98,6 +98,19 @@ mlx5_crypto_dev_configure(struct rte_cryptodev *dev,
 	return 0;
 }
 
+static void
+mlx5_crypto_dev_stop(struct rte_cryptodev *dev)
+{
+	RTE_SET_USED(dev);
+}
+
+static int
+mlx5_crypto_dev_start(struct rte_cryptodev *dev)
+{
+	RTE_SET_USED(dev);
+	return 0;
+}
+
 static int
 mlx5_crypto_dev_close(struct rte_cryptodev *dev)
 {
@@ -306,8 +319,8 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 
 static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_configure			= mlx5_crypto_dev_configure,
-	.dev_start			= NULL,
-	.dev_stop			= NULL,
+	.dev_start			= mlx5_crypto_dev_start,
+	.dev_stop			= mlx5_crypto_dev_stop,
 	.dev_close			= mlx5_crypto_dev_close,
 	.dev_infos_get			= mlx5_crypto_dev_infos_get,
 	.stats_get			= NULL,
-- 
2.25.1

[dpdk-dev] [PATCH v3 07/15] crypto/mlx5: add memory region management

[Matan Azrad]

From: Shiri Kuzin <shirik@nvidia.com>

Mellanox user space drivers don't deal with physical addresses as part
of a memory protection mechanism.
The device translates the given virtual address to a physical address
using the given memory key as an address space identifier.
That's why any mbuf virtual address is moved directly to the HW
descriptor(WQE).

The mapping between the virtual address to the physical address is saved
in MR configured by the kernel to the HW.

Each MR has a key that should also be moved to the WQE by the SW.

When the SW sees an unmapped address, it extends the address range and
creates a MR using a system call.

Add memory region cache management:
	- 2 level cache per queue-pair - no locks.
	- 1 shared cache between all the queues using a lock.

Using this way, the MR key search per data-path address is optimized.

Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/crypto/mlx5/mlx5_crypto.c | 20 ++++++++++++++++++++
 drivers/crypto/mlx5/mlx5_crypto.h |  3 +++
 2 files changed, 23 insertions(+)

diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index 538fe5ce56..b95aea0068 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -209,6 +209,7 @@ mlx5_crypto_queue_pair_release(struct rte_cryptodev *dev, uint16_t qp_id)
 		claim_zero(mlx5_glue->devx_umem_dereg(qp->umem_obj));
 	if (qp->umem_buf != NULL)
 		rte_free(qp->umem_buf);
+	mlx5_mr_btree_free(&qp->mr_ctrl.cache_bh);
 	mlx5_devx_cq_destroy(&qp->cq_obj);
 	rte_free(qp);
 	dev->data->queue_pairs[qp_id] = NULL;
@@ -288,6 +289,13 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 		DRV_LOG(ERR, "Failed to register QP umem.");
 		goto error;
 	}
+	if (mlx5_mr_btree_init(&qp->mr_ctrl.cache_bh, MLX5_MR_BTREE_CACHE_N,
+			       priv->dev_config.socket_id) != 0) {
+		DRV_LOG(ERR, "Cannot allocate MR Btree for qp %u.",
+			(uint32_t)qp_id);
+		rte_errno = ENOMEM;
+		goto error;
+	}
 	attr.pd = priv->pdn;
 	attr.uar_index = mlx5_os_get_devx_uar_page_id(priv->uar);
 	attr.cqn = qp->cq_obj.cq->id;
@@ -476,6 +484,17 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 		claim_zero(mlx5_glue->close_device(priv->ctx));
 		return -1;
 	}
+	if (mlx5_mr_btree_init(&priv->mr_scache.cache,
+			     MLX5_MR_BTREE_CACHE_N * 2, rte_socket_id()) != 0) {
+		DRV_LOG(ERR, "Failed to allocate shared cache MR memory.");
+		mlx5_crypto_hw_global_release(priv);
+		rte_cryptodev_pmd_destroy(priv->crypto_dev);
+		claim_zero(mlx5_glue->close_device(priv->ctx));
+		rte_errno = ENOMEM;
+		return -rte_errno;
+	}
+	priv->mr_scache.reg_mr_cb = mlx5_common_verbs_reg_mr;
+	priv->mr_scache.dereg_mr_cb = mlx5_common_verbs_dereg_mr;
 	pthread_mutex_lock(&priv_list_lock);
 	TAILQ_INSERT_TAIL(&mlx5_crypto_priv_list, priv, next);
 	pthread_mutex_unlock(&priv_list_lock);
@@ -495,6 +514,7 @@ mlx5_crypto_pci_remove(struct rte_pci_device *pdev)
 		TAILQ_REMOVE(&mlx5_crypto_priv_list, priv, next);
 	pthread_mutex_unlock(&priv_list_lock);
 	if (priv) {
+		mlx5_mr_release_cache(&priv->mr_scache);
 		mlx5_crypto_hw_global_release(priv);
 		rte_cryptodev_pmd_destroy(priv->crypto_dev);
 		claim_zero(mlx5_glue->close_device(priv->ctx));
diff --git a/drivers/crypto/mlx5/mlx5_crypto.h b/drivers/crypto/mlx5/mlx5_crypto.h
index f5313b89f2..397267d249 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.h
+++ b/drivers/crypto/mlx5/mlx5_crypto.h
@@ -12,6 +12,7 @@
 
 #include <mlx5_common_utils.h>
 #include <mlx5_common_devx.h>
+#include <mlx5_common_mr.h>
 
 #define MLX5_CRYPTO_DEK_HTABLE_SZ (1 << 11)
 #define MLX5_CRYPTO_KEY_LENGTH 80
@@ -27,6 +28,7 @@ struct mlx5_crypto_priv {
 	struct ibv_pd *pd;
 	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
 	struct rte_cryptodev_config dev_config;
+	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
 };
 
 struct mlx5_crypto_qp {
@@ -36,6 +38,7 @@ struct mlx5_crypto_qp {
 	void *umem_buf;
 	volatile uint32_t *db_rec;
 	struct rte_crypto_op **ops;
+	struct mlx5_mr_ctrl mr_ctrl;
 };
 
 struct mlx5_crypto_dek {
-- 
2.25.1

[dpdk-dev] [PATCH v3 08/15] crypto/mlx5: create login object using DevX

[Matan Azrad]

From: Shiri Kuzin <shirik@nvidia.com>

To work with crypto engines that are marked with wrapped_import_method,
a login session is required.
A crypto login object needs to be created using DevX.

The crypto login object contains:
	- The credential pointer.
	- The import_KEK pointer to be used for all secured information
	  communicated in crypto commands (key fields), including the
	  provided credential in this command.
	- The credential secret, wrapped by the import_KEK indicated in
	  this command. Size includes 8 bytes IV for wrapping.

Added devargs for the required login values:
	- wcs_file - path to the file containing the credential.
	- import_kek_id - the import KEK pointer.
	- credential_id - the credential pointer.

Create the login DevX object in pci_probe function and destroy it in
pci_remove.
Destroying the crypto login object means logout.

Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/crypto/mlx5/mlx5_crypto.c | 103 ++++++++++++++++++++++++++++++
 drivers/crypto/mlx5/mlx5_crypto.h |   7 ++
 2 files changed, 110 insertions(+)

diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index b95aea0068..18b1a6be88 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -402,6 +402,101 @@ mlx5_crypto_hw_global_prepare(struct mlx5_crypto_priv *priv)
 	return 0;
 }
 
+
+static int
+mlx5_crypto_args_check_handler(const char *key, const char *val, void *opaque)
+{
+	struct mlx5_crypto_devarg_params *devarg_prms = opaque;
+	struct mlx5_devx_crypto_login_attr *attr = &devarg_prms->login_attr;
+	unsigned long tmp;
+	FILE *file;
+	int ret;
+	int i;
+
+	if (strcmp(key, "class") == 0)
+		return 0;
+	if (strcmp(key, "wcs_file") == 0) {
+		file = fopen(val, "rb");
+		if (file == NULL) {
+			rte_errno = ENOTSUP;
+			return -rte_errno;
+		}
+		for (i = 0 ; i < MLX5_CRYPTO_CREDENTIAL_SIZE ; i++) {
+			ret = fscanf(file, "%02hhX", &attr->credential[i]);
+			if (ret <= 0) {
+				fclose(file);
+				DRV_LOG(ERR,
+					"Failed to read credential from file.");
+				rte_errno = EINVAL;
+				return -rte_errno;
+			}
+		}
+		fclose(file);
+		devarg_prms->login_devarg = true;
+		return 0;
+	}
+	errno = 0;
+	tmp = strtoul(val, NULL, 0);
+	if (errno) {
+		DRV_LOG(WARNING, "%s: \"%s\" is an invalid integer.", key, val);
+		return -errno;
+	}
+	if (strcmp(key, "import_kek_id") == 0)
+		attr->session_import_kek_ptr = (uint32_t)tmp;
+	else if (strcmp(key, "credential_id") == 0)
+		attr->credential_pointer = (uint32_t)tmp;
+	else
+		DRV_LOG(WARNING, "Invalid key %s.", key);
+	return 0;
+}
+
+static struct mlx5_devx_obj *
+mlx5_crypto_config_login(struct rte_devargs *devargs,
+			 struct ibv_context *ctx)
+{
+	/*
+	 * Set credential pointer and session import KEK pointer to a default
+	 * value of 0.
+	 */
+	struct mlx5_crypto_devarg_params login = {
+			.login_devarg = false,
+			.login_attr = {
+					.credential_pointer = 0,
+					.session_import_kek_ptr = 0,
+			}
+	};
+	struct rte_kvargs *kvlist;
+
+	if (devargs == NULL) {
+		DRV_LOG(ERR,
+	"No login devargs in order to enable crypto operations in the device.");
+		rte_errno = EINVAL;
+		return NULL;
+	}
+	kvlist = rte_kvargs_parse(devargs->args, NULL);
+	if (kvlist == NULL) {
+		DRV_LOG(ERR, "Failed to parse devargs.");
+		rte_errno = EINVAL;
+		return NULL;
+	}
+	if (rte_kvargs_process(kvlist, NULL, mlx5_crypto_args_check_handler,
+			   &login) != 0) {
+		DRV_LOG(ERR, "Devargs handler function Failed.");
+		rte_kvargs_free(kvlist);
+		rte_errno = EINVAL;
+		return NULL;
+	}
+	rte_kvargs_free(kvlist);
+	if (login.login_devarg == false) {
+		DRV_LOG(ERR,
+	"No login credential devarg in order to enable crypto operations "
+	"in the device.");
+		rte_errno = EINVAL;
+		return NULL;
+	}
+	return mlx5_devx_cmd_create_crypto_login_obj(ctx, &login.login_attr);
+}
+
 /**
  * DPDK callback to register a PCI device.
  *
@@ -423,6 +518,7 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 	struct ibv_device *ibv;
 	struct rte_cryptodev *crypto_dev;
 	struct ibv_context *ctx;
+	struct mlx5_devx_obj *login;
 	struct mlx5_crypto_priv *priv;
 	struct mlx5_hca_attr attr = { 0 };
 	struct rte_cryptodev_pmd_init_params init_params = {
@@ -461,6 +557,11 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 		rte_errno = ENOTSUP;
 		return -ENOTSUP;
 	}
+	login = mlx5_crypto_config_login(pci_dev->device.devargs, ctx);
+	if (login == NULL) {
+		DRV_LOG(ERR, "Failed to configure login.");
+		return -rte_errno;
+	}
 	crypto_dev = rte_cryptodev_pmd_create(ibv->name, &pci_dev->device,
 					&init_params);
 	if (crypto_dev == NULL) {
@@ -477,6 +578,7 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 	crypto_dev->driver_id = mlx5_crypto_driver_id;
 	priv = crypto_dev->data->dev_private;
 	priv->ctx = ctx;
+	priv->login_obj = login;
 	priv->pci_dev = pci_dev;
 	priv->crypto_dev = crypto_dev;
 	if (mlx5_crypto_hw_global_prepare(priv) != 0) {
@@ -517,6 +619,7 @@ mlx5_crypto_pci_remove(struct rte_pci_device *pdev)
 		mlx5_mr_release_cache(&priv->mr_scache);
 		mlx5_crypto_hw_global_release(priv);
 		rte_cryptodev_pmd_destroy(priv->crypto_dev);
+		claim_zero(mlx5_devx_cmd_destroy(priv->login_obj));
 		claim_zero(mlx5_glue->close_device(priv->ctx));
 	}
 	return 0;
diff --git a/drivers/crypto/mlx5/mlx5_crypto.h b/drivers/crypto/mlx5/mlx5_crypto.h
index 397267d249..0aef804b92 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.h
+++ b/drivers/crypto/mlx5/mlx5_crypto.h
@@ -29,6 +29,7 @@ struct mlx5_crypto_priv {
 	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
 	struct rte_cryptodev_config dev_config;
 	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
+	struct mlx5_devx_obj *login_obj;
 };
 
 struct mlx5_crypto_qp {
@@ -48,6 +49,12 @@ struct mlx5_crypto_dek {
 	bool size_is_48; /* Whether the key\data size is 48 bytes or not. */
 };
 
+
+struct mlx5_crypto_devarg_params {
+	bool login_devarg;
+	struct mlx5_devx_crypto_login_attr login_attr;
+};
+
 int
 mlx5_crypto_dek_destroy(struct mlx5_crypto_priv *priv,
 			struct mlx5_crypto_dek *dek);
-- 
2.25.1

[dpdk-dev] [PATCH v3 09/15] crypto/mlx5: adjust to the multiple data unit API

[Matan Azrad]

From: Shiri Kuzin <shirik@nvidia.com>

In AES-XTS the data to be encrypted\decrypted does not have to be
in multiples of 16B size, the unit of data is called data-unit.

As a result of patch [1] a new field is added to the cipher capability,
called dataunit_set, where the devices can report the range of
supported data-unit sizes.

The new field enables saving the data-unit size in the session
structure to the block size pointer variable in order to support
several data-unit sizes.

[1] https://www.mail-archive.com/dev@dpdk.org/msg205337.html

Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
---
 drivers/crypto/mlx5/mlx5_crypto.c | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index 18b1a6be88..8cc29ced21 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -48,6 +48,11 @@ struct mlx5_crypto_session {
 	 * bsf_size, bsf_p_type, encryption_order and encryption standard,
 	 * saved in big endian format.
 	 */
+	uint32_t bsp_res;
+	/*
+	 * crypto_block_size_pointer and reserved 24 bits saved in big endian
+	 * format.
+	 */
 	uint32_t iv_offset:16;
 	/* Starting point for Initialisation Vector. */
 	struct mlx5_crypto_dek *dek; /* Pointer to dek struct. */
@@ -171,6 +176,24 @@ mlx5_crypto_sym_session_configure(struct rte_cryptodev *dev,
 			 MLX5_BSF_P_TYPE_CRYPTO << MLX5_BSF_P_TYPE_OFFSET |
 			 encryption_order << MLX5_ENCRYPTION_ORDER_OFFSET |
 			 MLX5_ENCRYPTION_STANDARD_AES_XTS);
+	switch (xform->cipher.dataunit_len) {
+	case 0:
+		sess_private_data->bsp_res = 0;
+		break;
+	case 512:
+		sess_private_data->bsp_res = rte_cpu_to_be_32
+					     ((uint32_t)MLX5_BLOCK_SIZE_512B <<
+					     MLX5_BLOCK_SIZE_OFFSET);
+		break;
+	case 4096:
+		sess_private_data->bsp_res = rte_cpu_to_be_32
+					     ((uint32_t)MLX5_BLOCK_SIZE_4096B <<
+					     MLX5_BLOCK_SIZE_OFFSET);
+		break;
+	default:
+		DRV_LOG(ERR, "Cipher data unit length is not supported.");
+		return -ENOTSUP;
+	}
 	sess_private_data->iv_offset = cipher->iv.offset;
 	sess_private_data->dek_id =
 			rte_cpu_to_be_32(sess_private_data->dek->obj->id &
-- 
2.25.1

[dpdk-dev] [PATCH v3 10/15] crypto/mlx5: add keytag device argument

[Matan Azrad]

From: Suanming Mou <suanmingm@nvidia.com>

A keytag is a piece of data encrypted together with a DEK.

When a DEK is referenced by an MKEY.bsf through its index, the keytag is
also supplied in the BSF as plaintext. The HW will decrypt the DEK (and
the attached keytag) and will fail the operation if the keytags don't
match.

This commit adds the configuration of the keytag with devargs.

Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Signed-off-by: Matan Azrad <matan@nvidia.com>
---
 drivers/crypto/mlx5/mlx5_crypto.c | 50 +++++++++++++++++--------------
 drivers/crypto/mlx5/mlx5_crypto.h |  3 +-
 2 files changed, 30 insertions(+), 23 deletions(-)

diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index 8cc29ced21..73cca8136b 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -468,56 +468,52 @@ mlx5_crypto_args_check_handler(const char *key, const char *val, void *opaque)
 		attr->session_import_kek_ptr = (uint32_t)tmp;
 	else if (strcmp(key, "credential_id") == 0)
 		attr->credential_pointer = (uint32_t)tmp;
+	else if (strcmp(key, "keytag") == 0)
+		devarg_prms->keytag = tmp;
 	else
 		DRV_LOG(WARNING, "Invalid key %s.", key);
 	return 0;
 }
 
-static struct mlx5_devx_obj *
-mlx5_crypto_config_login(struct rte_devargs *devargs,
-			 struct ibv_context *ctx)
+static int
+mlx5_crypto_parse_devargs(struct rte_devargs *devargs,
+			  struct mlx5_crypto_devarg_params *devarg_prms)
 {
-	/*
-	 * Set credential pointer and session import KEK pointer to a default
-	 * value of 0.
-	 */
-	struct mlx5_crypto_devarg_params login = {
-			.login_devarg = false,
-			.login_attr = {
-					.credential_pointer = 0,
-					.session_import_kek_ptr = 0,
-			}
-	};
+	struct mlx5_devx_crypto_login_attr *attr = &devarg_prms->login_attr;
 	struct rte_kvargs *kvlist;
 
+	/* Default values. */
+	attr->credential_pointer = 0;
+	attr->session_import_kek_ptr = 0;
+	devarg_prms->keytag = 0;
 	if (devargs == NULL) {
 		DRV_LOG(ERR,
 	"No login devargs in order to enable crypto operations in the device.");
 		rte_errno = EINVAL;
-		return NULL;
+		return -1;
 	}
 	kvlist = rte_kvargs_parse(devargs->args, NULL);
 	if (kvlist == NULL) {
 		DRV_LOG(ERR, "Failed to parse devargs.");
 		rte_errno = EINVAL;
-		return NULL;
+		return -1;
 	}
 	if (rte_kvargs_process(kvlist, NULL, mlx5_crypto_args_check_handler,
-			   &login) != 0) {
+			   devarg_prms) != 0) {
 		DRV_LOG(ERR, "Devargs handler function Failed.");
 		rte_kvargs_free(kvlist);
 		rte_errno = EINVAL;
-		return NULL;
+		return -1;
 	}
 	rte_kvargs_free(kvlist);
-	if (login.login_devarg == false) {
+	if (devarg_prms->login_devarg == false) {
 		DRV_LOG(ERR,
 	"No login credential devarg in order to enable crypto operations "
 	"in the device.");
 		rte_errno = EINVAL;
-		return NULL;
+		return -1;
 	}
-	return mlx5_devx_cmd_create_crypto_login_obj(ctx, &login.login_attr);
+	return 0;
 }
 
 /**
@@ -543,6 +539,7 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 	struct ibv_context *ctx;
 	struct mlx5_devx_obj *login;
 	struct mlx5_crypto_priv *priv;
+	struct mlx5_crypto_devarg_params devarg_prms = { 0 };
 	struct mlx5_hca_attr attr = { 0 };
 	struct rte_cryptodev_pmd_init_params init_params = {
 		.name = "",
@@ -551,6 +548,8 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 		.max_nb_queue_pairs =
 				RTE_CRYPTODEV_PMD_DEFAULT_MAX_NB_QUEUE_PAIRS,
 	};
+	int ret;
+
 	RTE_SET_USED(pci_drv);
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
 		DRV_LOG(ERR, "Non-primary process type is not supported.");
@@ -580,7 +579,13 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 		rte_errno = ENOTSUP;
 		return -ENOTSUP;
 	}
-	login = mlx5_crypto_config_login(pci_dev->device.devargs, ctx);
+	ret = mlx5_crypto_parse_devargs(pci_dev->device.devargs, &devarg_prms);
+	if (ret) {
+		DRV_LOG(ERR, "Failed to parse devargs.");
+		return -rte_errno;
+	}
+	login = mlx5_devx_cmd_create_crypto_login_obj(ctx,
+						      &devarg_prms.login_attr);
 	if (login == NULL) {
 		DRV_LOG(ERR, "Failed to configure login.");
 		return -rte_errno;
@@ -620,6 +625,7 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 	}
 	priv->mr_scache.reg_mr_cb = mlx5_common_verbs_reg_mr;
 	priv->mr_scache.dereg_mr_cb = mlx5_common_verbs_dereg_mr;
+	priv->keytag = rte_cpu_to_be_64(devarg_prms.keytag);
 	pthread_mutex_lock(&priv_list_lock);
 	TAILQ_INSERT_TAIL(&mlx5_crypto_priv_list, priv, next);
 	pthread_mutex_unlock(&priv_list_lock);
diff --git a/drivers/crypto/mlx5/mlx5_crypto.h b/drivers/crypto/mlx5/mlx5_crypto.h
index 0aef804b92..34c65f9a24 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.h
+++ b/drivers/crypto/mlx5/mlx5_crypto.h
@@ -30,6 +30,7 @@ struct mlx5_crypto_priv {
 	struct rte_cryptodev_config dev_config;
 	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
 	struct mlx5_devx_obj *login_obj;
+	uint64_t keytag;
 };
 
 struct mlx5_crypto_qp {
@@ -49,10 +50,10 @@ struct mlx5_crypto_dek {
 	bool size_is_48; /* Whether the key\data size is 48 bytes or not. */
 };
 
-
 struct mlx5_crypto_devarg_params {
 	bool login_devarg;
 	struct mlx5_devx_crypto_login_attr login_attr;
+	uint64_t keytag;
 };
 
 int
-- 
2.25.1

[dpdk-dev] [PATCH v3 11/15] crypto/mlx5: add maximum segments device argument

[Matan Azrad]

From: Suanming Mou <suanmingm@nvidia.com>

The mlx5 HW crypto operations are done by attaching crypto property
to a memory region. Once done, every access to the memory via the
crypto-enabled memory region will result with in-line encryption or
decryption of the data.

As a result, the design choice is to provide two types of WQEs. One
is UMR WQE which sets the crypto property and the other is rdma write
WQE which sends DMA command to copy data from local MR to remote MR.

The size of the WQEs will be defined by a new devarg called
max_segs_num.

This devarg also defines the maximum segments in mbuf chain that will be
supported for crypto operations.

Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Signed-off-by: Matan Azrad <matan@nvidia.com>
---
 drivers/crypto/mlx5/mlx5_crypto.c | 35 +++++++++++++++++++++++++++----
 drivers/crypto/mlx5/mlx5_crypto.h |  7 +++++++
 2 files changed, 38 insertions(+), 4 deletions(-)

diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index 73cca8136b..6de44398bd 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -21,6 +21,7 @@
 #define MLX5_CRYPTO_DRIVER_NAME mlx5_crypto
 #define MLX5_CRYPTO_LOG_NAME pmd.crypto.mlx5
 #define MLX5_CRYPTO_MAX_QPS 1024
+#define MLX5_CRYPTO_MAX_SEGS 56
 
 TAILQ_HEAD(mlx5_crypto_privs, mlx5_crypto_priv) mlx5_crypto_priv_list =
 				TAILQ_HEAD_INITIALIZER(mlx5_crypto_priv_list);
@@ -464,14 +465,24 @@ mlx5_crypto_args_check_handler(const char *key, const char *val, void *opaque)
 		DRV_LOG(WARNING, "%s: \"%s\" is an invalid integer.", key, val);
 		return -errno;
 	}
-	if (strcmp(key, "import_kek_id") == 0)
+	if (strcmp(key, "max_segs_num") == 0) {
+		if (!tmp || tmp > MLX5_CRYPTO_MAX_SEGS) {
+			DRV_LOG(WARNING, "Invalid max_segs_num: %d, should"
+				" be less than %d.",
+				(uint32_t)tmp, MLX5_CRYPTO_MAX_SEGS);
+			rte_errno = EINVAL;
+			return -rte_errno;
+		}
+		devarg_prms->max_segs_num = (uint32_t)tmp;
+	} else if (strcmp(key, "import_kek_id") == 0) {
 		attr->session_import_kek_ptr = (uint32_t)tmp;
-	else if (strcmp(key, "credential_id") == 0)
+	} else if (strcmp(key, "credential_id") == 0) {
 		attr->credential_pointer = (uint32_t)tmp;
-	else if (strcmp(key, "keytag") == 0)
+	} else if (strcmp(key, "keytag") == 0) {
 		devarg_prms->keytag = tmp;
-	else
+	} else {
 		DRV_LOG(WARNING, "Invalid key %s.", key);
+	}
 	return 0;
 }
 
@@ -486,6 +497,7 @@ mlx5_crypto_parse_devargs(struct rte_devargs *devargs,
 	attr->credential_pointer = 0;
 	attr->session_import_kek_ptr = 0;
 	devarg_prms->keytag = 0;
+	devarg_prms->max_segs_num = 8;
 	if (devargs == NULL) {
 		DRV_LOG(ERR,
 	"No login devargs in order to enable crypto operations in the device.");
@@ -626,6 +638,21 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 	priv->mr_scache.reg_mr_cb = mlx5_common_verbs_reg_mr;
 	priv->mr_scache.dereg_mr_cb = mlx5_common_verbs_dereg_mr;
 	priv->keytag = rte_cpu_to_be_64(devarg_prms.keytag);
+	priv->max_segs_num = devarg_prms.max_segs_num;
+	priv->umr_wqe_size = sizeof(struct mlx5_wqe_umr_bsf_seg) +
+			     sizeof(struct mlx5_umr_wqe) +
+			     RTE_ALIGN(priv->max_segs_num, 4) *
+			     sizeof(struct mlx5_wqe_dseg);
+	priv->rdmw_wqe_size = sizeof(struct mlx5_rdma_write_wqe) +
+			      sizeof(struct mlx5_wqe_dseg) *
+			      (priv->max_segs_num <= 2 ? 2 : 2 +
+			       RTE_ALIGN(priv->max_segs_num - 2, 4));
+	priv->wqe_set_size = priv->umr_wqe_size + priv->rdmw_wqe_size;
+	priv->wqe_stride = (priv->umr_wqe_size + priv->rdmw_wqe_size) /
+							       MLX5_SEND_WQE_BB;
+	priv->max_rdmaw_klm_n = (priv->rdmw_wqe_size -
+				 sizeof(struct mlx5_rdma_write_wqe)) /
+				 sizeof(struct mlx5_wqe_dseg);
 	pthread_mutex_lock(&priv_list_lock);
 	TAILQ_INSERT_TAIL(&mlx5_crypto_priv_list, priv, next);
 	pthread_mutex_unlock(&priv_list_lock);
diff --git a/drivers/crypto/mlx5/mlx5_crypto.h b/drivers/crypto/mlx5/mlx5_crypto.h
index 34c65f9a24..81452bd700 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.h
+++ b/drivers/crypto/mlx5/mlx5_crypto.h
@@ -25,12 +25,18 @@ struct mlx5_crypto_priv {
 	struct rte_cryptodev *crypto_dev;
 	void *uar; /* User Access Region. */
 	uint32_t pdn; /* Protection Domain number. */
+	uint32_t max_segs_num; /* Maximum supported data segs. */
 	struct ibv_pd *pd;
 	struct mlx5_hlist *dek_hlist; /* Dek hash list. */
 	struct rte_cryptodev_config dev_config;
 	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
 	struct mlx5_devx_obj *login_obj;
 	uint64_t keytag;
+	uint16_t wqe_set_size;
+	uint16_t umr_wqe_size;
+	uint16_t rdmw_wqe_size;
+	uint16_t wqe_stride;
+	uint16_t max_rdmaw_klm_n;
 };
 
 struct mlx5_crypto_qp {
@@ -54,6 +60,7 @@ struct mlx5_crypto_devarg_params {
 	bool login_devarg;
 	struct mlx5_devx_crypto_login_attr login_attr;
 	uint64_t keytag;
+	uint32_t max_segs_num;
 };
 
 int
-- 
2.25.1

[dpdk-dev] [PATCH v3 12/15] crypto/mlx5: add WQE set initialization

[Matan Azrad]

From: Suanming Mou <suanmingm@nvidia.com>

Currently, HW handles the WQEs much faster than the software,
Using the constant WQE set layout can initialize most of the WQE
segments in advanced, and software only needs to configure very
limited segments in datapath. This accelerates the software WQE
organize in datapath.

This commit initializes the fixed WQE set segments.

Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Signed-off-by: Matan Azrad <matan@nvidia.com>
---
 drivers/crypto/mlx5/mlx5_crypto.c | 83 +++++++++++++++++++++++++++++--
 drivers/crypto/mlx5/mlx5_crypto.h | 10 +++-
 2 files changed, 87 insertions(+), 6 deletions(-)

diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index 6de44398bd..7bffe08bfe 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -268,6 +268,69 @@ mlx5_crypto_qp2rts(struct mlx5_crypto_qp *qp)
 	return 0;
 }
 
+static void
+mlx5_crypto_qp_init(struct mlx5_crypto_priv *priv, struct mlx5_crypto_qp *qp)
+{
+	uint32_t i;
+
+	for (i = 0 ; i < qp->entries_n; i++) {
+		struct mlx5_wqe_cseg *cseg = RTE_PTR_ADD(qp->umem_buf, i *
+							 priv->wqe_set_size);
+		struct mlx5_wqe_umr_cseg *ucseg = (struct mlx5_wqe_umr_cseg *)
+								     (cseg + 1);
+		struct mlx5_wqe_umr_bsf_seg *bsf =
+			(struct mlx5_wqe_umr_bsf_seg *)(RTE_PTR_ADD(cseg,
+						       priv->umr_wqe_size)) - 1;
+		struct mlx5_wqe_rseg *rseg;
+
+		/* Init UMR WQE. */
+		cseg->sq_ds = rte_cpu_to_be_32((qp->qp_obj->id << 8) |
+					 (priv->umr_wqe_size / MLX5_WSEG_SIZE));
+		cseg->flags = RTE_BE32(MLX5_COMP_ONLY_FIRST_ERR <<
+				       MLX5_COMP_MODE_OFFSET);
+		cseg->misc = rte_cpu_to_be_32(qp->mkey[i]->id);
+		ucseg->if_cf_toe_cq_res = RTE_BE32(1u << MLX5_UMRC_IF_OFFSET);
+		ucseg->mkey_mask = RTE_BE64(1u << 0); /* Mkey length bit. */
+		ucseg->ko_to_bs = rte_cpu_to_be_32
+			((RTE_ALIGN(priv->max_segs_num, 4u) <<
+			 MLX5_UMRC_KO_OFFSET) | (4 << MLX5_UMRC_TO_BS_OFFSET));
+		bsf->keytag = priv->keytag;
+		/* Init RDMA WRITE WQE. */
+		cseg = RTE_PTR_ADD(cseg, priv->umr_wqe_size);
+		cseg->flags = RTE_BE32((MLX5_COMP_ALWAYS <<
+				      MLX5_COMP_MODE_OFFSET) |
+				      MLX5_WQE_CTRL_INITIATOR_SMALL_FENCE);
+		rseg = (struct mlx5_wqe_rseg *)(cseg + 1);
+		rseg->rkey = rte_cpu_to_be_32(qp->mkey[i]->id);
+	}
+}
+
+static int
+mlx5_crypto_indirect_mkeys_prepare(struct mlx5_crypto_priv *priv,
+				  struct mlx5_crypto_qp *qp)
+{
+	struct mlx5_umr_wqe *umr;
+	uint32_t i;
+	struct mlx5_devx_mkey_attr attr = {
+		.pd = priv->pdn,
+		.umr_en = 1,
+		.crypto_en = 1,
+		.set_remote_rw = 1,
+		.klm_num = RTE_ALIGN(priv->max_segs_num, 4),
+	};
+
+	for (umr = (struct mlx5_umr_wqe *)qp->umem_buf, i = 0;
+	   i < qp->entries_n; i++, umr = RTE_PTR_ADD(umr, priv->wqe_set_size)) {
+		attr.klm_array = (struct mlx5_klm *)&umr->kseg[0];
+		qp->mkey[i] = mlx5_devx_cmd_mkey_create(priv->ctx, &attr);
+		if (!qp->mkey[i]) {
+			DRV_LOG(ERR, "Failed to allocate indirect mkey.");
+			return -1;
+		}
+	}
+	return 0;
+}
+
 static int
 mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 			     const struct rte_cryptodev_qp_conf *qp_conf,
@@ -278,7 +341,7 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	struct mlx5_crypto_qp *qp;
 	uint16_t log_nb_desc = rte_log2_u32(qp_conf->nb_descriptors);
 	uint32_t umem_size = RTE_BIT32(log_nb_desc) *
-			      MLX5_CRYPTO_WQE_SET_SIZE +
+			      priv->wqe_set_size +
 			      sizeof(*qp->db_rec) * 2;
 	uint32_t alloc_size = sizeof(*qp);
 	struct mlx5_devx_cq_attr cq_attr = {
@@ -286,7 +349,9 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	};
 
 	alloc_size = RTE_ALIGN(alloc_size, RTE_CACHE_LINE_SIZE);
-	alloc_size += sizeof(struct rte_crypto_op *) * RTE_BIT32(log_nb_desc);
+	alloc_size += (sizeof(struct rte_crypto_op *) +
+		       sizeof(struct mlx5_devx_obj *)) *
+		       RTE_BIT32(log_nb_desc);
 	qp = rte_zmalloc_socket(__func__, alloc_size, RTE_CACHE_LINE_SIZE,
 				socket_id);
 	if (qp == NULL) {
@@ -330,8 +395,7 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	attr.wq_umem_id = qp->umem_obj->umem_id;
 	attr.wq_umem_offset = 0;
 	attr.dbr_umem_id = qp->umem_obj->umem_id;
-	attr.dbr_address = RTE_BIT64(log_nb_desc) *
-			   MLX5_CRYPTO_WQE_SET_SIZE;
+	attr.dbr_address = RTE_BIT64(log_nb_desc) * priv->wqe_set_size;
 	qp->qp_obj = mlx5_devx_cmd_create_qp(priv->ctx, &attr);
 	if (qp->qp_obj == NULL) {
 		DRV_LOG(ERR, "Failed to create QP(%u).", rte_errno);
@@ -340,8 +404,17 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	qp->db_rec = RTE_PTR_ADD(qp->umem_buf, (uintptr_t)attr.dbr_address);
 	if (mlx5_crypto_qp2rts(qp))
 		goto error;
-	qp->ops = (struct rte_crypto_op **)RTE_ALIGN((uintptr_t)(qp + 1),
+	qp->mkey = (struct mlx5_devx_obj **)RTE_ALIGN((uintptr_t)(qp + 1),
 							   RTE_CACHE_LINE_SIZE);
+	qp->ops = (struct rte_crypto_op **)(qp->mkey + RTE_BIT32(log_nb_desc));
+	qp->entries_n = 1 << log_nb_desc;
+	if (mlx5_crypto_indirect_mkeys_prepare(priv, qp)) {
+		DRV_LOG(ERR, "Cannot allocate indirect memory regions.");
+		rte_errno = ENOMEM;
+		goto error;
+	}
+	mlx5_crypto_qp_init(priv, qp);
+	qp->priv = priv;
 	dev->data->queue_pairs[qp_id] = qp;
 	return 0;
 error:
diff --git a/drivers/crypto/mlx5/mlx5_crypto.h b/drivers/crypto/mlx5/mlx5_crypto.h
index 81452bd700..52fcf5217f 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.h
+++ b/drivers/crypto/mlx5/mlx5_crypto.h
@@ -16,7 +16,6 @@
 
 #define MLX5_CRYPTO_DEK_HTABLE_SZ (1 << 11)
 #define MLX5_CRYPTO_KEY_LENGTH 80
-#define MLX5_CRYPTO_WQE_SET_SIZE 1024
 
 struct mlx5_crypto_priv {
 	TAILQ_ENTRY(mlx5_crypto_priv) next;
@@ -24,6 +23,7 @@ struct mlx5_crypto_priv {
 	struct rte_pci_device *pci_dev;
 	struct rte_cryptodev *crypto_dev;
 	void *uar; /* User Access Region. */
+	volatile uint64_t *uar_addr;
 	uint32_t pdn; /* Protection Domain number. */
 	uint32_t max_segs_num; /* Maximum supported data segs. */
 	struct ibv_pd *pd;
@@ -40,13 +40,21 @@ struct mlx5_crypto_priv {
 };
 
 struct mlx5_crypto_qp {
+	struct mlx5_crypto_priv *priv;
 	struct mlx5_devx_cq cq_obj;
 	struct mlx5_devx_obj *qp_obj;
+	struct rte_cryptodev_stats stats;
 	struct mlx5dv_devx_umem *umem_obj;
 	void *umem_buf;
 	volatile uint32_t *db_rec;
 	struct rte_crypto_op **ops;
+	struct mlx5_devx_obj **mkey; /* WQE's indirect mekys. */
 	struct mlx5_mr_ctrl mr_ctrl;
+	uint8_t *wqe;
+	uint16_t entries_n;
+	uint16_t pi;
+	uint16_t ci;
+	uint16_t db_pi;
 };
 
 struct mlx5_crypto_dek {
-- 
2.25.1

[dpdk-dev] [PATCH v3 13/15] crypto/mlx5: add enqueue and dequeue operations

[Matan Azrad]

From: Suanming Mou <suanmingm@nvidia.com>

The crypto operations are done with the WQE set which contains
one UMR WQE and one rdma write WQE. Most segments of the WQE
set are initialized properly during queue setup, only limited
segments are initialized according to the crypto detail in the
datapath process.

This commit adds the enquue and dequeue operations and updates
the WQE set segments accordingly.

Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Signed-off-by: Matan Azrad <matan@nvidia.com>
---
 drivers/crypto/mlx5/mlx5_crypto.c | 243 +++++++++++++++++++++++++++++-
 drivers/crypto/mlx5/mlx5_crypto.h |   3 +
 2 files changed, 241 insertions(+), 5 deletions(-)

diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index 7bffe08bfe..fd0afb9fb2 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -268,6 +268,239 @@ mlx5_crypto_qp2rts(struct mlx5_crypto_qp *qp)
 	return 0;
 }
 
+static __rte_noinline uint32_t
+mlx5_crypto_get_block_size(struct rte_crypto_op *op)
+{
+	uint32_t bl = op->sym->cipher.data.length;
+
+	switch (bl) {
+	case (1 << 20):
+		return RTE_BE32(MLX5_BLOCK_SIZE_1MB << MLX5_BLOCK_SIZE_OFFSET);
+	case (1 << 12):
+		return RTE_BE32(MLX5_BLOCK_SIZE_4096B <<
+				MLX5_BLOCK_SIZE_OFFSET);
+	case (1 << 9):
+		return RTE_BE32(MLX5_BLOCK_SIZE_512B << MLX5_BLOCK_SIZE_OFFSET);
+	default:
+		DRV_LOG(ERR, "Unknown block size: %u.", bl);
+		return UINT32_MAX;
+	}
+}
+
+static __rte_always_inline uint32_t
+mlx5_crypto_klm_set(struct mlx5_crypto_priv *priv, struct mlx5_crypto_qp *qp,
+		      struct rte_mbuf *mbuf, struct mlx5_wqe_dseg *klm,
+		      uint32_t offset, uint32_t *remain)
+{
+	uint32_t data_len = (rte_pktmbuf_data_len(mbuf) - offset);
+	uintptr_t addr = rte_pktmbuf_mtod_offset(mbuf, uintptr_t, offset);
+
+	if (data_len > *remain)
+		data_len = *remain;
+	*remain -= data_len;
+	klm->bcount = rte_cpu_to_be_32(data_len);
+	klm->pbuf = rte_cpu_to_be_64(addr);
+	klm->lkey = mlx5_mr_addr2mr_bh(priv->pd, 0,
+		&priv->mr_scache, &qp->mr_ctrl, addr,
+		!!(mbuf->ol_flags & EXT_ATTACHED_MBUF));
+	return klm->lkey;
+
+}
+
+static __rte_always_inline uint32_t
+mlx5_crypto_klms_set(struct mlx5_crypto_priv *priv, struct mlx5_crypto_qp *qp,
+		     struct rte_crypto_op *op, struct rte_mbuf *mbuf,
+		     struct mlx5_wqe_dseg *klm)
+{
+	uint32_t remain_len = op->sym->cipher.data.length;
+	uint32_t nb_segs = mbuf->nb_segs;
+	uint32_t klm_n = 1;
+
+	/* First mbuf needs to take the cipher offset. */
+	if (unlikely(mlx5_crypto_klm_set(priv, qp, mbuf, klm,
+		     op->sym->cipher.data.offset, &remain_len) == UINT32_MAX)) {
+		op->status = RTE_CRYPTO_OP_STATUS_ERROR;
+		return 0;
+	}
+	while (remain_len) {
+		nb_segs--;
+		mbuf = mbuf->next;
+		if (unlikely(mbuf == NULL || nb_segs == 0)) {
+			op->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS;
+			return 0;
+		}
+		if (unlikely(mlx5_crypto_klm_set(priv, qp, mbuf, klm, 0,
+						 &remain_len) == UINT32_MAX)) {
+			op->status = RTE_CRYPTO_OP_STATUS_ERROR;
+			return 0;
+		}
+		klm_n++;
+	}
+	return klm_n;
+}
+
+static __rte_always_inline int
+mlx5_crypto_wqe_set(struct mlx5_crypto_priv *priv,
+			 struct mlx5_crypto_qp *qp,
+			 struct rte_crypto_op *op,
+			 struct mlx5_umr_wqe *umr)
+{
+	struct mlx5_crypto_session *sess = get_sym_session_private_data
+				(op->sym->session, mlx5_crypto_driver_id);
+	struct mlx5_wqe_cseg *cseg = &umr->ctr;
+	struct mlx5_wqe_mkey_cseg *mkc = &umr->mkc;
+	struct mlx5_wqe_dseg *klms = &umr->kseg[0];
+	struct mlx5_wqe_umr_bsf_seg *bsf = ((struct mlx5_wqe_umr_bsf_seg *)
+				      RTE_PTR_ADD(umr, priv->umr_wqe_size)) - 1;
+	uint16_t nop_ds;
+	/* Set UMR WQE. */
+	uint32_t klm_n = mlx5_crypto_klms_set(priv, qp, op,
+			op->sym->m_dst ? op->sym->m_dst : op->sym->m_src, klms);
+
+	if (unlikely(klm_n == 0))
+		return 0;
+	bsf->bs_bpt_eo_es = sess->bs_bpt_eo_es;
+	if (unlikely(!sess->bsp_res)) {
+		bsf->bsp_res = mlx5_crypto_get_block_size(op);
+		if (unlikely(bsf->bsp_res == UINT32_MAX)) {
+			op->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS;
+			return 0;
+		}
+	} else {
+		bsf->bsp_res = sess->bsp_res;
+	}
+	bsf->raw_data_size = rte_cpu_to_be_32(op->sym->cipher.data.length);
+	memcpy(bsf->xts_initial_tweak,
+	       rte_crypto_op_ctod_offset(op, uint8_t *, sess->iv_offset), 16);
+	bsf->res_dp = sess->dek_id;
+	cseg->opcode = rte_cpu_to_be_32((qp->db_pi << 8) | MLX5_OPCODE_UMR);
+	mkc->len = rte_cpu_to_be_64(op->sym->cipher.data.length);
+	/* Set RDMA_WRITE WQE. */
+	cseg = RTE_PTR_ADD(cseg, priv->umr_wqe_size);
+	klms = RTE_PTR_ADD(cseg, sizeof(struct mlx5_rdma_write_wqe));
+	cseg->opcode = rte_cpu_to_be_32((qp->db_pi << 8) |
+					MLX5_OPCODE_RDMA_WRITE);
+	if (op->sym->m_dst != op->sym->m_src) {
+		klm_n = mlx5_crypto_klms_set(priv, qp, op, op->sym->m_src,
+					     klms);
+		if (unlikely(klm_n == 0))
+			return 0;
+	} else {
+		memcpy(klms, &umr->kseg[0], sizeof(*klms) * klm_n);
+	}
+	cseg->sq_ds = rte_cpu_to_be_32((qp->qp_obj->id << 8) | (2 + klm_n));
+	qp->db_pi += priv->wqe_stride;
+	/* Set NOP WQE if needed. */
+	klm_n = RTE_ALIGN(klm_n + 2, 4) - 2;
+	nop_ds = priv->max_rdmaw_klm_n - klm_n;
+	if (nop_ds) {
+		cseg = (struct mlx5_wqe_cseg *)(klms + klm_n);
+		cseg->opcode = rte_cpu_to_be_32(((qp->db_pi - (nop_ds >> 2)) <<
+						 8) | MLX5_OPCODE_NOP);
+		cseg->sq_ds = rte_cpu_to_be_32((qp->qp_obj->id << 8) | nop_ds);
+	}
+	qp->wqe = (uint8_t *)cseg;
+	return 1;
+}
+
+static __rte_always_inline void
+mlx5_crypto_uar_write(uint64_t val, struct mlx5_crypto_priv *priv)
+{
+#ifdef RTE_ARCH_64
+	*priv->uar_addr = val;
+#else /* !RTE_ARCH_64 */
+	rte_spinlock_lock(&priv->uar32_sl);
+	*(volatile uint32_t *)priv->uar_addr = val;
+	rte_io_wmb();
+	*((volatile uint32_t *)priv->uar_addr + 1) = val >> 32;
+	rte_spinlock_unlock(&priv->uar32_sl);
+#endif
+}
+
+static uint16_t
+mlx5_crypto_enqueue_burst(void *queue_pair, struct rte_crypto_op **ops,
+			  uint16_t nb_ops)
+{
+	struct mlx5_crypto_qp *qp = queue_pair;
+	struct mlx5_crypto_priv *priv = qp->priv;
+	struct mlx5_umr_wqe *umr;
+	struct rte_crypto_op *op;
+	uint16_t mask = qp->entries_n - 1;
+	uint16_t remain = qp->entries_n - (qp->pi - qp->ci);
+
+	if (remain < nb_ops)
+		nb_ops = remain;
+	else
+		remain = nb_ops;
+	if (unlikely(remain == 0))
+		return 0;
+	do {
+		op = *ops++;
+		umr = RTE_PTR_ADD(qp->umem_buf, priv->wqe_set_size * qp->pi);
+		if (unlikely(mlx5_crypto_wqe_set(priv, qp, op, umr) == 0))
+			break;
+		qp->ops[qp->pi] = op;
+		qp->pi = (qp->pi + 1) & mask;
+	} while (--remain);
+	rte_io_wmb();
+	qp->db_rec[MLX5_SND_DBR] = rte_cpu_to_be_32(qp->db_pi);
+	rte_wmb();
+	mlx5_crypto_uar_write(*(volatile uint64_t *)qp->wqe, qp->priv);
+	rte_wmb();
+	return nb_ops;
+}
+
+static __rte_noinline void
+mlx5_crypto_cqe_err_handle(struct mlx5_crypto_qp *qp, struct rte_crypto_op *op)
+{
+	const uint32_t idx = qp->ci & (qp->entries_n - 1);
+	volatile struct mlx5_err_cqe *cqe = (volatile struct mlx5_err_cqe *)
+							&qp->cq_obj.cqes[idx];
+
+	op->status = RTE_CRYPTO_OP_STATUS_ERROR;
+	DRV_LOG(ERR, "CQE ERR:%x.\n", rte_be_to_cpu_32(cqe->syndrome));
+}
+
+static uint16_t
+mlx5_crypto_dequeue_burst(void *queue_pair, struct rte_crypto_op **ops,
+			  uint16_t nb_ops)
+{
+	struct mlx5_crypto_qp *qp = queue_pair;
+	volatile struct mlx5_cqe *restrict cqe;
+	struct rte_crypto_op *restrict op;
+	const unsigned int cq_size = qp->entries_n;
+	const unsigned int mask = cq_size - 1;
+	uint32_t idx;
+	uint32_t next_idx = qp->ci & mask;
+	const uint16_t max = RTE_MIN((uint16_t)(qp->pi - qp->ci), nb_ops);
+	uint16_t i = 0;
+	int ret;
+
+	if (unlikely(max == 0))
+		return 0;
+	do {
+		idx = next_idx;
+		next_idx = (qp->ci + 1) & mask;
+		op = qp->ops[idx];
+		cqe = &qp->cq_obj.cqes[idx];
+		ret = check_cqe(cqe, cq_size, qp->ci);
+		rte_io_rmb();
+		if (unlikely(ret != MLX5_CQE_STATUS_SW_OWN)) {
+			if (unlikely(ret != MLX5_CQE_STATUS_HW_OWN))
+				mlx5_crypto_cqe_err_handle(qp, op);
+			break;
+		}
+		op->status = RTE_CRYPTO_OP_STATUS_SUCCESS;
+		ops[i++] = op;
+		qp->ci++;
+	} while (i < max);
+	if (likely(i != 0)) {
+		rte_io_wmb();
+		qp->cq_obj.db_rec[0] = rte_cpu_to_be_32(qp->ci);
+	}
+	return i;
+}
+
 static void
 mlx5_crypto_qp_init(struct mlx5_crypto_priv *priv, struct mlx5_crypto_qp *qp)
 {
@@ -489,8 +722,9 @@ mlx5_crypto_hw_global_prepare(struct mlx5_crypto_priv *priv)
 	if (mlx5_crypto_pd_create(priv) != 0)
 		return -1;
 	priv->uar = mlx5_devx_alloc_uar(priv->ctx, -1);
-	if (priv->uar == NULL || mlx5_os_get_devx_uar_reg_addr(priv->uar) ==
-	    NULL) {
+	if (priv->uar)
+		priv->uar_addr = mlx5_os_get_devx_uar_reg_addr(priv->uar);
+	if (priv->uar == NULL || priv->uar_addr == NULL) {
 		rte_errno = errno;
 		claim_zero(mlx5_glue->dealloc_pd(priv->pd));
 		DRV_LOG(ERR, "Failed to allocate UAR.");
@@ -685,9 +919,8 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 	DRV_LOG(INFO,
 		"Crypto device %s was created successfully.", ibv->name);
 	crypto_dev->dev_ops = &mlx5_crypto_ops;
-	crypto_dev->dequeue_burst = NULL;
-	crypto_dev->enqueue_burst = NULL;
-	crypto_dev->feature_flags = RTE_CRYPTODEV_FF_HW_ACCELERATED;
+	crypto_dev->dequeue_burst = mlx5_crypto_dequeue_burst;
+	crypto_dev->enqueue_burst = mlx5_crypto_enqueue_burst;
 	crypto_dev->driver_id = mlx5_crypto_driver_id;
 	priv = crypto_dev->data->dev_private;
 	priv->ctx = ctx;
diff --git a/drivers/crypto/mlx5/mlx5_crypto.h b/drivers/crypto/mlx5/mlx5_crypto.h
index 52fcf5217f..ac4ad1834f 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.h
+++ b/drivers/crypto/mlx5/mlx5_crypto.h
@@ -37,6 +37,9 @@ struct mlx5_crypto_priv {
 	uint16_t rdmw_wqe_size;
 	uint16_t wqe_stride;
 	uint16_t max_rdmaw_klm_n;
+#ifndef RTE_ARCH_64
+	rte_spinlock_t uar32_sl;
+#endif /* RTE_ARCH_64 */
 };
 
 struct mlx5_crypto_qp {
-- 
2.25.1

[dpdk-dev] [PATCH v3 14/15] crypto/mlx5: add statistic get and reset operations

[Matan Azrad]

From: Suanming Mou <suanmingm@nvidia.com>

This commit adds mlx5 crypto statistic get and reset operations.

Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Signed-off-by: Matan Azrad <matan@nvidia.com>
---
 drivers/crypto/mlx5/mlx5_crypto.c | 39 ++++++++++++++++++++++++++++---
 1 file changed, 36 insertions(+), 3 deletions(-)

diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index fd0afb9fb2..896ca60866 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -437,11 +437,14 @@ mlx5_crypto_enqueue_burst(void *queue_pair, struct rte_crypto_op **ops,
 	do {
 		op = *ops++;
 		umr = RTE_PTR_ADD(qp->umem_buf, priv->wqe_set_size * qp->pi);
-		if (unlikely(mlx5_crypto_wqe_set(priv, qp, op, umr) == 0))
+		if (unlikely(mlx5_crypto_wqe_set(priv, qp, op, umr) == 0)) {
+			qp->stats.enqueue_err_count++;
 			break;
+		}
 		qp->ops[qp->pi] = op;
 		qp->pi = (qp->pi + 1) & mask;
 	} while (--remain);
+	qp->stats.enqueued_count += nb_ops;
 	rte_io_wmb();
 	qp->db_rec[MLX5_SND_DBR] = rte_cpu_to_be_32(qp->db_pi);
 	rte_wmb();
@@ -458,6 +461,7 @@ mlx5_crypto_cqe_err_handle(struct mlx5_crypto_qp *qp, struct rte_crypto_op *op)
 							&qp->cq_obj.cqes[idx];
 
 	op->status = RTE_CRYPTO_OP_STATUS_ERROR;
+	qp->stats.dequeue_err_count++;
 	DRV_LOG(ERR, "CQE ERR:%x.\n", rte_be_to_cpu_32(cqe->syndrome));
 }
 
@@ -497,6 +501,7 @@ mlx5_crypto_dequeue_burst(void *queue_pair, struct rte_crypto_op **ops,
 	if (likely(i != 0)) {
 		rte_io_wmb();
 		qp->cq_obj.db_rec[0] = rte_cpu_to_be_32(qp->ci);
+		qp->stats.dequeued_count += i;
 	}
 	return i;
 }
@@ -655,14 +660,42 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
 	return -1;
 }
 
+static void
+mlx5_crypto_stats_get(struct rte_cryptodev *dev,
+		      struct rte_cryptodev_stats *stats)
+{
+	int qp_id;
+
+	for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) {
+		struct mlx5_crypto_qp *qp = dev->data->queue_pairs[qp_id];
+
+		stats->enqueued_count += qp->stats.enqueued_count;
+		stats->dequeued_count += qp->stats.dequeued_count;
+		stats->enqueue_err_count += qp->stats.enqueue_err_count;
+		stats->dequeue_err_count += qp->stats.dequeue_err_count;
+	}
+}
+
+static void
+mlx5_crypto_stats_reset(struct rte_cryptodev *dev)
+{
+	int qp_id;
+
+	for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) {
+		struct mlx5_crypto_qp *qp = dev->data->queue_pairs[qp_id];
+
+		memset(&qp->stats, 0, sizeof(qp->stats));
+	}
+}
+
 static struct rte_cryptodev_ops mlx5_crypto_ops = {
 	.dev_configure			= mlx5_crypto_dev_configure,
 	.dev_start			= mlx5_crypto_dev_start,
 	.dev_stop			= mlx5_crypto_dev_stop,
 	.dev_close			= mlx5_crypto_dev_close,
 	.dev_infos_get			= mlx5_crypto_dev_infos_get,
-	.stats_get			= NULL,
-	.stats_reset			= NULL,
+	.stats_get			= mlx5_crypto_stats_get,
+	.stats_reset			= mlx5_crypto_stats_reset,
 	.queue_pair_setup		= mlx5_crypto_queue_pair_setup,
 	.queue_pair_release		= mlx5_crypto_queue_pair_release,
 	.sym_session_get_size		= mlx5_crypto_sym_session_get_size,
-- 
2.25.1

[dpdk-dev] [PATCH v3 15/15] crypto/mlx5: set feature flags and capabilities

[Matan Azrad]

From: Shiri Kuzin <shirik@nvidia.com>

Add the supported capabilities to the crypto driver.

Add supported feature flags.

Add crypto driver documentation.

Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
Signed-off-by: Matan Azrad <matan@nvidia.com>
---
 doc/guides/cryptodevs/features/mlx5.ini |  37 ++++++
 doc/guides/cryptodevs/index.rst         |   1 +
 doc/guides/cryptodevs/mlx5.rst          | 152 ++++++++++++++++++++++++
 doc/guides/rel_notes/release_21_05.rst  |   5 +
 drivers/crypto/mlx5/mlx5_crypto.c       |  39 +++++-
 5 files changed, 231 insertions(+), 3 deletions(-)
 create mode 100644 doc/guides/cryptodevs/features/mlx5.ini
 create mode 100644 doc/guides/cryptodevs/mlx5.rst

diff --git a/doc/guides/cryptodevs/features/mlx5.ini b/doc/guides/cryptodevs/features/mlx5.ini
new file mode 100644
index 0000000000..a89526add0
--- /dev/null
+++ b/doc/guides/cryptodevs/features/mlx5.ini
@@ -0,0 +1,37 @@
+;
+; Features of a mlx5 crypto driver.
+;
+; Refer to default.ini for the full list of available PMD features.
+;
+[Features]
+Symmetric crypto       = Y
+HW Accelerated         = Y
+In Place SGL           = Y
+OOP SGL In SGL Out     = Y
+OOP SGL In LB  Out     = Y
+OOP LB  In SGL Out     = Y
+OOP LB  In LB  Out     = Y
+Cipher multiple data units = Y
+Cipher wrapped key     = Y
+
+;
+; Supported crypto algorithms of a mlx5 crypto driver.
+;
+[Cipher]
+AES XTS (128)  = Y
+AES XTS (256)  = Y
+
+;
+; Supported authentication algorithms of a mlx5 crypto driver.
+;
+[Auth]
+
+;
+; Supported AEAD algorithms of a mlx5 crypto driver.
+;
+[AEAD]
+
+;
+; Supported Asymmetric algorithms of a mlx5 crypto driver.
+;
+[Asymmetric]
diff --git a/doc/guides/cryptodevs/index.rst b/doc/guides/cryptodevs/index.rst
index 279f56a002..747409c441 100644
--- a/doc/guides/cryptodevs/index.rst
+++ b/doc/guides/cryptodevs/index.rst
@@ -22,6 +22,7 @@ Crypto Device Drivers
     octeontx
     octeontx2
     openssl
+    mlx5
     mvsam
     nitrox
     null
diff --git a/doc/guides/cryptodevs/mlx5.rst b/doc/guides/cryptodevs/mlx5.rst
new file mode 100644
index 0000000000..4ccec78be8
--- /dev/null
+++ b/doc/guides/cryptodevs/mlx5.rst
@@ -0,0 +1,152 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+   Copyright 2021 Mellanox Technologies, Ltd
+
+.. include:: <isonum.txt>
+
+MLX5 Crypto Driver
+==================
+
+The MLX5 crypto driver library
+(**librte_crypto_mlx5**) provides support for **Mellanox ConnectX-6**
+family adapters.
+
+Overview
+--------
+
+The device can provide disk encryption services, allowing data encryption
+and decryption towards a disk. Having all encryption/decryption
+operations done in a single device can reduce cost and overheads of the related
+FIPS certification, as ConnectX-6 is FIPS 140-2 level-2 ready.
+The encryption cipher is AES-XTS of 256/512 key size.
+
+MKEY is a memory region object in the hardware, that holds address translation information and
+attributes per memory area. Its ID must be tied to addresses provided to the hardware.
+The encryption operations are performed with MKEY read\write transactions, when
+the MKEY is configured to perform crypto operations.
+
+The encryption does not require text to be aligned to the AES block size (128b).
+
+In order to move the device to crypto operational mode, credential and KEK
+(Key Encrypting Key) should be set as the first step.
+The credential will be used by the software in order to perform crypto login, and the KEK is
+the AES Key Wrap Algorithm (rfc3394) key that will be used for sensitive data
+wrapping.
+The credential and the AES-XTS keys should be provided to the hardware, as ciphertext
+encrypted by the KEK.
+
+A keytag (64 bits) should be appended to the AES-XTS keys (before wrapping),
+and will be validated when the hardware attempts to access it.
+
+For security reasons and to increase robustness, this driver only deals with virtual
+memory addresses. The way resources allocations are handled by the kernel,
+combined with hardware specifications that allow handling virtual memory
+addresses directly, ensure that DPDK applications cannot access random
+physical memory (or memory that does not belong to the current process).
+
+The PMD uses libibverbs and libmlx5 to access the device firmware or to
+access the hardware components directly.
+There are different levels of objects and bypassing abilities.
+To get the best performances:
+
+- Verbs is a complete high-level generic API.
+- Direct Verbs is a device-specific API.
+- DevX allows to access firmware objects.
+
+Enabling librte_crypto_mlx5 causes DPDK applications to be linked against
+libibverbs.
+
+Mellanox mlx5 PCI device can be probed by a number of different PCI devices, such as
+net / vDPA / RegEx. To select the crypto PMD, ``class=crypto``
+should be specified as a device parameter. The crypto device can be probed and
+used with other Mellanox classes by adding more options in the class.
+For example: ``class=net:crypto`` will probe both the net PMD and the crypto
+PMD.
+
+When crypto engines are defined to work in wrapped import method, they come out
+of the factory in Commissioning mode, and thus, cannot be used for crypto operations
+yet. A dedicated tool is used for changing the mode from Commissioning to
+Operational, while setting the first import_KEK and credential in plaintext.
+The mlxreg dedicated tool should be used as follows:
+
+- Set CRYPTO_OPERATIONAL register to set the device in crypto operational mode.
+
+  The input to this tool is:
+    The first credential in plaintext, 40B.
+    The first import_KEK in plaintext: kek size 0 for 16B or 1 for 32B, kek data.
+
+  Example:
+  mlxreg -d /dev/mst/mt4123_pciconf0 --reg_name CRYPTO_OPERATIONAL --get
+
+  The "wrapped_crypto_operational" value will be "0x00000000".
+  The command to set the register should be executed only once, and all the
+  values mentioned above should be specified in the same command.
+
+  Example:
+  mlxreg -d /dev/mst/mt4123_pciconf0 --reg_name CRYPTO_OPERATIONAL
+  --set "credential[0]=0x10000000, credential[1]=0x10000000, kek[0]=0x00000000"
+
+  All values not specified will remain 0.
+  "wrapped_crypto_going_to_commissioning" and  "wrapped_crypto_operational"
+  should not be specified.
+
+  All the device ports should set it in order to move to operational mode.
+
+- Query CRYPTO_OPERATIONAL register to make sure the device is in Operational
+  mode.
+
+  Example:
+  mlxreg -d /dev/mst/mt4123_pciconf0 --reg_name CRYPTO_OPERATIONAL --get
+  The "wrapped_crypto_operational" value will be "0x00000001" if the mode was
+  successfully changed to operational mode.
+
+
+Driver options
+--------------
+
+- ``class`` parameter [string]
+
+  Select the class of the driver that should probe the device.
+  `crypto` for the mlx5 crypto driver.
+
+- ``wcs_file`` parameter [string] - mandatory
+
+  File path including only the wrapped credential in string format of hexadecimal
+  numbers, represent 48 bytes (8 bytes IV added by the AES key wrap algorithm).
+
+- ``import_kek_id`` parameter [int]
+
+  The identifier of the KEK, default value is 0 represents the operational
+  register import_kek..
+
+- ``credential_id`` parameter [int]
+
+  The identifier of the credential, default value is 0 represents the operational
+  register credential.
+
+- ``max_segs_num`` parameter [int]
+
+  Maximum number of mbuf chain segments(src or dest), default value is 8.
+
+- ``keytag`` parameter [int]
+
+  The plaintext of the keytag appanded to the AES-XTS keys, default value is 0.
+
+
+Limitations
+-----------
+
+- AES-XTS keys provided in Xform must include keytag and should be wrappend.
+- The supported data-unit lengths are: 512B, 1KB, 1MB. In case the `dataunit_len`
+  is not provided in the cipher Xform, the OP length is limitted to the above values.
+
+
+Supported NICs
+--------------
+
+* Mellanox\ |reg| ConnectX\ |reg|-6 200G MCX654106A-HCAT (2x200G)
+
+Prerequisites
+-------------
+
+- Mellanox OFED version: **5.3**
+  see :doc:`../../nics/mlx5` guide for more Mellanox OFED details.
diff --git a/doc/guides/rel_notes/release_21_05.rst b/doc/guides/rel_notes/release_21_05.rst
index 0970a2331a..9030fd8b98 100644
--- a/doc/guides/rel_notes/release_21_05.rst
+++ b/doc/guides/rel_notes/release_21_05.rst
@@ -275,6 +275,11 @@ New Features
   * Added support for crypto adapter forward mode in octeontx2 event and crypto
     device driver.
 
+* **Added support for Nvidia crypto device driver.**
+
+  * Added mlx5 crypto driver to support AES-XTS cipher operations.
+    the first device to support it is ConnectX-6.
+
 
 Removed Items
 -------------
diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index 896ca60866..f318ff4682 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -22,6 +22,14 @@
 #define MLX5_CRYPTO_LOG_NAME pmd.crypto.mlx5
 #define MLX5_CRYPTO_MAX_QPS 1024
 #define MLX5_CRYPTO_MAX_SEGS 56
+#define MLX5_CRYPTO_FEATURE_FLAGS \
+	(RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO | RTE_CRYPTODEV_FF_HW_ACCELERATED | \
+	RTE_CRYPTODEV_FF_IN_PLACE_SGL | RTE_CRYPTODEV_FF_OOP_SGL_IN_SGL_OUT | \
+	RTE_CRYPTODEV_FF_OOP_SGL_IN_LB_OUT | \
+	RTE_CRYPTODEV_FF_OOP_LB_IN_SGL_OUT | \
+	RTE_CRYPTODEV_FF_OOP_LB_IN_LB_OUT | \
+	RTE_CRYPTODEV_FF_CIPHER_WRAPPED_KEY | \
+	RTE_CRYPTODEV_FF_CIPHER_MULTIPLE_DATA_UNITS)
 
 TAILQ_HEAD(mlx5_crypto_privs, mlx5_crypto_priv) mlx5_crypto_priv_list =
 				TAILQ_HEAD_INITIALIZER(mlx5_crypto_priv_list);
@@ -31,8 +39,32 @@ int mlx5_crypto_logtype;
 
 uint8_t mlx5_crypto_driver_id;
 
-const struct rte_cryptodev_capabilities
-		mlx5_crypto_caps[RTE_CRYPTO_OP_TYPE_UNDEFINED];
+const struct rte_cryptodev_capabilities mlx5_crypto_caps[] = {
+	{		/* AES XTS */
+		.op = RTE_CRYPTO_OP_TYPE_SYMMETRIC,
+		{.sym = {
+			.xform_type = RTE_CRYPTO_SYM_XFORM_CIPHER,
+			{.cipher = {
+				.algo = RTE_CRYPTO_CIPHER_AES_XTS,
+				.block_size = 16,
+				.key_size = {
+					.min = 32,
+					.max = 64,
+					.increment = 32
+				},
+				.iv_size = {
+					.min = 16,
+					.max = 16,
+					.increment = 0
+				},
+				.dataunit_set =
+				RTE_CRYPTO_CIPHER_DATA_UNIT_LEN_512_BYTES |
+				RTE_CRYPTO_CIPHER_DATA_UNIT_LEN_4096_BYTES,
+			}, }
+		}, }
+	},
+};
+
 
 static const char mlx5_crypto_drv_name[] = RTE_STR(MLX5_CRYPTO_DRIVER_NAME);
 
@@ -67,7 +99,7 @@ mlx5_crypto_dev_infos_get(struct rte_cryptodev *dev,
 	RTE_SET_USED(dev);
 	if (dev_info != NULL) {
 		dev_info->driver_id = mlx5_crypto_driver_id;
-		dev_info->feature_flags = 0;
+		dev_info->feature_flags = MLX5_CRYPTO_FEATURE_FLAGS;
 		dev_info->capabilities = mlx5_crypto_caps;
 		dev_info->max_nb_queue_pairs = MLX5_CRYPTO_MAX_QPS;
 		dev_info->min_mbuf_headroom_req = 0;
@@ -954,6 +986,7 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
 	crypto_dev->dev_ops = &mlx5_crypto_ops;
 	crypto_dev->dequeue_burst = mlx5_crypto_dequeue_burst;
 	crypto_dev->enqueue_burst = mlx5_crypto_enqueue_burst;
+	crypto_dev->feature_flags = MLX5_CRYPTO_FEATURE_FLAGS;
 	crypto_dev->driver_id = mlx5_crypto_driver_id;
 	priv = crypto_dev->data->dev_private;
 	priv->ctx = ctx;
-- 
2.25.1

Re: [dpdk-dev] [PATCH v3 00/15] mlx5 common part for crypto driver

[Thomas Monjalon]

04/05/2021 19:54, Matan Azrad:
> Dekel Peled (12):
>   common/mlx5: remove redundant spaces in header file
>   common/mlx5: update GENEVE TLV OPT obj name
>   common/mlx5: optimize read of general obj type caps
>   common/mlx5: add HCA cap for AES-XTS crypto
>   common/mlx5: support general object DEK create op
>   common/mlx5: adjust DevX MKEY fields for crypto
>   common/mlx5: support general obj IMPORT KEK create
>   common/mlx5: support general obj CRYPTO LOGIN create
>   common/mlx5: add crypto BSF struct and defines
>   common/mlx5: support general obj CREDENTIAL create
>   common/mlx5: add crypto register structs and defs
>   common/mlx5: support register write access
> 
> Shiri Kuzin (2):
>   common/mlx5: share hash list tool
>   common/mlx5: share get ib device match function
> 
> Suanming Mou (1):
>   common/mlx5: add UMR and RDMA write WQE defines

Applied, thanks.

Re: [dpdk-dev] [EXT] [PATCH v3 01/15] drivers: introduce mlx5 crypto PMD

[Akhil Goyal]

> diff --git a/drivers/crypto/mlx5/meson.build
> b/drivers/crypto/mlx5/meson.build
> new file mode 100644
> index 0000000000..fd00283665
> --- /dev/null
> +++ b/drivers/crypto/mlx5/meson.build
> @@ -0,0 +1,26 @@
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright 2021 Mellanox Technologies, Ltd
> +
> +if not is_linux
> +	build = false
> +	reason = 'only supported on Linux'
> +	subdir_done()
> +endif
> +
> +fmt_name = 'mlx5_crypto'
> +deps += ['common_mlx5', 'eal', 'cryptodev']

I don't see a reason for eal dependency. Do you really need it?

> +sources = files(
> +	'mlx5_crypto.c',
> +)
> +cflags_options = [
> +	'-std=c11',
> +	'-Wno-strict-prototypes',
> +	'-D_BSD_SOURCE',
> +	'-D_DEFAULT_SOURCE',
> +	'-D_XOPEN_SOURCE=600',
> +]

Any specific reason to use '-Wno-strict-prototypes'
Compilation works without this as well.

> +foreach option:cflags_options
> +	if cc.has_argument(option)
> +		cflags += option
> +	endif
> +endforeach

Re: [dpdk-dev] [EXT] [PATCH v3 15/15] crypto/mlx5: set feature flags and capabilities

[Akhil Goyal]

> +
> +Overview
> +--------
> +
> +The device can provide disk encryption services, allowing data encryption
> +and decryption towards a disk. Having all encryption/decryption
> +operations done in a single device can reduce cost and overheads of the
> related
> +FIPS certification, as ConnectX-6 is FIPS 140-2 level-2 ready.
> +The encryption cipher is AES-XTS of 256/512 key size.

256/512 bit key size

> +
> +MKEY is a memory region object in the hardware, that holds address
> translation information and
> +attributes per memory area. Its ID must be tied to addresses provided to
> the hardware.
> +The encryption operations are performed with MKEY read\write

read/write

> transactions, when
> +the MKEY is configured to perform crypto operations.
> +
> +The encryption does not require text to be aligned to the AES block size
> (128b).
> +
> +In order to move the device to crypto operational mode, credential and KEK
> +(Key Encrypting Key) should be set as the first step.
> +The credential will be used by the software in order to perform crypto login,
> and the KEK is
> +the AES Key Wrap Algorithm (rfc3394) key that will be used for sensitive
> data
> +wrapping.
> +The credential and the AES-XTS keys should be provided to the hardware, as
> ciphertext
> +encrypted by the KEK.
> +
> +A keytag (64 bits) should be appended to the AES-XTS keys (before
> wrapping),
> +and will be validated when the hardware attempts to access it.
> +
> +For security reasons and to increase robustness, this driver only deals with
> virtual
> +memory addresses. The way resources allocations are handled by the
> kernel,
> +combined with hardware specifications that allow handling virtual memory
> +addresses directly, ensure that DPDK applications cannot access random
> +physical memory (or memory that does not belong to the current process).
> +
> +The PMD uses libibverbs and libmlx5 to access the device firmware or to
> +access the hardware components directly.
> +There are different levels of objects and bypassing abilities.
> +To get the best performances:
> +
> +- Verbs is a complete high-level generic API.
> +- Direct Verbs is a device-specific API.
> +- DevX allows to access firmware objects.
> +
> +Enabling librte_crypto_mlx5 causes DPDK applications to be linked against
> +libibverbs.
> +
> +Mellanox mlx5 PCI device can be probed by a number of different PCI
> devices, such as
> +net / vDPA / RegEx. To select the crypto PMD, ``class=crypto``
> +should be specified as a device parameter. The crypto device can be probed
> and
> +used with other Mellanox classes by adding more options in the class.
> +For example: ``class=net:crypto`` will probe both the net PMD and the
> crypto
> +PMD.
> +
> +When crypto engines are defined to work in wrapped import method, they
> come out
> +of the factory in Commissioning mode, and thus, cannot be used for crypto
> operations
> +yet. A dedicated tool is used for changing the mode from Commissioning to
> +Operational, while setting the first import_KEK and credential in plaintext.
> +The mlxreg dedicated tool should be used as follows:
> +
> +- Set CRYPTO_OPERATIONAL register to set the device in crypto operational
> mode.
> +
> +  The input to this tool is:
> +    The first credential in plaintext, 40B.
> +    The first import_KEK in plaintext: kek size 0 for 16B or 1 for 32B, kek data.
> +
> +  Example:
> +  mlxreg -d /dev/mst/mt4123_pciconf0 --reg_name CRYPTO_OPERATIONAL -
> -get
> +
> +  The "wrapped_crypto_operational" value will be "0x00000000".
> +  The command to set the register should be executed only once, and all the
> +  values mentioned above should be specified in the same command.
> +
> +  Example:
> +  mlxreg -d /dev/mst/mt4123_pciconf0 --reg_name CRYPTO_OPERATIONAL
> +  --set "credential[0]=0x10000000, credential[1]=0x10000000,
> kek[0]=0x00000000"
> +
> +  All values not specified will remain 0.
> +  "wrapped_crypto_going_to_commissioning" and
> "wrapped_crypto_operational"
> +  should not be specified.
> +
> +  All the device ports should set it in order to move to operational mode.
> +
> +- Query CRYPTO_OPERATIONAL register to make sure the device is in
> Operational
> +  mode.
> +
> +  Example:
> +  mlxreg -d /dev/mst/mt4123_pciconf0 --reg_name CRYPTO_OPERATIONAL -
> -get
> +  The "wrapped_crypto_operational" value will be "0x00000001" if the
> mode was
> +  successfully changed to operational mode.
> +
> +
> +Driver options
> +--------------
> +
> +- ``class`` parameter [string]
> +
> +  Select the class of the driver that should probe the device.
> +  `crypto` for the mlx5 crypto driver.
> +
> +- ``wcs_file`` parameter [string] - mandatory
> +
> +  File path including only the wrapped credential in string format of
> hexadecimal
> +  numbers, represent 48 bytes (8 bytes IV added by the AES key wrap
> algorithm).
> +
> +- ``import_kek_id`` parameter [int]
> +
> +  The identifier of the KEK, default value is 0 represents the operational
> +  register import_kek..
> +
> +- ``credential_id`` parameter [int]
> +
> +  The identifier of the credential, default value is 0 represents the
> operational
> +  register credential.
> +
> +- ``max_segs_num`` parameter [int]
> +
> +  Maximum number of mbuf chain segments(src or dest), default value is 8.
> +
> +- ``keytag`` parameter [int]
> +
> +  The plaintext of the keytag appanded to the AES-XTS keys, default value is
> 0.
> +
> +
> +Limitations
> +-----------
> +
> +- AES-XTS keys provided in Xform must include keytag and should be
> wrappend.
> +- The supported data-unit lengths are: 512B, 1KB, 1MB. In case the
> `dataunit_len`
> +  is not provided in the cipher Xform, the OP length is limitted to the above

Xform should be xform

Spell check  - limited

> values.
> +
> +
> +Supported NICs
> +--------------
> +
> +* Mellanox\ |reg| ConnectX\ |reg|-6 200G MCX654106A-HCAT (2x200G)
> +
> +Prerequisites
> +-------------
> +
> +- Mellanox OFED version: **5.3**
> +  see :doc:`../../nics/mlx5` guide for more Mellanox OFED details.

Since the driver is by default compiled off due to the dependency on external
Libraries, I would recommend to add few lines here as well for compilation.
Like to compile rdma-core and set PKG_CONFIG_LIBDIR.

And I do not see any updates to the test application for testing this driver.
Is this driver really tested or is it work in progress? If it is work in progress,
We should defer this PMD to next release.

> diff --git a/doc/guides/rel_notes/release_21_05.rst
> b/doc/guides/rel_notes/release_21_05.rst
> index 0970a2331a..9030fd8b98 100644
> --- a/doc/guides/rel_notes/release_21_05.rst
> +++ b/doc/guides/rel_notes/release_21_05.rst
> @@ -275,6 +275,11 @@ New Features
>    * Added support for crypto adapter forward mode in octeontx2 event and
> crypto
>      device driver.
> 
> +* **Added support for Nvidia crypto device driver.**
> +
> +  * Added mlx5 crypto driver to support AES-XTS cipher operations.
> +    the first device to support it is ConnectX-6.

'The' 

> +
> 
>  Removed Items
>  -------------
> diff --git a/drivers/crypto/mlx5/mlx5_crypto.c
> b/drivers/crypto/mlx5/mlx5_crypto.c
> index 896ca60866..f318ff4682 100644
> --- a/drivers/crypto/mlx5/mlx5_crypto.c
> +++ b/drivers/crypto/mlx5/mlx5_crypto.c
> @@ -22,6 +22,14 @@
>  #define MLX5_CRYPTO_LOG_NAME pmd.crypto.mlx5
>  #define MLX5_CRYPTO_MAX_QPS 1024
>  #define MLX5_CRYPTO_MAX_SEGS 56
> +#define MLX5_CRYPTO_FEATURE_FLAGS \
> +	(RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO |
> RTE_CRYPTODEV_FF_HW_ACCELERATED | \
> +	RTE_CRYPTODEV_FF_IN_PLACE_SGL |
> RTE_CRYPTODEV_FF_OOP_SGL_IN_SGL_OUT | \
> +	RTE_CRYPTODEV_FF_OOP_SGL_IN_LB_OUT | \
> +	RTE_CRYPTODEV_FF_OOP_LB_IN_SGL_OUT | \
> +	RTE_CRYPTODEV_FF_OOP_LB_IN_LB_OUT | \
> +	RTE_CRYPTODEV_FF_CIPHER_WRAPPED_KEY | \
> +	RTE_CRYPTODEV_FF_CIPHER_MULTIPLE_DATA_UNITS)
> 
>  TAILQ_HEAD(mlx5_crypto_privs, mlx5_crypto_priv) mlx5_crypto_priv_list =
> 
> 	TAILQ_HEAD_INITIALIZER(mlx5_crypto_priv_list);
> @@ -31,8 +39,32 @@ int mlx5_crypto_logtype;
> 
>  uint8_t mlx5_crypto_driver_id;
> 
> -const struct rte_cryptodev_capabilities
> -		mlx5_crypto_caps[RTE_CRYPTO_OP_TYPE_UNDEFINED];
> +const struct rte_cryptodev_capabilities mlx5_crypto_caps[] = {
> +	{		/* AES XTS */
> +		.op = RTE_CRYPTO_OP_TYPE_SYMMETRIC,
> +		{.sym = {
> +			.xform_type = RTE_CRYPTO_SYM_XFORM_CIPHER,
> +			{.cipher = {
> +				.algo = RTE_CRYPTO_CIPHER_AES_XTS,
> +				.block_size = 16,
> +				.key_size = {
> +					.min = 32,
> +					.max = 64,
> +					.increment = 32
> +				},
> +				.iv_size = {
> +					.min = 16,
> +					.max = 16,
> +					.increment = 0
> +				},
> +				.dataunit_set =
> +
> 	RTE_CRYPTO_CIPHER_DATA_UNIT_LEN_512_BYTES |
> +
> 	RTE_CRYPTO_CIPHER_DATA_UNIT_LEN_4096_BYTES,
> +			}, }
> +		}, }
> +	},
> +};
> +
> 
>  static const char mlx5_crypto_drv_name[] =
> RTE_STR(MLX5_CRYPTO_DRIVER_NAME);
> 
> @@ -67,7 +99,7 @@ mlx5_crypto_dev_infos_get(struct rte_cryptodev *dev,
>  	RTE_SET_USED(dev);
>  	if (dev_info != NULL) {
>  		dev_info->driver_id = mlx5_crypto_driver_id;
> -		dev_info->feature_flags = 0;
> +		dev_info->feature_flags = MLX5_CRYPTO_FEATURE_FLAGS;
>  		dev_info->capabilities = mlx5_crypto_caps;
>  		dev_info->max_nb_queue_pairs = MLX5_CRYPTO_MAX_QPS;
>  		dev_info->min_mbuf_headroom_req = 0;
> @@ -954,6 +986,7 @@ mlx5_crypto_pci_probe(struct rte_pci_driver
> *pci_drv,
>  	crypto_dev->dev_ops = &mlx5_crypto_ops;
>  	crypto_dev->dequeue_burst = mlx5_crypto_dequeue_burst;
>  	crypto_dev->enqueue_burst = mlx5_crypto_enqueue_burst;
> +	crypto_dev->feature_flags = MLX5_CRYPTO_FEATURE_FLAGS;
>  	crypto_dev->driver_id = mlx5_crypto_driver_id;
>  	priv = crypto_dev->data->dev_private;
>  	priv->ctx = ctx;
> --
> 2.25.1

Re: [dpdk-dev] [EXT] [PATCH v3 13/15] crypto/mlx5: add enqueue and dequeue operations

[Akhil Goyal]

> @@ -685,9 +919,8 @@ mlx5_crypto_pci_probe(struct rte_pci_driver
> *pci_drv,
>  	DRV_LOG(INFO,
>  		"Crypto device %s was created successfully.", ibv->name);
>  	crypto_dev->dev_ops = &mlx5_crypto_ops;
> -	crypto_dev->dequeue_burst = NULL;
> -	crypto_dev->enqueue_burst = NULL;
> -	crypto_dev->feature_flags =
> RTE_CRYPTODEV_FF_HW_ACCELERATED;
It should be set once in the patchset.
I see this getting set in 1/15 and removed here and again set in 15/15.
And whenever you are setting it, it should be added in the .ini file as well.

> +	crypto_dev->dequeue_burst = mlx5_crypto_dequeue_burst;
> +	crypto_dev->enqueue_burst = mlx5_crypto_enqueue_burst;
>  	crypto_dev->driver_id = mlx5_crypto_driver_id;
>  	priv = crypto_dev->data->dev_private;
>  	priv->ctx = ctx;
> diff --git a/drivers/crypto/mlx5/mlx5_crypto.h
> b/drivers/crypto/mlx5/mlx5_crypto.h
> index 52fcf5217f..ac4ad1834f 100644
> --- a/drivers/crypto/mlx5/mlx5_crypto.h
> +++ b/drivers/crypto/mlx5/mlx5_crypto.h
> @@ -37,6 +37,9 @@ struct mlx5_crypto_priv {
>  	uint16_t rdmw_wqe_size;
>  	uint16_t wqe_stride;
>  	uint16_t max_rdmaw_klm_n;
> +#ifndef RTE_ARCH_64
> +	rte_spinlock_t uar32_sl;
> +#endif /* RTE_ARCH_64 */
>  };
> 
>  struct mlx5_crypto_qp {
> --
> 2.25.1

Re: [dpdk-dev] [EXT] [PATCH v3 09/15] crypto/mlx5: adjust to the multiple data unit API

[Akhil Goyal]

> From: Shiri Kuzin <shirik@nvidia.com>
> 
> In AES-XTS the data to be encrypted\decrypted does not have to be
> in multiples of 16B size, the unit of data is called data-unit.
> 
> As a result of patch [1] a new field is added to the cipher capability,
> called dataunit_set, where the devices can report the range of
> supported data-unit sizes.
> 
> The new field enables saving the data-unit size in the session
> structure to the block size pointer variable in order to support
> several data-unit sizes.
> 
> [1] https://www.mail-archive.com/dev@dpdk.org/msg205337.html 
> 
Since [1] patch is already merged, no need to refer it in the patch description.
Title can be rephrased as
Crypto/mlx5: support multiple data units

Also set feature flag in the code and the documentation in this patch.
I see that you are setting all of them in a single patch in the end.
This is not correct. It should be added where the feature is supported.
Please fix this for all the feature flags.


> Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
> ---
>  drivers/crypto/mlx5/mlx5_crypto.c | 23 +++++++++++++++++++++++
>  1 file changed, 23 insertions(+)
> 
> diff --git a/drivers/crypto/mlx5/mlx5_crypto.c
> b/drivers/crypto/mlx5/mlx5_crypto.c
> index 18b1a6be88..8cc29ced21 100644
> --- a/drivers/crypto/mlx5/mlx5_crypto.c
> +++ b/drivers/crypto/mlx5/mlx5_crypto.c
> @@ -48,6 +48,11 @@ struct mlx5_crypto_session {
>  	 * bsf_size, bsf_p_type, encryption_order and encryption standard,
>  	 * saved in big endian format.
>  	 */
> +	uint32_t bsp_res;
> +	/*
> +	 * crypto_block_size_pointer and reserved 24 bits saved in big endian
> +	 * format.
> +	 */
>  	uint32_t iv_offset:16;
>  	/* Starting point for Initialisation Vector. */
>  	struct mlx5_crypto_dek *dek; /* Pointer to dek struct. */
> @@ -171,6 +176,24 @@ mlx5_crypto_sym_session_configure(struct
> rte_cryptodev *dev,
>  			 MLX5_BSF_P_TYPE_CRYPTO <<
> MLX5_BSF_P_TYPE_OFFSET |
>  			 encryption_order <<
> MLX5_ENCRYPTION_ORDER_OFFSET |
>  			 MLX5_ENCRYPTION_STANDARD_AES_XTS);
> +	switch (xform->cipher.dataunit_len) {
> +	case 0:
> +		sess_private_data->bsp_res = 0;
> +		break;
> +	case 512:
> +		sess_private_data->bsp_res = rte_cpu_to_be_32
> +
> ((uint32_t)MLX5_BLOCK_SIZE_512B <<
> +					     MLX5_BLOCK_SIZE_OFFSET);
> +		break;
> +	case 4096:
> +		sess_private_data->bsp_res = rte_cpu_to_be_32
> +
> ((uint32_t)MLX5_BLOCK_SIZE_4096B <<
> +					     MLX5_BLOCK_SIZE_OFFSET);
> +		break;
> +	default:
> +		DRV_LOG(ERR, "Cipher data unit length is not supported.");
> +		return -ENOTSUP;
> +	}
>  	sess_private_data->iv_offset = cipher->iv.offset;
>  	sess_private_data->dek_id =
>  			rte_cpu_to_be_32(sess_private_data->dek->obj->id
> &
> --
> 2.25.1

Re: [dpdk-dev] [EXT] [PATCH v3 10/15] crypto/mlx5: add keytag device argument

[Akhil Goyal]

> From: Suanming Mou <suanmingm@nvidia.com>
> 
> A keytag is a piece of data encrypted together with a DEK.
> 
> When a DEK is referenced by an MKEY.bsf through its index, the keytag is
> also supplied in the BSF as plaintext. The HW will decrypt the DEK (and
> the attached keytag) and will fail the operation if the keytags don't
> match.
> 
> This commit adds the configuration of the keytag with devargs.
> 
> Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
> Signed-off-by: Matan Azrad <matan@nvidia.com>
> ---
Documentation for devargs should be part of this patch.
Please split the last patch accordingly.

Title can be shortened to
Crypto/mlx5: add keytag devarg

Fix other patches of devargs accordingly.

Re: [dpdk-dev] [EXT] [PATCH v3 03/15] crypto/mlx5: support session operations

[Akhil Goyal]

> +
> +static void
> +mlx5_crypto_sym_session_clear(struct rte_cryptodev *dev,
> +			      struct rte_cryptodev_sym_session *sess)
> +{
> +	struct mlx5_crypto_priv *priv = dev->data->dev_private;
> +	struct mlx5_crypto_session *sess_private_data =
> +			get_sym_session_private_data(sess, dev->driver_id);
> +
> +	if (unlikely(sess_private_data == NULL)) {
> +		DRV_LOG(ERR, "Failed to get session %p private data.",
> +				sess_private_data);
> +		return;
> +	}
> +	mlx5_crypto_dek_destroy(priv, sess_private_data->dek);
> +	DRV_LOG(DEBUG, "Session %p was cleared.", sess_private_data);
> +}

Memory leakage, mempool is not freed.
IMO, this driver is not properly tested with the unit test app.
Please add a note in the documentation that it is tested with autotest.

Re: [dpdk-dev] [EXT] [PATCH v3 01/15] drivers: introduce mlx5 crypto PMD

[Matan Azrad]

Hi Akhil

Thanks for the review, I know it was too tied for you to review, so, I really appreciate..

From: Akhil Goyal
> > diff --git a/drivers/crypto/mlx5/meson.build
> > b/drivers/crypto/mlx5/meson.build new file mode 100644 index
> > 0000000000..fd00283665
> > --- /dev/null
> > +++ b/drivers/crypto/mlx5/meson.build
> > @@ -0,0 +1,26 @@
> > +# SPDX-License-Identifier: BSD-3-Clause # Copyright 2021 Mellanox
> > +Technologies, Ltd
> > +
> > +if not is_linux
> > +     build = false
> > +     reason = 'only supported on Linux'
> > +     subdir_done()
> > +endif
> > +
> > +fmt_name = 'mlx5_crypto'
> > +deps += ['common_mlx5', 'eal', 'cryptodev']
> 
> I don't see a reason for eal dependency. Do you really need it?

See RTE_LOG_REGISTER.

> > +sources = files(
> > +     'mlx5_crypto.c',
> > +)
> > +cflags_options = [
> > +     '-std=c11',
> > +     '-Wno-strict-prototypes',
> > +     '-D_BSD_SOURCE',
> > +     '-D_DEFAULT_SOURCE',
> > +     '-D_XOPEN_SOURCE=600',
> > +]
> 
> Any specific reason to use '-Wno-strict-prototypes'
> Compilation works without this as well.

We use it in Mellanox driver usually.
This is internal mlx5 code convention.


 
> > +foreach option:cflags_options
> > +     if cc.has_argument(option)
> > +             cflags += option
> > +     endif
> > +endforeach

Re: [dpdk-dev] [EXT] [PATCH v3 03/15] crypto/mlx5: support session operations

[Matan Azrad]

From: Akhil Goyal
> > +static void
> > +mlx5_crypto_sym_session_clear(struct rte_cryptodev *dev,
> > +                           struct rte_cryptodev_sym_session *sess) {
> > +     struct mlx5_crypto_priv *priv = dev->data->dev_private;
> > +     struct mlx5_crypto_session *sess_private_data =
> > +                     get_sym_session_private_data(sess,
> > +dev->driver_id);
> > +
> > +     if (unlikely(sess_private_data == NULL)) {
> > +             DRV_LOG(ERR, "Failed to get session %p private data.",
> > +                             sess_private_data);
> > +             return;
> > +     }
> > +     mlx5_crypto_dek_destroy(priv, sess_private_data->dek);
> > +     DRV_LOG(DEBUG, "Session %p was cleared.", sess_private_data); }
> 
> Memory leakage, mempool is not freed.

Yes, good catch, this part was missed.

> IMO, this driver is not properly tested with the unit test app.

The only app we tested until now is l2fwd_crypto and it works fine!
We can add it to doc.

> Please add a note in the documentation that it is tested with autotest.


The next app we want to test with, is test-crypto-perf.
 

Re: [dpdk-dev] [EXT] [PATCH v3 09/15] crypto/mlx5: adjust to the multiple data unit API

[Matan Azrad]

From: Akhil Goyal
> > From: Shiri Kuzin <shirik@nvidia.com>
> >
> > In AES-XTS the data to be encrypted\decrypted does not have to be in
> > multiples of 16B size, the unit of data is called data-unit.
> >
> > As a result of patch [1] a new field is added to the cipher
> > capability, called dataunit_set, where the devices can report the
> > range of supported data-unit sizes.
> >
> > The new field enables saving the data-unit size in the session
> > structure to the block size pointer variable in order to support
> > several data-unit sizes.
> >
> > [1] https://www.mail-archive.com/dev@dpdk.org/msg205337.html
> >
> Since [1] patch is already merged, no need to refer it in the patch description.
> Title can be rephrased as
> Crypto/mlx5: support multiple data units

Ok.

 
> Also set feature flag in the code and the documentation in this patch.
> I see that you are setting all of them in a single patch in the end.
> This is not correct. It should be added where the feature is supported.
> Please fix this for all the feature flags.

No, in this stage no feature is really supported, the actual time it will be supported is after the datapath patches and capabilities set.

> 
> > Signed-off-by: Shiri Kuzin <shirik@nvidia.com>
> > ---
> >  drivers/crypto/mlx5/mlx5_crypto.c | 23 +++++++++++++++++++++++
> >  1 file changed, 23 insertions(+)
> >
> > diff --git a/drivers/crypto/mlx5/mlx5_crypto.c
> > b/drivers/crypto/mlx5/mlx5_crypto.c
> > index 18b1a6be88..8cc29ced21 100644
> > --- a/drivers/crypto/mlx5/mlx5_crypto.c
> > +++ b/drivers/crypto/mlx5/mlx5_crypto.c
> > @@ -48,6 +48,11 @@ struct mlx5_crypto_session {
> >        * bsf_size, bsf_p_type, encryption_order and encryption standard,
> >        * saved in big endian format.
> >        */
> > +     uint32_t bsp_res;
> > +     /*
> > +      * crypto_block_size_pointer and reserved 24 bits saved in big endian
> > +      * format.
> > +      */
> >       uint32_t iv_offset:16;
> >       /* Starting point for Initialisation Vector. */
> >       struct mlx5_crypto_dek *dek; /* Pointer to dek struct. */ @@
> > -171,6 +176,24 @@ mlx5_crypto_sym_session_configure(struct
> > rte_cryptodev *dev,
> >                        MLX5_BSF_P_TYPE_CRYPTO <<
> > MLX5_BSF_P_TYPE_OFFSET |
> >                        encryption_order <<
> > MLX5_ENCRYPTION_ORDER_OFFSET |
> >                        MLX5_ENCRYPTION_STANDARD_AES_XTS);
> > +     switch (xform->cipher.dataunit_len) {
> > +     case 0:
> > +             sess_private_data->bsp_res = 0;
> > +             break;
> > +     case 512:
> > +             sess_private_data->bsp_res = rte_cpu_to_be_32
> > +
> > ((uint32_t)MLX5_BLOCK_SIZE_512B <<
> > +                                          MLX5_BLOCK_SIZE_OFFSET);
> > +             break;
> > +     case 4096:
> > +             sess_private_data->bsp_res = rte_cpu_to_be_32
> > +
> > ((uint32_t)MLX5_BLOCK_SIZE_4096B <<
> > +                                          MLX5_BLOCK_SIZE_OFFSET);
> > +             break;
> > +     default:
> > +             DRV_LOG(ERR, "Cipher data unit length is not supported.");
> > +             return -ENOTSUP;
> > +     }
> >       sess_private_data->iv_offset = cipher->iv.offset;
> >       sess_private_data->dek_id =
> >                       rte_cpu_to_be_32(sess_private_data->dek->obj->id
> > &
> > --
> > 2.25.1

Re: [dpdk-dev] [EXT] [PATCH v3 10/15] crypto/mlx5: add keytag device argument

[Matan Azrad]

From: Akhil Goyal
> > From: Suanming Mou <suanmingm@nvidia.com>
> >
> > A keytag is a piece of data encrypted together with a DEK.
> >
> > When a DEK is referenced by an MKEY.bsf through its index, the keytag
> > is also supplied in the BSF as plaintext. The HW will decrypt the DEK
> > (and the attached keytag) and will fail the operation if the keytags
> > don't match.
> >
> > This commit adds the configuration of the keytag with devargs.
> >
> > Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
> > Signed-off-by: Matan Azrad <matan@nvidia.com>
> > ---
> Documentation for devargs should be part of this patch.
> Please split the last patch accordingly.
> 
> Title can be shortened to
> Crypto/mlx5: add keytag devarg
> 
> Fix other patches of devargs accordingly.

As I said before, no devargs is really active before adding datapath patches.
The option to add all the supported features \ documentations in the patch which actually adds the support is correct.

The last patch adds the capabilities and docs when all of them are really supported.

Ok for title.

Re: [dpdk-dev] [EXT] [PATCH v3 13/15] crypto/mlx5: add enqueue and dequeue operations

[Matan Azrad]

From: Akhil Goyal
> > @@ -685,9 +919,8 @@ mlx5_crypto_pci_probe(struct rte_pci_driver
> > *pci_drv,
> >       DRV_LOG(INFO,
> >               "Crypto device %s was created successfully.", ibv->name);
> >       crypto_dev->dev_ops = &mlx5_crypto_ops;
> > -     crypto_dev->dequeue_burst = NULL;
> > -     crypto_dev->enqueue_burst = NULL;
> > -     crypto_dev->feature_flags =
> > RTE_CRYPTODEV_FF_HW_ACCELERATED;
> It should be set once in the patchset.
> I see this getting set in 1/15 and removed here and again set in 15/15.
> And whenever you are setting it, it should be added in the .ini file as well.

Moved it to last patch too

> > +     crypto_dev->dequeue_burst = mlx5_crypto_dequeue_burst;
> > +     crypto_dev->enqueue_burst = mlx5_crypto_enqueue_burst;
> >       crypto_dev->driver_id = mlx5_crypto_driver_id;
> >       priv = crypto_dev->data->dev_private;
> >       priv->ctx = ctx;
> > diff --git a/drivers/crypto/mlx5/mlx5_crypto.h
> > b/drivers/crypto/mlx5/mlx5_crypto.h
> > index 52fcf5217f..ac4ad1834f 100644
> > --- a/drivers/crypto/mlx5/mlx5_crypto.h
> > +++ b/drivers/crypto/mlx5/mlx5_crypto.h
> > @@ -37,6 +37,9 @@ struct mlx5_crypto_priv {
> >       uint16_t rdmw_wqe_size;
> >       uint16_t wqe_stride;
> >       uint16_t max_rdmaw_klm_n;
> > +#ifndef RTE_ARCH_64
> > +     rte_spinlock_t uar32_sl;
> > +#endif /* RTE_ARCH_64 */
> >  };
> >
> >  struct mlx5_crypto_qp {
> > --
> > 2.25.1

Re: [dpdk-dev] [EXT] [PATCH v3 15/15] crypto/mlx5: set feature flags and capabilities

[Matan Azrad]

From: Akhil Goyal
> > +
> > +Overview
> > +--------
> > +
> > +The device can provide disk encryption services, allowing data
> > +encryption and decryption towards a disk. Having all
> > +encryption/decryption operations done in a single device can reduce
> > +cost and overheads of the
> > related
> > +FIPS certification, as ConnectX-6 is FIPS 140-2 level-2 ready.
> > +The encryption cipher is AES-XTS of 256/512 key size.
> 
> 256/512 bit key size
> 
> > +
> > +MKEY is a memory region object in the hardware, that holds address
> > translation information and
> > +attributes per memory area. Its ID must be tied to addresses provided
> > +to
> > the hardware.
> > +The encryption operations are performed with MKEY read\write
> 
> read/write
> 
> > transactions, when
> > +the MKEY is configured to perform crypto operations.
> > +
> > +The encryption does not require text to be aligned to the AES block
> > +size
> > (128b).
> > +
> > +In order to move the device to crypto operational mode, credential
> > +and KEK (Key Encrypting Key) should be set as the first step.
> > +The credential will be used by the software in order to perform
> > +crypto login,
> > and the KEK is
> > +the AES Key Wrap Algorithm (rfc3394) key that will be used for
> > +sensitive
> > data
> > +wrapping.
> > +The credential and the AES-XTS keys should be provided to the
> > +hardware, as
> > ciphertext
> > +encrypted by the KEK.
> > +
> > +A keytag (64 bits) should be appended to the AES-XTS keys (before
> > wrapping),
> > +and will be validated when the hardware attempts to access it.
> > +
> > +For security reasons and to increase robustness, this driver only
> > +deals with
> > virtual
> > +memory addresses. The way resources allocations are handled by the
> > kernel,
> > +combined with hardware specifications that allow handling virtual
> > +memory addresses directly, ensure that DPDK applications cannot
> > +access random physical memory (or memory that does not belong to the
> current process).
> > +
> > +The PMD uses libibverbs and libmlx5 to access the device firmware or
> > +to access the hardware components directly.
> > +There are different levels of objects and bypassing abilities.
> > +To get the best performances:
> > +
> > +- Verbs is a complete high-level generic API.
> > +- Direct Verbs is a device-specific API.
> > +- DevX allows to access firmware objects.
> > +
> > +Enabling librte_crypto_mlx5 causes DPDK applications to be linked
> > +against libibverbs.
> > +
> > +Mellanox mlx5 PCI device can be probed by a number of different PCI
> > devices, such as
> > +net / vDPA / RegEx. To select the crypto PMD, ``class=crypto`` should
> > +be specified as a device parameter. The crypto device can be probed
> > and
> > +used with other Mellanox classes by adding more options in the class.
> > +For example: ``class=net:crypto`` will probe both the net PMD and the
> > crypto
> > +PMD.
> > +
> > +When crypto engines are defined to work in wrapped import method,
> > +they
> > come out
> > +of the factory in Commissioning mode, and thus, cannot be used for
> > +crypto
> > operations
> > +yet. A dedicated tool is used for changing the mode from
> > +Commissioning to Operational, while setting the first import_KEK and
> credential in plaintext.
> > +The mlxreg dedicated tool should be used as follows:
> > +
> > +- Set CRYPTO_OPERATIONAL register to set the device in crypto
> > +operational
> > mode.
> > +
> > +  The input to this tool is:
> > +    The first credential in plaintext, 40B.
> > +    The first import_KEK in plaintext: kek size 0 for 16B or 1 for 32B, kek
> data.
> > +
> > +  Example:
> > +  mlxreg -d /dev/mst/mt4123_pciconf0 --reg_name
> CRYPTO_OPERATIONAL -
> > -get
> > +
> > +  The "wrapped_crypto_operational" value will be "0x00000000".
> > +  The command to set the register should be executed only once, and
> > + all the  values mentioned above should be specified in the same
> command.
> > +
> > +  Example:
> > +  mlxreg -d /dev/mst/mt4123_pciconf0 --reg_name
> CRYPTO_OPERATIONAL
> > + --set "credential[0]=0x10000000, credential[1]=0x10000000,
> > kek[0]=0x00000000"
> > +
> > +  All values not specified will remain 0.
> > +  "wrapped_crypto_going_to_commissioning" and
> > "wrapped_crypto_operational"
> > +  should not be specified.
> > +
> > +  All the device ports should set it in order to move to operational mode.
> > +
> > +- Query CRYPTO_OPERATIONAL register to make sure the device is in
> > Operational
> > +  mode.
> > +
> > +  Example:
> > +  mlxreg -d /dev/mst/mt4123_pciconf0 --reg_name
> CRYPTO_OPERATIONAL -
> > -get
> > +  The "wrapped_crypto_operational" value will be "0x00000001" if the
> > mode was
> > +  successfully changed to operational mode.
> > +
> > +
> > +Driver options
> > +--------------
> > +
> > +- ``class`` parameter [string]
> > +
> > +  Select the class of the driver that should probe the device.
> > +  `crypto` for the mlx5 crypto driver.
> > +
> > +- ``wcs_file`` parameter [string] - mandatory
> > +
> > +  File path including only the wrapped credential in string format of
> > hexadecimal
> > +  numbers, represent 48 bytes (8 bytes IV added by the AES key wrap
> > algorithm).
> > +
> > +- ``import_kek_id`` parameter [int]
> > +
> > +  The identifier of the KEK, default value is 0 represents the
> > + operational  register import_kek..
> > +
> > +- ``credential_id`` parameter [int]
> > +
> > +  The identifier of the credential, default value is 0 represents the
> > operational
> > +  register credential.
> > +
> > +- ``max_segs_num`` parameter [int]
> > +
> > +  Maximum number of mbuf chain segments(src or dest), default value is
> 8.
> > +
> > +- ``keytag`` parameter [int]
> > +
> > +  The plaintext of the keytag appanded to the AES-XTS keys, default
> > + value is
> > 0.
> > +
> > +
> > +Limitations
> > +-----------
> > +
> > +- AES-XTS keys provided in Xform must include keytag and should be
> > wrappend.
> > +- The supported data-unit lengths are: 512B, 1KB, 1MB. In case the
> > `dataunit_len`
> > +  is not provided in the cipher Xform, the OP length is limitted to
> > + the above
> 
> Xform should be xform
> 
> Spell check  - limited


Ok, for all the above comments. Will adjust in v4.
 
> > values.
> > +
> > +
> > +Supported NICs
> > +--------------
> > +
> > +* Mellanox\ |reg| ConnectX\ |reg|-6 200G MCX654106A-HCAT (2x200G)
> > +
> > +Prerequisites
> > +-------------
> > +
> > +- Mellanox OFED version: **5.3**
> > +  see :doc:`../../nics/mlx5` guide for more Mellanox OFED details.
> 
> Since the driver is by default compiled off due to the dependency on
> external Libraries, I would recommend to add few lines here as well for
> compilation.
> Like to compile rdma-core and set PKG_CONFIG_LIBDIR.

Why? all Mellanox drivers has the same external dependencies.
I added here link for the doc explains it well.

> And I do not see any updates to the test application for testing this driver.

You can see update to l2fwd_crypto, we tested with this example for the first stage.
Everything looks ok there.

> Is this driver really tested or is it work in progress? If it is work in progress, We
> should defer this PMD to next release.

We can continue test this driver for a long time.
The basics were tested in the above crypto example what proofs that driver works well.
If we will find issues in other tests, we will send fixes.

I'll send v4 with your comments adjustment soon.

> 
> > diff --git a/doc/guides/rel_notes/release_21_05.rst
> > b/doc/guides/rel_notes/release_21_05.rst
> > index 0970a2331a..9030fd8b98 100644
> > --- a/doc/guides/rel_notes/release_21_05.rst
> > +++ b/doc/guides/rel_notes/release_21_05.rst
> > @@ -275,6 +275,11 @@ New Features
> >    * Added support for crypto adapter forward mode in octeontx2 event
> > and crypto
> >      device driver.
> >
> > +* **Added support for Nvidia crypto device driver.**
> > +
> > +  * Added mlx5 crypto driver to support AES-XTS cipher operations.
> > +    the first device to support it is ConnectX-6.
> 
> 'The'
> 
> > +
> >
> >  Removed Items
> >  -------------
> > diff --git a/drivers/crypto/mlx5/mlx5_crypto.c
> > b/drivers/crypto/mlx5/mlx5_crypto.c
> > index 896ca60866..f318ff4682 100644
> > --- a/drivers/crypto/mlx5/mlx5_crypto.c
> > +++ b/drivers/crypto/mlx5/mlx5_crypto.c
> > @@ -22,6 +22,14 @@
> >  #define MLX5_CRYPTO_LOG_NAME pmd.crypto.mlx5  #define
> > MLX5_CRYPTO_MAX_QPS 1024  #define MLX5_CRYPTO_MAX_SEGS 56
> > +#define MLX5_CRYPTO_FEATURE_FLAGS \
> > +     (RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO |
> > RTE_CRYPTODEV_FF_HW_ACCELERATED | \
> > +     RTE_CRYPTODEV_FF_IN_PLACE_SGL |
> > RTE_CRYPTODEV_FF_OOP_SGL_IN_SGL_OUT | \
> > +     RTE_CRYPTODEV_FF_OOP_SGL_IN_LB_OUT | \
> > +     RTE_CRYPTODEV_FF_OOP_LB_IN_SGL_OUT | \
> > +     RTE_CRYPTODEV_FF_OOP_LB_IN_LB_OUT | \
> > +     RTE_CRYPTODEV_FF_CIPHER_WRAPPED_KEY | \
> > +     RTE_CRYPTODEV_FF_CIPHER_MULTIPLE_DATA_UNITS)
> >
> >  TAILQ_HEAD(mlx5_crypto_privs, mlx5_crypto_priv) mlx5_crypto_priv_list
> > =
> >
> >       TAILQ_HEAD_INITIALIZER(mlx5_crypto_priv_list);
> > @@ -31,8 +39,32 @@ int mlx5_crypto_logtype;
> >
> >  uint8_t mlx5_crypto_driver_id;
> >
> > -const struct rte_cryptodev_capabilities
> > -             mlx5_crypto_caps[RTE_CRYPTO_OP_TYPE_UNDEFINED];
> > +const struct rte_cryptodev_capabilities mlx5_crypto_caps[] = {
> > +     {               /* AES XTS */
> > +             .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC,
> > +             {.sym = {
> > +                     .xform_type = RTE_CRYPTO_SYM_XFORM_CIPHER,
> > +                     {.cipher = {
> > +                             .algo = RTE_CRYPTO_CIPHER_AES_XTS,
> > +                             .block_size = 16,
> > +                             .key_size = {
> > +                                     .min = 32,
> > +                                     .max = 64,
> > +                                     .increment = 32
> > +                             },
> > +                             .iv_size = {
> > +                                     .min = 16,
> > +                                     .max = 16,
> > +                                     .increment = 0
> > +                             },
> > +                             .dataunit_set =
> > +
> >       RTE_CRYPTO_CIPHER_DATA_UNIT_LEN_512_BYTES |
> > +
> >       RTE_CRYPTO_CIPHER_DATA_UNIT_LEN_4096_BYTES,
> > +                     }, }
> > +             }, }
> > +     },
> > +};
> > +
> >
> >  static const char mlx5_crypto_drv_name[] =
> > RTE_STR(MLX5_CRYPTO_DRIVER_NAME);
> >
> > @@ -67,7 +99,7 @@ mlx5_crypto_dev_infos_get(struct rte_cryptodev
> *dev,
> >       RTE_SET_USED(dev);
> >       if (dev_info != NULL) {
> >               dev_info->driver_id = mlx5_crypto_driver_id;
> > -             dev_info->feature_flags = 0;
> > +             dev_info->feature_flags = MLX5_CRYPTO_FEATURE_FLAGS;
> >               dev_info->capabilities = mlx5_crypto_caps;
> >               dev_info->max_nb_queue_pairs = MLX5_CRYPTO_MAX_QPS;
> >               dev_info->min_mbuf_headroom_req = 0; @@ -954,6 +986,7 @@
> > mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv,
> >       crypto_dev->dev_ops = &mlx5_crypto_ops;
> >       crypto_dev->dequeue_burst = mlx5_crypto_dequeue_burst;
> >       crypto_dev->enqueue_burst = mlx5_crypto_enqueue_burst;
> > +     crypto_dev->feature_flags = MLX5_CRYPTO_FEATURE_FLAGS;
> >       crypto_dev->driver_id = mlx5_crypto_driver_id;
> >       priv = crypto_dev->data->dev_private;
> >       priv->ctx = ctx;
> > --
> > 2.25.1

Re: [dpdk-dev] [EXT] [PATCH v3 10/15] crypto/mlx5: add keytag device argument

[Thomas Monjalon]

09/05/2021 10:31, Matan Azrad:
> 
> From: Akhil Goyal
> > > From: Suanming Mou <suanmingm@nvidia.com>
> > >
> > > A keytag is a piece of data encrypted together with a DEK.
> > >
> > > When a DEK is referenced by an MKEY.bsf through its index, the keytag
> > > is also supplied in the BSF as plaintext. The HW will decrypt the DEK
> > > (and the attached keytag) and will fail the operation if the keytags
> > > don't match.
> > >
> > > This commit adds the configuration of the keytag with devargs.
> > >
> > > Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
> > > Signed-off-by: Matan Azrad <matan@nvidia.com>
> > > ---
> > Documentation for devargs should be part of this patch.
> > Please split the last patch accordingly.
> > 
> > Title can be shortened to
> > Crypto/mlx5: add keytag devarg
> > 
> > Fix other patches of devargs accordingly.
> 
> As I said before, no devargs is really active before adding datapath patches.
> The option to add all the supported features \ documentations in the patch which actually adds the support is correct.
> 
> The last patch adds the capabilities and docs when all of them are really supported.

I think it is better for git history (blame, etc) to have doc
added at the same time of the code, even if the datapath is not active.
I agree with Akhil.

Re: [dpdk-dev] [EXT] [PATCH v3 15/15] crypto/mlx5: set feature flags and capabilities

[Akhil Goyal]

> > > +Supported NICs
> > > +--------------
> > > +
> > > +* Mellanox\ |reg| ConnectX\ |reg|-6 200G MCX654106A-HCAT (2x200G)
> > > +
> > > +Prerequisites
> > > +-------------
> > > +
> > > +- Mellanox OFED version: **5.3**
> > > +  see :doc:`../../nics/mlx5` guide for more Mellanox OFED details.
> >
> > Since the driver is by default compiled off due to the dependency on
> > external Libraries, I would recommend to add few lines here as well for
> > compilation.
> > Like to compile rdma-core and set PKG_CONFIG_LIBDIR.
> 
> Why? all Mellanox drivers has the same external dependencies.
> I added here link for the doc explains it well.

This is a crypto PMD, not a NIC PMD. Somebody working on crypto PMDs,
do not really care about the NIC PMDs.
Hence it would be convenient to have compilation information here as well.
You can refer to other document for details, but basic info should be added
here as well.

> 
> > And I do not see any updates to the test application for testing this driver.
> 
> You can see update to l2fwd_crypto, we tested with this example for the first
> stage.
> Everything looks ok there.

L2fwd-crypto is an app which only test data path with no packet validation.
It does not tell if your encryption is correctly done as per standards or not.
Did you test interoperability with l2fwd-crypto?
All basic configuration tests are also not done, like cleanup etc of the PMD.
I haven't seen a driver getting merge without the unit test application run.
Test app helps you comply with the way dpdk drivers are meant to be written.

> 
> > Is this driver really tested or is it work in progress? If it is work in progress,
> We
> > should defer this PMD to next release.
> 
> We can continue test this driver for a long time.
> The basics were tested in the above crypto example what proofs that driver
> works well.
> If we will find issues in other tests, we will send fixes.
> 
> I'll send v4 with your comments adjustment soon.
> 

Re: [dpdk-dev] [EXT] [PATCH v3 10/15] crypto/mlx5: add keytag device argument

[Akhil Goyal]

> From: Akhil Goyal
> > > From: Suanming Mou <suanmingm@nvidia.com>
> > >
> > > A keytag is a piece of data encrypted together with a DEK.
> > >
> > > When a DEK is referenced by an MKEY.bsf through its index, the keytag
> > > is also supplied in the BSF as plaintext. The HW will decrypt the DEK
> > > (and the attached keytag) and will fail the operation if the keytags
> > > don't match.
> > >
> > > This commit adds the configuration of the keytag with devargs.
> > >
> > > Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
> > > Signed-off-by: Matan Azrad <matan@nvidia.com>
> > > ---
> > Documentation for devargs should be part of this patch.
> > Please split the last patch accordingly.
> >
> > Title can be shortened to
> > Crypto/mlx5: add keytag devarg
> >
> > Fix other patches of devargs accordingly.
> 
> As I said before, no devargs is really active before adding datapath patches.
> The option to add all the supported features \ documentations in the patch
> which actually adds the support is correct.
> 
> The last patch adds the capabilities and docs when all of them are really
> supported.
> 
In that case split the patches in such a manner that data path is added
Before devargs patch with some dummy values and add the devargs with
Appropriate documentation update in a later patch.