[PATCH 0/4] new options and fixes for xnvme ioengine

[PATCH 0/4] new options and fixes for xnvme ioengine

[Ankit Kumar]

This patch series adds
1. Few minor fixes and cleanups to the xnvme I/O engine.
2. xNVMe recently added a user-space VFIO based backend implemented
   using libvfn. This allow users to select that instead of SPDK.
3. Add subnqn option for fabrics target with multiple systems.
4. Add an option to select memory backend.

Ankit Kumar (4):
  engines/xnvme: fixes for xnvme ioengine
  engines/xnvme: user space vfio based backend
  engines/xnvme: add subnqn to fio-options
  engines/xnvme: add support for picking mem backend

 HOWTO.rst       | 27 ++++++++++++++++++++++++++-
 engines/xnvme.c | 44 ++++++++++++++++++++++++++++++++++++++------
 fio.1           | 32 +++++++++++++++++++++++++++++++-
 3 files changed, 95 insertions(+), 8 deletions(-)

-- 
2.17.1

[PATCH 1/4] engines/xnvme: fixes for xnvme ioengine

[Ankit Kumar]

1. fix error-handling in xnvme_fioe_queue()
2. fix resource-leak in xnvme_fioe_init()

Signed-off-by: Simon A. F. Lund <simon.lund@samsung.com>
Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>
---
 engines/xnvme.c | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)

diff --git a/engines/xnvme.c b/engines/xnvme.c
index d8647481..dcc54998 100644
--- a/engines/xnvme.c
+++ b/engines/xnvme.c
@@ -322,12 +322,15 @@ static int xnvme_fioe_init(struct thread_data *td)
 
 	xd->iocq = calloc(td->o.iodepth, sizeof(struct io_u *));
 	if (!xd->iocq) {
-		log_err("ioeng->init(): !calloc(), err(%d)\n", errno);
+		free(xd);
+		log_err("ioeng->init(): !calloc(xd->iocq), err(%d)\n", errno);
 		return 1;
 	}
 
 	xd->iovec = calloc(td->o.iodepth, sizeof(*xd->iovec));
 	if (!xd->iovec) {
+		free(xd->iocq);
+		free(xd);
 		log_err("ioeng->init(): !calloc(xd->iovec), err(%d)\n", errno);
 		return 1;
 	}
@@ -338,6 +341,10 @@ static int xnvme_fioe_init(struct thread_data *td)
 	for_each_file(td, f, i)
 	{
 		if (_dev_open(td, f)) {
+			/*
+			 * Note: We are not freeing xd, iocq and iovec. This
+			 * will be done as part of cleanup routine.
+			 */
 			log_err("ioeng->init(): failed; _dev_open(%s)\n", f->file_name);
 			return 1;
 		}
@@ -506,9 +513,11 @@ static enum fio_q_status xnvme_fioe_queue(struct thread_data *td, struct io_u *i
 
 	default:
 		log_err("ioeng->queue(): ENOSYS: %u\n", io_u->ddir);
-		err = -1;
+		xnvme_queue_put_cmd_ctx(ctx->async.queue, ctx);
+
+		io_u->error = ENOSYS;
 		assert(false);
-		break;
+		return FIO_Q_COMPLETED;
 	}
 
 	if (vectored_io) {
-- 
2.17.1

[PATCH 2/4] engines/xnvme: user space vfio based backend

[Ankit Kumar]

Add an option to use user-space VFIO-based backend,
implemented using libvfn.
Update xnvme engine options for missing backends.

Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>
---
 HOWTO.rst       | 5 ++++-
 engines/xnvme.c | 7 ++++---
 fio.1           | 6 +++++-
 3 files changed, 13 insertions(+), 5 deletions(-)

diff --git a/HOWTO.rst b/HOWTO.rst
index 97fe5350..aba6c9b3 100644
--- a/HOWTO.rst
+++ b/HOWTO.rst
@@ -2845,6 +2845,9 @@ with the caveat that when used on the command line, they must come after the
 	**posix**
 		Use the posix asynchronous I/O interface to perform one or
 		more I/O operations asynchronously.
+	**vfio**
+		Use the user-space VFIO-based backend, implemented using
+		libvfn instead of SPDK.
 	**nil**
 		Do not transfer any data; just pretend to. This is mainly used
 		for introspective performance evaluation.
@@ -2875,7 +2878,7 @@ with the caveat that when used on the command line, they must come after the
 
 .. option:: xnvme_dev_nsid=int : [xnvme]
 
-	xnvme namespace identifier for userspace NVMe driver, such as SPDK.
+	xnvme namespace identifier for userspace NVMe driver, SPDK or vfio.
 
 .. option:: xnvme_iovec=int : [xnvme]
 
diff --git a/engines/xnvme.c b/engines/xnvme.c
index dcc54998..ee6b67c1 100644
--- a/engines/xnvme.c
+++ b/engines/xnvme.c
@@ -113,7 +113,8 @@ static struct fio_option options[] = {
 		.lname = "xNVMe Asynchronous command-interface",
 		.type = FIO_OPT_STR_STORE,
 		.off1 = offsetof(struct xnvme_fioe_options, xnvme_async),
-		.help = "Select xNVMe async. interface: [emu,thrpool,io_uring,libaio,posix,nil]",
+		.help = "Select xNVMe async. interface: "
+			"[emu,thrpool,io_uring,io_uring_cmd,libaio,posix,vfio,nil]",
 		.category = FIO_OPT_C_ENGINE,
 		.group = FIO_OPT_G_XNVME,
 	},
@@ -122,7 +123,7 @@ static struct fio_option options[] = {
 		.lname = "xNVMe Synchronous. command-interface",
 		.type = FIO_OPT_STR_STORE,
 		.off1 = offsetof(struct xnvme_fioe_options, xnvme_sync),
-		.help = "Select xNVMe sync. interface: [nvme,psync]",
+		.help = "Select xNVMe sync. interface: [nvme,psync,block]",
 		.category = FIO_OPT_C_ENGINE,
 		.group = FIO_OPT_G_XNVME,
 	},
@@ -131,7 +132,7 @@ static struct fio_option options[] = {
 		.lname = "xNVMe Admin command-interface",
 		.type = FIO_OPT_STR_STORE,
 		.off1 = offsetof(struct xnvme_fioe_options, xnvme_admin),
-		.help = "Select xNVMe admin. cmd-interface: [nvme,block,file_as_ns]",
+		.help = "Select xNVMe admin. cmd-interface: [nvme,block]",
 		.category = FIO_OPT_C_ENGINE,
 		.group = FIO_OPT_G_XNVME,
 	},
diff --git a/fio.1 b/fio.1
index 1074b52a..004d3ba0 100644
--- a/fio.1
+++ b/fio.1
@@ -2584,6 +2584,10 @@ Use Linux aio for Asynchronous I/O
 Use the posix asynchronous I/O interface to perform one or more I/O operations
 asynchronously.
 .TP
+.BI vfio
+Use the user-space VFIO-based backend, implemented using libvfn instead of
+SPDK.
+.TP
 .BI nil
 Do not transfer any data; just pretend to. This is mainly used for
 introspective performance evaluation.
@@ -2621,7 +2625,7 @@ Use Linux Block Layer ioctl() and sysfs for admin commands.
 .RE
 .TP
 .BI (xnvme)xnvme_dev_nsid\fR=\fPint
-xnvme namespace identifier for userspace NVMe driver such as SPDK.
+xnvme namespace identifier for userspace NVMe driver SPDK or vfio.
 .TP
 .BI (xnvme)xnvme_iovec
 If this option is set, xnvme will use vectored read/write commands.
-- 
2.17.1

[PATCH 3/4] engines/xnvme: add subnqn to fio-options

[Ankit Kumar]

For fio to utilize a fabrics target with multiple systems, it needs a
way for the user to specify which subsystem to use. This is done by
providing 'subnqn' as fio-option.

Signed-off-by: Simon A. F. Lund <simon.lund@samsung.com>
Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>
---
 HOWTO.rst       |  5 +++++
 engines/xnvme.c | 11 +++++++++++
 fio.1           |  4 ++++
 3 files changed, 20 insertions(+)

diff --git a/HOWTO.rst b/HOWTO.rst
index aba6c9b3..0fec38a7 100644
--- a/HOWTO.rst
+++ b/HOWTO.rst
@@ -2880,6 +2880,11 @@ with the caveat that when used on the command line, they must come after the
 
 	xnvme namespace identifier for userspace NVMe driver, SPDK or vfio.
 
+.. option:: xnvme_dev_subnqn=str : [xnvme]
+
+	Sets the subsystem NQN for fabrics. This is for xNVMe to utilize a
+	fabrics target with multiple systems.
+
 .. option:: xnvme_iovec=int : [xnvme]
 
 	If this option is set. xnvme will use vectored read/write commands.
diff --git a/engines/xnvme.c b/engines/xnvme.c
index ee6b67c1..208d917d 100644
--- a/engines/xnvme.c
+++ b/engines/xnvme.c
@@ -78,6 +78,7 @@ struct xnvme_fioe_options {
 	char *xnvme_async;
 	char *xnvme_sync;
 	char *xnvme_admin;
+	char *xnvme_dev_subnqn;
 };
 
 static struct fio_option options[] = {
@@ -145,6 +146,15 @@ static struct fio_option options[] = {
 		.category = FIO_OPT_C_ENGINE,
 		.group = FIO_OPT_G_XNVME,
 	},
+	{
+		.name = "xnvme_dev_subnqn",
+		.lname = "Subsystem nqn for Fabrics",
+		.type = FIO_OPT_STR_STORE,
+		.off1 = offsetof(struct xnvme_fioe_options, xnvme_dev_subnqn),
+		.help = "Subsystem NQN for Fabrics",
+		.category = FIO_OPT_C_ENGINE,
+		.group = FIO_OPT_G_XNVME,
+	},
 	{
 		.name = "xnvme_iovec",
 		.lname = "Vectored IOs",
@@ -181,6 +191,7 @@ static struct xnvme_opts xnvme_opts_from_fioe(struct thread_data *td)
 	struct xnvme_opts opts = xnvme_opts_default();
 
 	opts.nsid = o->xnvme_dev_nsid;
+	opts.subnqn = o->xnvme_dev_subnqn;
 	opts.be = o->xnvme_be;
 	opts.async = o->xnvme_async;
 	opts.sync = o->xnvme_sync;
diff --git a/fio.1 b/fio.1
index 004d3ba0..134aed54 100644
--- a/fio.1
+++ b/fio.1
@@ -2627,6 +2627,10 @@ Use Linux Block Layer ioctl() and sysfs for admin commands.
 .BI (xnvme)xnvme_dev_nsid\fR=\fPint
 xnvme namespace identifier for userspace NVMe driver SPDK or vfio.
 .TP
+.BI (xnvme)xnvme_dev_subnqn\fR=\fPstr
+Sets the subsystem NQN for fabrics. This is for xNVMe to utilize a fabrics
+target with multiple systems.
+.TP
 .BI (xnvme)xnvme_iovec
 If this option is set, xnvme will use vectored read/write commands.
 .TP
-- 
2.17.1

[PATCH 4/4] engines/xnvme: add support for picking mem backend

[Ankit Kumar]

Add option to the xnvme fio engine for picking a
memory backend. Update the fio document.

Signed-off-by: Mads Ynddal <m.ynddal@samsung.com>
Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>
---
 HOWTO.rst       | 17 +++++++++++++++++
 engines/xnvme.c | 11 +++++++++++
 fio.1           | 22 ++++++++++++++++++++++
 3 files changed, 50 insertions(+)

diff --git a/HOWTO.rst b/HOWTO.rst
index 0fec38a7..0a48a453 100644
--- a/HOWTO.rst
+++ b/HOWTO.rst
@@ -2885,6 +2885,23 @@ with the caveat that when used on the command line, they must come after the
 	Sets the subsystem NQN for fabrics. This is for xNVMe to utilize a
 	fabrics target with multiple systems.
 
+.. option:: xnvme_mem=str : [xnvme]
+
+	Select the xnvme memory backend. This can take these values.
+
+	**posix**
+		This is the default posix memory backend for linux NVMe driver.
+	**hugepage**
+		Use hugepages, instead of existing posix memory backend. The
+		memory backend uses hugetlbfs. This require users to allocate
+		hugepages, mount hugetlbfs and set an enviornment variable for
+		XNVME_HUGETLB_PATH.
+	**spdk**
+		Uses SPDK's memory allocator.
+	**vfio**
+		Uses libvfn's memory allocator. This also specifies the use
+		of libvfn backend instead of SPDK.
+
 .. option:: xnvme_iovec=int : [xnvme]
 
 	If this option is set. xnvme will use vectored read/write commands.
diff --git a/engines/xnvme.c b/engines/xnvme.c
index 208d917d..bb92a121 100644
--- a/engines/xnvme.c
+++ b/engines/xnvme.c
@@ -75,6 +75,7 @@ struct xnvme_fioe_options {
 	unsigned int xnvme_dev_nsid;
 	unsigned int xnvme_iovec;
 	char *xnvme_be;
+	char *xnvme_mem;
 	char *xnvme_async;
 	char *xnvme_sync;
 	char *xnvme_admin;
@@ -109,6 +110,15 @@ static struct fio_option options[] = {
 		.category = FIO_OPT_C_ENGINE,
 		.group = FIO_OPT_G_XNVME,
 	},
+	{
+		.name = "xnvme_mem",
+		.lname = "xNVMe Memory Backend",
+		.type = FIO_OPT_STR_STORE,
+		.off1 = offsetof(struct xnvme_fioe_options, xnvme_mem),
+		.help = "Select xNVMe memory backend",
+		.category = FIO_OPT_C_ENGINE,
+		.group = FIO_OPT_G_XNVME,
+	},
 	{
 		.name = "xnvme_async",
 		.lname = "xNVMe Asynchronous command-interface",
@@ -193,6 +203,7 @@ static struct xnvme_opts xnvme_opts_from_fioe(struct thread_data *td)
 	opts.nsid = o->xnvme_dev_nsid;
 	opts.subnqn = o->xnvme_dev_subnqn;
 	opts.be = o->xnvme_be;
+	opts.mem = o->xnvme_mem;
 	opts.async = o->xnvme_async;
 	opts.sync = o->xnvme_sync;
 	opts.admin = o->xnvme_admin;
diff --git a/fio.1 b/fio.1
index 134aed54..eb87533f 100644
--- a/fio.1
+++ b/fio.1
@@ -2631,6 +2631,28 @@ xnvme namespace identifier for userspace NVMe driver SPDK or vfio.
 Sets the subsystem NQN for fabrics. This is for xNVMe to utilize a fabrics
 target with multiple systems.
 .TP
+.BI (xnvme)xnvme_mem\fR=\fPstr
+Select the xnvme memory backend. This can take these values.
+.RS
+.RS
+.TP
+.B posix
+This is the default posix memory backend for linux NVMe driver.
+.TP
+.BI hugepage
+Use hugepages, instead of existing posix memory backend. The memory backend
+uses hugetlbfs. This require users to allocate hugepages, mount hugetlbfs and
+set an enviornment variable for XNVME_HUGETLB_PATH.
+.TP
+.BI spdk
+Uses SPDK's memory allocator.
+.TP
+.BI vfio
+Uses libvfn's memory allocator. This also specifies the use of libvfn backend
+instead of SPDK.
+.RE
+.RE
+.TP
 .BI (xnvme)xnvme_iovec
 If this option is set, xnvme will use vectored read/write commands.
 .TP
-- 
2.17.1

Re: [PATCH 0/4] new options and fixes for xnvme ioengine

[Vincent Fu]

On 12/21/22 23:39, Ankit Kumar wrote:
> This patch series adds
> 1. Few minor fixes and cleanups to the xnvme I/O engine.
> 2. xNVMe recently added a user-space VFIO based backend implemented
>     using libvfn. This allow users to select that instead of SPDK.
> 3. Add subnqn option for fabrics target with multiple systems.
> 4. Add an option to select memory backend.
> 
> Ankit Kumar (4):
>    engines/xnvme: fixes for xnvme ioengine
>    engines/xnvme: user space vfio based backend
>    engines/xnvme: add subnqn to fio-options
>    engines/xnvme: add support for picking mem backend
> 
>   HOWTO.rst       | 27 ++++++++++++++++++++++++++-
>   engines/xnvme.c | 44 ++++++++++++++++++++++++++++++++++++++------
>   fio.1           | 32 +++++++++++++++++++++++++++++++-
>   3 files changed, 95 insertions(+), 8 deletions(-)
> 

Applied. Thanks.