[PATCH 0/3] v3 Patchset : Zone Append command support

[PATCH 0/3] v3 Patchset : Zone Append command support

[Krishna Kanth Reddy]

This is a v3 Patchset for the Zone Append command support
The v1 Pull Request is https://github.com/axboe/fio/pull/1021
The v2 Pull Request is https://github.com/axboe/fio/pull/1026

Review comments for the v2 Patchset have been taken care of in the v3 Patchset.
Corresponding Linux Kernel Patchset for the Zone Append command support :
https://lore.kernel.org/linux-block/1595605762-17010-1-git-send-email-joshi.k@samsung.com/

1. Added a new FIO option zone_append.
	When zone_append option is enabled, for Zone mode ZBD,
	the existing write path will send Zone Append command with offset as start of the Zone.
2. libaio: support for Zone Append command in libaio IO engine
3. t/zbd: Add support to verify Zone Append command with libaio IO engine tests

This is a RFC and the Linux Kernel io_uring interface is being discussed in the LKML.
io_uring engine's Zone Append command support will be submitted for review after the kernel interface is finalized.

Feedback / Comments will help in improving this further.

Ankit Kumar (2):
  Add Zone Append command support
  libaio: support for Zone Append command

Krishna Kanth Reddy (1):
  t/zbd: Add support to verify Zone Append command with libaio tests

 HOWTO                  |  7 +++++++
 engines/libaio.c       | 32 ++++++++++++++++++++++++++++----
 fio.1                  |  7 +++++++
 ioengines.c            |  6 ++++++
 ioengines.h            |  3 ++-
 options.c              | 10 ++++++++++
 t/zbd/test-zbd-support | 29 +++++++++++++++++++++++++++++
 thread_options.h       |  2 ++
 8 files changed, 91 insertions(+), 5 deletions(-)

-- 
2.17.1

[PATCH 1/3] Add Zone Append command support

[Krishna Kanth Reddy]

From: Ankit Kumar <ankit.kumar@samsung.com>

Added a new FIO option zone_append. In ZBD zone mode,
when zone_append option is enabled, the existing write path will
send Zone Append command with offset as start of the Zone.
Added a new io engine flag FIO_ZONE_APPEND that can be used
in engine declaration to indicate that it supports zone append.

Signed-off-by: Krishna Kanth Reddy <krish.reddy@samsung.com>
---
 HOWTO            |  7 +++++++
 fio.1            |  7 +++++++
 ioengines.c      |  6 ++++++
 ioengines.h      |  3 ++-
 options.c        | 10 ++++++++++
 thread_options.h |  2 ++
 6 files changed, 34 insertions(+), 1 deletion(-)

diff --git a/HOWTO b/HOWTO
index 35ead0cb..a46b9da0 100644
--- a/HOWTO
+++ b/HOWTO
@@ -1022,6 +1022,13 @@ Target file/device
 	:option:`zonesize` bytes of data have been transferred. This parameter
 	must be zero for :option:`zonemode` =zbd.
 
+.. option:: zone_append=bool
+
+	For :option:`rw` =write or :option: `rw` =randwrite and
+	:option:`zonemode` =zbd, if zone_append is enabled the offset to write
+	will be the starting offset of a zone. On successful completion the device
+	returns the offset where the data has been placed.
+
 .. option:: read_beyond_wp=bool
 
 	This parameter applies to :option:`zonemode` =zbd only.
diff --git a/fio.1 b/fio.1
index a3d348b2..00122808 100644
--- a/fio.1
+++ b/fio.1
@@ -790,6 +790,13 @@ zone have been read (read workloads). This parameter is valid only for
 sequential workloads and ignored for random workloads. For read workloads,
 see also \fBread_beyond_wp\fR.
 
+.TP
+.BI zone_append
+For :option:`rw` =write or :option: `rw` =randwrite and for
+:option:`zonemode` =zbd, if zone_append is enabled the offset to write will be
+the starting offset of a zone. On successful completion the device returns the
+offset where the data has been placed.
+
 .TP
 .BI read_beyond_wp \fR=\fPbool
 This parameter applies to \fBzonemode=zbd\fR only.
diff --git a/ioengines.c b/ioengines.c
index 1c5970a4..288306a2 100644
--- a/ioengines.c
+++ b/ioengines.c
@@ -448,6 +448,12 @@ int td_io_init(struct thread_data *td)
 			td->error = ret;
 	}
 
+	if (td->o.zone_append && !td_ioengine_flagged(td, FIO_ZONE_APPEND)) {
+		log_err("fio: io engine %s doesn't support zone append\n",
+			td->io_ops->name);
+		return 1;
+	}
+
 	return ret;
 }
 
diff --git a/ioengines.h b/ioengines.h
index 54dadba2..5ba6b7f2 100644
--- a/ioengines.h
+++ b/ioengines.h
@@ -77,7 +77,8 @@ enum fio_ioengine_flags {
 	FIO_NOSTATS	= 1 << 12,	/* don't do IO stats */
 	FIO_NOFILEHASH	= 1 << 13,	/* doesn't hash the files for lookup later. */
 	FIO_ASYNCIO_SYNC_TRIM
-			= 1 << 14	/* io engine has async ->queue except for trim */
+			= 1 << 14,	/* io engine has async ->queue except for trim */
+	FIO_ZONE_APPEND = 1 << 15	/* engine supports zone append */
 };
 
 /*
diff --git a/options.c b/options.c
index 251ad2c1..0c9cd910 100644
--- a/options.c
+++ b/options.c
@@ -3316,6 +3316,16 @@ struct fio_option fio_options[FIO_MAX_OPTS] = {
 			   },
 		},
 	},
+	{
+		.name	= "zone_append",
+		.lname	= "zone_append",
+		.type	= FIO_OPT_BOOL,
+		.off1	= offsetof(struct thread_options, zone_append),
+		.help	= "Use zone append for writing zones of a zoned block device",
+		.def	= "0",
+		.category = FIO_OPT_C_IO,
+		.group	= FIO_OPT_G_ZONE,
+	},
 	{
 		.name	= "zonesize",
 		.lname	= "Zone size",
diff --git a/thread_options.h b/thread_options.h
index 3fe48ecc..b03b35a7 100644
--- a/thread_options.h
+++ b/thread_options.h
@@ -196,6 +196,7 @@ struct thread_options {
 	unsigned long long zone_capacity;
 	unsigned long long zone_skip;
 	enum fio_zone_mode zone_mode;
+	unsigned int zone_append;
 	unsigned long long lockmem;
 	enum fio_memtype mem_type;
 	unsigned int mem_align;
@@ -633,6 +634,7 @@ struct thread_options_pack {
 	uint32_t allow_mounted_write;
 
 	uint32_t zone_mode;
+	uint32_t zone_append;
 } __attribute__((packed));
 
 extern void convert_thread_options_to_cpu(struct thread_options *o, struct thread_options_pack *top);
-- 
2.17.1

[PATCH 2/3] libaio: support for Zone Append command

[Krishna Kanth Reddy]

From: Ankit Kumar <ankit.kumar@samsung.com>

The zone append command uses the write path with offset as
start of the zone, when ZBD zone mode is enabled.
Upon successful completion, offset  of the location where
the data has been placed is returned in res2 field of the
io event. If data verification is enabled, the offset returned
is used for read and verify.

Signed-off-by: Krishna Kanth Reddy <krish.reddy@samsung.com>
---
 engines/libaio.c | 32 ++++++++++++++++++++++++++++----
 1 file changed, 28 insertions(+), 4 deletions(-)

diff --git a/engines/libaio.c b/engines/libaio.c
index b909b79e..5c10f69d 100644
--- a/engines/libaio.c
+++ b/engines/libaio.c
@@ -12,6 +12,7 @@
 #include <sys/resource.h>
 
 #include "../fio.h"
+#include "../zbd.h"
 #include "../lib/pow2.h"
 #include "../optgroup.h"
 #include "../lib/memalign.h"
@@ -123,7 +124,14 @@ static int fio_libaio_prep(struct thread_data *td, struct io_u *io_u)
 		if (o->nowait)
 			iocb->aio_rw_flags |= RWF_NOWAIT;
 	} else if (io_u->ddir == DDIR_WRITE) {
-		io_prep_pwrite(iocb, f->fd, io_u->xfer_buf, io_u->xfer_buflen, io_u->offset);
+		if ((td->o.zone_mode == ZONE_MODE_ZBD) && td->o.zone_append && f->zbd_info->model != ZBD_NONE) {
+			unsigned long long zone_start_offset = io_u->offset & ~(f->zbd_info->zone_size - 1);
+			io_prep_pwrite(iocb, f->fd, io_u->xfer_buf, io_u->xfer_buflen, zone_start_offset);
+		}
+		else
+			io_prep_pwrite(iocb, f->fd, io_u->xfer_buf, io_u->xfer_buflen, io_u->offset);
+		if (td->o.zone_append)
+			iocb->aio_rw_flags |= RWF_APPEND;
 		if (o->nowait)
 			iocb->aio_rw_flags |= RWF_NOWAIT;
 	} else if (ddir_sync(io_u->ddir))
@@ -229,9 +237,25 @@ static int fio_libaio_getevents(struct thread_data *td, unsigned int min,
 			r = io_getevents(ld->aio_ctx, actual_min,
 				max, ld->aio_events + events, lt);
 		}
-		if (r > 0)
+		if (r > 0) {
+			if ((td->o.zone_mode == ZONE_MODE_ZBD) && td->o.zone_append
+			    && td->o.do_verify && td->o.verify) {
+				struct io_event *ev;
+				struct io_u *io_u;
+				struct fio_file *f;
+				unsigned event;
+
+				for (event = 0; event < r; event++) {
+					ev = ld->aio_events + event;
+					io_u = container_of(ev->obj, struct io_u, iocb);
+					f = io_u->file;
+					if ((io_u->ddir == DDIR_WRITE)
+					    && (f->zbd_info->model != ZBD_NONE))
+						io_u->ipo->offset = ev->res2;
+				}
+			}
 			events += r;
-		else if ((min && r == 0) || r == -EAGAIN) {
+		} else if ((min && r == 0) || r == -EAGAIN) {
 			fio_libaio_commit(td);
 			if (actual_min)
 				usleep(10);
@@ -448,7 +472,7 @@ static int fio_libaio_init(struct thread_data *td)
 FIO_STATIC struct ioengine_ops ioengine = {
 	.name			= "libaio",
 	.version		= FIO_IOOPS_VERSION,
-	.flags			= FIO_ASYNCIO_SYNC_TRIM,
+	.flags			= FIO_ASYNCIO_SYNC_TRIM | FIO_ZONE_APPEND,
 	.init			= fio_libaio_init,
 	.post_init		= fio_libaio_post_init,
 	.prep			= fio_libaio_prep,
-- 
2.17.1

[PATCH 3/3] t/zbd: Add support to verify Zone Append command with libaio tests

[Krishna Kanth Reddy]

Added a new FIO option zone_append.
When zone_append option is enabled and when the zonemode is ZBD,
the existing write path will send Zone Append command with offset as start of the Zone.

Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>
---
 t/zbd/test-zbd-support | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)

diff --git a/t/zbd/test-zbd-support b/t/zbd/test-zbd-support
index 471a3487..048daaf5 100755
--- a/t/zbd/test-zbd-support
+++ b/t/zbd/test-zbd-support
@@ -860,6 +860,35 @@ test49() {
     check_read $((capacity * 2)) || return $?
 }
 
+# Zone append to sequential zones, libaio, 1 job, queue depth 1
+test50() {
+    local size capacity
+
+    [ -n "$use_libzbc" ] && return 0
+    size=$((8 * zone_size))
+    off=$((first_sequential_zone_sector * 512))
+    capacity=$(total_zone_capacity 8 $off $dev)
+
+    run_fio_on_seq --ioengine=libaio --iodepth=1 --rw=write --zone_append=1 \
+                   --bs=65536 --size=$size --do_verify=1 --verify=md5 \
+                   >>"${logfile}.${test_number}" 2>&1 || return $?
+    check_written $capacity || return $?
+    check_read $capacity || return $?
+}
+
+# Random zone append to sequential zones, libaio, 8 jobs, queue depth 64 per job
+test51() {
+    local capacity
+
+    [ -n "$use_libzbc" ] && return 0
+    off=$((first_sequential_zone_sector * 512))
+    capacity=$(total_zone_capacity 4 $off $dev)
+    run_fio_on_seq --ioengine=libaio --iodepth=64 --rw=randwrite \
+		   --bs=4096 --group_reporting=1 --numjobs=8 --zone_append=1 \
+                   >> "${logfile}.${test_number}" 2>&1 || return $?
+    check_written $((capacity * 8)) || return $?
+}
+
 tests=()
 dynamic_analyzer=()
 reset_all_zones=
-- 
2.17.1

Re: [PATCH 0/3] v3 Patchset : Zone Append command support

[Damien Le Moal]

On 2020/07/28 21:40, Krishna Kanth Reddy wrote:
> This is a v3 Patchset for the Zone Append command support
> The v1 Pull Request is https://github.com/axboe/fio/pull/1021
> The v2 Pull Request is https://github.com/axboe/fio/pull/1026
> 
> Review comments for the v2 Patchset have been taken care of in the v3 Patchset.
> Corresponding Linux Kernel Patchset for the Zone Append command support :
> https://lore.kernel.org/linux-block/1595605762-17010-1-git-send-email-joshi.k@samsung.com/
> 
> 1. Added a new FIO option zone_append.
> 	When zone_append option is enabled, for Zone mode ZBD,
> 	the existing write path will send Zone Append command with offset as start of the Zone.
> 2. libaio: support for Zone Append command in libaio IO engine
> 3. t/zbd: Add support to verify Zone Append command with libaio IO engine tests
> 
> This is a RFC and the Linux Kernel io_uring interface is being discussed in the LKML.
> io_uring engine's Zone Append command support will be submitted for review after the kernel interface is finalized.
> 
> Feedback / Comments will help in improving this further.

Krishna,

The kernel side interface for zone append to raw block devices is not yet sorted
out. That interface needs to be fixed first and we can then think of fio
patching. Without a clear idea of what the kernel interface and its semantic
will be, I do not think that commenting on your current code will be useful.

> 
> Ankit Kumar (2):
>   Add Zone Append command support
>   libaio: support for Zone Append command
> 
> Krishna Kanth Reddy (1):
>   t/zbd: Add support to verify Zone Append command with libaio tests
> 
>  HOWTO                  |  7 +++++++
>  engines/libaio.c       | 32 ++++++++++++++++++++++++++++----
>  fio.1                  |  7 +++++++
>  ioengines.c            |  6 ++++++
>  ioengines.h            |  3 ++-
>  options.c              | 10 ++++++++++
>  t/zbd/test-zbd-support | 29 +++++++++++++++++++++++++++++
>  thread_options.h       |  2 ++
>  8 files changed, 91 insertions(+), 5 deletions(-)
> 


-- 
Damien Le Moal
Western Digital Research