[PATCH v2] t/io_uring: allow flexible IO threads assignment

[PATCH v2] t/io_uring: allow flexible IO threads assignment

[Andrzej Jakowski]

This patch allows to flexibly assign IO threads to fileset. When
you specify:

t/io_uring -n 5 /dev/dev1 dev/dev2

First file/device will get 3 IO threads and second file/device
remaining 2 IO threads. When there is more files then IO threads,
IO thread may get assigned multiple files/devices.

Signed-off-by: Andrzej Jakowski <andrzej.jakowski@intel.com>
---
Changes from v1:
 * Handling crashing case when no files are specified
---
 t/io_uring.c | 55 ++++++++++++++++++++++++++++++++++++++-----------------
 1 file changed, 38 insertions(+), 17 deletions(-)

diff --git a/t/io_uring.c b/t/io_uring.c
index 77d2785..e7ff142 100644
--- a/t/io_uring.c
+++ b/t/io_uring.c
@@ -563,7 +563,8 @@ int main(int argc, char *argv[])
 {
 	struct submitter *s;
 	unsigned long done, calls, reap;
-	int err, i, j, flags, fd, opt;
+	int err, i, j, flags, fd, opt, threads_per_f, threads_rem = 0, nfiles;
+	struct file f;
 	char *fdepths;
 	void *ret;
 
@@ -620,37 +621,57 @@ int main(int argc, char *argv[])
 
 	j = 0;
 	i = optind;
+	nfiles = argc - i;
+	if (!nfiles) {
+		printf("no files specified\n");
+		return 1;
+	}
+	threads_per_f = nthreads / nfiles;
+	/* make sure each thread gets assigned files */
+	if (threads_per_f == 0) {
+		threads_per_f = 1;
+	} else {
+		threads_rem = nthreads - threads_per_f * nfiles;
+	}
 	while (!do_nop && i < argc) {
-		struct file *f;
+		int k, limit;
+
+		memset(&f, 0, sizeof(f));
 
-		s = get_submitter(j);
-		if (s->nr_files == MAX_FDS) {
-			printf("Max number of files (%d) reached\n", MAX_FDS);
-			break;
-		}
 		fd = open(argv[i], flags);
 		if (fd < 0) {
 			perror("open");
 			return 1;
 		}
-
-		f = &s->files[s->nr_files];
-		f->real_fd = fd;
-		if (get_file_size(f)) {
+		f.real_fd = fd;
+		if (get_file_size(&f)) {
 			printf("failed getting size of device/file\n");
 			return 1;
 		}
-		if (f->max_blocks <= 1) {
+		if (f.max_blocks <= 1) {
 			printf("Zero file/device size?\n");
 			return 1;
 		}
-		f->max_blocks--;
+		f.max_blocks--;
+
+		limit = threads_per_f;
+		limit += threads_rem > 0 ? 1 : 0;
+		for (k = 0; k < limit; k++) {
+			s = get_submitter((j + k) % nthreads);
+
+			if (s->nr_files == MAX_FDS) {
+				printf("Max number of files (%d) reached\n", MAX_FDS);
+				break;
+			}
 
-		printf("Added file %s (submitter %d)\n", argv[i], s->index);
-		s->nr_files++;
+			memcpy(&s->files[s->nr_files], &f, sizeof(f));
+
+			printf("Added file %s (submitter %d)\n", argv[i], s->index);
+			s->nr_files++;
+		}
+		threads_rem--;
 		i++;
-		if (++j >= nthreads)
-			j = 0;
+		j += limit;
 	}
 
 	arm_sig_int();
-- 
1.8.3.1

Re: [PATCH v2] t/io_uring: allow flexible IO threads assignment

[Jens Axboe]

On 9/8/21 9:07 AM, Andrzej Jakowski wrote:
> This patch allows to flexibly assign IO threads to fileset. When
> you specify:
> 
> t/io_uring -n 5 /dev/dev1 dev/dev2
> 
> First file/device will get 3 IO threads and second file/device
> remaining 2 IO threads. When there is more files then IO threads,
> IO thread may get assigned multiple files/devices.

Not to nit pick, but:
  
> @@ -620,37 +621,57 @@ int main(int argc, char *argv[])
>  
>  	j = 0;
>  	i = optind;
> +	nfiles = argc - i;
> +	if (!nfiles) {
> +		printf("no files specified\n");
> +		return 1;
> +	}

I think this looks correct now, do we want to call usage() here?
Seems common practice for any kind of invocation error.

usage() should probably exit() with non-zero though, in general.

-- 
Jens Axboe