Re: [RFC 2/2] io_uring: mark REQ_NOWAIT for a non-mq queue as unspported

[Jeff Moyer <jmoyer@redhat.com>]

Bijan Mottahedeh <bijan.mottahedeh@oracle.com> writes:

> Mark a REQ_NOWAIT request for a non-mq queue as unspported instead of
> retryable since otherwise the io_uring layer will keep resubmitting
> the request.

Getting back to this...

Jens, right now (using your io_uring-5.7 or linus' tree) fio's
t/io_uring will never get io completions when run against a file on a
file system that is backed by lvm.  The system will have one workqueue
per sqe submitted, all spinning, eating up CPU time.

# ./t/io_uring /mnt/test/poo 
Added file /mnt/test/poo
sq_ring ptr = 0x0x7fbed40ae000
sqes ptr    = 0x0x7fbed40ac000
cq_ring ptr = 0x0x7fbed40aa000
polled=1, fixedbufs=1, buffered=0 QD=128, sq_ring=128, cq_ring=256
submitter=3851
IOPS=128, IOS/call=6/0, inflight=128 (128)
IOPS=0, IOS/call=0/0, inflight=128 (128)
IOPS=0, IOS/call=0/0, inflight=128 (128)
IOPS=0, IOS/call=0/0, inflight=128 (128)
IOPS=0, IOS/call=0/0, inflight=128 (128)
IOPS=0, IOS/call=0/0, inflight=128 (128)
...

# ps auxw | grep io_wqe
root      3849 80.1  0.0      0     0 ?        R    14:32   0:40 [io_wqe_worker-0]
root      3850  0.0  0.0      0     0 ?        S    14:32   0:00 [io_wqe_worker-0]
root      3853 72.8  0.0      0     0 ?        R    14:32   0:36 [io_wqe_worker-0]
root      3854 81.4  0.0      0     0 ?        R    14:32   0:40 [io_wqe_worker-1]
root      3855 74.8  0.0      0     0 ?        R    14:32   0:37 [io_wqe_worker-0]
root      3856 74.8  0.0      0     0 ?        R    14:32   0:37 [io_wqe_worker-1]
...

# ps auxw | grep io_wqe | grep -v grep | wc -l
129

With this patch applied, the test program will exit without doing I/O
(which I don't think is the right behavior either, right?):

# t/io_uring /mnt/test/poo
Added file /mnt/test/poo
sq_ring ptr = 0x0x7fdb98f00000
sqes ptr    = 0x0x7fdb98efe000
cq_ring ptr = 0x0x7fdb98efc000
polled=1, fixedbufs=1, buffered=0 QD=128, sq_ring=128, cq_ring=256
submitter=33233
io: unexpected ret=-95
Your filesystem/driver/kernel doesn't support polled IO
IOPS=128, IOS/call=32/0, inflight=128 (127)

/mnt/test is an xfs file system on top of a linear LVM volume on an nvme
device (with 8 poll queues configured).

-Jeff

>
> Signed-off-by: Bijan Mottahedeh <bijan.mottahedeh@oracle.com>
> ---
>  block/blk-core.c | 10 +++-------
>  1 file changed, 3 insertions(+), 7 deletions(-)
>
> diff --git a/block/blk-core.c b/block/blk-core.c
> index 5847993..3807140 100644
> --- a/block/blk-core.c
> +++ b/block/blk-core.c
> @@ -962,14 +962,10 @@ static inline blk_status_t blk_check_zone_append(struct request_queue *q,
>  	}
>  
>  	/*
> -	 * Non-mq queues do not honor REQ_NOWAIT, so complete a bio
> -	 * with BLK_STS_AGAIN status in order to catch -EAGAIN and
> -	 * to give a chance to the caller to repeat request gracefully.
> +	 * Non-mq queues do not honor REQ_NOWAIT, return -EOPNOTSUPP.
>  	 */
> -	if ((bio->bi_opf & REQ_NOWAIT) && !queue_is_mq(q)) {
> -		status = BLK_STS_AGAIN;
> -		goto end_io;
> -	}
> +	if ((bio->bi_opf & REQ_NOWAIT) && !queue_is_mq(q))
> +		goto not_supported;
>  
>  	if (should_fail_bio(bio))
>  		goto end_io;