Re: [RFC 2/2] io_uring: mark REQ_NOWAIT for a non-mq queue as unspported

[Jens Axboe <axboe@kernel.dk>]

On 5/28/20 1:01 PM, Jens Axboe wrote:
> On 5/28/20 12:35 PM, Jeff Moyer wrote:
>> Bijan Mottahedeh <bijan.mottahedeh@oracle.com> writes:
>>
>>> Mark a REQ_NOWAIT request for a non-mq queue as unspported instead of
>>> retryable since otherwise the io_uring layer will keep resubmitting
>>> the request.
>>
>> Getting back to this...
>>
>> Jens, right now (using your io_uring-5.7 or linus' tree) fio's
>> t/io_uring will never get io completions when run against a file on a
>> file system that is backed by lvm.  The system will have one workqueue
>> per sqe submitted, all spinning, eating up CPU time.
>>
>> # ./t/io_uring /mnt/test/poo 
>> Added file /mnt/test/poo
>> sq_ring ptr = 0x0x7fbed40ae000
>> sqes ptr    = 0x0x7fbed40ac000
>> cq_ring ptr = 0x0x7fbed40aa000
>> polled=1, fixedbufs=1, buffered=0 QD=128, sq_ring=128, cq_ring=256
>> submitter=3851
>> IOPS=128, IOS/call=6/0, inflight=128 (128)
>> IOPS=0, IOS/call=0/0, inflight=128 (128)
>> IOPS=0, IOS/call=0/0, inflight=128 (128)
>> IOPS=0, IOS/call=0/0, inflight=128 (128)
>> IOPS=0, IOS/call=0/0, inflight=128 (128)
>> IOPS=0, IOS/call=0/0, inflight=128 (128)
>> ...
>>
>> # ps auxw | grep io_wqe
>> root      3849 80.1  0.0      0     0 ?        R    14:32   0:40 [io_wqe_worker-0]
>> root      3850  0.0  0.0      0     0 ?        S    14:32   0:00 [io_wqe_worker-0]
>> root      3853 72.8  0.0      0     0 ?        R    14:32   0:36 [io_wqe_worker-0]
>> root      3854 81.4  0.0      0     0 ?        R    14:32   0:40 [io_wqe_worker-1]
>> root      3855 74.8  0.0      0     0 ?        R    14:32   0:37 [io_wqe_worker-0]
>> root      3856 74.8  0.0      0     0 ?        R    14:32   0:37 [io_wqe_worker-1]
>> ...
>>
>> # ps auxw | grep io_wqe | grep -v grep | wc -l
>> 129
>>
>> With this patch applied, the test program will exit without doing I/O
>> (which I don't think is the right behavior either, right?):
>>
>> # t/io_uring /mnt/test/poo
>> Added file /mnt/test/poo
>> sq_ring ptr = 0x0x7fdb98f00000
>> sqes ptr    = 0x0x7fdb98efe000
>> cq_ring ptr = 0x0x7fdb98efc000
>> polled=1, fixedbufs=1, buffered=0 QD=128, sq_ring=128, cq_ring=256
>> submitter=33233
>> io: unexpected ret=-95
>> Your filesystem/driver/kernel doesn't support polled IO
>> IOPS=128, IOS/call=32/0, inflight=128 (127)
>>
>> /mnt/test is an xfs file system on top of a linear LVM volume on an nvme
>> device (with 8 poll queues configured).
> 
> poll won't work over dm, so that looks correct. What happens if you edit
> it and disable poll? Would be curious to see both buffered = 0 and
> buffered = 1 runs with that.
> 
> I'll try this here too.

I checked, and with the offending commit reverted, it behaves exactly
like it should - io_uring doesn't hit endless retries, and we still
return -EAGAIN to userspace for preadv2(..., RFW_NOWAIT) if not supported.
I've queued up the revert.

Jeff, the poll test above is supposed to fail as we can't poll on dm.
So that part is expected.

-- 
Jens Axboe