From: JeffleXu <jefflexu@linux.alibaba.com>
To: Ming Lei <ming.lei@redhat.com>
Cc: axboe@kernel.dk, hch@infradead.org, viro@zeniv.linux.org.uk,
linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org,
joseph.qi@linux.alibaba.com, xiaoguang.wang@linux.alibaba.com
Subject: Re: [PATCH v3 2/2] block,iomap: disable iopoll when split needed
Date: Fri, 16 Oct 2020 19:02:44 +0800 [thread overview]
Message-ID: <d6d6b80b-6b16-637a-fac3-7f5a161b8f51@linux.alibaba.com> (raw)
In-Reply-To: <20201016102625.GA1218835@T590>
On 10/16/20 6:26 PM, Ming Lei wrote:
> On Fri, Oct 16, 2020 at 05:18:51PM +0800, Jeffle Xu wrote:
>> Both blkdev fs and iomap-based fs (ext4, xfs, etc.) currently support
>> sync iopoll. One single bio can contain at most BIO_MAX_PAGES, i.e. 256
>> bio_vec. If the input iov_iter contains more than 256 segments, then
>> one dio will be split into multiple bios, which may cause potential
>> deadlock for sync iopoll.
>>
>> When it comes to sync iopoll, the bio is submitted without REQ_NOWAIT
>> flag set and the process may hang in blk_mq_get_tag() if the dio needs
>> to be split into multiple bios and thus can rapidly exhausts the queue
>> depth. The process has to wait for the completion of the previously
>> allocated requests, which should be reaped by the following sync
>> polling, and thus causing a deadlock.
>>
>> In fact there's a subtle difference of handling of HIPRI IO between
>> blkdev fs and iomap-based fs, when dio need to be split into multiple
>> bios. blkdev fs will set REQ_HIPRI for only the last split bio, leaving
>> the previous bios queued into normal hardware queues, and not causing
>> the trouble described above. iomap-based fs will set REQ_HIPRI for all
>> split bios, and thus may cause the potential deadlock decribed above.
>>
>> Thus disable iopoll when one dio need to be split into multiple bios.
>> Though blkdev fs may not suffer this issue, still it may not make much
>> sense to iopoll for big IO, since iopoll is initially for small size,
>> latency sensitive IO.
>>
>> Signed-off-by: Jeffle Xu <jefflexu@linux.alibaba.com>
>> ---
>> fs/block_dev.c | 7 +++++++
>> fs/iomap/direct-io.c | 8 ++++++++
>> 2 files changed, 15 insertions(+)
>>
>> diff --git a/fs/block_dev.c b/fs/block_dev.c
>> index 9e84b1928b94..1b56b39e35b5 100644
>> --- a/fs/block_dev.c
>> +++ b/fs/block_dev.c
>> @@ -436,6 +436,13 @@ __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, int nr_pages)
>> break;
>> }
>>
>> + /*
>> + * The current dio need to be split into multiple bios here.
>> + * iopoll is initially for small size, latency sensitive IO,
>> + * and thus disable iopoll if split needed.
>> + */
>> + iocb->ki_flags &= ~IOCB_HIPRI;
>> +
> Not sure if it is good to clear IOCB_HIPRI of iocb, since it is usually
> maintained by upper layer code(io_uring, aio, ...) and we shouldn't
> touch this flag here.
If we queue bios into the DEFAULT hardware queue, but leaving the
corresponding kiocb->ki_flags's
IOCB_HIPRI set (exactly what the first patch does), is this another
inconsistency?
Please consider the following code snippet from __blkdev_direct_IO()
```
for (;;) {
set_current_state(TASK_UNINTERRUPTIBLE);
if (!READ_ONCE(dio->waiter))
break;
if (!(iocb->ki_flags & IOCB_HIPRI) ||
!blk_poll(bdev_get_queue(bdev), qc, true))
blk_io_schedule();
}
```
The IOCB_HIPRI flag is still set in iocb->ki_flags, but the corresponding
bios are queued into DEFAULT hardware queue since the first patch.
blk_poll() is still called in this case.
>
>> if (!dio->multi_bio) {
>> /*
>> * AIO needs an extra reference to ensure the dio
>> diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c
>> index c1aafb2ab990..46668cceefd2 100644
>> --- a/fs/iomap/direct-io.c
>> +++ b/fs/iomap/direct-io.c
>> @@ -308,6 +308,14 @@ iomap_dio_bio_actor(struct inode *inode, loff_t pos, loff_t length,
>> copied += n;
>>
>> nr_pages = iov_iter_npages(dio->submit.iter, BIO_MAX_PAGES);
>> + /*
>> + * The current dio need to be split into multiple bios here.
>> + * iopoll is initially for small size, latency sensitive IO,
>> + * and thus disable iopoll if split needed.
>> + */
>> + if (nr_pages)
>> + dio->iocb->ki_flags &= ~IOCB_HIPRI;
> Same concern as above.
>
> Thanks,
> Ming
next prev parent reply other threads:[~2020-10-16 11:02 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-10-16 9:18 [PATCH v3 0/2] block, iomap: disable iopoll for split bio Jeffle Xu
2020-10-16 9:18 ` [PATCH v3 1/2] block: " Jeffle Xu
2020-10-16 12:51 ` Ming Lei
2020-10-16 9:18 ` [PATCH v3 2/2] block,iomap: disable iopoll when split needed Jeffle Xu
2020-10-16 10:26 ` Ming Lei
2020-10-16 11:02 ` JeffleXu [this message]
2020-10-16 12:39 ` Ming Lei
2020-10-16 13:30 ` JeffleXu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=d6d6b80b-6b16-637a-fac3-7f5a161b8f51@linux.alibaba.com \
--to=jefflexu@linux.alibaba.com \
--cc=axboe@kernel.dk \
--cc=hch@infradead.org \
--cc=joseph.qi@linux.alibaba.com \
--cc=linux-block@vger.kernel.org \
--cc=linux-fsdevel@vger.kernel.org \
--cc=ming.lei@redhat.com \
--cc=viro@zeniv.linux.org.uk \
--cc=xiaoguang.wang@linux.alibaba.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).