Linux-Block Archive on lore.kernel.org
 help / color / Atom feed
From: JeffleXu <jefflexu@linux.alibaba.com>
To: Ming Lei <ming.lei@redhat.com>
Cc: axboe@kernel.dk, hch@infradead.org, viro@zeniv.linux.org.uk,
	linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org,
	joseph.qi@linux.alibaba.com, xiaoguang.wang@linux.alibaba.com
Subject: Re: [PATCH v3 2/2] block,iomap: disable iopoll when split needed
Date: Fri, 16 Oct 2020 21:30:27 +0800
Message-ID: <e9d477ed-df87-d46a-5a81-f3fb377e4233@linux.alibaba.com> (raw)
In-Reply-To: <20201016123925.GB1218835@T590>


On 10/16/20 8:39 PM, Ming Lei wrote:
> On Fri, Oct 16, 2020 at 07:02:44PM +0800, JeffleXu wrote:
>> On 10/16/20 6:26 PM, Ming Lei wrote:
>>> On Fri, Oct 16, 2020 at 05:18:51PM +0800, Jeffle Xu wrote:
>>>> Both blkdev fs and iomap-based fs (ext4, xfs, etc.) currently support
>>>> sync iopoll. One single bio can contain at most BIO_MAX_PAGES, i.e. 256
>>>> bio_vec. If the input iov_iter contains more than 256 segments, then
>>>> one dio will be split into multiple bios, which may cause potential
>>>> deadlock for sync iopoll.
>>>>
>>>> When it comes to sync iopoll, the bio is submitted without REQ_NOWAIT
>>>> flag set and the process may hang in blk_mq_get_tag() if the dio needs
>>>> to be split into multiple bios and thus can rapidly exhausts the queue
>>>> depth. The process has to wait for the completion of the previously
>>>> allocated requests, which should be reaped by the following sync
>>>> polling, and thus causing a deadlock.
>>>>
>>>> In fact there's a subtle difference of handling of HIPRI IO between
>>>> blkdev fs and iomap-based fs, when dio need to be split into multiple
>>>> bios. blkdev fs will set REQ_HIPRI for only the last split bio, leaving
>>>> the previous bios queued into normal hardware queues, and not causing
>>>> the trouble described above. iomap-based fs will set REQ_HIPRI for all
>>>> split bios, and thus may cause the potential deadlock decribed above.
>>>>
>>>> Thus disable iopoll when one dio need to be split into multiple bios.
>>>> Though blkdev fs may not suffer this issue, still it may not make much
>>>> sense to iopoll for big IO, since iopoll is initially for small size,
>>>> latency sensitive IO.
>>>>
>>>> Signed-off-by: Jeffle Xu <jefflexu@linux.alibaba.com>
>>>> ---
>>>>    fs/block_dev.c       | 7 +++++++
>>>>    fs/iomap/direct-io.c | 8 ++++++++
>>>>    2 files changed, 15 insertions(+)
>>>>
>>>> diff --git a/fs/block_dev.c b/fs/block_dev.c
>>>> index 9e84b1928b94..1b56b39e35b5 100644
>>>> --- a/fs/block_dev.c
>>>> +++ b/fs/block_dev.c
>>>> @@ -436,6 +436,13 @@ __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, int nr_pages)
>>>>    			break;
>>>>    		}
>>>> +		/*
>>>> +		 * The current dio need to be split into multiple bios here.
>>>> +		 * iopoll is initially for small size, latency sensitive IO,
>>>> +		 * and thus disable iopoll if split needed.
>>>> +		 */
>>>> +		iocb->ki_flags &= ~IOCB_HIPRI;
>>>> +
>>> Not sure if it is good to clear IOCB_HIPRI of iocb, since it is usually
>>> maintained by upper layer code(io_uring, aio, ...) and we shouldn't
>>> touch this flag here.
>> If we queue bios into the DEFAULT hardware queue, but leaving the
>> corresponding kiocb->ki_flags's
>>
>> IOCB_HIPRI set (exactly what the first patch does), is this another
>> inconsistency?
> My question is that if it is good for this code to clear IOCB_HIPRI of iocb,
> given this is the 1st such usage. And does io_uring implementation expect
> the flag to be cleared by lower layer?

I know your point. I will check code in io_uring later.


>
>> Please consider the following code snippet from __blkdev_direct_IO()
>>
>> ```
>> 	for (;;) {
>> 		set_current_state(TASK_UNINTERRUPTIBLE);
>> 		if (!READ_ONCE(dio->waiter))
>> 			break;
>>
>> 		if (!(iocb->ki_flags & IOCB_HIPRI) ||
>> 		    !blk_poll(bdev_get_queue(bdev), qc, true))
>> 			blk_io_schedule();
>> 	}
>> ```
>>
>> The IOCB_HIPRI flag is still set in iocb->ki_flags, but the corresponding
>> bios are queued into DEFAULT hardware queue since the first patch.
>> blk_poll() is still called in this case.
> It may be handled in the following way:
>
>   		if (!((iocb->ki_flags & IOCB_HIPRI) && !dio->multi_bio) ||
>   		    !blk_poll(bdev_get_queue(bdev), qc, true))
>   				blk_io_schedule();
>
> BTW, even for single bio with IOCB_HIPRI, the single fs bio can still be
> splitted, and blk_poll() will be called too.
Yes that's exactly I'm concerned and I've seen your comments in patch 1. 
Thanks.
>
>
> Thanks,
> Ming

      reply index

Thread overview: 8+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-10-16  9:18 [PATCH v3 0/2] block, iomap: disable iopoll for split bio Jeffle Xu
2020-10-16  9:18 ` [PATCH v3 1/2] block: " Jeffle Xu
2020-10-16 12:51   ` Ming Lei
2020-10-16  9:18 ` [PATCH v3 2/2] block,iomap: disable iopoll when split needed Jeffle Xu
2020-10-16 10:26   ` Ming Lei
2020-10-16 11:02     ` JeffleXu
2020-10-16 12:39       ` Ming Lei
2020-10-16 13:30         ` JeffleXu [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=e9d477ed-df87-d46a-5a81-f3fb377e4233@linux.alibaba.com \
    --to=jefflexu@linux.alibaba.com \
    --cc=axboe@kernel.dk \
    --cc=hch@infradead.org \
    --cc=joseph.qi@linux.alibaba.com \
    --cc=linux-block@vger.kernel.org \
    --cc=linux-fsdevel@vger.kernel.org \
    --cc=ming.lei@redhat.com \
    --cc=viro@zeniv.linux.org.uk \
    --cc=xiaoguang.wang@linux.alibaba.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-Block Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-block/0 linux-block/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-block linux-block/ https://lore.kernel.org/linux-block \
		linux-block@vger.kernel.org
	public-inbox-index linux-block

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-block


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git