Re: question about relative control for sync io using bfq

From: Paolo Valente <paolo.valente@linaro.org>
To: "yukuai (C)" <yukuai3@huawei.com>
Cc: Jens Axboe <axboe@kernel.dk>, Ming Lei <ming.lei@redhat.com>,
	hch@lst.de, linux-block@vger.kernel.org,
	chenzhou <chenzhou10@huawei.com>,
	"houtao (A)" <houtao1@huawei.com>
Subject: Re: question about relative control for sync io using bfq
Date: Fri, 22 Jan 2021 11:09:40 +0100	[thread overview]
Message-ID: <7E41BC22-33EA-4D0F-9EBD-3AB0824E3F2E@linaro.org> (raw)
In-Reply-To: <b4163392-0462-ff6f-b958-1f96f33d69e6@huawei.com>

> Il giorno 11 gen 2021, alle ore 14:15, yukuai (C) <yukuai3@huawei.com> ha scritto:
> 
> Hi,
> 
> We found a performance problem:
> 
> kernel version: 5.10
> disk: ssd
> scheduler: bfq
> arch: arm64 / x86_64
> test param: direct=1, ioengine=psync, bs=4k, rw=randread, numjobs=32
> 
> We are using 32 threads here, test results showed that iops is equal
> to single thread.
> 
> After digging into the problem, I found root cause of the problem is strange:
> 
> bfq_add_request
> bfq_bfqq_handle_idle_busy_switch
>  bfq_add_bfqq_busy
>   bfq_activate_bfq
>    bfq_activate_requeue_entity
>     __bfq_activate_requeue_entity
>      __bfq_activate_entity
>       if (!bfq_entity_to_bfqq(entity))
>        if (!entity->in_groups_with_pending_reqs)
>         entity->in_groups_with_pending_reqs = true;
>         bfqd->num_groups_with_pending_reqs++
> 
> If test process is not in root cgroup, num_groups_with_pending_reqs will
> be increased after request was instered to bfq.
> 
> bfq_select_queue
> bfq_better_to_idle
>  idling_needed_for_service_guarantees
>   bfq_asymmetric_scenario
>    return varied_queue_weights || multiple_classes_busy || bfqd->num_groups_with_pending_reqs > 0
> 
> After issuing IO to driver, num_groups_with_pending_reqs is ensured to
> be nonzero, thus bfq won't expire the queue. This is the root cause of
> degradating to single-process performance.
> 
> One the other hand, if I set slice_idle to zero, bfq_better_to_idle will
> return false early, and the problem will disapear. However, relative
> control will be inactive.
> 
> My question is that, is this a known flaw for bfq? If not, as cfq don't
> have such problem, is there a suitable solution?
> 

Hi,
this is a core problem, not of BFQ but of any possible solution that
has to provide bandwidth isolation with sync I/O.  One of the examples
is the one I made for you in my other email.  At any rate, the problem
that you report seems to occur with just one group.  We may think of
simply changing my condition

bfqd->num_groups_with_pending_reqs > 0

to

bfqd->num_groups_with_pending_reqs > 1

If this simple solution does solve the problem you report, then I
could run my batch of tests to check whether it causes some
regression.

What do you think?

Thanks.
Paolo

> Thanks!
> Yu Kuai
> such problem,