question about relative control for sync io using bfq

From: "yukuai (C)" <yukuai3@huawei.com>
To: <axboe@kernel.dk>, Ming Lei <ming.lei@redhat.com>, <hch@lst.de>,
	<linux-block@vger.kernel.org>, chenzhou <chenzhou10@huawei.com>,
	"houtao (A)" <houtao1@huawei.com>
Subject: question about relative control for sync io using bfq
Date: Mon, 11 Jan 2021 21:15:13 +0800	[thread overview]
Message-ID: <b4163392-0462-ff6f-b958-1f96f33d69e6@huawei.com> (raw)

Hi,

We found a performance problem:

kernel version: 5.10
disk: ssd
scheduler: bfq
arch: arm64 / x86_64
test param: direct=1, ioengine=psync, bs=4k, rw=randread, numjobs=32

We are using 32 threads here, test results showed that iops is equal
to single thread.

After digging into the problem, I found root cause of the problem is 
strange:

bfq_add_request
  bfq_bfqq_handle_idle_busy_switch
   bfq_add_bfqq_busy
    bfq_activate_bfq
     bfq_activate_requeue_entity
      __bfq_activate_requeue_entity
       __bfq_activate_entity
        if (!bfq_entity_to_bfqq(entity))
         if (!entity->in_groups_with_pending_reqs)
          entity->in_groups_with_pending_reqs = true;
          bfqd->num_groups_with_pending_reqs++

If test process is not in root cgroup, num_groups_with_pending_reqs will
be increased after request was instered to bfq.

bfq_select_queue
  bfq_better_to_idle
   idling_needed_for_service_guarantees
    bfq_asymmetric_scenario
     return varied_queue_weights || multiple_classes_busy || 
bfqd->num_groups_with_pending_reqs > 0

After issuing IO to driver, num_groups_with_pending_reqs is ensured to
be nonzero, thus bfq won't expire the queue. This is the root cause of
degradating to single-process performance.

One the other hand, if I set slice_idle to zero, bfq_better_to_idle will
return false early, and the problem will disapear. However, relative
control will be inactive.

My question is that, is this a known flaw for bfq? If not, as cfq don't
have such problem, is there a suitable solution?

Thanks!
Yu Kuai
such problem,