From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 734ADC433EF for ; Wed, 27 Apr 2022 12:33:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234192AbiD0MgZ (ORCPT ); Wed, 27 Apr 2022 08:36:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58316 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233982AbiD0MgX (ORCPT ); Wed, 27 Apr 2022 08:36:23 -0400 Received: from szxga03-in.huawei.com (szxga03-in.huawei.com [45.249.212.189]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D46A0108C; Wed, 27 Apr 2022 05:33:11 -0700 (PDT) Received: from kwepemi100006.china.huawei.com (unknown [172.30.72.57]) by szxga03-in.huawei.com (SkyGuard) with ESMTP id 4KpJ2F0h6lzCs4T; Wed, 27 Apr 2022 20:28:37 +0800 (CST) Received: from kwepemm600009.china.huawei.com (7.193.23.164) by kwepemi100006.china.huawei.com (7.221.188.165) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Wed, 27 Apr 2022 20:33:09 +0800 Received: from huawei.com (10.175.127.227) by kwepemm600009.china.huawei.com (7.193.23.164) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Wed, 27 Apr 2022 20:33:09 +0800 From: Yu Kuai To: , , , CC: , , , , Subject: [PATCH -next v3 0/3] support concurrent sync io for bfq on a specail occasion Date: Wed, 27 Apr 2022 20:47:19 +0800 Message-ID: <20220427124722.48465-1-yukuai3@huawei.com> X-Mailer: git-send-email 2.31.1 MIME-Version: 1.0 Content-Transfer-Encoding: 7BIT Content-Type: text/plain; charset=US-ASCII X-Originating-IP: [10.175.127.227] X-ClientProxiedBy: dggems704-chm.china.huawei.com (10.3.19.181) To kwepemm600009.china.huawei.com (7.193.23.164) X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Changes in v3: - remove the cleanup patch that is irrelevant now(I'll post it separately). - instead of hacking wr queues and using weights tree insertion/removal, using bfq_add/del_bfqq_busy() to count the number of groups (suggested by Jan Kara). Changes in v2: - Use a different approch to count root group, which is much simple. Currently, bfq can't handle sync io concurrently as long as they are not issued from root group. This is because 'bfqd->num_groups_with_pending_reqs > 0' is always true in bfq_asymmetric_scenario(). The way that bfqg is counted into 'num_groups_with_pending_reqs': Before this patchset: 1) root group will never be counted. 2) Count if bfqg or it's child bfqgs have pending requests. 3) Don't count if bfqg and it's child bfqgs complete all the requests. After this patchset: 1) root group is counted. 2) Count if bfqg have at least one bfqq that is marked busy. 3) Don't count if bfqg doesn't have any busy bfqqs. The main reason to use busy state of bfqq instead of 'pending requests' is that bfqq can stay busy after dispatching the last request if idling is needed for service guarantees. With the above changes, concurrent sync io can be supported if only one group is activated. fio test script(startdelay is used to avoid queue merging): [global] filename=/dev/nvme0n1 allow_mounted_write=0 ioengine=psync direct=1 ioscheduler=bfq offset_increment=10g group_reporting rw=randwrite bs=4k [test1] numjobs=1 [test2] startdelay=1 numjobs=1 [test3] startdelay=2 numjobs=1 [test4] startdelay=3 numjobs=1 [test5] startdelay=4 numjobs=1 [test6] startdelay=5 numjobs=1 [test7] startdelay=6 numjobs=1 [test8] startdelay=7 numjobs=1 test result: running fio on root cgroup v5.18-rc1: 550 Mib/s v5.18-rc1-patched: 550 Mib/s running fio on non-root cgroup v5.18-rc1: 349 Mib/s v5.18-rc1-patched: 550 Mib/s Note that I also test null_blk with "irqmode=2 completion_nsec=100000000(100ms) hw_queue_depth=1", and tests show that service guarantees are still preserved. Previous versions: RFC: https://lore.kernel.org/all/20211127101132.486806-1-yukuai3@huawei.com/ v1: https://lore.kernel.org/all/20220305091205.4188398-1-yukuai3@huawei.com/ v2: https://lore.kernel.org/all/20220416093753.3054696-1-yukuai3@huawei.com/ Yu Kuai (3): block, bfq: record how many queues are busy in bfq_group block, bfq: refactor the counting of 'num_groups_with_pending_reqs' block, bfq: do not idle if only one group is activated block/bfq-cgroup.c | 1 + block/bfq-iosched.c | 46 ++------------------------------------ block/bfq-iosched.h | 54 ++++++++------------------------------------- block/bfq-wf2q.c | 29 +++++++++++++----------- 4 files changed, 28 insertions(+), 102 deletions(-) -- 2.31.1 From mboxrd@z Thu Jan 1 00:00:00 1970 From: Yu Kuai Subject: [PATCH -next v3 0/3] support concurrent sync io for bfq on a specail occasion Date: Wed, 27 Apr 2022 20:47:19 +0800 Message-ID: <20220427124722.48465-1-yukuai3@huawei.com> Mime-Version: 1.0 Content-Transfer-Encoding: 7BIT Return-path: List-ID: Content-Type: text/plain; charset="us-ascii" To: jack@suse.cz, tj@kernel.org, axboe@kernel.dk, paolo.valente@linaro.org Cc: cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, yukuai3@huawei.com, yi.zhang@huawei.com Changes in v3: - remove the cleanup patch that is irrelevant now(I'll post it separately). - instead of hacking wr queues and using weights tree insertion/removal, using bfq_add/del_bfqq_busy() to count the number of groups (suggested by Jan Kara). Changes in v2: - Use a different approch to count root group, which is much simple. Currently, bfq can't handle sync io concurrently as long as they are not issued from root group. This is because 'bfqd->num_groups_with_pending_reqs > 0' is always true in bfq_asymmetric_scenario(). The way that bfqg is counted into 'num_groups_with_pending_reqs': Before this patchset: 1) root group will never be counted. 2) Count if bfqg or it's child bfqgs have pending requests. 3) Don't count if bfqg and it's child bfqgs complete all the requests. After this patchset: 1) root group is counted. 2) Count if bfqg have at least one bfqq that is marked busy. 3) Don't count if bfqg doesn't have any busy bfqqs. The main reason to use busy state of bfqq instead of 'pending requests' is that bfqq can stay busy after dispatching the last request if idling is needed for service guarantees. With the above changes, concurrent sync io can be supported if only one group is activated. fio test script(startdelay is used to avoid queue merging): [global] filename=/dev/nvme0n1 allow_mounted_write=0 ioengine=psync direct=1 ioscheduler=bfq offset_increment=10g group_reporting rw=randwrite bs=4k [test1] numjobs=1 [test2] startdelay=1 numjobs=1 [test3] startdelay=2 numjobs=1 [test4] startdelay=3 numjobs=1 [test5] startdelay=4 numjobs=1 [test6] startdelay=5 numjobs=1 [test7] startdelay=6 numjobs=1 [test8] startdelay=7 numjobs=1 test result: running fio on root cgroup v5.18-rc1: 550 Mib/s v5.18-rc1-patched: 550 Mib/s running fio on non-root cgroup v5.18-rc1: 349 Mib/s v5.18-rc1-patched: 550 Mib/s Note that I also test null_blk with "irqmode=2 completion_nsec=100000000(100ms) hw_queue_depth=1", and tests show that service guarantees are still preserved. Previous versions: RFC: https://lore.kernel.org/all/20211127101132.486806-1-yukuai3@huawei.com/ v1: https://lore.kernel.org/all/20220305091205.4188398-1-yukuai3@huawei.com/ v2: https://lore.kernel.org/all/20220416093753.3054696-1-yukuai3@huawei.com/ Yu Kuai (3): block, bfq: record how many queues are busy in bfq_group block, bfq: refactor the counting of 'num_groups_with_pending_reqs' block, bfq: do not idle if only one group is activated block/bfq-cgroup.c | 1 + block/bfq-iosched.c | 46 ++------------------------------------ block/bfq-iosched.h | 54 ++++++++------------------------------------- block/bfq-wf2q.c | 29 +++++++++++++----------- 4 files changed, 28 insertions(+), 102 deletions(-) -- 2.31.1