From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8A69CC433FE for ; Fri, 25 Mar 2022 07:30:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1352116AbiCYHbm (ORCPT ); Fri, 25 Mar 2022 03:31:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53458 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348234AbiCYHbj (ORCPT ); Fri, 25 Mar 2022 03:31:39 -0400 Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [45.249.212.187]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9F7B6BF510; Fri, 25 Mar 2022 00:30:05 -0700 (PDT) Received: from kwepemi100022.china.huawei.com (unknown [172.30.72.56]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4KPtym50JZzcbs1; Fri, 25 Mar 2022 15:29:52 +0800 (CST) Received: from kwepemm600009.china.huawei.com (7.193.23.164) by kwepemi100022.china.huawei.com (7.221.188.126) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Fri, 25 Mar 2022 15:30:03 +0800 Received: from [10.174.176.73] (10.174.176.73) by kwepemm600009.china.huawei.com (7.193.23.164) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.21; Fri, 25 Mar 2022 15:30:03 +0800 Subject: Re: [PATCH RFC -next 0/3] improve fairness for sbitmap waitqueues To: , , , , CC: , , References: <20220318082505.3025427-1-yukuai3@huawei.com> From: "yukuai (C)" Message-ID: Date: Fri, 25 Mar 2022 15:30:02 +0800 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.8.0 MIME-Version: 1.0 In-Reply-To: <20220318082505.3025427-1-yukuai3@huawei.com> Content-Type: text/plain; charset="gbk"; format=flowed Content-Transfer-Encoding: 8bit X-Originating-IP: [10.174.176.73] X-ClientProxiedBy: dggems701-chm.china.huawei.com (10.3.19.178) To kwepemm600009.china.huawei.com (7.193.23.164) X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org friendly ping ... ÔÚ 2022/03/18 16:25, Yu Kuai дµÀ: > During some io test, I found that waitqueues can be extremly unbalanced, > especially when tags are little. > > For example: > test cmd: nr_requests is set to 64, and queue_depth is set to 32 > [global] > filename=/dev/sdh > ioengine=libaio > direct=1 > allow_mounted_write=0 > group_reporting > > [test] > rw=randwrite > bs=4k > numjobs=512 > iodepth=2 > > With patch 1 applied, I observe the following status: > ws_active=484 > ws={ > {.wait_cnt=8, .waiters_cnt=117}, > {.wait_cnt=8, .waiters_cnt=59}, > {.wait_cnt=8, .waiters_cnt=76}, > {.wait_cnt=8, .waiters_cnt=0}, > {.wait_cnt=5, .waiters_cnt=24}, > {.wait_cnt=8, .waiters_cnt=12}, > {.wait_cnt=8, .waiters_cnt=21}, > {.wait_cnt=8, .waiters_cnt=175}, > } > > 'waiters_cnt' means how many threads are waitng for tags in the 'ws', > and such extremely unbalanced status is very frequent. After reading the > sbitmap code, I found there are two situations that might cause the > problem: > > 1) blk_mq_get_tag() can call 'bt_wait_ptr()' while the threads might get > tag successfully before going to wait. - patch 2 > > 2) After a 'ws' is woken up, following blk_mq_put_tag() might wake up > the same 'ws' again instead of the next one. - patch 3 > > I'm not sure if the unbalanced status is really a *problem* and need to > be fixed, this patchset is just to improve fairness and not a thorough > fix. Any comments and suggestions are welcome. > > Yu Kuai (3): > sbitmap: record the number of waiters for each waitqueue > blk-mq: call 'bt_wait_ptr()' later in blk_mq_get_tag() > sbitmap: improve the fairness of waitqueues' wake up > > block/blk-mq-tag.c | 6 ++--- > include/linux/sbitmap.h | 5 ++++ > lib/sbitmap.c | 57 ++++++++++++++++++++++------------------- > 3 files changed, 39 insertions(+), 29 deletions(-) >