From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E61D7C47089 for ; Mon, 5 Dec 2022 09:35:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231190AbiLEJfd (ORCPT ); Mon, 5 Dec 2022 04:35:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51296 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230092AbiLEJfN (ORCPT ); Mon, 5 Dec 2022 04:35:13 -0500 Received: from dggsgout12.his.huawei.com (dggsgout12.his.huawei.com [45.249.212.56]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A161D18B07; Mon, 5 Dec 2022 01:35:11 -0800 (PST) Received: from mail02.huawei.com (unknown [172.30.67.153]) by dggsgout12.his.huawei.com (SkyGuard) with ESMTP id 4NQdgZ1Q0tz4f3kpH; Mon, 5 Dec 2022 17:35:06 +0800 (CST) Received: from [10.174.176.73] (unknown [10.174.176.73]) by APP1 (Coremail) with SMTP id cCh0CgCHL65Lu41jRqNiBg--.49659S3; Mon, 05 Dec 2022 17:35:08 +0800 (CST) Subject: Re: [PATCH -next v2 9/9] blk-iocost: fix walk_list corruption To: Tejun Heo , Li Nan Cc: josef@toxicpanda.com, axboe@kernel.dk, cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, yi.zhang@huawei.com, "yukuai (C)" References: <20221130132156.2836184-1-linan122@huawei.com> <20221130132156.2836184-10-linan122@huawei.com> From: Yu Kuai Message-ID: <2f9495ff-130a-ce1d-5887-0448bdbceedd@huaweicloud.com> Date: Mon, 5 Dec 2022 17:35:07 +0800 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=gbk; format=flowed Content-Transfer-Encoding: 8bit X-CM-TRANSID: cCh0CgCHL65Lu41jRqNiBg--.49659S3 X-Coremail-Antispam: 1UD129KBjvJXoW7tFW7Cr4DGF1kCw4UGF18AFb_yoW8AFyrpF WfKana9rW8tw1I9F10q3Z0q3WSyF40vry7JrWfW340y3W2yw17Jr1qyF48WF98WFW8A3yU Xa1UK3WkXw4DAaDanT9S1TB71UUUUUUqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUyEb4IE77IF4wAFF20E14v26r4j6ryUM7CY07I20VC2zVCF04k2 6cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28lY4IEw2IIxxk0rwA2F7IY1VAKz4 vEj48ve4kI8wA2z4x0Y4vE2Ix0cI8IcVAFwI0_tr0E3s1l84ACjcxK6xIIjxv20xvEc7Cj xVAFwI0_Gr1j6F4UJwA2z4x0Y4vEx4A2jsIE14v26rxl6s0DM28EF7xvwVC2z280aVCY1x 0267AKxVW0oVCq3wAS0I0E0xvYzxvE52x082IY62kv0487Mc02F40EFcxC0VAKzVAqx4xG 6I80ewAv7VC0I7IYx2IY67AKxVWUGVWUXwAv7VC2z280aVAFwI0_Jr0_Gr1lOx8S6xCaFV Cjc4AY6r1j6r4UM4x0Y48IcVAKI48JMxk0xIA0c2IEe2xFo4CEbIxvr21l42xK82IYc2Ij 64vIr41l4I8I3I0E4IkC6x0Yz7v_Jr0_Gr1lx2IqxVAqx4xG67AKxVWUJVWUGwC20s026x 8GjcxK67AKxVWUGVWUWwC2zVAF1VAY17CE14v26r1q6r43MIIYrxkI7VAKI48JMIIF0xvE 2Ix0cI8IcVAFwI0_Jr0_JF4lIxAIcVC0I7IYx2IY6xkF7I0E14v26r4j6F4UMIIF0xvE42 xK8VAvwI8IcIk0rVW3JVWrJr1lIxAIcVC2z280aVAFwI0_Jr0_Gr1lIxAIcVC2z280aVCY 1x0267AKxVW8JVW8JrUvcSsGvfC2KfnxnUUI43ZEXa7IUbPEf5UUUUU== X-CM-SenderInfo: 51xn3trlr6x35dzhxuhorxvhhfrp/ X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Hi, Tejun ÔÚ 2022/12/01 4:59, Tejun Heo дµÀ: > On Wed, Nov 30, 2022 at 09:21:56PM +0800, Li Nan wrote: >> From: Yu Kuai >> >> Our test report a problem: >> >> ------------[ cut here ]------------ >> list_del corruption. next->prev should be ffff888127e0c4b0, but was ffff888127e090b0 >> WARNING: CPU: 2 PID: 3117789 at lib/list_debug.c:62 __list_del_entry_valid+0x119/0x130 >> RIP: 0010:__list_del_entry_valid+0x119/0x130 >> RIP: 0010:__list_del_entry_valid+0x119/0x130 >> Call Trace: >> >> iocg_flush_stat.isra.0+0x11e/0x230 >> ? ioc_rqos_done+0x230/0x230 >> ? ioc_now+0x14f/0x180 >> ioc_timer_fn+0x569/0x1640 >> >> We haven't reporduced it yet, but we think this is due to parent iocg is >> freed before child iocg, and then in ioc_timer_fn, walk_list is >> corrupted. >> >> 1) Remove child cgroup can concurrent with remove parent cgroup, and >> ioc_pd_free for parent iocg can be called before child iocg. This can be >> fixed by moving the handle of walk_list to ioc_pd_offline, since that >> offline from child is ensured to be called before parent. > > Which you already did in a previous patch, right? > >> 2) ioc_pd_free can be triggered from both removing device and removing >> cgroup, this patch fix the problem by deleting timer before deactivating >> policy, so that free parent iocg first in this case won't matter. > > Okay, so, yeah, css's pin parents but blkg's don't. I think the right thing > to do here is making sure that a child blkg pins its parent (and eventually > ioc). Sorry about this, actually it's can be ensured that pd_offline from child will be called before parent. Hence just moving he handle of walk_list to ioc_pd_offline can fix this problem thoroughly. Thanks, Kuai From mboxrd@z Thu Jan 1 00:00:00 1970 From: Yu Kuai Subject: Re: [PATCH -next v2 9/9] blk-iocost: fix walk_list corruption Date: Mon, 5 Dec 2022 17:35:07 +0800 Message-ID: <2f9495ff-130a-ce1d-5887-0448bdbceedd@huaweicloud.com> References: <20221130132156.2836184-1-linan122@huawei.com> <20221130132156.2836184-10-linan122@huawei.com> Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: List-ID: Content-Type: text/plain; charset="iso-8859-1"; format="flowed" To: Tejun Heo , Li Nan Cc: josef-DigfWCa+lFGyeJad7bwFQA@public.gmane.org, axboe-tSWWG44O7X1aa/9Udqfwiw@public.gmane.org, cgroups-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-block-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, yi.zhang-hv44wF8Li93QT0dZR+AlfA@public.gmane.org, "yukuai (C)" Hi, Tejun =D4=DA 2022/12/01 4:59, Tejun Heo =D0=B4=B5=C0: > On Wed, Nov 30, 2022 at 09:21:56PM +0800, Li Nan wrote: >> From: Yu Kuai >> >> Our test report a problem: >> >> ------------[ cut here ]------------ >> list_del corruption. next->prev should be ffff888127e0c4b0, but was ffff= 888127e090b0 >> WARNING: CPU: 2 PID: 3117789 at lib/list_debug.c:62 __list_del_entry_val= id+0x119/0x130 >> RIP: 0010:__list_del_entry_valid+0x119/0x130 >> RIP: 0010:__list_del_entry_valid+0x119/0x130 >> Call Trace: >> >> iocg_flush_stat.isra.0+0x11e/0x230 >> ? ioc_rqos_done+0x230/0x230 >> ? ioc_now+0x14f/0x180 >> ioc_timer_fn+0x569/0x1640 >> >> We haven't reporduced it yet, but we think this is due to parent iocg is >> freed before child iocg, and then in ioc_timer_fn, walk_list is >> corrupted. >> >> 1) Remove child cgroup can concurrent with remove parent cgroup, and >> ioc_pd_free for parent iocg can be called before child iocg. This can be >> fixed by moving the handle of walk_list to ioc_pd_offline, since that >> offline from child is ensured to be called before parent. >=20 > Which you already did in a previous patch, right? >=20 >> 2) ioc_pd_free can be triggered from both removing device and removing >> cgroup, this patch fix the problem by deleting timer before deactivating >> policy, so that free parent iocg first in this case won't matter. >=20 > Okay, so, yeah, css's pin parents but blkg's don't. I think the right thi= ng > to do here is making sure that a child blkg pins its parent (and eventual= ly > ioc). Sorry about this, actually it's can be ensured that pd_offline from child will be called before parent. Hence just moving he handle of walk_list to ioc_pd_offline can fix this problem thoroughly. Thanks, Kuai