From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-11.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA99CC12002 for ; Fri, 16 Jul 2021 20:04:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C1A9A61289 for ; Fri, 16 Jul 2021 20:04:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233227AbhGPUHd (ORCPT ); Fri, 16 Jul 2021 16:07:33 -0400 Received: from mx0a-00082601.pphosted.com ([67.231.145.42]:55298 "EHLO mx0a-00082601.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230256AbhGPUHc (ORCPT ); Fri, 16 Jul 2021 16:07:32 -0400 Received: from pps.filterd (m0109334.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16GK2kIc027743; Fri, 16 Jul 2021 13:04:03 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=date : from : to : cc : subject : message-id : references : content-type : in-reply-to : mime-version; s=facebook; bh=AXJ+ll7jaisStOs8vMtpqD9TN8bhj2wEJ1TvhKlbogw=; b=p5B2wkE8iIShIXZjj7bIdn3JRSgkFu5+FY7lGerXujAfVpzUGAbBXOefh9kFyaBp600V kKGI9qWAdQvH9Tykv9FAW3Skxld7osh28+/J2IOaVhk0j9Ufl16sxtvdDL5/Do4/Q/60 JAjZtREIrfpM8L9eO20O27u4I3ZgJn5XsV0= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 39tw37p8xb-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Fri, 16 Jul 2021 13:04:03 -0700 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (100.104.31.183) by o365-in.thefacebook.com (100.104.36.101) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.2; Fri, 16 Jul 2021 13:04:03 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=RP31xeEIzfXv0AotFk4dDce8IZOO4O02/rq+bUuLB68YxgXPU7KUEfw5bmg5Hwir1RQu0AcbNQEWdgfPyHStymKl7gd7ajqnabP7/naurhibpnJT+VaS+LAjO/b3QT/iRsdigdZIpwIxFganIQZoehg6oOjyxkXm4tvEEZp007OT+xfGoM10lYcW11q6TUOl+zmKN9lMDJkrX5Bi4i9hgh2zvm9DqZSuPgyVOmwsulNGSfh3o5FNIO7QO7viev6Q2EthhbARddnLWOyePW5s6/e67w4KF1Zx88x8EZaQvU3OsdzmRn8r2BJ514q381m+OeB/IfTPW9lZGnMQ4lYphA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=AXJ+ll7jaisStOs8vMtpqD9TN8bhj2wEJ1TvhKlbogw=; b=SByZ6JbsIBp3Dp2PEpNrttzw8E3NIcmDQwyfViYjxu8Tgs7ngP/oT0/M3wAk2Pm1Kdga8vQX6qUy+5nf1P2zclWAdomyn9GI0Vu6Cd/q5Xe8hjWI5g72pkNG0YbbMyIzDJkMnnYfSnfWiGoAbmSXIJoqneSmdEIQX+tB56zTkkxdtwqYgRtwAw1ePEA7ov4Hx0EerIJO+Jz1DLvWqS6sXxS2xW3ubW5GAwqxX9D78lobtOyEwrC58g6C5DovuCkZKF/x80sDtUl5ZIL7jM2v9a0t5sMqobqOyI+iR2AO6gZvu5q3SpzeYiVZ3fdD8quRM6g04xY1j4xmMKpvaSnLJA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=fb.com; dmarc=pass action=none header.from=fb.com; dkim=pass header.d=fb.com; arc=none Authentication-Results: kernel.org; dkim=none (message not signed) header.d=none;kernel.org; dmarc=none action=none header.from=fb.com; Received: from BYAPR15MB4136.namprd15.prod.outlook.com (2603:10b6:a03:96::24) by BY3PR15MB5012.namprd15.prod.outlook.com (2603:10b6:a03:3cb::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.21; Fri, 16 Jul 2021 20:04:01 +0000 Received: from BYAPR15MB4136.namprd15.prod.outlook.com ([fe80::9520:2bcd:e6fd:1dc7]) by BYAPR15MB4136.namprd15.prod.outlook.com ([fe80::9520:2bcd:e6fd:1dc7%6]) with mapi id 15.20.4308.027; Fri, 16 Jul 2021 20:04:01 +0000 Date: Fri, 16 Jul 2021 13:03:59 -0700 From: Roman Gushchin To: "Darrick J. Wong" CC: Boyang Xue , Matthew Wilcox , Jan Kara , Subject: Re: Patch 'writeback, cgroup: release dying cgwbs by switching attached inodes' leads to kernel crash Message-ID: References: <20210714092639.GB9457@quack2.suse.cz> <20210715171050.GB22357@magnolia> <20210715222812.GW22357@magnolia> <20210716162340.GY22357@magnolia> Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: <20210716162340.GY22357@magnolia> X-ClientProxiedBy: BY5PR13CA0031.namprd13.prod.outlook.com (2603:10b6:a03:180::44) To BYAPR15MB4136.namprd15.prod.outlook.com (2603:10b6:a03:96::24) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from carbon.dhcp.thefacebook.com (2620:10d:c090:400::5:2929) by BY5PR13CA0031.namprd13.prod.outlook.com (2603:10b6:a03:180::44) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4352.11 via Frontend Transport; Fri, 16 Jul 2021 20:04:01 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: c67650a1-ab8a-4f58-3c6d-08d94894dad6 X-MS-TrafficTypeDiagnostic: BY3PR15MB5012: X-Microsoft-Antispam-PRVS: X-FB-Source: Internal X-MS-Oob-TLC-OOBClassifiers: OLM:7691; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 9vqzjFnXYhLs+LMWqXpNanB2iSEyI9AtY+DpsiDfnyhrNUBALT7/BtTCxSkI1kxjhqdKBicUAbxAP/GtPJn5b3UmlwnWs5KdlXKUEmaM9bAu2LwJobRhncw7yRuO5RvfLu9Tvym2CPyIm5DUS0vLkX+PVnqVH1Hz7LH4YMAEEahSC3M3TbMaN5wUdWxx6leXhDYu0/Xu7YIbEveAIyijEaQI2S15LNrcLUkR0XoXEaKxuRZ15zKqiQVhW4XHtPsFSujSVP+WGBvDFMlnpmmZT4+xG6yZqNu2kwfZVjBhntWCbrz0X94NrTC8BPCSqM1kPd0KY9KHlDd61oX9VSz+RlcvxiZ7o9TdwDtow8Rh1PD7KY0Y5YS4IXfuDcEtCRNGHhMYAhzLMKD0okz1G+2hNcyzCGWX7v2z+NpqxC4M6kEj9iesvKhIoJVTZHEwMxVqy5xPog/hH5gde0HFnEyoO8oA66ne48f6DRMd9QtoOTtU4GiDAlrBleIb8jxVkYGGjMUqvtUysGadOxI11/RUHsb0v1AQ/Rr3MGEUdCe2s8zWQvTVogDnbKXHUyVGNq117Hdvlr7W+4ibrfVDYeT4FacsqZchpLXGgs1oZ8UAOymGPetGzmi/7NU/roUEt6ZpXsqa/FDsmNSq4l6YYoyETw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR15MB4136.namprd15.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(39860400002)(396003)(366004)(346002)(136003)(376002)(186003)(9686003)(52116002)(7696005)(8936002)(38100700002)(5660300002)(66946007)(66476007)(53546011)(4326008)(2906002)(83380400001)(8676002)(86362001)(6506007)(478600001)(316002)(6916009)(54906003)(66556008)(55016002);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?VC5AmrpBJTc0ENzKzov1cIneqhjXtGnrNeTuZyKs469pQuA+VJQNDsTU+jRU?= =?us-ascii?Q?Xzd6duLCuZ5ydGoZnOYgKH6nnZYtht9xXImQCR/zgUuSz8Asqgc3rtUqY8UO?= =?us-ascii?Q?1fgqFBdF4wkUxMo/s31jrhIY/zarUYttFkXIpeywrTT/n8X7Z4/QOl0PJgho?= =?us-ascii?Q?kUuM3t/QVyKVOd2c3lsjCpeRO3U4j/zMKiBN6lKWJsYwEUIoYVbJge/Jl/Cr?= =?us-ascii?Q?/4PwqzdWmmXnDsJiQHCKDx5hErBI82R6dl6nQ74zlF5egpLsKlv0ta0Dm1Dn?= =?us-ascii?Q?mYCgceHnwXPpgC/d2gVZQdxHY4o7rYmJa6Sb1QPpxQqxG3jzn5F1jSm9roQg?= =?us-ascii?Q?hicfmpt7zMkVenSAqgO55JaqyzL75IwheEYWA4E79h5nwkllDUvB/s3hbMyi?= =?us-ascii?Q?ntHfjdYroB08ml7BHwwJgPxjRazSmbDA4PwrwHriS0zHU5Ky+UX9wZOhlF/f?= =?us-ascii?Q?v03YYurVeHYQ6gVLiWeST4Fepz/KzDrhkO6Qwe/W/5EkiSaqgVeUdKb5oMw8?= =?us-ascii?Q?Z5DmEnCQlMxx5Rx4XDkdgJxSM0mj7VNKSup+CYGwD8vs0BGyqkTnV+Sdu3jC?= =?us-ascii?Q?mNlIak+eKeD2YIJ8FRthROf7II+NBv2/f0JxRwrp3wgOjsMFYdMniLExEU+m?= =?us-ascii?Q?wVGujJGZ3lraL/5TlirVFhPL819wyhC/SP2Rab77LQn0j0rqx8WsVaC7p3BK?= =?us-ascii?Q?u809guPNB/Y9Eiaoi+d7G7Z1blsxZrFKIOGh0FV+STNYuFsSjzvvkEcaHY/C?= =?us-ascii?Q?MmPE/mp0tSzEM/uhN078XgnmL2BpFxuc1mzpkgGCeNa7FeDpLpFH4HfJO8f+?= =?us-ascii?Q?1E1pj8q2l9dze3vTljx7hFZjT+QMslWvdhpf+QwEyWEfrG01vuLpatf3JBW7?= =?us-ascii?Q?VXFJ57ym+04NxttCnEazBoHZXV7j5QrdZ7P1IEHPprrRTL4oVasnWJN8OFwC?= =?us-ascii?Q?Y5vZySDJV5l9JNUNvvvmej+BjPqF37m54lKGZ2omtHRVzxZPkorOJuAb2ET0?= =?us-ascii?Q?EMN/FUQ4pKn+//yHD1ndugJIHCo7+JxxlMK8K+2EOHWbr4f4LjLtaDVyHE4z?= =?us-ascii?Q?j1t0mgb8R059bBgwlYihXt+RaU6HE2uIEM8qlpagA3Cpq963uydphI6VvOqe?= =?us-ascii?Q?aOFKmEsKtUB5t2YYAlRoTQdd/q6Ye/qxxk3RfzAseL3AOnPzk5T2Ga2X2Ylb?= =?us-ascii?Q?5icCQAG3OlGfjCEssrrJu/sm8ocX0iDuly1ILgeMcBhZAHAlpKIYMFkTRYCm?= =?us-ascii?Q?4mdtwU4QaGUizNRnK14xcK/ggA71WEDSEh1CgoJsz939pmP2PKDkq8wrcmKD?= =?us-ascii?Q?cZduAvHqn7IkDHrjetgn8Xz1yXBW5WZZkSOD3TniP6sG3A=3D=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: c67650a1-ab8a-4f58-3c6d-08d94894dad6 X-MS-Exchange-CrossTenant-AuthSource: BYAPR15MB4136.namprd15.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Jul 2021 20:04:01.4748 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: ihGHeAvpzarD+oe144KYCXKr0ztFRVtIbDCYYuk6/6IOORizF/up+0nhvqyhnBVS X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY3PR15MB5012 X-OriginatorOrg: fb.com X-Proofpoint-GUID: BNYZyjnWskUwWPM142qR2tu4My5l3JFt X-Proofpoint-ORIG-GUID: BNYZyjnWskUwWPM142qR2tu4My5l3JFt X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391,18.0.790 definitions=2021-07-16_09:2021-07-16,2021-07-16 signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 bulkscore=0 impostorscore=0 priorityscore=1501 lowpriorityscore=0 clxscore=1015 mlxscore=0 phishscore=0 malwarescore=0 spamscore=0 mlxlogscore=901 adultscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107160126 X-FB-Internal: deliver Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org On Fri, Jul 16, 2021 at 09:23:40AM -0700, Darrick J. Wong wrote: > On Thu, Jul 15, 2021 at 03:28:12PM -0700, Darrick J. Wong wrote: > > On Thu, Jul 15, 2021 at 01:08:15PM -0700, Roman Gushchin wrote: > > > On Thu, Jul 15, 2021 at 10:10:50AM -0700, Darrick J. Wong wrote: > > > > On Thu, Jul 15, 2021 at 11:51:50AM +0800, Boyang Xue wrote: > > > > > On Thu, Jul 15, 2021 at 10:36 AM Matthew Wilcox wrote: > > > > > > > > > > > > On Thu, Jul 15, 2021 at 12:22:28AM +0800, Boyang Xue wrote: > > > > > > > It's unclear to me that where to find the required address in the > > > > > > > addr2line command line, i.e. > > > > > > > > > > > > > > addr2line -e /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux > > > > > > > > > > > > > > > > > > > ./scripts/faddr2line /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux cleanup_offline_cgwbs_workfn+0x320/0x394 > > > > > > > > > > > > > > > > Thanks! The result is the same as the > > > > > > > > > > addr2line -i -e > > > > > /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux > > > > > FFFF8000102D6DD0 > > > > > > > > > > But this script is very handy. > > > > > > > > > > # /usr/src/kernels/5.14.0-0.rc1.15.bx.el9.aarch64/scripts/faddr2line > > > > > /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux > > > > > cleanup_offlin > > > > > e_cgwbs_workfn+0x320/0x394 > > > > > cleanup_offline_cgwbs_workfn+0x320/0x394: > > > > > arch_atomic64_fetch_add_unless at > > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/atomic-arch-fallback.h:2265 > > > > > (inlined by) arch_atomic64_add_unless at > > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/atomic-arch-fallback.h:2290 > > > > > (inlined by) atomic64_add_unless at > > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/asm-generic/atomic-instrumented.h:1149 > > > > > (inlined by) atomic_long_add_unless at > > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/asm-generic/atomic-long.h:491 > > > > > (inlined by) percpu_ref_tryget_many at > > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/percpu-refcount.h:247 > > > > > (inlined by) percpu_ref_tryget at > > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/percpu-refcount.h:266 > > > > > (inlined by) wb_tryget at > > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/backing-dev-defs.h:227 > > > > > (inlined by) wb_tryget at > > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/backing-dev-defs.h:224 > > > > > (inlined by) cleanup_offline_cgwbs_workfn at > > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/mm/backing-dev.c:679 > > > > > > > > > > # vi /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/mm/backing-dev.c > > > > > ``` > > > > > static void cleanup_offline_cgwbs_workfn(struct work_struct *work) > > > > > { > > > > > struct bdi_writeback *wb; > > > > > LIST_HEAD(processed); > > > > > > > > > > spin_lock_irq(&cgwb_lock); > > > > > > > > > > while (!list_empty(&offline_cgwbs)) { > > > > > wb = list_first_entry(&offline_cgwbs, struct bdi_writeback, > > > > > offline_node); > > > > > list_move(&wb->offline_node, &processed); > > > > > > > > > > /* > > > > > * If wb is dirty, cleaning up the writeback by switching > > > > > * attached inodes will result in an effective removal of any > > > > > * bandwidth restrictions, which isn't the goal. Instead, > > > > > * it can be postponed until the next time, when all io > > > > > * will be likely completed. If in the meantime some inodes > > > > > * will get re-dirtied, they should be eventually switched to > > > > > * a new cgwb. > > > > > */ > > > > > if (wb_has_dirty_io(wb)) > > > > > continue; > > > > > > > > > > if (!wb_tryget(wb)) <=== line#679 > > > > > continue; > > > > > > > > > > spin_unlock_irq(&cgwb_lock); > > > > > while (cleanup_offline_cgwb(wb)) > > > > > cond_resched(); > > > > > spin_lock_irq(&cgwb_lock); > > > > > > > > > > wb_put(wb); > > > > > } > > > > > > > > > > if (!list_empty(&processed)) > > > > > list_splice_tail(&processed, &offline_cgwbs); > > > > > > > > > > spin_unlock_irq(&cgwb_lock); > > > > > } > > > > > ``` > > > > > > > > > > BTW, this bug can be only reproduced on a non-debug production built > > > > > kernel (a.k.a kernel rpm package), it's not reproducible on a debug > > > > > build with various debug configuration enabled (a.k.a kernel-debug rpm > > > > > package) > > > > > > > > FWIW I've also seen this regularly on x86_64 kernels on ext4 with all > > > > default mkfs settings when running generic/256. > > > > > > Oh, that's a useful information, thank you! > > > > > > Btw, would you mind to give a patch from an earlier message in the thread > > > a test? I'd highly appreciate it. > > > > > > Thanks! > > > > Will do. > > fstests passed here, so > > Tested-by: Darrick J. Wong Great, thank you!