From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 24160C433F5 for ; Mon, 27 Dec 2021 03:12:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235035AbhL0DMH (ORCPT ); Sun, 26 Dec 2021 22:12:07 -0500 Received: from szxga02-in.huawei.com ([45.249.212.188]:29291 "EHLO szxga02-in.huawei.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234875AbhL0DMF (ORCPT ); Sun, 26 Dec 2021 22:12:05 -0500 Received: from kwepemi500002.china.huawei.com (unknown [172.30.72.54]) by szxga02-in.huawei.com (SkyGuard) with ESMTP id 4JMjPN05b5zbjfd; Mon, 27 Dec 2021 11:11:35 +0800 (CST) Received: from kwepemm600013.china.huawei.com (7.193.23.68) by kwepemi500002.china.huawei.com (7.221.188.171) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.20; Mon, 27 Dec 2021 11:12:02 +0800 Received: from huawei.com (10.175.127.227) by kwepemm600013.china.huawei.com (7.193.23.68) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.20; Mon, 27 Dec 2021 11:12:02 +0800 From: Zhihao Cheng To: , , , , , CC: , Subject: [PATCH v6 02/15] ubifs: Fix deadlock in concurrent rename whiteout and inode writeback Date: Mon, 27 Dec 2021 11:22:33 +0800 Message-ID: <20211227032246.2886878-3-chengzhihao1@huawei.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211227032246.2886878-1-chengzhihao1@huawei.com> References: <20211227032246.2886878-1-chengzhihao1@huawei.com> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit X-Originating-IP: [10.175.127.227] X-ClientProxiedBy: dggems706-chm.china.huawei.com (10.3.19.183) To kwepemm600013.china.huawei.com (7.193.23.68) X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Following hung tasks: [ 77.028764] task:kworker/u8:4 state:D stack: 0 pid: 132 [ 77.028820] Call Trace: [ 77.029027] schedule+0x8c/0x1b0 [ 77.029067] mutex_lock+0x50/0x60 [ 77.029074] ubifs_write_inode+0x68/0x1f0 [ubifs] [ 77.029117] __writeback_single_inode+0x43c/0x570 [ 77.029128] writeback_sb_inodes+0x259/0x740 [ 77.029148] wb_writeback+0x107/0x4d0 [ 77.029163] wb_workfn+0x162/0x7b0 [ 92.390442] task:aa state:D stack: 0 pid: 1506 [ 92.390448] Call Trace: [ 92.390458] schedule+0x8c/0x1b0 [ 92.390461] wb_wait_for_completion+0x82/0xd0 [ 92.390469] __writeback_inodes_sb_nr+0xb2/0x110 [ 92.390472] writeback_inodes_sb_nr+0x14/0x20 [ 92.390476] ubifs_budget_space+0x705/0xdd0 [ubifs] [ 92.390503] do_rename.cold+0x7f/0x187 [ubifs] [ 92.390549] ubifs_rename+0x8b/0x180 [ubifs] [ 92.390571] vfs_rename+0xdb2/0x1170 [ 92.390580] do_renameat2+0x554/0x770 , are caused by concurrent rename whiteout and inode writeback processes: rename_whiteout(Thread 1) wb_workfn(Thread2) ubifs_rename do_rename lock_4_inodes (Hold ui_mutex) ubifs_budget_space make_free_space shrink_liability __writeback_inodes_sb_nr bdi_split_work_to_wbs (Queue new wb work) wb_do_writeback(wb work) __writeback_single_inode ubifs_write_inode LOCK(ui_mutex) ↑ wb_wait_for_completion (Wait wb work) <-- deadlock! Reproducer (Detail program in [Link]): 1. SYS_renameat2("/mp/dir/file", "/mp/dir/whiteout", RENAME_WHITEOUT) 2. Consume out of space before kernel(mdelay) doing budget for whiteout Fix it by doing whiteout space budget before locking ubifs inodes. BTW, it also fixes wrong goto tag 'out_release' in whiteout budget error handling path(It should at least recover dir i_size and unlock 4 ubifs inodes). Fixes: 9e0a1fff8db56ea ("ubifs: Implement RENAME_WHITEOUT") Link: https://bugzilla.kernel.org/show_bug.cgi?id=214733 Signed-off-by: Zhihao Cheng --- fs/ubifs/dir.c | 25 +++++++++++++++---------- 1 file changed, 15 insertions(+), 10 deletions(-) diff --git a/fs/ubifs/dir.c b/fs/ubifs/dir.c index cfa8881d8cca..2735ad1affed 100644 --- a/fs/ubifs/dir.c +++ b/fs/ubifs/dir.c @@ -1324,6 +1324,7 @@ static int do_rename(struct inode *old_dir, struct dentry *old_dentry, if (flags & RENAME_WHITEOUT) { union ubifs_dev_desc *dev = NULL; + struct ubifs_budget_req wht_req; dev = kmalloc(sizeof(union ubifs_dev_desc), GFP_NOFS); if (!dev) { @@ -1345,6 +1346,20 @@ static int do_rename(struct inode *old_dir, struct dentry *old_dentry, whiteout_ui->data = dev; whiteout_ui->data_len = ubifs_encode_dev(dev, MKDEV(0, 0)); ubifs_assert(c, !whiteout_ui->dirty); + + memset(&wht_req, 0, sizeof(struct ubifs_budget_req)); + wht_req.dirtied_ino = 1; + wht_req.dirtied_ino_d = ALIGN(whiteout_ui->data_len, 8); + /* + * To avoid deadlock between space budget (holds ui_mutex and + * waits wb work) and writeback work(waits ui_mutex), do space + * budget before ubifs inodes locked. + */ + err = ubifs_budget_space(c, &wht_req); + if (err) { + iput(whiteout); + goto out_release; + } } lock_4_inodes(old_dir, new_dir, new_inode, whiteout); @@ -1419,16 +1434,6 @@ static int do_rename(struct inode *old_dir, struct dentry *old_dentry, } if (whiteout) { - struct ubifs_budget_req wht_req = { .dirtied_ino = 1, - .dirtied_ino_d = \ - ALIGN(ubifs_inode(whiteout)->data_len, 8) }; - - err = ubifs_budget_space(c, &wht_req); - if (err) { - iput(whiteout); - goto out_release; - } - inc_nlink(whiteout); mark_inode_dirty(whiteout); -- 2.31.1 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 004C3C433EF for ; Mon, 27 Dec 2021 03:20:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:CC:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=4VN0R1Xj/69gyoFJkO7Nz6Ru/oEYaPvPtEzDxOjbwgc=; b=QaKG5ZBI0hYXAB zC8vSB+VxZdC5JEmcrvyqXRUMqsRUPHY/JEIxx88XgPx8HbTw8xw1I8en/yYsu6sMCW0zts3fNbpF EgQFXKHuyy4ULDv0v03C9AdGWlTc7dMVPEolN72j4RGlNGR2e/ZA5F3RZ6vCcdoOMDm2+uYsrzcFb cuIkTA+EplRCvS7PzPTHcA1N4dTnkH342HcXPujfr+GNEilWBkNHksDxE3Wa0PBd59SXbhOsFbFSg lED+y9dOJO31x7WBRGC51btrPwmQdnpoUYyRAySk75Et00iHBDEi9oBI9VC4oE94ZG5HbXMgQBDBt ltNYF261P49bZQys4cWA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1n1gXv-00G5mc-Jh; Mon, 27 Dec 2021 03:19:35 +0000 Received: from szxga02-in.huawei.com ([45.249.212.188]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1n1gQm-00G3yd-KB for linux-mtd@lists.infradead.org; Mon, 27 Dec 2021 03:12:19 +0000 Received: from kwepemi500002.china.huawei.com (unknown [172.30.72.54]) by szxga02-in.huawei.com (SkyGuard) with ESMTP id 4JMjPN05b5zbjfd; Mon, 27 Dec 2021 11:11:35 +0800 (CST) Received: from kwepemm600013.china.huawei.com (7.193.23.68) by kwepemi500002.china.huawei.com (7.221.188.171) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.20; Mon, 27 Dec 2021 11:12:02 +0800 Received: from huawei.com (10.175.127.227) by kwepemm600013.china.huawei.com (7.193.23.68) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.20; Mon, 27 Dec 2021 11:12:02 +0800 From: Zhihao Cheng To: , , , , , CC: , Subject: [PATCH v6 02/15] ubifs: Fix deadlock in concurrent rename whiteout and inode writeback Date: Mon, 27 Dec 2021 11:22:33 +0800 Message-ID: <20211227032246.2886878-3-chengzhihao1@huawei.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211227032246.2886878-1-chengzhihao1@huawei.com> References: <20211227032246.2886878-1-chengzhihao1@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.127.227] X-ClientProxiedBy: dggems706-chm.china.huawei.com (10.3.19.183) To kwepemm600013.china.huawei.com (7.193.23.68) X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211226_191213_013400_F27C2226 X-CRM114-Status: UNSURE ( 9.55 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-mtd@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Linux MTD discussion mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-mtd" Errors-To: linux-mtd-bounces+linux-mtd=archiver.kernel.org@lists.infradead.org Rm9sbG93aW5nIGh1bmcgdGFza3M6ClsgICA3Ny4wMjg3NjRdIHRhc2s6a3dvcmtlci91ODo0ICAg IHN0YXRlOkQgc3RhY2s6ICAgIDAgcGlkOiAgMTMyClsgICA3Ny4wMjg4MjBdIENhbGwgVHJhY2U6 ClsgICA3Ny4wMjkwMjddICBzY2hlZHVsZSsweDhjLzB4MWIwClsgICA3Ny4wMjkwNjddICBtdXRl eF9sb2NrKzB4NTAvMHg2MApbICAgNzcuMDI5MDc0XSAgdWJpZnNfd3JpdGVfaW5vZGUrMHg2OC8w eDFmMCBbdWJpZnNdClsgICA3Ny4wMjkxMTddICBfX3dyaXRlYmFja19zaW5nbGVfaW5vZGUrMHg0 M2MvMHg1NzAKWyAgIDc3LjAyOTEyOF0gIHdyaXRlYmFja19zYl9pbm9kZXMrMHgyNTkvMHg3NDAK WyAgIDc3LjAyOTE0OF0gIHdiX3dyaXRlYmFjaysweDEwNy8weDRkMApbICAgNzcuMDI5MTYzXSAg d2Jfd29ya2ZuKzB4MTYyLzB4N2IwCgpbICAgOTIuMzkwNDQyXSB0YXNrOmFhICAgICAgICAgICAg ICBzdGF0ZTpEIHN0YWNrOiAgICAwIHBpZDogMTUwNgpbICAgOTIuMzkwNDQ4XSBDYWxsIFRyYWNl OgpbICAgOTIuMzkwNDU4XSAgc2NoZWR1bGUrMHg4Yy8weDFiMApbICAgOTIuMzkwNDYxXSAgd2Jf d2FpdF9mb3JfY29tcGxldGlvbisweDgyLzB4ZDAKWyAgIDkyLjM5MDQ2OV0gIF9fd3JpdGViYWNr X2lub2Rlc19zYl9ucisweGIyLzB4MTEwClsgICA5Mi4zOTA0NzJdICB3cml0ZWJhY2tfaW5vZGVz X3NiX25yKzB4MTQvMHgyMApbICAgOTIuMzkwNDc2XSAgdWJpZnNfYnVkZ2V0X3NwYWNlKzB4NzA1 LzB4ZGQwIFt1Ymlmc10KWyAgIDkyLjM5MDUwM10gIGRvX3JlbmFtZS5jb2xkKzB4N2YvMHgxODcg W3ViaWZzXQpbICAgOTIuMzkwNTQ5XSAgdWJpZnNfcmVuYW1lKzB4OGIvMHgxODAgW3ViaWZzXQpb ICAgOTIuMzkwNTcxXSAgdmZzX3JlbmFtZSsweGRiMi8weDExNzAKWyAgIDkyLjM5MDU4MF0gIGRv X3JlbmFtZWF0MisweDU1NC8weDc3MAoKLCBhcmUgY2F1c2VkIGJ5IGNvbmN1cnJlbnQgcmVuYW1l IHdoaXRlb3V0IGFuZCBpbm9kZSB3cml0ZWJhY2sgcHJvY2Vzc2VzOgoJcmVuYW1lX3doaXRlb3V0 KFRocmVhZCAxKQkgICAgICAgIHdiX3dvcmtmbihUaHJlYWQyKQp1Ymlmc19yZW5hbWUKICBkb19y ZW5hbWUKICAgIGxvY2tfNF9pbm9kZXMgKEhvbGQgdWlfbXV0ZXgpCiAgICB1Ymlmc19idWRnZXRf c3BhY2UKICAgICAgbWFrZV9mcmVlX3NwYWNlCiAgICAgICAgc2hyaW5rX2xpYWJpbGl0eQoJICBf X3dyaXRlYmFja19pbm9kZXNfc2JfbnIKCSAgICBiZGlfc3BsaXRfd29ya190b193YnMgKFF1ZXVl IG5ldyB3YiB3b3JrKQoJCQkJCSAgICAgIHdiX2RvX3dyaXRlYmFjayh3YiB3b3JrKQoJCQkJCQlf X3dyaXRlYmFja19zaW5nbGVfaW5vZGUKCQkJCQkgICAgICAgICAgdWJpZnNfd3JpdGVfaW5vZGUK CQkJCQkgICAgICAgICAgICBMT0NLKHVpX211dGV4KQoJCQkJCQkJICAg4oaRCgkgICAgICB3Yl93 YWl0X2Zvcl9jb21wbGV0aW9uIChXYWl0IHdiIHdvcmspIDwtLSBkZWFkbG9jayEKClJlcHJvZHVj ZXIgKERldGFpbCBwcm9ncmFtIGluIFtMaW5rXSk6CiAgMS4gU1lTX3JlbmFtZWF0MigiL21wL2Rp ci9maWxlIiwgIi9tcC9kaXIvd2hpdGVvdXQiLCBSRU5BTUVfV0hJVEVPVVQpCiAgMi4gQ29uc3Vt ZSBvdXQgb2Ygc3BhY2UgYmVmb3JlIGtlcm5lbChtZGVsYXkpIGRvaW5nIGJ1ZGdldCBmb3Igd2hp dGVvdXQKCkZpeCBpdCBieSBkb2luZyB3aGl0ZW91dCBzcGFjZSBidWRnZXQgYmVmb3JlIGxvY2tp bmcgdWJpZnMgaW5vZGVzLgpCVFcsIGl0IGFsc28gZml4ZXMgd3JvbmcgZ290byB0YWcgJ291dF9y ZWxlYXNlJyBpbiB3aGl0ZW91dCBidWRnZXQKZXJyb3IgaGFuZGxpbmcgcGF0aChJdCBzaG91bGQg YXQgbGVhc3QgcmVjb3ZlciBkaXIgaV9zaXplIGFuZCB1bmxvY2sKNCB1YmlmcyBpbm9kZXMpLgoK Rml4ZXM6IDllMGExZmZmOGRiNTZlYSAoInViaWZzOiBJbXBsZW1lbnQgUkVOQU1FX1dISVRFT1VU IikKTGluazogaHR0cHM6Ly9idWd6aWxsYS5rZXJuZWwub3JnL3Nob3dfYnVnLmNnaT9pZD0yMTQ3 MzMKU2lnbmVkLW9mZi1ieTogWmhpaGFvIENoZW5nIDxjaGVuZ3poaWhhbzFAaHVhd2VpLmNvbT4K LS0tCiBmcy91Ymlmcy9kaXIuYyB8IDI1ICsrKysrKysrKysrKysrKy0tLS0tLS0tLS0KIDEgZmls ZSBjaGFuZ2VkLCAxNSBpbnNlcnRpb25zKCspLCAxMCBkZWxldGlvbnMoLSkKCmRpZmYgLS1naXQg YS9mcy91Ymlmcy9kaXIuYyBiL2ZzL3ViaWZzL2Rpci5jCmluZGV4IGNmYTg4ODFkOGNjYS4uMjcz NWFkMWFmZmVkIDEwMDY0NAotLS0gYS9mcy91Ymlmcy9kaXIuYworKysgYi9mcy91Ymlmcy9kaXIu YwpAQCAtMTMyNCw2ICsxMzI0LDcgQEAgc3RhdGljIGludCBkb19yZW5hbWUoc3RydWN0IGlub2Rl ICpvbGRfZGlyLCBzdHJ1Y3QgZGVudHJ5ICpvbGRfZGVudHJ5LAogCiAJaWYgKGZsYWdzICYgUkVO QU1FX1dISVRFT1VUKSB7CiAJCXVuaW9uIHViaWZzX2Rldl9kZXNjICpkZXYgPSBOVUxMOworCQlz dHJ1Y3QgdWJpZnNfYnVkZ2V0X3JlcSB3aHRfcmVxOwogCiAJCWRldiA9IGttYWxsb2Moc2l6ZW9m KHVuaW9uIHViaWZzX2Rldl9kZXNjKSwgR0ZQX05PRlMpOwogCQlpZiAoIWRldikgewpAQCAtMTM0 NSw2ICsxMzQ2LDIwIEBAIHN0YXRpYyBpbnQgZG9fcmVuYW1lKHN0cnVjdCBpbm9kZSAqb2xkX2Rp ciwgc3RydWN0IGRlbnRyeSAqb2xkX2RlbnRyeSwKIAkJd2hpdGVvdXRfdWktPmRhdGEgPSBkZXY7 CiAJCXdoaXRlb3V0X3VpLT5kYXRhX2xlbiA9IHViaWZzX2VuY29kZV9kZXYoZGV2LCBNS0RFVigw LCAwKSk7CiAJCXViaWZzX2Fzc2VydChjLCAhd2hpdGVvdXRfdWktPmRpcnR5KTsKKworCQltZW1z ZXQoJndodF9yZXEsIDAsIHNpemVvZihzdHJ1Y3QgdWJpZnNfYnVkZ2V0X3JlcSkpOworCQl3aHRf cmVxLmRpcnRpZWRfaW5vID0gMTsKKwkJd2h0X3JlcS5kaXJ0aWVkX2lub19kID0gQUxJR04od2hp dGVvdXRfdWktPmRhdGFfbGVuLCA4KTsKKwkJLyoKKwkJICogVG8gYXZvaWQgZGVhZGxvY2sgYmV0 d2VlbiBzcGFjZSBidWRnZXQgKGhvbGRzIHVpX211dGV4IGFuZAorCQkgKiB3YWl0cyB3YiB3b3Jr KSBhbmQgd3JpdGViYWNrIHdvcmsod2FpdHMgdWlfbXV0ZXgpLCBkbyBzcGFjZQorCQkgKiBidWRn ZXQgYmVmb3JlIHViaWZzIGlub2RlcyBsb2NrZWQuCisJCSAqLworCQllcnIgPSB1Ymlmc19idWRn ZXRfc3BhY2UoYywgJndodF9yZXEpOworCQlpZiAoZXJyKSB7CisJCQlpcHV0KHdoaXRlb3V0KTsK KwkJCWdvdG8gb3V0X3JlbGVhc2U7CisJCX0KIAl9CiAKIAlsb2NrXzRfaW5vZGVzKG9sZF9kaXIs IG5ld19kaXIsIG5ld19pbm9kZSwgd2hpdGVvdXQpOwpAQCAtMTQxOSwxNiArMTQzNCw2IEBAIHN0 YXRpYyBpbnQgZG9fcmVuYW1lKHN0cnVjdCBpbm9kZSAqb2xkX2Rpciwgc3RydWN0IGRlbnRyeSAq b2xkX2RlbnRyeSwKIAl9CiAKIAlpZiAod2hpdGVvdXQpIHsKLQkJc3RydWN0IHViaWZzX2J1ZGdl dF9yZXEgd2h0X3JlcSA9IHsgLmRpcnRpZWRfaW5vID0gMSwKLQkJCQkuZGlydGllZF9pbm9fZCA9 IFwKLQkJCQlBTElHTih1Ymlmc19pbm9kZSh3aGl0ZW91dCktPmRhdGFfbGVuLCA4KSB9OwotCi0J CWVyciA9IHViaWZzX2J1ZGdldF9zcGFjZShjLCAmd2h0X3JlcSk7Ci0JCWlmIChlcnIpIHsKLQkJ CWlwdXQod2hpdGVvdXQpOwotCQkJZ290byBvdXRfcmVsZWFzZTsKLQkJfQotCiAJCWluY19ubGlu ayh3aGl0ZW91dCk7CiAJCW1hcmtfaW5vZGVfZGlydHkod2hpdGVvdXQpOwogCi0tIAoyLjMxLjEK CgpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18K TGludXggTVREIGRpc2N1c3Npb24gbWFpbGluZyBsaXN0Cmh0dHA6Ly9saXN0cy5pbmZyYWRlYWQu b3JnL21haWxtYW4vbGlzdGluZm8vbGludXgtbXRkLwo=