From: Zhihao Cheng <chengzhihao1@huawei.com> To: <richard@nod.at>, <miquel.raynal@bootlin.com>, <vigneshr@ti.com>, <mcoquelin.stm32@gmail.com>, <kirill.shutemov@linux.intel.com> Cc: <linux-mtd@lists.infradead.org>, <linux-kernel@vger.kernel.org>, <chengzhihao1@huawei.com> Subject: [PATCH v5 02/14] ubifs: Fix deadlock in concurrent rename whiteout and inode writeback Date: Mon, 20 Dec 2021 21:59:39 +0800 [thread overview] Message-ID: <20211220135951.4075801-3-chengzhihao1@huawei.com> (raw) In-Reply-To: <20211220135951.4075801-1-chengzhihao1@huawei.com> Following hung tasks: [ 77.028764] task:kworker/u8:4 state:D stack: 0 pid: 132 [ 77.028820] Call Trace: [ 77.029027] schedule+0x8c/0x1b0 [ 77.029067] mutex_lock+0x50/0x60 [ 77.029074] ubifs_write_inode+0x68/0x1f0 [ubifs] [ 77.029117] __writeback_single_inode+0x43c/0x570 [ 77.029128] writeback_sb_inodes+0x259/0x740 [ 77.029148] wb_writeback+0x107/0x4d0 [ 77.029163] wb_workfn+0x162/0x7b0 [ 92.390442] task:aa state:D stack: 0 pid: 1506 [ 92.390448] Call Trace: [ 92.390458] schedule+0x8c/0x1b0 [ 92.390461] wb_wait_for_completion+0x82/0xd0 [ 92.390469] __writeback_inodes_sb_nr+0xb2/0x110 [ 92.390472] writeback_inodes_sb_nr+0x14/0x20 [ 92.390476] ubifs_budget_space+0x705/0xdd0 [ubifs] [ 92.390503] do_rename.cold+0x7f/0x187 [ubifs] [ 92.390549] ubifs_rename+0x8b/0x180 [ubifs] [ 92.390571] vfs_rename+0xdb2/0x1170 [ 92.390580] do_renameat2+0x554/0x770 , are caused by concurrent rename whiteout and inode writeback processes: rename_whiteout(Thread 1) wb_workfn(Thread2) ubifs_rename do_rename lock_4_inodes (Hold ui_mutex) ubifs_budget_space make_free_space shrink_liability __writeback_inodes_sb_nr bdi_split_work_to_wbs (Queue new wb work) wb_do_writeback(wb work) __writeback_single_inode ubifs_write_inode LOCK(ui_mutex) ↑ wb_wait_for_completion (Wait wb work) <-- deadlock! Reproducer (Detail program in [Link]): 1. SYS_renameat2("/mp/dir/file", "/mp/dir/whiteout", RENAME_WHITEOUT) 2. Consume out of space before kernel(mdelay) doing budget for whiteout Fix it by doing whiteout space budget before locking ubifs inodes. BTW, it also fixes wrong goto tag 'out_release' in whiteout budget error handling path(It should at least recover dir i_size and unlock 4 ubifs inodes). Fixes: 9e0a1fff8db56ea ("ubifs: Implement RENAME_WHITEOUT") Link: https://bugzilla.kernel.org/show_bug.cgi?id=214733 Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> --- fs/ubifs/dir.c | 25 +++++++++++++++---------- 1 file changed, 15 insertions(+), 10 deletions(-) diff --git a/fs/ubifs/dir.c b/fs/ubifs/dir.c index cfa8881d8cca..2735ad1affed 100644 --- a/fs/ubifs/dir.c +++ b/fs/ubifs/dir.c @@ -1324,6 +1324,7 @@ static int do_rename(struct inode *old_dir, struct dentry *old_dentry, if (flags & RENAME_WHITEOUT) { union ubifs_dev_desc *dev = NULL; + struct ubifs_budget_req wht_req; dev = kmalloc(sizeof(union ubifs_dev_desc), GFP_NOFS); if (!dev) { @@ -1345,6 +1346,20 @@ static int do_rename(struct inode *old_dir, struct dentry *old_dentry, whiteout_ui->data = dev; whiteout_ui->data_len = ubifs_encode_dev(dev, MKDEV(0, 0)); ubifs_assert(c, !whiteout_ui->dirty); + + memset(&wht_req, 0, sizeof(struct ubifs_budget_req)); + wht_req.dirtied_ino = 1; + wht_req.dirtied_ino_d = ALIGN(whiteout_ui->data_len, 8); + /* + * To avoid deadlock between space budget (holds ui_mutex and + * waits wb work) and writeback work(waits ui_mutex), do space + * budget before ubifs inodes locked. + */ + err = ubifs_budget_space(c, &wht_req); + if (err) { + iput(whiteout); + goto out_release; + } } lock_4_inodes(old_dir, new_dir, new_inode, whiteout); @@ -1419,16 +1434,6 @@ static int do_rename(struct inode *old_dir, struct dentry *old_dentry, } if (whiteout) { - struct ubifs_budget_req wht_req = { .dirtied_ino = 1, - .dirtied_ino_d = \ - ALIGN(ubifs_inode(whiteout)->data_len, 8) }; - - err = ubifs_budget_space(c, &wht_req); - if (err) { - iput(whiteout); - goto out_release; - } - inc_nlink(whiteout); mark_inode_dirty(whiteout); -- 2.31.1
WARNING: multiple messages have this Message-ID (diff)
From: Zhihao Cheng <chengzhihao1@huawei.com> To: <richard@nod.at>, <miquel.raynal@bootlin.com>, <vigneshr@ti.com>, <mcoquelin.stm32@gmail.com>, <kirill.shutemov@linux.intel.com> Cc: <linux-mtd@lists.infradead.org>, <linux-kernel@vger.kernel.org>, <chengzhihao1@huawei.com> Subject: [PATCH v5 02/14] ubifs: Fix deadlock in concurrent rename whiteout and inode writeback Date: Mon, 20 Dec 2021 21:59:39 +0800 [thread overview] Message-ID: <20211220135951.4075801-3-chengzhihao1@huawei.com> (raw) In-Reply-To: <20211220135951.4075801-1-chengzhihao1@huawei.com> Following hung tasks: [ 77.028764] task:kworker/u8:4 state:D stack: 0 pid: 132 [ 77.028820] Call Trace: [ 77.029027] schedule+0x8c/0x1b0 [ 77.029067] mutex_lock+0x50/0x60 [ 77.029074] ubifs_write_inode+0x68/0x1f0 [ubifs] [ 77.029117] __writeback_single_inode+0x43c/0x570 [ 77.029128] writeback_sb_inodes+0x259/0x740 [ 77.029148] wb_writeback+0x107/0x4d0 [ 77.029163] wb_workfn+0x162/0x7b0 [ 92.390442] task:aa state:D stack: 0 pid: 1506 [ 92.390448] Call Trace: [ 92.390458] schedule+0x8c/0x1b0 [ 92.390461] wb_wait_for_completion+0x82/0xd0 [ 92.390469] __writeback_inodes_sb_nr+0xb2/0x110 [ 92.390472] writeback_inodes_sb_nr+0x14/0x20 [ 92.390476] ubifs_budget_space+0x705/0xdd0 [ubifs] [ 92.390503] do_rename.cold+0x7f/0x187 [ubifs] [ 92.390549] ubifs_rename+0x8b/0x180 [ubifs] [ 92.390571] vfs_rename+0xdb2/0x1170 [ 92.390580] do_renameat2+0x554/0x770 , are caused by concurrent rename whiteout and inode writeback processes: rename_whiteout(Thread 1) wb_workfn(Thread2) ubifs_rename do_rename lock_4_inodes (Hold ui_mutex) ubifs_budget_space make_free_space shrink_liability __writeback_inodes_sb_nr bdi_split_work_to_wbs (Queue new wb work) wb_do_writeback(wb work) __writeback_single_inode ubifs_write_inode LOCK(ui_mutex) ↑ wb_wait_for_completion (Wait wb work) <-- deadlock! Reproducer (Detail program in [Link]): 1. SYS_renameat2("/mp/dir/file", "/mp/dir/whiteout", RENAME_WHITEOUT) 2. Consume out of space before kernel(mdelay) doing budget for whiteout Fix it by doing whiteout space budget before locking ubifs inodes. BTW, it also fixes wrong goto tag 'out_release' in whiteout budget error handling path(It should at least recover dir i_size and unlock 4 ubifs inodes). Fixes: 9e0a1fff8db56ea ("ubifs: Implement RENAME_WHITEOUT") Link: https://bugzilla.kernel.org/show_bug.cgi?id=214733 Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> --- fs/ubifs/dir.c | 25 +++++++++++++++---------- 1 file changed, 15 insertions(+), 10 deletions(-) diff --git a/fs/ubifs/dir.c b/fs/ubifs/dir.c index cfa8881d8cca..2735ad1affed 100644 --- a/fs/ubifs/dir.c +++ b/fs/ubifs/dir.c @@ -1324,6 +1324,7 @@ static int do_rename(struct inode *old_dir, struct dentry *old_dentry, if (flags & RENAME_WHITEOUT) { union ubifs_dev_desc *dev = NULL; + struct ubifs_budget_req wht_req; dev = kmalloc(sizeof(union ubifs_dev_desc), GFP_NOFS); if (!dev) { @@ -1345,6 +1346,20 @@ static int do_rename(struct inode *old_dir, struct dentry *old_dentry, whiteout_ui->data = dev; whiteout_ui->data_len = ubifs_encode_dev(dev, MKDEV(0, 0)); ubifs_assert(c, !whiteout_ui->dirty); + + memset(&wht_req, 0, sizeof(struct ubifs_budget_req)); + wht_req.dirtied_ino = 1; + wht_req.dirtied_ino_d = ALIGN(whiteout_ui->data_len, 8); + /* + * To avoid deadlock between space budget (holds ui_mutex and + * waits wb work) and writeback work(waits ui_mutex), do space + * budget before ubifs inodes locked. + */ + err = ubifs_budget_space(c, &wht_req); + if (err) { + iput(whiteout); + goto out_release; + } } lock_4_inodes(old_dir, new_dir, new_inode, whiteout); @@ -1419,16 +1434,6 @@ static int do_rename(struct inode *old_dir, struct dentry *old_dentry, } if (whiteout) { - struct ubifs_budget_req wht_req = { .dirtied_ino = 1, - .dirtied_ino_d = \ - ALIGN(ubifs_inode(whiteout)->data_len, 8) }; - - err = ubifs_budget_space(c, &wht_req); - if (err) { - iput(whiteout); - goto out_release; - } - inc_nlink(whiteout); mark_inode_dirty(whiteout); -- 2.31.1 ______________________________________________________ Linux MTD discussion mailing list http://lists.infradead.org/mailman/listinfo/linux-mtd/
next prev parent reply other threads:[~2021-12-20 13:48 UTC|newest] Thread overview: 30+ messages / expand[flat|nested] mbox.gz Atom feed top 2021-12-20 13:59 [PATCH v5 00/14] Some bugfixs for ubi/ubifs Zhihao Cheng 2021-12-20 13:59 ` Zhihao Cheng 2021-12-20 13:59 ` [PATCH v5 01/14] ubifs: rename_whiteout: Fix double free for whiteout_ui->data Zhihao Cheng 2021-12-20 13:59 ` Zhihao Cheng 2021-12-20 13:59 ` Zhihao Cheng [this message] 2021-12-20 13:59 ` [PATCH v5 02/14] ubifs: Fix deadlock in concurrent rename whiteout and inode writeback Zhihao Cheng 2021-12-20 13:59 ` [PATCH v5 03/14] ubifs: Fix wrong number of inodes locked by ui_mutex in ubifs_inode comment Zhihao Cheng 2021-12-20 13:59 ` Zhihao Cheng 2021-12-20 13:59 ` [PATCH v5 04/14] ubifs: Add missing iput if do_tmpfile() failed in rename whiteout Zhihao Cheng 2021-12-20 13:59 ` Zhihao Cheng 2021-12-20 13:59 ` [PATCH v5 05/14] ubifs: Rename whiteout atomically Zhihao Cheng 2021-12-20 13:59 ` Zhihao Cheng 2021-12-20 13:59 ` [PATCH v5 06/14] ubifs: Fix 'ui->dirty' race between do_tmpfile() and writeback work Zhihao Cheng 2021-12-20 13:59 ` Zhihao Cheng 2021-12-20 13:59 ` [PATCH v5 07/14] ubifs: Rectify space amount budget for mkdir/tmpfile operations Zhihao Cheng 2021-12-20 13:59 ` Zhihao Cheng 2021-12-20 13:59 ` [PATCH v5 08/14] ubifs: setflags: Make dirtied_ino_d 8 bytes aligned Zhihao Cheng 2021-12-20 13:59 ` Zhihao Cheng 2021-12-20 13:59 ` [PATCH v5 09/14] ubifs: Fix read out-of-bounds in ubifs_wbuf_write_nolock() Zhihao Cheng 2021-12-20 13:59 ` Zhihao Cheng 2021-12-20 13:59 ` [PATCH v5 10/14] ubifs: Fix to add refcount once page is set private Zhihao Cheng 2021-12-20 13:59 ` Zhihao Cheng 2021-12-20 13:59 ` [PATCH v5 11/14] ubi: fastmap: Return error code if memory allocation fails in add_aeb() Zhihao Cheng 2021-12-20 13:59 ` Zhihao Cheng 2021-12-20 13:59 ` [PATCH v5 12/14] ubi: fastmap: Add all fastmap pebs into 'ai->fastmap' when fm->used_blocks>=2 Zhihao Cheng 2021-12-20 13:59 ` Zhihao Cheng 2021-12-20 13:59 ` [PATCH v5 13/14] ubifs: ubifs_writepage: Mark page dirty after writing inode failed Zhihao Cheng 2021-12-20 13:59 ` Zhihao Cheng 2021-12-20 13:59 ` [PATCH v5 14/14] ubifs: ubifs_releasepage: Remove ubifs_assert(0) to valid this process Zhihao Cheng 2021-12-20 13:59 ` Zhihao Cheng
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20211220135951.4075801-3-chengzhihao1@huawei.com \ --to=chengzhihao1@huawei.com \ --cc=kirill.shutemov@linux.intel.com \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mtd@lists.infradead.org \ --cc=mcoquelin.stm32@gmail.com \ --cc=miquel.raynal@bootlin.com \ --cc=richard@nod.at \ --cc=vigneshr@ti.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.