Re: ubifs: read bad node type in ubifs_tnc_read_wbuf

* ubifs: read bad node type in ubifs_tnc_read_wbuf
@ 2020-01-10  9:10 李傲傲 (Carson Li1/9542)
  0 siblings, 0 replies; 19+ messages in thread
From: 李傲傲 (Carson Li1/9542) @ 2020-01-10  9:10 UTC (permalink / raw)
  To: linux-mtd

Hi mtd-list:
I have a problem recently on ubifs: kernel reports to read a bad node type
in ubifs_tnc_read_wbuf.

I guess the problem occurs because of tnc_mutex is freed in ubifs_tnc_locate.
Though it marks the journal heads will not be GCed. But if there is a commit
just finished, LEBs in journal heads will not be in bud rbtree anymore. And
it might be GCed. Here is the kernel log:

------------------------------------------------kernel log ------------------------------------
[   30.072901] c0 UBIFS error (ubi0:23 pid 944): ubifs_read_node_wbuf: bad node type (232 but expected 0)
[   30.085890] c1 UBIFS error (ubi0:23 pid 944): ubifs_read_node_wbuf: wbuf=cd008f40,buf=d4cd9000,
wbuf->buf=c90ec000,offs=164320, len=160, rlen=0
[   30.109436] c1 UBIFS error (ubi0:23 pid 944): ubifs_read_node_wbuf: wxf bad node at LEB 139:164320
[   30.121841] c0 Not a node, first 24 bytes:
[   30.125697] c1 00000000: 65 72 2e 78 6d 6c 2e 62 61 6b 00 b0 4d 14 c0 20 31 18 10 06 e8 3a 41 27
[   30.141302] c1 Kernel panic - not syncing: ch address is d4cd9000
[   30.152510] c1 CPU: 1 PID: 944 Comm: http-thread Tainted: G           O    4.4.83 #2
[   30.152522] c1 Hardware name: Generic DT based system
[   30.152557] c1 [<c001624c>] (unwind_backtrace) from [<c0012f8c>] (show_stack+0x10/0x14)
[   30.152574] c1 [<c0012f8c>] (show_stack) from [<c0287188>] (dump_stack+0x80/0xa0)
[   30.152590] c1 [<c0287188>] (dump_stack) from [<c00acff0>] (panic+0x70/0x1e4)
[   30.152609] c1 [<c00acff0>] (panic) from [<c01e3274>] (ubifs_dump_node+0x6c/0x9a0)
[   30.152626] c1 [<c01e3274>] (ubifs_dump_node) from [<c01c3200>] (ubifs_read_node_wbuf+0x350/0x384)
[   30.152641] c1 [<c01c3200>] (ubifs_read_node_wbuf) from [<c01e0d28>] (ubifs_tnc_read_node+0x54/0x214)
[   30.152658] c1 [<c01e0d28>] (ubifs_tnc_read_node) from [<c01c65ac>] (ubifs_tnc_locate+0x118/0x1b4)
[   30.152675] c1 [<c01c65ac>] (ubifs_tnc_locate) from [<c01bd2c8>] (ubifs_iget+0xb8/0x68c)
[   30.152691] c1 [<c01bd2c8>] (ubifs_iget) from [<c01b9b90>] (ubifs_lookup+0x1b4/0x258)
[   30.152706] c1 [<c01b9b90>] (ubifs_lookup) from [<c00f8134>] (lookup_real+0x30/0x4c)
[   30.152720] c1 [<c00f8134>] (lookup_real) from [<c00f8e68>] (__lookup_hash+0x34/0x3c)
[   30.152732] c1 [<c00f8e68>] (__lookup_hash) from [<c00f8f5c>] (walk_component+0xec/0x2a0)
[   30.152745] c1 [<c00f8f5c>] (walk_component) from [<c00fa600>] (path_lookupat+0x80/0xfc)
[   30.152761] c1 [<c00fa600>] (path_lookupat) from [<c00fd358>] (filename_lookup+0x5c/0xfc)
[   30.152776] c1 [<c00fd358>] (filename_lookup) from [<c00ef114>] (SyS_faccessat+0xa0/0x1c0)
[   30.152792] c1 [<c00ef114>] (SyS_faccessat) from [<c000f8c0>] (ret_fast_syscall+0x0/0x34)
------------------------------------------------kernel log ------------------------------------
And here is my analysis:
--------------------------zbranch passed to ubifs_tnc_read_node -------------------
crash_arm> ubifs_zbranch  c612bd90 struct ubifs_zbranch {
  key = {
    u8 = "]\003\000\000\000\000\000",
    u32 = {861, 0},
    u64 = {861},
    j32 = {861, 0}
  },
  {
    znode = 0x0,
    leaf = 0x0
  },
  lnum = 139,                               //jhead[DATAHD]
  offs = 164320,
  len = 160
}
--------------------------zbranch matches the key {861,0} in TNC-------------------
crash_arm> ubifs_zbranch 0xc8fc2230 struct ubifs_zbranch {
  key = {
    u8 = "]\003\000\000\000\000\000",
    u32 = {861, 0},
    u64 = {861},
    j32 = {861, 0}
  },
  {
    znode = 0x0,
    leaf = 0x0
  },
  lnum = 195,                            //jhead[GCHD]
  offs = 246080,
  len = 160
}
The zbr->lnum if different, and I found that the lnum = 139 is gced_lnum and also the
jhead[DATAHD].wbuf.lnum. The lnum = 195 is jhead[GCHD].wbuf.lnum.

crash_arm> ubifs_jhead.wbuf.lnum  0xcd008e00
  wbuf.lnum = 195,               //jhead[GCHD]
crash_arm> ubifs_jhead.wbuf.lnum  0xcd008ea0
  wbuf.lnum = 279,               //jhead[BASEHD]
crash_arm> ubifs_jhead.wbuf.lnum  0xcd008f40
  wbuf.lnum = 139,               //jhead[DATAHD]

so it can somehow prove my guess that when tnc_mutex is dropped, the zbr->lnum = 139
is GCed and taken to be the leb used by jhead[DATAHD]. In the GC progress, the nodes
lay in the GCing leb are moved to jhead[GCHD].wbuf, at the same time the nodes in the
TNC will be uptodated to point to the jhead[GCHD].wbuf.lnum = 195.

All progress above happened while ubifs_get_wbuf returns true in ubifs_tnc_locate.
So I doubt that marks saying “We do not GC journal heads” and delete some codes as a
temporary solution for this problem. This deletion is just take every leb found
as the same to execute fallible_read_node.

------------------------------------temporary solution--------------------------------

--- a/fs/ubifs/tnc.c
+++ b/fs/ubifs/tnc.c
@@ -1482,12 +1482,6 @@ again:
 gc_seq1 = c->gc_seq;
 mutex_unlock(&c->tnc_mutex);

-if (ubifs_get_wbuf(c, zbr.lnum)) {
-/* We do not GC journal heads */
-err = ubifs_tnc_read_node(c, &zbr, node);
-return err;
-}
-
Actually, compared to that solution described above, I more suggest to modify the
ubifs_get_wbuf.ubifs_get_wbuf is to check if the LEB is on the jhead, but
ubifs_tnc_read_wbuf only read node from wbuf when the lnum is equal to wbuf.lnum
and the others still need to read on flash. It seems to be better to just make
ubifs_get_wbuf to check if the LEB is equal to the wbufs.lnum, and then there is
no need to have a double check in ubifs_tnc_read_wbuf.
-----------------------------more suggested but not tested solution---------------------
--- a/fs/ubifs/log.c
+++ b/fs/ubifs/log.c
@@ -70,28 +70,16 @@ struct ubifs_bud *ubifs_search_bud(struct ubifs_info *c, int lnum)
  */
 struct ubifs_wbuf *ubifs_get_wbuf(struct ubifs_info *c, int lnum)  {
-struct rb_node *p;
-struct ubifs_bud *bud;
 int jhead;

 if (!c->jheads)
 return NULL;

-spin_lock(&c->buds_lock);
-p = c->buds.rb_node;
-while (p) {
-bud = rb_entry(p, struct ubifs_bud, rb);
-if (lnum < bud->lnum)
-p = p->rb_left;
-else if (lnum > bud->lnum)
-p = p->rb_right;
-else {
-jhead = bud->jhead;
-spin_unlock(&c->buds_lock);
+for(jhead = 0; jhead < c->jhead_cnt; jhead++){
+if(lnum == c->jheads[jhead].wbuf.lnum)
 return &c->jheads[jhead].wbuf;
-}
 }
-spin_unlock(&c->buds_lock);
+
 return NULL;
 }

--- a/fs/ubifs/io.c
+++ b/fs/ubifs/io.c
@@ -906,9 +906,10 @@ int ubifs_read_node_wbuf(struct ubifs_wbuf *wbuf, void *buf, int type, int len,
 ubifs_assert(wbuf && lnum >= 0 && lnum < c->leb_cnt && offs >= 0);
 ubifs_assert(!(offs & 7) && offs < c->leb_size);
 ubifs_assert(type >= 0 && type < UBIFS_NODE_TYPES_CNT);
+ubifs_assert(wbuf->lnum == lnum);

 spin_lock(&wbuf->lock);
-overlap = (lnum == wbuf->lnum && offs + len > wbuf->offs);
+overlap = (offs + len > wbuf->offs);
 if (!overlap) {
 /* We may safely unlock the write-buffer and read the data */
 spin_unlock(&wbuf->lock);

after the modification, the LEB contains node will not be GCed since even though
there is a commit, the wbuf.lnum is still on the bud rbtree as a journal head leb.

I am not sure if I have made things right, and it is a problem with low probability.
so I haven’t applied the modification yet. could you please kindly check for me
if I am on the wrong direction? If so, is there any relevant information for me to
fix that?

Thanks
Carson.li

________________________________
 This email (including its attachments) is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. Unauthorized use, dissemination, distribution or copying of this email or the information herein or taking any action in reliance on the contents of this email or the information herein, by anyone other than the intended recipient, or an employee or agent responsible for delivering the message to the intended recipient, is strictly prohibited. If you are not the intended recipient, please do not read, copy, use or disclose any part of this e-mail to others. Please notify the sender immediately and permanently delete this e-mail and any attachments if you received it in error. Internet communications cannot be guaranteed to be timely, secure, error-free or virus-free. The sender does not accept liability for any errors or omissions.
本邮件及其附件具有保密性质，受法律保护不得泄露，仅发送给本邮件所指特定收件人。严禁非经授权使用、宣传、发布或复制本邮件或其内容。若非该特定收件人，请勿阅读、复制、 使用或披露本邮件的任何内容。若误收本邮件，请从系统中永久性删除本邮件及所有附件，并以回复邮件的方式即刻告知发件人。无法保证互联网通信及时、安全、无误或防毒。发件人对任何错漏均不承担责任。
______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

^ permalink raw reply	[flat|nested] 19+ messages in thread