* kernel BUG at fs/jffs2/gc.c:395! @ 2019-08-20 23:09 Tao Ren 2019-08-20 23:11 ` Tao Ren 0 siblings, 1 reply; 11+ messages in thread From: Tao Ren @ 2019-08-20 23:09 UTC (permalink / raw) To: linux-mtd, OpenBMC Maillist Hi, I hit following jffs2 bug while running Linux 5.0.3 on CMM (ASPEED2500) BMC platform. Has anyone seen the issue before? Any suggestions? [ 46.024017] ------------[ cut here ]------------ [ 46.079178] kernel BUG at /data/users/taoren/openbmc/build-cmm/tmp/work-shared/cmm/kernel-source/fs/jffs2/gc.c:395! [ 46.204076] Internal error: Oops - BUG: 0 [#1] ARM [ 46.261378] Modules linked in: ext4 mbcache jbd2 crypto_hash [ 46.329093] CPU: 0 PID: 1181 Comm: jffs2_gcd_mtd3 Not tainted 5.0.3-cmm #1 [ 46.411343] Hardware name: Generic DT based system [ 46.468685] PC is at jffs2_garbage_collect_pass+0x6f4/0x734 [ 46.535322] LR is at jffs2_garbage_collect_pass+0x6f4/0x734 [ 46.601977] pc : [<802c292c>] lr : [<802c292c>] psr: 60000013 [ 46.676959] sp : af3cded0 ip : b56a75c0 fp : af3cdf24 [ 46.739463] r10: b4061140 r9 : b57a3900 r8 : b555d4ac [ 46.801968] r7 : b555d4ac r6 : b403502c r5 : 00000000 r4 : b4035000 [ 46.880073] r3 : b56a75c0 r2 : 00000000 r1 : 00000000 r0 : b403502c [ 46.958177] Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none [ 47.043561] Control: 00c5387d Table: b5774008 DAC: 00000051 [ 47.112319] Process jffs2_gcd_mtd3 (pid: 1181, stack limit = 0x54372ffe) [ 47.192490] Stack: (0xaf3cded0 to 0xaf3ce000) [ 47.244601] dec0: 00000000 80a07028 800ad6c9 0000ff2c [ 47.342468] dee0: af3cdefc af3cdef0 80125cd4 8012313c af3cdf24 800ad6c9 8012614c b4035000 [ 47.440331] df00: ffffe000 af3cc000 af3cc000 b4035000 802c509c b419dd18 af3cdf74 af3cdf28 [ 47.538196] df20: 802c5174 802c2244 ffffe000 00000001 00000000 ffffe000 b57b0940 00000000 [ 47.636058] df40: ffffe000 b4035000 802c509c b419dd18 af3cdf74 800ad6c9 b5753980 b5753980 [ 47.733923] df60: b57b0940 00000000 af3cdfac af3cdf78 80134d58 802c50a8 b5753998 b5753998 [ 47.831787] df80: af3cdfac b57b0940 80134c0c 00000000 00000000 00000000 00000000 00000000 [ 47.929648] dfa0: 00000000 af3cdfb0 801010e8 80134c18 00000000 00000000 00000000 00000000 [ 48.027512] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 48.125376] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000 [ 48.223230] Backtrace: [ 48.252489] [<802c2238>] (jffs2_garbage_collect_pass) from [<802c5174>] (jffs2_garbage_collect_thread+0xd8/0x1ac) [ 48.375294] r10:b419dd18 r9:802c509c r8:b4035000 r7:af3cc000 r6:af3cc000 r5:ffffe000 [ 48.468985] r4:b4035000 [ 48.499281] [<802c509c>] (jffs2_garbage_collect_thread) from [<80134d58>] (kthread+0x14c/0x164) [ 48.603358] r6:00000000 r5:b57b0940 r4:b5753980 [ 48.658590] [<80134c0c>] (kthread) from [<801010e8>] (ret_from_fork+0x14/0x2c) [ 48.745001] Exception stack(0xaf3cdfb0 to 0xaf3cdff8) [ 48.805428] dfa0: 00000000 00000000 00000000 00000000 [ 48.903296] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 49.001157] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000 [ 49.080305] r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:80134c0c [ 49.174000] r4:b57b0940 [ 49.204275] Code: e59f0044 ebfa25cb e1a00006 eb0e888d (e7f001f2) [ 49.277188] ---[ end trace 6baa7af0a90d15ab ]--- [ 49.332395] Kernel panic - not syncing: Fatal exception Thanks, Tao ______________________________________________________ Linux MTD discussion mailing list http://lists.infradead.org/mailman/listinfo/linux-mtd/ ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: kernel BUG at fs/jffs2/gc.c:395! 2019-08-20 23:09 kernel BUG at fs/jffs2/gc.c:395! Tao Ren @ 2019-08-20 23:11 ` Tao Ren 2019-08-21 0:06 ` Andrew Jeffery 0 siblings, 1 reply; 11+ messages in thread From: Tao Ren @ 2019-08-20 23:11 UTC (permalink / raw) To: linux-mtd, OpenBMC Maillist On 8/20/19 4:09 PM, Tao Ren wrote: > Hi, > > I hit following jffs2 bug while running Linux 5.0.3 on CMM (ASPEED2500) BMC platform. Has anyone seen the issue before? Any suggestions? > > [ 46.024017] ------------[ cut here ]------------ > [ 46.079178] kernel BUG at /data/users/taoren/openbmc/build-cmm/tmp/work-shared/cmm/kernel-source/fs/jffs2/gc.c:395! > [ 46.204076] Internal error: Oops - BUG: 0 [#1] ARM > [ 46.261378] Modules linked in: ext4 mbcache jbd2 crypto_hash > [ 46.329093] CPU: 0 PID: 1181 Comm: jffs2_gcd_mtd3 Not tainted 5.0.3-cmm #1 > [ 46.411343] Hardware name: Generic DT based system > [ 46.468685] PC is at jffs2_garbage_collect_pass+0x6f4/0x734 > [ 46.535322] LR is at jffs2_garbage_collect_pass+0x6f4/0x734 > [ 46.601977] pc : [<802c292c>] lr : [<802c292c>] psr: 60000013 > [ 46.676959] sp : af3cded0 ip : b56a75c0 fp : af3cdf24 > [ 46.739463] r10: b4061140 r9 : b57a3900 r8 : b555d4ac > [ 46.801968] r7 : b555d4ac r6 : b403502c r5 : 00000000 r4 : b4035000 > [ 46.880073] r3 : b56a75c0 r2 : 00000000 r1 : 00000000 r0 : b403502c > [ 46.958177] Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none > [ 47.043561] Control: 00c5387d Table: b5774008 DAC: 00000051 > [ 47.112319] Process jffs2_gcd_mtd3 (pid: 1181, stack limit = 0x54372ffe) > [ 47.192490] Stack: (0xaf3cded0 to 0xaf3ce000) > [ 47.244601] dec0: 00000000 80a07028 800ad6c9 0000ff2c > [ 47.342468] dee0: af3cdefc af3cdef0 80125cd4 8012313c af3cdf24 800ad6c9 8012614c b4035000 > [ 47.440331] df00: ffffe000 af3cc000 af3cc000 b4035000 802c509c b419dd18 af3cdf74 af3cdf28 > [ 47.538196] df20: 802c5174 802c2244 ffffe000 00000001 00000000 ffffe000 b57b0940 00000000 > [ 47.636058] df40: ffffe000 b4035000 802c509c b419dd18 af3cdf74 800ad6c9 b5753980 b5753980 > [ 47.733923] df60: b57b0940 00000000 af3cdfac af3cdf78 80134d58 802c50a8 b5753998 b5753998 > [ 47.831787] df80: af3cdfac b57b0940 80134c0c 00000000 00000000 00000000 00000000 00000000 > [ 47.929648] dfa0: 00000000 af3cdfb0 801010e8 80134c18 00000000 00000000 00000000 00000000 > [ 48.027512] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 > [ 48.125376] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000 > [ 48.223230] Backtrace: > [ 48.252489] [<802c2238>] (jffs2_garbage_collect_pass) from [<802c5174>] (jffs2_garbage_collect_thread+0xd8/0x1ac) > [ 48.375294] r10:b419dd18 r9:802c509c r8:b4035000 r7:af3cc000 r6:af3cc000 r5:ffffe000 > [ 48.468985] r4:b4035000 > [ 48.499281] [<802c509c>] (jffs2_garbage_collect_thread) from [<80134d58>] (kthread+0x14c/0x164) > [ 48.603358] r6:00000000 r5:b57b0940 r4:b5753980 > [ 48.658590] [<80134c0c>] (kthread) from [<801010e8>] (ret_from_fork+0x14/0x2c) > [ 48.745001] Exception stack(0xaf3cdfb0 to 0xaf3cdff8) > [ 48.805428] dfa0: 00000000 00000000 00000000 00000000 > [ 48.903296] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 > [ 49.001157] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000 > [ 49.080305] r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:80134c0c > [ 49.174000] r4:b57b0940 > [ 49.204275] Code: e59f0044 ebfa25cb e1a00006 eb0e888d (e7f001f2) > [ 49.277188] ---[ end trace 6baa7af0a90d15ab ]--- > [ 49.332395] Kernel panic - not syncing: Fatal exception BTW, below are all the messages printed by jffs2 before system crash: [ 21.078433] jffs2: version 2.2. (SUMMARY) © 2001-2006 Red Hat, Inc. [ 39.776207] jffs2: notice: (1180) jffs2_build_xattr_subsystem: complete building xattr subsystem, 0 of xdatum (0 unchecked, 0 orphan) and 0 of xref (0 dead, 0 orphan) found. [ 40.016574] jffs2: warning: (1181) jffs2_do_read_inode_internal: no data nodes found for ino #140 [ 40.122964] jffs2: notice: (1181) jffs2_do_read_inode_internal: but it has children so we fake some modes for it [ 43.579361] jffs2: warning: (1181) jffs2_get_inode_nodes: Eep. No valid nodes for ino #141. [ 43.679404] jffs2: warning: (1181) jffs2_do_read_inode_internal: no data nodes found for ino #141 [ 43.785661] jffs2: Returned error for crccheck of ino #141. Expect badness... [ 44.021825] jffs2: warning: (1181) jffs2_do_read_inode_internal: no data nodes found for ino #154 [ 44.128191] jffs2: notice: (1181) jffs2_do_read_inode_internal: but it has children so we fake some modes for it [ 44.314862] jffs2: warning: (1181) jffs2_do_read_inode_internal: no data nodes found for ino #155 [ 44.421152] jffs2: notice: (1181) jffs2_do_read_inode_internal: but it has children so we fake some modes for it [ 44.607378] jffs2: warning: (1181) jffs2_do_read_inode_internal: no data nodes found for ino #163 [ 44.713692] jffs2: notice: (1181) jffs2_do_read_inode_internal: but it has children so we fake some modes for it [ 44.899991] jffs2: warning: (1181) jffs2_get_inode_nodes: Eep. No valid nodes for ino #164. [ 45.000107] jffs2: warning: (1181) jffs2_do_read_inode_internal: no data nodes found for ino #164 [ 45.106370] jffs2: Returned error for crccheck of ino #164. Expect badness... [ 45.934282] jffs2: Inode #106 already in state 0 in jffs2_garbage_collect_pass()! Thanks, Tao ______________________________________________________ Linux MTD discussion mailing list http://lists.infradead.org/mailman/listinfo/linux-mtd/ ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: kernel BUG at fs/jffs2/gc.c:395! 2019-08-20 23:11 ` Tao Ren @ 2019-08-21 0:06 ` Andrew Jeffery 2019-08-21 0:20 ` Tao Ren 2019-08-25 19:22 ` Richard Weinberger 0 siblings, 2 replies; 11+ messages in thread From: Andrew Jeffery @ 2019-08-21 0:06 UTC (permalink / raw) To: Tao Ren, linux-mtd, OpenBMC Maillist On Wed, 21 Aug 2019, at 08:42, Tao Ren wrote: > On 8/20/19 4:09 PM, Tao Ren wrote: > > Hi, > > > > I hit following jffs2 bug while running Linux 5.0.3 on CMM (ASPEED2500) BMC platform. Has anyone seen the issue before? Any suggestions? > > > > [ 46.024017] ------------[ cut here ]------------ > > [ 46.079178] kernel BUG at /data/users/taoren/openbmc/build-cmm/tmp/work-shared/cmm/kernel-source/fs/jffs2/gc.c:395! > > [ 46.204076] Internal error: Oops - BUG: 0 [#1] ARM > > [ 46.261378] Modules linked in: ext4 mbcache jbd2 crypto_hash > > [ 46.329093] CPU: 0 PID: 1181 Comm: jffs2_gcd_mtd3 Not tainted 5.0.3-cmm #1 > > [ 46.411343] Hardware name: Generic DT based system > > [ 46.468685] PC is at jffs2_garbage_collect_pass+0x6f4/0x734 > > [ 46.535322] LR is at jffs2_garbage_collect_pass+0x6f4/0x734 > > [ 46.601977] pc : [<802c292c>] lr : [<802c292c>] psr: 60000013 > > [ 46.676959] sp : af3cded0 ip : b56a75c0 fp : af3cdf24 > > [ 46.739463] r10: b4061140 r9 : b57a3900 r8 : b555d4ac > > [ 46.801968] r7 : b555d4ac r6 : b403502c r5 : 00000000 r4 : b4035000 > > [ 46.880073] r3 : b56a75c0 r2 : 00000000 r1 : 00000000 r0 : b403502c > > [ 46.958177] Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none > > [ 47.043561] Control: 00c5387d Table: b5774008 DAC: 00000051 > > [ 47.112319] Process jffs2_gcd_mtd3 (pid: 1181, stack limit = 0x54372ffe) > > [ 47.192490] Stack: (0xaf3cded0 to 0xaf3ce000) > > [ 47.244601] dec0: 00000000 80a07028 800ad6c9 0000ff2c > > [ 47.342468] dee0: af3cdefc af3cdef0 80125cd4 8012313c af3cdf24 800ad6c9 8012614c b4035000 > > [ 47.440331] df00: ffffe000 af3cc000 af3cc000 b4035000 802c509c b419dd18 af3cdf74 af3cdf28 > > [ 47.538196] df20: 802c5174 802c2244 ffffe000 00000001 00000000 ffffe000 b57b0940 00000000 > > [ 47.636058] df40: ffffe000 b4035000 802c509c b419dd18 af3cdf74 800ad6c9 b5753980 b5753980 > > [ 47.733923] df60: b57b0940 00000000 af3cdfac af3cdf78 80134d58 802c50a8 b5753998 b5753998 > > [ 47.831787] df80: af3cdfac b57b0940 80134c0c 00000000 00000000 00000000 00000000 00000000 > > [ 47.929648] dfa0: 00000000 af3cdfb0 801010e8 80134c18 00000000 00000000 00000000 00000000 > > [ 48.027512] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 > > [ 48.125376] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000 > > [ 48.223230] Backtrace: > > [ 48.252489] [<802c2238>] (jffs2_garbage_collect_pass) from [<802c5174>] (jffs2_garbage_collect_thread+0xd8/0x1ac) > > [ 48.375294] r10:b419dd18 r9:802c509c r8:b4035000 r7:af3cc000 r6:af3cc000 r5:ffffe000 > > [ 48.468985] r4:b4035000 > > [ 48.499281] [<802c509c>] (jffs2_garbage_collect_thread) from [<80134d58>] (kthread+0x14c/0x164) > > [ 48.603358] r6:00000000 r5:b57b0940 r4:b5753980 > > [ 48.658590] [<80134c0c>] (kthread) from [<801010e8>] (ret_from_fork+0x14/0x2c) > > [ 48.745001] Exception stack(0xaf3cdfb0 to 0xaf3cdff8) > > [ 48.805428] dfa0: 00000000 00000000 00000000 00000000 > > [ 48.903296] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 > > [ 49.001157] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000 > > [ 49.080305] r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:80134c0c > > [ 49.174000] r4:b57b0940 > > [ 49.204275] Code: e59f0044 ebfa25cb e1a00006 eb0e888d (e7f001f2) > > [ 49.277188] ---[ end trace 6baa7af0a90d15ab ]--- > > [ 49.332395] Kernel panic - not syncing: Fatal exception > > BTW, below are all the messages printed by jffs2 before system crash: > > [ 21.078433] jffs2: version 2.2. (SUMMARY) © 2001-2006 Red Hat, Inc. > [ 39.776207] jffs2: notice: (1180) jffs2_build_xattr_subsystem: > complete building xattr subsystem, 0 of xdatum (0 unchecked, 0 orphan) > and 0 of xref (0 dead, 0 orphan) found. > [ 40.016574] jffs2: warning: (1181) jffs2_do_read_inode_internal: no > data nodes found for ino #140 > [ 40.122964] jffs2: notice: (1181) jffs2_do_read_inode_internal: but > it has children so we fake some modes for it > [ 43.579361] jffs2: warning: (1181) jffs2_get_inode_nodes: Eep. No > valid nodes for ino #141. > [ 43.679404] jffs2: warning: (1181) jffs2_do_read_inode_internal: no > data nodes found for ino #141 > [ 43.785661] jffs2: Returned error for crccheck of ino #141. Expect > badness... > [ 44.021825] jffs2: warning: (1181) jffs2_do_read_inode_internal: no > data nodes found for ino #154 > [ 44.128191] jffs2: notice: (1181) jffs2_do_read_inode_internal: but > it has children so we fake some modes for it > [ 44.314862] jffs2: warning: (1181) jffs2_do_read_inode_internal: no > data nodes found for ino #155 > [ 44.421152] jffs2: notice: (1181) jffs2_do_read_inode_internal: but > it has children so we fake some modes for it > [ 44.607378] jffs2: warning: (1181) jffs2_do_read_inode_internal: no > data nodes found for ino #163 > [ 44.713692] jffs2: notice: (1181) jffs2_do_read_inode_internal: but > it has children so we fake some modes for it > [ 44.899991] jffs2: warning: (1181) jffs2_get_inode_nodes: Eep. No > valid nodes for ino #164. > [ 45.000107] jffs2: warning: (1181) jffs2_do_read_inode_internal: no > data nodes found for ino #164 > [ 45.106370] jffs2: Returned error for crccheck of ino #164. Expect > badness... > [ 45.934282] jffs2: Inode #106 already in state 0 in > jffs2_garbage_collect_pass()! Looks like a lack of robustness to filesystem corruption to me. LWN published an article on the topic just yesterday! https://lwn.net/Articles/796687/ Andrew ______________________________________________________ Linux MTD discussion mailing list http://lists.infradead.org/mailman/listinfo/linux-mtd/ ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: kernel BUG at fs/jffs2/gc.c:395! 2019-08-21 0:06 ` Andrew Jeffery @ 2019-08-21 0:20 ` Tao Ren 2019-08-25 19:22 ` Richard Weinberger 1 sibling, 0 replies; 11+ messages in thread From: Tao Ren @ 2019-08-21 0:20 UTC (permalink / raw) To: Andrew Jeffery, linux-mtd, OpenBMC Maillist On 8/20/19 5:06 PM, Andrew Jeffery wrote: > > > On Wed, 21 Aug 2019, at 08:42, Tao Ren wrote: >> On 8/20/19 4:09 PM, Tao Ren wrote: >>> Hi, >>> >>> I hit following jffs2 bug while running Linux 5.0.3 on CMM (ASPEED2500) BMC platform. Has anyone seen the issue before? Any suggestions? >>> >>> [ 46.024017] ------------[ cut here ]------------ >>> [ 46.079178] kernel BUG at /data/users/taoren/openbmc/build-cmm/tmp/work-shared/cmm/kernel-source/fs/jffs2/gc.c:395! >>> [ 46.204076] Internal error: Oops - BUG: 0 [#1] ARM >>> [ 46.261378] Modules linked in: ext4 mbcache jbd2 crypto_hash >>> [ 46.329093] CPU: 0 PID: 1181 Comm: jffs2_gcd_mtd3 Not tainted 5.0.3-cmm #1 >>> [ 46.411343] Hardware name: Generic DT based system >>> [ 46.468685] PC is at jffs2_garbage_collect_pass+0x6f4/0x734 >>> [ 46.535322] LR is at jffs2_garbage_collect_pass+0x6f4/0x734 >>> [ 46.601977] pc : [<802c292c>] lr : [<802c292c>] psr: 60000013 >>> [ 46.676959] sp : af3cded0 ip : b56a75c0 fp : af3cdf24 >>> [ 46.739463] r10: b4061140 r9 : b57a3900 r8 : b555d4ac >>> [ 46.801968] r7 : b555d4ac r6 : b403502c r5 : 00000000 r4 : b4035000 >>> [ 46.880073] r3 : b56a75c0 r2 : 00000000 r1 : 00000000 r0 : b403502c >>> [ 46.958177] Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none >>> [ 47.043561] Control: 00c5387d Table: b5774008 DAC: 00000051 >>> [ 47.112319] Process jffs2_gcd_mtd3 (pid: 1181, stack limit = 0x54372ffe) >>> [ 47.192490] Stack: (0xaf3cded0 to 0xaf3ce000) >>> [ 47.244601] dec0: 00000000 80a07028 800ad6c9 0000ff2c >>> [ 47.342468] dee0: af3cdefc af3cdef0 80125cd4 8012313c af3cdf24 800ad6c9 8012614c b4035000 >>> [ 47.440331] df00: ffffe000 af3cc000 af3cc000 b4035000 802c509c b419dd18 af3cdf74 af3cdf28 >>> [ 47.538196] df20: 802c5174 802c2244 ffffe000 00000001 00000000 ffffe000 b57b0940 00000000 >>> [ 47.636058] df40: ffffe000 b4035000 802c509c b419dd18 af3cdf74 800ad6c9 b5753980 b5753980 >>> [ 47.733923] df60: b57b0940 00000000 af3cdfac af3cdf78 80134d58 802c50a8 b5753998 b5753998 >>> [ 47.831787] df80: af3cdfac b57b0940 80134c0c 00000000 00000000 00000000 00000000 00000000 >>> [ 47.929648] dfa0: 00000000 af3cdfb0 801010e8 80134c18 00000000 00000000 00000000 00000000 >>> [ 48.027512] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 >>> [ 48.125376] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000 >>> [ 48.223230] Backtrace: >>> [ 48.252489] [<802c2238>] (jffs2_garbage_collect_pass) from [<802c5174>] (jffs2_garbage_collect_thread+0xd8/0x1ac) >>> [ 48.375294] r10:b419dd18 r9:802c509c r8:b4035000 r7:af3cc000 r6:af3cc000 r5:ffffe000 >>> [ 48.468985] r4:b4035000 >>> [ 48.499281] [<802c509c>] (jffs2_garbage_collect_thread) from [<80134d58>] (kthread+0x14c/0x164) >>> [ 48.603358] r6:00000000 r5:b57b0940 r4:b5753980 >>> [ 48.658590] [<80134c0c>] (kthread) from [<801010e8>] (ret_from_fork+0x14/0x2c) >>> [ 48.745001] Exception stack(0xaf3cdfb0 to 0xaf3cdff8) >>> [ 48.805428] dfa0: 00000000 00000000 00000000 00000000 >>> [ 48.903296] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 >>> [ 49.001157] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000 >>> [ 49.080305] r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:80134c0c >>> [ 49.174000] r4:b57b0940 >>> [ 49.204275] Code: e59f0044 ebfa25cb e1a00006 eb0e888d (e7f001f2) >>> [ 49.277188] ---[ end trace 6baa7af0a90d15ab ]--- >>> [ 49.332395] Kernel panic - not syncing: Fatal exception >> >> BTW, below are all the messages printed by jffs2 before system crash: >> >> [ 21.078433] jffs2: version 2.2. (SUMMARY) © 2001-2006 Red Hat, Inc. >> [ 39.776207] jffs2: notice: (1180) jffs2_build_xattr_subsystem: >> complete building xattr subsystem, 0 of xdatum (0 unchecked, 0 orphan) >> and 0 of xref (0 dead, 0 orphan) found. >> [ 40.016574] jffs2: warning: (1181) jffs2_do_read_inode_internal: no >> data nodes found for ino #140 >> [ 40.122964] jffs2: notice: (1181) jffs2_do_read_inode_internal: but >> it has children so we fake some modes for it >> [ 43.579361] jffs2: warning: (1181) jffs2_get_inode_nodes: Eep. No >> valid nodes for ino #141. >> [ 43.679404] jffs2: warning: (1181) jffs2_do_read_inode_internal: no >> data nodes found for ino #141 >> [ 43.785661] jffs2: Returned error for crccheck of ino #141. Expect >> badness... >> [ 44.021825] jffs2: warning: (1181) jffs2_do_read_inode_internal: no >> data nodes found for ino #154 >> [ 44.128191] jffs2: notice: (1181) jffs2_do_read_inode_internal: but >> it has children so we fake some modes for it >> [ 44.314862] jffs2: warning: (1181) jffs2_do_read_inode_internal: no >> data nodes found for ino #155 >> [ 44.421152] jffs2: notice: (1181) jffs2_do_read_inode_internal: but >> it has children so we fake some modes for it >> [ 44.607378] jffs2: warning: (1181) jffs2_do_read_inode_internal: no >> data nodes found for ino #163 >> [ 44.713692] jffs2: notice: (1181) jffs2_do_read_inode_internal: but >> it has children so we fake some modes for it >> [ 44.899991] jffs2: warning: (1181) jffs2_get_inode_nodes: Eep. No >> valid nodes for ino #164. >> [ 45.000107] jffs2: warning: (1181) jffs2_do_read_inode_internal: no >> data nodes found for ino #164 >> [ 45.106370] jffs2: Returned error for crccheck of ino #164. Expect >> badness... >> [ 45.934282] jffs2: Inode #106 already in state 0 in >> jffs2_garbage_collect_pass()! > > Looks like a lack of robustness to filesystem corruption to me. LWN > published an article on the topic just yesterday! > > https://urldefense.proofpoint.com/v2/url?u=https-3A__lwn.net_Articles_796687_&d=DwIFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=iYElT7HC77pRZ3byVvW8ng&m=VI7s7FnQP0Ooe6tEJPtpjajwGDNh4FhQjTgiRbW8hmk&s=obrUgP3wo7zCtMTk8cnEMXhxaF9VSJmy1iJeSxKHmt8&e= > > Andrew Thank you for the sharing, Andrew. Let me check out the article.. Cheers, Tao ______________________________________________________ Linux MTD discussion mailing list http://lists.infradead.org/mailman/listinfo/linux-mtd/ ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: kernel BUG at fs/jffs2/gc.c:395! 2019-08-21 0:06 ` Andrew Jeffery 2019-08-21 0:20 ` Tao Ren @ 2019-08-25 19:22 ` Richard Weinberger 2019-08-25 22:08 ` Tao Ren 2019-08-26 0:33 ` Andrew Jeffery 1 sibling, 2 replies; 11+ messages in thread From: Richard Weinberger @ 2019-08-25 19:22 UTC (permalink / raw) To: Andrew Jeffery; +Cc: Tao Ren, OpenBMC Maillist, linux-mtd On Wed, Aug 21, 2019 at 2:06 AM Andrew Jeffery <andrew@aj.id.au> wrote: > Looks like a lack of robustness to filesystem corruption to me. LWN What exactly makes you think so? The inode cache entry is in state INO_STATE_UNCHECKED while GC run, which is not allowed. Tao, is the error persistent or did it happen only once? -- Thanks, //richard ______________________________________________________ Linux MTD discussion mailing list http://lists.infradead.org/mailman/listinfo/linux-mtd/ ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: kernel BUG at fs/jffs2/gc.c:395! 2019-08-25 19:22 ` Richard Weinberger @ 2019-08-25 22:08 ` Tao Ren 2019-08-25 22:29 ` Richard Weinberger 2019-08-26 0:33 ` Andrew Jeffery 1 sibling, 1 reply; 11+ messages in thread From: Tao Ren @ 2019-08-25 22:08 UTC (permalink / raw) To: Richard Weinberger, Andrew Jeffery; +Cc: OpenBMC Maillist, linux-mtd On 8/25/19 12:22 PM, Richard Weinberger wrote: > On Wed, Aug 21, 2019 at 2:06 AM Andrew Jeffery <andrew@aj.id.au> wrote: >> Looks like a lack of robustness to filesystem corruption to me. LWN > > What exactly makes you think so? > The inode cache entry is in state INO_STATE_UNCHECKED while GC run, > which is not allowed. > > Tao, is the error persistent or did it happen only once? Hi Richard, It rarely happens (~1 out of 1000 machines in my environment), but once it happens, it's persistent: the machine will fall into reboot loop due to the crash. Thanks, Tao ______________________________________________________ Linux MTD discussion mailing list http://lists.infradead.org/mailman/listinfo/linux-mtd/ ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: kernel BUG at fs/jffs2/gc.c:395! 2019-08-25 22:08 ` Tao Ren @ 2019-08-25 22:29 ` Richard Weinberger 2019-08-27 4:42 ` Tao Ren 0 siblings, 1 reply; 11+ messages in thread From: Richard Weinberger @ 2019-08-25 22:29 UTC (permalink / raw) To: Tao Ren; +Cc: Andrew Jeffery, OpenBMC Maillist, linux-mtd ----- Ursprüngliche Mail ----- > Von: "Tao Ren" <taoren@fb.com> > An: "Richard Weinberger" <richard.weinberger@gmail.com>, "Andrew Jeffery" <andrew@aj.id.au> > CC: "linux-mtd" <linux-mtd@lists.infradead.org>, "OpenBMC Maillist" <openbmc@lists.ozlabs.org> > Gesendet: Montag, 26. August 2019 00:08:08 > Betreff: Re: kernel BUG at fs/jffs2/gc.c:395! > On 8/25/19 12:22 PM, Richard Weinberger wrote: >> On Wed, Aug 21, 2019 at 2:06 AM Andrew Jeffery <andrew@aj.id.au> wrote: >>> Looks like a lack of robustness to filesystem corruption to me. LWN >> >> What exactly makes you think so? >> The inode cache entry is in state INO_STATE_UNCHECKED while GC run, >> which is not allowed. >> >> Tao, is the error persistent or did it happen only once? > > Hi Richard, > > It rarely happens (~1 out of 1000 machines in my environment), but once it > happens, it's persistent: the machine will fall into reboot loop due to the > crash. Can you provide me an image of the filesystem such that I can have a look? An image where the issue is persistent... Thanks, //richard ______________________________________________________ Linux MTD discussion mailing list http://lists.infradead.org/mailman/listinfo/linux-mtd/ ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: kernel BUG at fs/jffs2/gc.c:395! 2019-08-25 22:29 ` Richard Weinberger @ 2019-08-27 4:42 ` Tao Ren 2019-08-28 0:08 ` [Potential Spoof] " Tao Ren 0 siblings, 1 reply; 11+ messages in thread From: Tao Ren @ 2019-08-27 4:42 UTC (permalink / raw) To: Richard Weinberger; +Cc: Andrew Jeffery, OpenBMC Maillist, linux-mtd On 8/25/19 3:29 PM, Richard Weinberger wrote: > ----- Ursprüngliche Mail ----- >> Von: "Tao Ren" <taoren@fb.com> >> An: "Richard Weinberger" <richard.weinberger@gmail.com>, "Andrew Jeffery" <andrew@aj.id.au> >> CC: "linux-mtd" <linux-mtd@lists.infradead.org>, "OpenBMC Maillist" <openbmc@lists.ozlabs.org> >> Gesendet: Montag, 26. August 2019 00:08:08 >> Betreff: Re: kernel BUG at fs/jffs2/gc.c:395! > >> On 8/25/19 12:22 PM, Richard Weinberger wrote: >>> On Wed, Aug 21, 2019 at 2:06 AM Andrew Jeffery <andrew@aj.id.au> wrote: >>>> Looks like a lack of robustness to filesystem corruption to me. LWN >>> >>> What exactly makes you think so? >>> The inode cache entry is in state INO_STATE_UNCHECKED while GC run, >>> which is not allowed. >>> >>> Tao, is the error persistent or did it happen only once? >> >> Hi Richard, >> >> It rarely happens (~1 out of 1000 machines in my environment), but once it >> happens, it's persistent: the machine will fall into reboot loop due to the >> crash. > > Can you provide me an image of the filesystem such that I can have a look? > An image where the issue is persistent... Hi Richard, I tried kernel image with jffs2 summary enabled and disabled, and it looks to me the result is similar: I can reach login screen now, but the same kernel panic happens after "reboot" command. The behavior is a little different from what I saw yesterday: previously kernel panic happened at boot time, and now it's after "reboot" command. I guess it's because more node being written to the flash? I understand it's helpful to share the file system image, but unfortunately I cannot do it because it contains confidential data. Sorry about that.. Thank you again for the help, and kindly let me know if you have further suggestions. Cheers, Tao ______________________________________________________ Linux MTD discussion mailing list http://lists.infradead.org/mailman/listinfo/linux-mtd/ ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Potential Spoof] Re: kernel BUG at fs/jffs2/gc.c:395! 2019-08-27 4:42 ` Tao Ren @ 2019-08-28 0:08 ` Tao Ren 0 siblings, 0 replies; 11+ messages in thread From: Tao Ren @ 2019-08-28 0:08 UTC (permalink / raw) To: Richard Weinberger; +Cc: Andrew Jeffery, OpenBMC Maillist, linux-mtd On 8/26/19 9:42 PM, Tao Ren wrote: > On 8/25/19 3:29 PM, Richard Weinberger wrote: >> ----- Ursprüngliche Mail ----- >>> Von: "Tao Ren" <taoren@fb.com> >>> An: "Richard Weinberger" <richard.weinberger@gmail.com>, "Andrew Jeffery" <andrew@aj.id.au> >>> CC: "linux-mtd" <linux-mtd@lists.infradead.org>, "OpenBMC Maillist" <openbmc@lists.ozlabs.org> >>> Gesendet: Montag, 26. August 2019 00:08:08 >>> Betreff: Re: kernel BUG at fs/jffs2/gc.c:395! >> >>> On 8/25/19 12:22 PM, Richard Weinberger wrote: >>>> On Wed, Aug 21, 2019 at 2:06 AM Andrew Jeffery <andrew@aj.id.au> wrote: >>>>> Looks like a lack of robustness to filesystem corruption to me. LWN >>>> >>>> What exactly makes you think so? >>>> The inode cache entry is in state INO_STATE_UNCHECKED while GC run, >>>> which is not allowed. >>>> >>>> Tao, is the error persistent or did it happen only once? >>> >>> Hi Richard, >>> >>> It rarely happens (~1 out of 1000 machines in my environment), but once it >>> happens, it's persistent: the machine will fall into reboot loop due to the >>> crash. >> >> Can you provide me an image of the filesystem such that I can have a look? >> An image where the issue is persistent... > > Hi Richard, > > I tried kernel image with jffs2 summary enabled and disabled, and it looks to > me the result is similar: I can reach login screen now, but the same kernel > panic happens after "reboot" command. > > The behavior is a little different from what I saw yesterday: previously kernel > panic happened at boot time, and now it's after "reboot" command. I guess it's > because more node being written to the flash? > > I understand it's helpful to share the file system image, but unfortunately I > cannot do it because it contains confidential data. Sorry about that.. > > Thank you again for the help, and kindly let me know if you have further > suggestions. Some interesting findings: I checked 3 impacted machines and all of them are caused by the same set of files, and these files have been on the flash for years without modification. - Tao ______________________________________________________ Linux MTD discussion mailing list http://lists.infradead.org/mailman/listinfo/linux-mtd/ ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: kernel BUG at fs/jffs2/gc.c:395! 2019-08-25 19:22 ` Richard Weinberger 2019-08-25 22:08 ` Tao Ren @ 2019-08-26 0:33 ` Andrew Jeffery 2019-08-26 7:42 ` Richard Weinberger 1 sibling, 1 reply; 11+ messages in thread From: Andrew Jeffery @ 2019-08-26 0:33 UTC (permalink / raw) To: Richard Weinberger; +Cc: Tao Ren, OpenBMC Maillist, linux-mtd On Mon, 26 Aug 2019, at 04:53, Richard Weinberger wrote: > On Wed, Aug 21, 2019 at 2:06 AM Andrew Jeffery <andrew@aj.id.au> wrote: > > Looks like a lack of robustness to filesystem corruption to me. LWN > > What exactly makes you think so? It was a bit of guess from a brief inspection of the stack trace (that was probably unhelpful in the overall scheme of things). Sorry for the noise. Andrew ______________________________________________________ Linux MTD discussion mailing list http://lists.infradead.org/mailman/listinfo/linux-mtd/ ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: kernel BUG at fs/jffs2/gc.c:395! 2019-08-26 0:33 ` Andrew Jeffery @ 2019-08-26 7:42 ` Richard Weinberger 0 siblings, 0 replies; 11+ messages in thread From: Richard Weinberger @ 2019-08-26 7:42 UTC (permalink / raw) To: Andrew Jeffery; +Cc: Tao Ren, OpenBMC Maillist, linux-mtd On Mon, Aug 26, 2019 at 2:32 AM Andrew Jeffery <andrew@aj.id.au> wrote: > On Mon, 26 Aug 2019, at 04:53, Richard Weinberger wrote: > > On Wed, Aug 21, 2019 at 2:06 AM Andrew Jeffery <andrew@aj.id.au> wrote: > > > Looks like a lack of robustness to filesystem corruption to me. LWN > > > > What exactly makes you think so? > > It was a bit of guess from a brief inspection of the stack trace (that was probably > unhelpful in the overall scheme of things). Sorry for the noise. No need to worry. :-) Without inspecting the image it is hard to say what the root cause is. But I suspect a runtime bug in jffs2. -- Thanks, //richard ______________________________________________________ Linux MTD discussion mailing list http://lists.infradead.org/mailman/listinfo/linux-mtd/ ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2019-08-28 0:09 UTC | newest] Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-08-20 23:09 kernel BUG at fs/jffs2/gc.c:395! Tao Ren 2019-08-20 23:11 ` Tao Ren 2019-08-21 0:06 ` Andrew Jeffery 2019-08-21 0:20 ` Tao Ren 2019-08-25 19:22 ` Richard Weinberger 2019-08-25 22:08 ` Tao Ren 2019-08-25 22:29 ` Richard Weinberger 2019-08-27 4:42 ` Tao Ren 2019-08-28 0:08 ` [Potential Spoof] " Tao Ren 2019-08-26 0:33 ` Andrew Jeffery 2019-08-26 7:42 ` Richard Weinberger
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).