* kernel BUG at fs/jffs2/gc.c:395!
@ 2019-08-20 23:09 Tao Ren
2019-08-20 23:11 ` Tao Ren
0 siblings, 1 reply; 11+ messages in thread
From: Tao Ren @ 2019-08-20 23:09 UTC (permalink / raw)
To: linux-mtd, OpenBMC Maillist
Hi,
I hit following jffs2 bug while running Linux 5.0.3 on CMM (ASPEED2500) BMC platform. Has anyone seen the issue before? Any suggestions?
[ 46.024017] ------------[ cut here ]------------
[ 46.079178] kernel BUG at /data/users/taoren/openbmc/build-cmm/tmp/work-shared/cmm/kernel-source/fs/jffs2/gc.c:395!
[ 46.204076] Internal error: Oops - BUG: 0 [#1] ARM
[ 46.261378] Modules linked in: ext4 mbcache jbd2 crypto_hash
[ 46.329093] CPU: 0 PID: 1181 Comm: jffs2_gcd_mtd3 Not tainted 5.0.3-cmm #1
[ 46.411343] Hardware name: Generic DT based system
[ 46.468685] PC is at jffs2_garbage_collect_pass+0x6f4/0x734
[ 46.535322] LR is at jffs2_garbage_collect_pass+0x6f4/0x734
[ 46.601977] pc : [<802c292c>] lr : [<802c292c>] psr: 60000013
[ 46.676959] sp : af3cded0 ip : b56a75c0 fp : af3cdf24
[ 46.739463] r10: b4061140 r9 : b57a3900 r8 : b555d4ac
[ 46.801968] r7 : b555d4ac r6 : b403502c r5 : 00000000 r4 : b4035000
[ 46.880073] r3 : b56a75c0 r2 : 00000000 r1 : 00000000 r0 : b403502c
[ 46.958177] Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none
[ 47.043561] Control: 00c5387d Table: b5774008 DAC: 00000051
[ 47.112319] Process jffs2_gcd_mtd3 (pid: 1181, stack limit = 0x54372ffe)
[ 47.192490] Stack: (0xaf3cded0 to 0xaf3ce000)
[ 47.244601] dec0: 00000000 80a07028 800ad6c9 0000ff2c
[ 47.342468] dee0: af3cdefc af3cdef0 80125cd4 8012313c af3cdf24 800ad6c9 8012614c b4035000
[ 47.440331] df00: ffffe000 af3cc000 af3cc000 b4035000 802c509c b419dd18 af3cdf74 af3cdf28
[ 47.538196] df20: 802c5174 802c2244 ffffe000 00000001 00000000 ffffe000 b57b0940 00000000
[ 47.636058] df40: ffffe000 b4035000 802c509c b419dd18 af3cdf74 800ad6c9 b5753980 b5753980
[ 47.733923] df60: b57b0940 00000000 af3cdfac af3cdf78 80134d58 802c50a8 b5753998 b5753998
[ 47.831787] df80: af3cdfac b57b0940 80134c0c 00000000 00000000 00000000 00000000 00000000
[ 47.929648] dfa0: 00000000 af3cdfb0 801010e8 80134c18 00000000 00000000 00000000 00000000
[ 48.027512] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 48.125376] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
[ 48.223230] Backtrace:
[ 48.252489] [<802c2238>] (jffs2_garbage_collect_pass) from [<802c5174>] (jffs2_garbage_collect_thread+0xd8/0x1ac)
[ 48.375294] r10:b419dd18 r9:802c509c r8:b4035000 r7:af3cc000 r6:af3cc000 r5:ffffe000
[ 48.468985] r4:b4035000
[ 48.499281] [<802c509c>] (jffs2_garbage_collect_thread) from [<80134d58>] (kthread+0x14c/0x164)
[ 48.603358] r6:00000000 r5:b57b0940 r4:b5753980
[ 48.658590] [<80134c0c>] (kthread) from [<801010e8>] (ret_from_fork+0x14/0x2c)
[ 48.745001] Exception stack(0xaf3cdfb0 to 0xaf3cdff8)
[ 48.805428] dfa0: 00000000 00000000 00000000 00000000
[ 48.903296] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 49.001157] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000
[ 49.080305] r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:80134c0c
[ 49.174000] r4:b57b0940
[ 49.204275] Code: e59f0044 ebfa25cb e1a00006 eb0e888d (e7f001f2)
[ 49.277188] ---[ end trace 6baa7af0a90d15ab ]---
[ 49.332395] Kernel panic - not syncing: Fatal exception
Thanks,
Tao
______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: kernel BUG at fs/jffs2/gc.c:395!
2019-08-20 23:09 kernel BUG at fs/jffs2/gc.c:395! Tao Ren
@ 2019-08-20 23:11 ` Tao Ren
2019-08-21 0:06 ` Andrew Jeffery
0 siblings, 1 reply; 11+ messages in thread
From: Tao Ren @ 2019-08-20 23:11 UTC (permalink / raw)
To: linux-mtd, OpenBMC Maillist
On 8/20/19 4:09 PM, Tao Ren wrote:
> Hi,
>
> I hit following jffs2 bug while running Linux 5.0.3 on CMM (ASPEED2500) BMC platform. Has anyone seen the issue before? Any suggestions?
>
> [ 46.024017] ------------[ cut here ]------------
> [ 46.079178] kernel BUG at /data/users/taoren/openbmc/build-cmm/tmp/work-shared/cmm/kernel-source/fs/jffs2/gc.c:395!
> [ 46.204076] Internal error: Oops - BUG: 0 [#1] ARM
> [ 46.261378] Modules linked in: ext4 mbcache jbd2 crypto_hash
> [ 46.329093] CPU: 0 PID: 1181 Comm: jffs2_gcd_mtd3 Not tainted 5.0.3-cmm #1
> [ 46.411343] Hardware name: Generic DT based system
> [ 46.468685] PC is at jffs2_garbage_collect_pass+0x6f4/0x734
> [ 46.535322] LR is at jffs2_garbage_collect_pass+0x6f4/0x734
> [ 46.601977] pc : [<802c292c>] lr : [<802c292c>] psr: 60000013
> [ 46.676959] sp : af3cded0 ip : b56a75c0 fp : af3cdf24
> [ 46.739463] r10: b4061140 r9 : b57a3900 r8 : b555d4ac
> [ 46.801968] r7 : b555d4ac r6 : b403502c r5 : 00000000 r4 : b4035000
> [ 46.880073] r3 : b56a75c0 r2 : 00000000 r1 : 00000000 r0 : b403502c
> [ 46.958177] Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none
> [ 47.043561] Control: 00c5387d Table: b5774008 DAC: 00000051
> [ 47.112319] Process jffs2_gcd_mtd3 (pid: 1181, stack limit = 0x54372ffe)
> [ 47.192490] Stack: (0xaf3cded0 to 0xaf3ce000)
> [ 47.244601] dec0: 00000000 80a07028 800ad6c9 0000ff2c
> [ 47.342468] dee0: af3cdefc af3cdef0 80125cd4 8012313c af3cdf24 800ad6c9 8012614c b4035000
> [ 47.440331] df00: ffffe000 af3cc000 af3cc000 b4035000 802c509c b419dd18 af3cdf74 af3cdf28
> [ 47.538196] df20: 802c5174 802c2244 ffffe000 00000001 00000000 ffffe000 b57b0940 00000000
> [ 47.636058] df40: ffffe000 b4035000 802c509c b419dd18 af3cdf74 800ad6c9 b5753980 b5753980
> [ 47.733923] df60: b57b0940 00000000 af3cdfac af3cdf78 80134d58 802c50a8 b5753998 b5753998
> [ 47.831787] df80: af3cdfac b57b0940 80134c0c 00000000 00000000 00000000 00000000 00000000
> [ 47.929648] dfa0: 00000000 af3cdfb0 801010e8 80134c18 00000000 00000000 00000000 00000000
> [ 48.027512] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
> [ 48.125376] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
> [ 48.223230] Backtrace:
> [ 48.252489] [<802c2238>] (jffs2_garbage_collect_pass) from [<802c5174>] (jffs2_garbage_collect_thread+0xd8/0x1ac)
> [ 48.375294] r10:b419dd18 r9:802c509c r8:b4035000 r7:af3cc000 r6:af3cc000 r5:ffffe000
> [ 48.468985] r4:b4035000
> [ 48.499281] [<802c509c>] (jffs2_garbage_collect_thread) from [<80134d58>] (kthread+0x14c/0x164)
> [ 48.603358] r6:00000000 r5:b57b0940 r4:b5753980
> [ 48.658590] [<80134c0c>] (kthread) from [<801010e8>] (ret_from_fork+0x14/0x2c)
> [ 48.745001] Exception stack(0xaf3cdfb0 to 0xaf3cdff8)
> [ 48.805428] dfa0: 00000000 00000000 00000000 00000000
> [ 48.903296] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
> [ 49.001157] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000
> [ 49.080305] r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:80134c0c
> [ 49.174000] r4:b57b0940
> [ 49.204275] Code: e59f0044 ebfa25cb e1a00006 eb0e888d (e7f001f2)
> [ 49.277188] ---[ end trace 6baa7af0a90d15ab ]---
> [ 49.332395] Kernel panic - not syncing: Fatal exception
BTW, below are all the messages printed by jffs2 before system crash:
[ 21.078433] jffs2: version 2.2. (SUMMARY) © 2001-2006 Red Hat, Inc.
[ 39.776207] jffs2: notice: (1180) jffs2_build_xattr_subsystem: complete building xattr subsystem, 0 of xdatum (0 unchecked, 0 orphan) and 0 of xref (0 dead, 0 orphan) found.
[ 40.016574] jffs2: warning: (1181) jffs2_do_read_inode_internal: no data nodes found for ino #140
[ 40.122964] jffs2: notice: (1181) jffs2_do_read_inode_internal: but it has children so we fake some modes for it
[ 43.579361] jffs2: warning: (1181) jffs2_get_inode_nodes: Eep. No valid nodes for ino #141.
[ 43.679404] jffs2: warning: (1181) jffs2_do_read_inode_internal: no data nodes found for ino #141
[ 43.785661] jffs2: Returned error for crccheck of ino #141. Expect badness...
[ 44.021825] jffs2: warning: (1181) jffs2_do_read_inode_internal: no data nodes found for ino #154
[ 44.128191] jffs2: notice: (1181) jffs2_do_read_inode_internal: but it has children so we fake some modes for it
[ 44.314862] jffs2: warning: (1181) jffs2_do_read_inode_internal: no data nodes found for ino #155
[ 44.421152] jffs2: notice: (1181) jffs2_do_read_inode_internal: but it has children so we fake some modes for it
[ 44.607378] jffs2: warning: (1181) jffs2_do_read_inode_internal: no data nodes found for ino #163
[ 44.713692] jffs2: notice: (1181) jffs2_do_read_inode_internal: but it has children so we fake some modes for it
[ 44.899991] jffs2: warning: (1181) jffs2_get_inode_nodes: Eep. No valid nodes for ino #164.
[ 45.000107] jffs2: warning: (1181) jffs2_do_read_inode_internal: no data nodes found for ino #164
[ 45.106370] jffs2: Returned error for crccheck of ino #164. Expect badness...
[ 45.934282] jffs2: Inode #106 already in state 0 in jffs2_garbage_collect_pass()!
Thanks,
Tao
______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: kernel BUG at fs/jffs2/gc.c:395!
2019-08-20 23:11 ` Tao Ren
@ 2019-08-21 0:06 ` Andrew Jeffery
2019-08-21 0:20 ` Tao Ren
2019-08-25 19:22 ` Richard Weinberger
0 siblings, 2 replies; 11+ messages in thread
From: Andrew Jeffery @ 2019-08-21 0:06 UTC (permalink / raw)
To: Tao Ren, linux-mtd, OpenBMC Maillist
On Wed, 21 Aug 2019, at 08:42, Tao Ren wrote:
> On 8/20/19 4:09 PM, Tao Ren wrote:
> > Hi,
> >
> > I hit following jffs2 bug while running Linux 5.0.3 on CMM (ASPEED2500) BMC platform. Has anyone seen the issue before? Any suggestions?
> >
> > [ 46.024017] ------------[ cut here ]------------
> > [ 46.079178] kernel BUG at /data/users/taoren/openbmc/build-cmm/tmp/work-shared/cmm/kernel-source/fs/jffs2/gc.c:395!
> > [ 46.204076] Internal error: Oops - BUG: 0 [#1] ARM
> > [ 46.261378] Modules linked in: ext4 mbcache jbd2 crypto_hash
> > [ 46.329093] CPU: 0 PID: 1181 Comm: jffs2_gcd_mtd3 Not tainted 5.0.3-cmm #1
> > [ 46.411343] Hardware name: Generic DT based system
> > [ 46.468685] PC is at jffs2_garbage_collect_pass+0x6f4/0x734
> > [ 46.535322] LR is at jffs2_garbage_collect_pass+0x6f4/0x734
> > [ 46.601977] pc : [<802c292c>] lr : [<802c292c>] psr: 60000013
> > [ 46.676959] sp : af3cded0 ip : b56a75c0 fp : af3cdf24
> > [ 46.739463] r10: b4061140 r9 : b57a3900 r8 : b555d4ac
> > [ 46.801968] r7 : b555d4ac r6 : b403502c r5 : 00000000 r4 : b4035000
> > [ 46.880073] r3 : b56a75c0 r2 : 00000000 r1 : 00000000 r0 : b403502c
> > [ 46.958177] Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none
> > [ 47.043561] Control: 00c5387d Table: b5774008 DAC: 00000051
> > [ 47.112319] Process jffs2_gcd_mtd3 (pid: 1181, stack limit = 0x54372ffe)
> > [ 47.192490] Stack: (0xaf3cded0 to 0xaf3ce000)
> > [ 47.244601] dec0: 00000000 80a07028 800ad6c9 0000ff2c
> > [ 47.342468] dee0: af3cdefc af3cdef0 80125cd4 8012313c af3cdf24 800ad6c9 8012614c b4035000
> > [ 47.440331] df00: ffffe000 af3cc000 af3cc000 b4035000 802c509c b419dd18 af3cdf74 af3cdf28
> > [ 47.538196] df20: 802c5174 802c2244 ffffe000 00000001 00000000 ffffe000 b57b0940 00000000
> > [ 47.636058] df40: ffffe000 b4035000 802c509c b419dd18 af3cdf74 800ad6c9 b5753980 b5753980
> > [ 47.733923] df60: b57b0940 00000000 af3cdfac af3cdf78 80134d58 802c50a8 b5753998 b5753998
> > [ 47.831787] df80: af3cdfac b57b0940 80134c0c 00000000 00000000 00000000 00000000 00000000
> > [ 47.929648] dfa0: 00000000 af3cdfb0 801010e8 80134c18 00000000 00000000 00000000 00000000
> > [ 48.027512] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
> > [ 48.125376] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
> > [ 48.223230] Backtrace:
> > [ 48.252489] [<802c2238>] (jffs2_garbage_collect_pass) from [<802c5174>] (jffs2_garbage_collect_thread+0xd8/0x1ac)
> > [ 48.375294] r10:b419dd18 r9:802c509c r8:b4035000 r7:af3cc000 r6:af3cc000 r5:ffffe000
> > [ 48.468985] r4:b4035000
> > [ 48.499281] [<802c509c>] (jffs2_garbage_collect_thread) from [<80134d58>] (kthread+0x14c/0x164)
> > [ 48.603358] r6:00000000 r5:b57b0940 r4:b5753980
> > [ 48.658590] [<80134c0c>] (kthread) from [<801010e8>] (ret_from_fork+0x14/0x2c)
> > [ 48.745001] Exception stack(0xaf3cdfb0 to 0xaf3cdff8)
> > [ 48.805428] dfa0: 00000000 00000000 00000000 00000000
> > [ 48.903296] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
> > [ 49.001157] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000
> > [ 49.080305] r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:80134c0c
> > [ 49.174000] r4:b57b0940
> > [ 49.204275] Code: e59f0044 ebfa25cb e1a00006 eb0e888d (e7f001f2)
> > [ 49.277188] ---[ end trace 6baa7af0a90d15ab ]---
> > [ 49.332395] Kernel panic - not syncing: Fatal exception
>
> BTW, below are all the messages printed by jffs2 before system crash:
>
> [ 21.078433] jffs2: version 2.2. (SUMMARY) © 2001-2006 Red Hat, Inc.
> [ 39.776207] jffs2: notice: (1180) jffs2_build_xattr_subsystem:
> complete building xattr subsystem, 0 of xdatum (0 unchecked, 0 orphan)
> and 0 of xref (0 dead, 0 orphan) found.
> [ 40.016574] jffs2: warning: (1181) jffs2_do_read_inode_internal: no
> data nodes found for ino #140
> [ 40.122964] jffs2: notice: (1181) jffs2_do_read_inode_internal: but
> it has children so we fake some modes for it
> [ 43.579361] jffs2: warning: (1181) jffs2_get_inode_nodes: Eep. No
> valid nodes for ino #141.
> [ 43.679404] jffs2: warning: (1181) jffs2_do_read_inode_internal: no
> data nodes found for ino #141
> [ 43.785661] jffs2: Returned error for crccheck of ino #141. Expect
> badness...
> [ 44.021825] jffs2: warning: (1181) jffs2_do_read_inode_internal: no
> data nodes found for ino #154
> [ 44.128191] jffs2: notice: (1181) jffs2_do_read_inode_internal: but
> it has children so we fake some modes for it
> [ 44.314862] jffs2: warning: (1181) jffs2_do_read_inode_internal: no
> data nodes found for ino #155
> [ 44.421152] jffs2: notice: (1181) jffs2_do_read_inode_internal: but
> it has children so we fake some modes for it
> [ 44.607378] jffs2: warning: (1181) jffs2_do_read_inode_internal: no
> data nodes found for ino #163
> [ 44.713692] jffs2: notice: (1181) jffs2_do_read_inode_internal: but
> it has children so we fake some modes for it
> [ 44.899991] jffs2: warning: (1181) jffs2_get_inode_nodes: Eep. No
> valid nodes for ino #164.
> [ 45.000107] jffs2: warning: (1181) jffs2_do_read_inode_internal: no
> data nodes found for ino #164
> [ 45.106370] jffs2: Returned error for crccheck of ino #164. Expect
> badness...
> [ 45.934282] jffs2: Inode #106 already in state 0 in
> jffs2_garbage_collect_pass()!
Looks like a lack of robustness to filesystem corruption to me. LWN
published an article on the topic just yesterday!
https://lwn.net/Articles/796687/
Andrew
______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: kernel BUG at fs/jffs2/gc.c:395!
2019-08-21 0:06 ` Andrew Jeffery
@ 2019-08-21 0:20 ` Tao Ren
2019-08-25 19:22 ` Richard Weinberger
1 sibling, 0 replies; 11+ messages in thread
From: Tao Ren @ 2019-08-21 0:20 UTC (permalink / raw)
To: Andrew Jeffery, linux-mtd, OpenBMC Maillist
On 8/20/19 5:06 PM, Andrew Jeffery wrote:
>
>
> On Wed, 21 Aug 2019, at 08:42, Tao Ren wrote:
>> On 8/20/19 4:09 PM, Tao Ren wrote:
>>> Hi,
>>>
>>> I hit following jffs2 bug while running Linux 5.0.3 on CMM (ASPEED2500) BMC platform. Has anyone seen the issue before? Any suggestions?
>>>
>>> [ 46.024017] ------------[ cut here ]------------
>>> [ 46.079178] kernel BUG at /data/users/taoren/openbmc/build-cmm/tmp/work-shared/cmm/kernel-source/fs/jffs2/gc.c:395!
>>> [ 46.204076] Internal error: Oops - BUG: 0 [#1] ARM
>>> [ 46.261378] Modules linked in: ext4 mbcache jbd2 crypto_hash
>>> [ 46.329093] CPU: 0 PID: 1181 Comm: jffs2_gcd_mtd3 Not tainted 5.0.3-cmm #1
>>> [ 46.411343] Hardware name: Generic DT based system
>>> [ 46.468685] PC is at jffs2_garbage_collect_pass+0x6f4/0x734
>>> [ 46.535322] LR is at jffs2_garbage_collect_pass+0x6f4/0x734
>>> [ 46.601977] pc : [<802c292c>] lr : [<802c292c>] psr: 60000013
>>> [ 46.676959] sp : af3cded0 ip : b56a75c0 fp : af3cdf24
>>> [ 46.739463] r10: b4061140 r9 : b57a3900 r8 : b555d4ac
>>> [ 46.801968] r7 : b555d4ac r6 : b403502c r5 : 00000000 r4 : b4035000
>>> [ 46.880073] r3 : b56a75c0 r2 : 00000000 r1 : 00000000 r0 : b403502c
>>> [ 46.958177] Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none
>>> [ 47.043561] Control: 00c5387d Table: b5774008 DAC: 00000051
>>> [ 47.112319] Process jffs2_gcd_mtd3 (pid: 1181, stack limit = 0x54372ffe)
>>> [ 47.192490] Stack: (0xaf3cded0 to 0xaf3ce000)
>>> [ 47.244601] dec0: 00000000 80a07028 800ad6c9 0000ff2c
>>> [ 47.342468] dee0: af3cdefc af3cdef0 80125cd4 8012313c af3cdf24 800ad6c9 8012614c b4035000
>>> [ 47.440331] df00: ffffe000 af3cc000 af3cc000 b4035000 802c509c b419dd18 af3cdf74 af3cdf28
>>> [ 47.538196] df20: 802c5174 802c2244 ffffe000 00000001 00000000 ffffe000 b57b0940 00000000
>>> [ 47.636058] df40: ffffe000 b4035000 802c509c b419dd18 af3cdf74 800ad6c9 b5753980 b5753980
>>> [ 47.733923] df60: b57b0940 00000000 af3cdfac af3cdf78 80134d58 802c50a8 b5753998 b5753998
>>> [ 47.831787] df80: af3cdfac b57b0940 80134c0c 00000000 00000000 00000000 00000000 00000000
>>> [ 47.929648] dfa0: 00000000 af3cdfb0 801010e8 80134c18 00000000 00000000 00000000 00000000
>>> [ 48.027512] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
>>> [ 48.125376] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
>>> [ 48.223230] Backtrace:
>>> [ 48.252489] [<802c2238>] (jffs2_garbage_collect_pass) from [<802c5174>] (jffs2_garbage_collect_thread+0xd8/0x1ac)
>>> [ 48.375294] r10:b419dd18 r9:802c509c r8:b4035000 r7:af3cc000 r6:af3cc000 r5:ffffe000
>>> [ 48.468985] r4:b4035000
>>> [ 48.499281] [<802c509c>] (jffs2_garbage_collect_thread) from [<80134d58>] (kthread+0x14c/0x164)
>>> [ 48.603358] r6:00000000 r5:b57b0940 r4:b5753980
>>> [ 48.658590] [<80134c0c>] (kthread) from [<801010e8>] (ret_from_fork+0x14/0x2c)
>>> [ 48.745001] Exception stack(0xaf3cdfb0 to 0xaf3cdff8)
>>> [ 48.805428] dfa0: 00000000 00000000 00000000 00000000
>>> [ 48.903296] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
>>> [ 49.001157] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000
>>> [ 49.080305] r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:80134c0c
>>> [ 49.174000] r4:b57b0940
>>> [ 49.204275] Code: e59f0044 ebfa25cb e1a00006 eb0e888d (e7f001f2)
>>> [ 49.277188] ---[ end trace 6baa7af0a90d15ab ]---
>>> [ 49.332395] Kernel panic - not syncing: Fatal exception
>>
>> BTW, below are all the messages printed by jffs2 before system crash:
>>
>> [ 21.078433] jffs2: version 2.2. (SUMMARY) © 2001-2006 Red Hat, Inc.
>> [ 39.776207] jffs2: notice: (1180) jffs2_build_xattr_subsystem:
>> complete building xattr subsystem, 0 of xdatum (0 unchecked, 0 orphan)
>> and 0 of xref (0 dead, 0 orphan) found.
>> [ 40.016574] jffs2: warning: (1181) jffs2_do_read_inode_internal: no
>> data nodes found for ino #140
>> [ 40.122964] jffs2: notice: (1181) jffs2_do_read_inode_internal: but
>> it has children so we fake some modes for it
>> [ 43.579361] jffs2: warning: (1181) jffs2_get_inode_nodes: Eep. No
>> valid nodes for ino #141.
>> [ 43.679404] jffs2: warning: (1181) jffs2_do_read_inode_internal: no
>> data nodes found for ino #141
>> [ 43.785661] jffs2: Returned error for crccheck of ino #141. Expect
>> badness...
>> [ 44.021825] jffs2: warning: (1181) jffs2_do_read_inode_internal: no
>> data nodes found for ino #154
>> [ 44.128191] jffs2: notice: (1181) jffs2_do_read_inode_internal: but
>> it has children so we fake some modes for it
>> [ 44.314862] jffs2: warning: (1181) jffs2_do_read_inode_internal: no
>> data nodes found for ino #155
>> [ 44.421152] jffs2: notice: (1181) jffs2_do_read_inode_internal: but
>> it has children so we fake some modes for it
>> [ 44.607378] jffs2: warning: (1181) jffs2_do_read_inode_internal: no
>> data nodes found for ino #163
>> [ 44.713692] jffs2: notice: (1181) jffs2_do_read_inode_internal: but
>> it has children so we fake some modes for it
>> [ 44.899991] jffs2: warning: (1181) jffs2_get_inode_nodes: Eep. No
>> valid nodes for ino #164.
>> [ 45.000107] jffs2: warning: (1181) jffs2_do_read_inode_internal: no
>> data nodes found for ino #164
>> [ 45.106370] jffs2: Returned error for crccheck of ino #164. Expect
>> badness...
>> [ 45.934282] jffs2: Inode #106 already in state 0 in
>> jffs2_garbage_collect_pass()!
>
> Looks like a lack of robustness to filesystem corruption to me. LWN
> published an article on the topic just yesterday!
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lwn.net_Articles_796687_&d=DwIFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=iYElT7HC77pRZ3byVvW8ng&m=VI7s7FnQP0Ooe6tEJPtpjajwGDNh4FhQjTgiRbW8hmk&s=obrUgP3wo7zCtMTk8cnEMXhxaF9VSJmy1iJeSxKHmt8&e=
>
> Andrew
Thank you for the sharing, Andrew. Let me check out the article..
Cheers,
Tao
______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: kernel BUG at fs/jffs2/gc.c:395!
2019-08-21 0:06 ` Andrew Jeffery
2019-08-21 0:20 ` Tao Ren
@ 2019-08-25 19:22 ` Richard Weinberger
2019-08-25 22:08 ` Tao Ren
2019-08-26 0:33 ` Andrew Jeffery
1 sibling, 2 replies; 11+ messages in thread
From: Richard Weinberger @ 2019-08-25 19:22 UTC (permalink / raw)
To: Andrew Jeffery; +Cc: Tao Ren, OpenBMC Maillist, linux-mtd
On Wed, Aug 21, 2019 at 2:06 AM Andrew Jeffery <andrew@aj.id.au> wrote:
> Looks like a lack of robustness to filesystem corruption to me. LWN
What exactly makes you think so?
The inode cache entry is in state INO_STATE_UNCHECKED while GC run,
which is not allowed.
Tao, is the error persistent or did it happen only once?
--
Thanks,
//richard
______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: kernel BUG at fs/jffs2/gc.c:395!
2019-08-25 19:22 ` Richard Weinberger
@ 2019-08-25 22:08 ` Tao Ren
2019-08-25 22:29 ` Richard Weinberger
2019-08-26 0:33 ` Andrew Jeffery
1 sibling, 1 reply; 11+ messages in thread
From: Tao Ren @ 2019-08-25 22:08 UTC (permalink / raw)
To: Richard Weinberger, Andrew Jeffery; +Cc: OpenBMC Maillist, linux-mtd
On 8/25/19 12:22 PM, Richard Weinberger wrote:
> On Wed, Aug 21, 2019 at 2:06 AM Andrew Jeffery <andrew@aj.id.au> wrote:
>> Looks like a lack of robustness to filesystem corruption to me. LWN
>
> What exactly makes you think so?
> The inode cache entry is in state INO_STATE_UNCHECKED while GC run,
> which is not allowed.
>
> Tao, is the error persistent or did it happen only once?
Hi Richard,
It rarely happens (~1 out of 1000 machines in my environment), but once it happens, it's persistent: the machine will fall into reboot loop due to the crash.
Thanks,
Tao
______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: kernel BUG at fs/jffs2/gc.c:395!
2019-08-25 22:08 ` Tao Ren
@ 2019-08-25 22:29 ` Richard Weinberger
2019-08-27 4:42 ` Tao Ren
0 siblings, 1 reply; 11+ messages in thread
From: Richard Weinberger @ 2019-08-25 22:29 UTC (permalink / raw)
To: Tao Ren; +Cc: Andrew Jeffery, OpenBMC Maillist, linux-mtd
----- Ursprüngliche Mail -----
> Von: "Tao Ren" <taoren@fb.com>
> An: "Richard Weinberger" <richard.weinberger@gmail.com>, "Andrew Jeffery" <andrew@aj.id.au>
> CC: "linux-mtd" <linux-mtd@lists.infradead.org>, "OpenBMC Maillist" <openbmc@lists.ozlabs.org>
> Gesendet: Montag, 26. August 2019 00:08:08
> Betreff: Re: kernel BUG at fs/jffs2/gc.c:395!
> On 8/25/19 12:22 PM, Richard Weinberger wrote:
>> On Wed, Aug 21, 2019 at 2:06 AM Andrew Jeffery <andrew@aj.id.au> wrote:
>>> Looks like a lack of robustness to filesystem corruption to me. LWN
>>
>> What exactly makes you think so?
>> The inode cache entry is in state INO_STATE_UNCHECKED while GC run,
>> which is not allowed.
>>
>> Tao, is the error persistent or did it happen only once?
>
> Hi Richard,
>
> It rarely happens (~1 out of 1000 machines in my environment), but once it
> happens, it's persistent: the machine will fall into reboot loop due to the
> crash.
Can you provide me an image of the filesystem such that I can have a look?
An image where the issue is persistent...
Thanks,
//richard
______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: kernel BUG at fs/jffs2/gc.c:395!
2019-08-25 19:22 ` Richard Weinberger
2019-08-25 22:08 ` Tao Ren
@ 2019-08-26 0:33 ` Andrew Jeffery
2019-08-26 7:42 ` Richard Weinberger
1 sibling, 1 reply; 11+ messages in thread
From: Andrew Jeffery @ 2019-08-26 0:33 UTC (permalink / raw)
To: Richard Weinberger; +Cc: Tao Ren, OpenBMC Maillist, linux-mtd
On Mon, 26 Aug 2019, at 04:53, Richard Weinberger wrote:
> On Wed, Aug 21, 2019 at 2:06 AM Andrew Jeffery <andrew@aj.id.au> wrote:
> > Looks like a lack of robustness to filesystem corruption to me. LWN
>
> What exactly makes you think so?
It was a bit of guess from a brief inspection of the stack trace (that was probably
unhelpful in the overall scheme of things). Sorry for the noise.
Andrew
______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: kernel BUG at fs/jffs2/gc.c:395!
2019-08-26 0:33 ` Andrew Jeffery
@ 2019-08-26 7:42 ` Richard Weinberger
0 siblings, 0 replies; 11+ messages in thread
From: Richard Weinberger @ 2019-08-26 7:42 UTC (permalink / raw)
To: Andrew Jeffery; +Cc: Tao Ren, OpenBMC Maillist, linux-mtd
On Mon, Aug 26, 2019 at 2:32 AM Andrew Jeffery <andrew@aj.id.au> wrote:
> On Mon, 26 Aug 2019, at 04:53, Richard Weinberger wrote:
> > On Wed, Aug 21, 2019 at 2:06 AM Andrew Jeffery <andrew@aj.id.au> wrote:
> > > Looks like a lack of robustness to filesystem corruption to me. LWN
> >
> > What exactly makes you think so?
>
> It was a bit of guess from a brief inspection of the stack trace (that was probably
> unhelpful in the overall scheme of things). Sorry for the noise.
No need to worry. :-)
Without inspecting the image it is hard to say what the root cause is.
But I suspect a runtime bug in jffs2.
--
Thanks,
//richard
______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: kernel BUG at fs/jffs2/gc.c:395!
2019-08-25 22:29 ` Richard Weinberger
@ 2019-08-27 4:42 ` Tao Ren
2019-08-28 0:08 ` [Potential Spoof] " Tao Ren
0 siblings, 1 reply; 11+ messages in thread
From: Tao Ren @ 2019-08-27 4:42 UTC (permalink / raw)
To: Richard Weinberger; +Cc: Andrew Jeffery, OpenBMC Maillist, linux-mtd
On 8/25/19 3:29 PM, Richard Weinberger wrote:
> ----- Ursprüngliche Mail -----
>> Von: "Tao Ren" <taoren@fb.com>
>> An: "Richard Weinberger" <richard.weinberger@gmail.com>, "Andrew Jeffery" <andrew@aj.id.au>
>> CC: "linux-mtd" <linux-mtd@lists.infradead.org>, "OpenBMC Maillist" <openbmc@lists.ozlabs.org>
>> Gesendet: Montag, 26. August 2019 00:08:08
>> Betreff: Re: kernel BUG at fs/jffs2/gc.c:395!
>
>> On 8/25/19 12:22 PM, Richard Weinberger wrote:
>>> On Wed, Aug 21, 2019 at 2:06 AM Andrew Jeffery <andrew@aj.id.au> wrote:
>>>> Looks like a lack of robustness to filesystem corruption to me. LWN
>>>
>>> What exactly makes you think so?
>>> The inode cache entry is in state INO_STATE_UNCHECKED while GC run,
>>> which is not allowed.
>>>
>>> Tao, is the error persistent or did it happen only once?
>>
>> Hi Richard,
>>
>> It rarely happens (~1 out of 1000 machines in my environment), but once it
>> happens, it's persistent: the machine will fall into reboot loop due to the
>> crash.
>
> Can you provide me an image of the filesystem such that I can have a look?
> An image where the issue is persistent...
Hi Richard,
I tried kernel image with jffs2 summary enabled and disabled, and it looks to
me the result is similar: I can reach login screen now, but the same kernel
panic happens after "reboot" command.
The behavior is a little different from what I saw yesterday: previously kernel
panic happened at boot time, and now it's after "reboot" command. I guess it's
because more node being written to the flash?
I understand it's helpful to share the file system image, but unfortunately I
cannot do it because it contains confidential data. Sorry about that..
Thank you again for the help, and kindly let me know if you have further
suggestions.
Cheers,
Tao
______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Potential Spoof] Re: kernel BUG at fs/jffs2/gc.c:395!
2019-08-27 4:42 ` Tao Ren
@ 2019-08-28 0:08 ` Tao Ren
0 siblings, 0 replies; 11+ messages in thread
From: Tao Ren @ 2019-08-28 0:08 UTC (permalink / raw)
To: Richard Weinberger; +Cc: Andrew Jeffery, OpenBMC Maillist, linux-mtd
On 8/26/19 9:42 PM, Tao Ren wrote:
> On 8/25/19 3:29 PM, Richard Weinberger wrote:
>> ----- Ursprüngliche Mail -----
>>> Von: "Tao Ren" <taoren@fb.com>
>>> An: "Richard Weinberger" <richard.weinberger@gmail.com>, "Andrew Jeffery" <andrew@aj.id.au>
>>> CC: "linux-mtd" <linux-mtd@lists.infradead.org>, "OpenBMC Maillist" <openbmc@lists.ozlabs.org>
>>> Gesendet: Montag, 26. August 2019 00:08:08
>>> Betreff: Re: kernel BUG at fs/jffs2/gc.c:395!
>>
>>> On 8/25/19 12:22 PM, Richard Weinberger wrote:
>>>> On Wed, Aug 21, 2019 at 2:06 AM Andrew Jeffery <andrew@aj.id.au> wrote:
>>>>> Looks like a lack of robustness to filesystem corruption to me. LWN
>>>>
>>>> What exactly makes you think so?
>>>> The inode cache entry is in state INO_STATE_UNCHECKED while GC run,
>>>> which is not allowed.
>>>>
>>>> Tao, is the error persistent or did it happen only once?
>>>
>>> Hi Richard,
>>>
>>> It rarely happens (~1 out of 1000 machines in my environment), but once it
>>> happens, it's persistent: the machine will fall into reboot loop due to the
>>> crash.
>>
>> Can you provide me an image of the filesystem such that I can have a look?
>> An image where the issue is persistent...
>
> Hi Richard,
>
> I tried kernel image with jffs2 summary enabled and disabled, and it looks to
> me the result is similar: I can reach login screen now, but the same kernel
> panic happens after "reboot" command.
>
> The behavior is a little different from what I saw yesterday: previously kernel
> panic happened at boot time, and now it's after "reboot" command. I guess it's
> because more node being written to the flash?
>
> I understand it's helpful to share the file system image, but unfortunately I
> cannot do it because it contains confidential data. Sorry about that..
>
> Thank you again for the help, and kindly let me know if you have further
> suggestions.
Some interesting findings: I checked 3 impacted machines and all of them are
caused by the same set of files, and these files have been on the flash for
years without modification.
- Tao
______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2019-08-28 0:09 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-08-20 23:09 kernel BUG at fs/jffs2/gc.c:395! Tao Ren
2019-08-20 23:11 ` Tao Ren
2019-08-21 0:06 ` Andrew Jeffery
2019-08-21 0:20 ` Tao Ren
2019-08-25 19:22 ` Richard Weinberger
2019-08-25 22:08 ` Tao Ren
2019-08-25 22:29 ` Richard Weinberger
2019-08-27 4:42 ` Tao Ren
2019-08-28 0:08 ` [Potential Spoof] " Tao Ren
2019-08-26 0:33 ` Andrew Jeffery
2019-08-26 7:42 ` Richard Weinberger
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).