NAND Flash Issue Need Help!

* NAND Flash Issue Need Help!
@ 2021-03-05 18:38 Justin Mitchell
  2021-03-05 19:36 ` Willy Tarreau
  0 siblings, 1 reply; 3+ messages in thread
From: Justin Mitchell @ 2021-03-05 18:38 UTC (permalink / raw)
  To: linux-kernel

I have a bug that keeps coming back! We use CRE9G25 (Corwind module with Mircon Flash) that is using linux 4.4. We upgraded from linux 3.16 back in 2018. Please see issue below, any feedback would be greatly appreciate. I don't want to go thru hassle of upgrading linux  and introduce new issues (months of development) if there is another path to fix. If it is a known issue and an upgrade will fix, then so be it. 
Thank you!

Issue:
Intermittent occurrence of failure to program new boards from CM. Occurs in approximately 1 out of 10 boards from our CM going through conversion. The issue seems to occur when the SYNC portion of the USB update that is setting up the backup partition encounters some number of bad nodes on the flash. After going through programming, the unit will boot, but on the NEXT power cycle, it finds another bad node, error reading kernel/uImage, and gets stuck at u-boot. The attached logs show the UBIFS failures. (Current hardware is corewind with micron Nand Flash)

Primary partition mounted OK
Loading file 'dtb/at91sam9g25ek.dtb' to addr 0x21000000 with size 25836 (0x000064ec)...
Done
DTB file loaded OK
Loading file 'kernel/uImage' to addr 0x22000000 with size 2968536 (0x002d4bd8)...
UBIFS error (pid 0): ubifs_read_node: bad node type (255 but expected 1) UBIFS error (pid 0): ubifs_read_node: bad node at LEB 215:0 UBIFS error (pid 0): do_readpage: cannot read page 178 of inode 1244, error -22 Error reading file 'kernel/uImage'
kernel/uImage not found!

But if it will do ubifsls for SystremA or SystemB all files and folders are there:

U-Boot> ubifsls
<DIR>        5688  Tue Aug 11 13:48:46 2020  bin
<DIR>         352  Sat Jul 02 03:48:56 2016  dev
<DIR>         320  Mon Jan 04 15:59:27 2021  dtb
<DIR>        3848  Wed Jan 27 18:13:08 2021  etc
<DIR>        3184  Mon Jan 04 15:59:27 2021  lib
<DIR>         232  Wed Jan 27 18:11:22 2021  mnt
<DIR>         160  Sat Jul 02 03:48:56 2016  opt
<DIR>         288  Wed Jan 27 18:11:20 2021  run
<DIR>         224  Mon Nov 11 20:54:43 2019  tmp
<DIR>         160  Sat Jul 02 03:48:56 2016  sys
<DIR>         864  Mon Jan 04 15:59:27 2021  var
<DIR>         672  Mon Jan 04 15:59:27 2021  usr
<DIR>         160  Sat Jul 02 03:48:56 2016  proc
<DIR>        4312  Tue Aug 11 13:48:46 2020  sbin
<DIR>         296  Mon Jan 04 15:59:28 2021  root
<LNK>          11  Mon Nov 11 20:33:46 2019  linuxrc
<DIR>         224  Mon Jan 04 15:59:27 2021  kernel
<LNK>           3  Mon Nov 11 20:27:44 2019  lib32
<DIR>         160  Sat Jul 02 03:48:56 2016  media

I can even see uImage there:

U-Boot> ubifsls kernel
          2968536  Mon Jan 04 15:59:27 2021  uImage

If I will do ubifscat kernel/uImage it starts to read the file and then I see this error:

P▒@ ▒▒▒P▒@ ▒▒▒P▒@ ▒▒Q▒▒ ▒
aB Q▒'
0▒▒0`BS▒&▒(
▒ ▒ ▒Q▒Q1▒1"▒1▒▒▒:Q▒Q1▒1 ▒1▒▒▒:▒▒S▒0C !▒S▒0C ▒!!S▒!1C "!▒S▒1C ▒!S▒""▒!▒▒▒▒\▒`B▒▒<▒`B▒▒3̠▒▒Q▒!▒! ▒# ▒3Q▒!▒! "Q▒!▒! "Q▒ ▒ \▒3▒▒`B▒▒Q▒> aB▒▒`B Q▒P▒& ▒▒Q▒Q1▒1 2▒▒▒:Q▒Q1▒1 2▒▒▒: R▒P▒@ ▒P▒@ !P▒!@ ▒P▒@ P▒!▒▒ R▒▒▒▒▒▒0

r▒
P▒@ ▒▒▒P▒@ ▒▒▒P▒@ \▒`B▒▒P-▒▒▒▒P▒▒▒A▒▒▒P-▒|▒▒▒P▒▒▒A▒▒▒▒-▒▒▒▒▒▒▒ 0R▒ ▒b▒A▒Q0A▒▒▒▒`8 ▒#4▒▒▒<▒▒`#▒▒▒▒▒l8,▒` ▒!▒▒#4▒▒▒<▒▒▒▒▒`!▒l#▒▒▒▒▒ #+3;CScs▒▒▒@@!1Aa▒ 0@``Psp0▒ ` ▒@▒X;x8▒h(▒H▒T▒+t4▒d$▒D▒\S|<▒l,▒L▒R▒#r2▒b"▒B▒ZCz:▒j*▒
J▒V@3v6▒f&▒F▒^c~>▒n.▒N▒`Qq1▒
a!▒A▒Y;y9▒i)▒I▒U+u5▒e%▒E▒]S}=▒m-▒M▒S▒#s3▒c#▒C▒[C{;▒k+▒K▒W@3w7▒g'▒G▒_c?▒o/▒O▒`Psp0▒
` ▒@▒X;x8▒h(▒H▒T▒+t4▒d$▒D▒\S|<▒l,▒L▒R▒#r2▒b"▒B▒ZCz:▒j*▒
J▒V@3v6▒f&▒F▒^c~>▒n.▒N▒`Qq1▒
a!▒A▒Y;y9▒i)▒I▒U+u5▒e%▒E▒]S}=▒m-▒M▒S▒#s3▒c#▒C▒[C{;▒k+▒K▒W@3w7▒g'▒G▒_c?▒o/▒O▒A@! @a`10▒@

-- System haltedAttempting division by 0!stack-protector: Kernel stack is corrupted Uncompressing Linux...decompressor returned an error done, booting the kernel.
invalid distance too far backinvalid distance codeinvalid literal/length codeincorrect header checkunknown compression methodinvalid window sizeinvalid block typeinvalid stored block lengthstoo many length or distance symbolsinvalid code lengths setinvalid bit length repeatinvalid literal/lengths setinvalid distances setincorrect data checkOut of memory while allocating output bufferOut of memory while allocating input bufferOut of memory while allocating z_streamOut of memory while allocating workspaceNot a gzi <CTRL-C>

Just wanted to show if this may help. Also if u will do ubifsmount SystemB

U-Boot> ubifsmount SystemB
UBIFS error (pid 0): ubifs_get_sb: cannot open "ubi:SystemB", error -19 UBIFS error (pid 0): ubifs_mount: Error reading superblock on volume 'ubi:SystemB' errno=-19!

ubifsmount - mount UBIFS volume

Usage:
ubifsmount <volume-name>
    - mount 'volume-name' volume

And if I will do the same second time then it starts but getting no uImage error:

DTB file loaded OK
Loading file 'kernel/uImage' to addr 0x22000000 with size 2968536 (0x002d4bd8)...
UBIFS error (pid 0): ubifs_read_node: bad node type (255 but expected 1) UBIFS error (pid 0): ubifs_read_node: bad node at LEB 215:0 UBIFS error (pid 0): do_readpage: cannot read page 178 of inode 1244, error -22 Error reading file 'kernel/uImage'
kernel/uImage not found!

Strange that I need to run same command twice and that uImage also can not be loaded from SystemB

Log file for all I did is attached.

Justin Mitchell
Thank you !!!!!!

^ permalink raw reply	[flat|nested] 3+ messages in thread